From: Michael Walle <hidden> Date: 2020-12-07 15:11:25
Hi,
The problem I'm having is that I'm trying to install debian on
an embedded system onto an sdcard. During installation it will
format the target filesystem, but the "mkfs.ext4 -F /dev/mmcblk0p2"
takes ages.
What I've found out so far:
- mkfs.ext4 tries to discard all blocks on the target device
- with my target device being an sdcard it seems to fallback
to normal erase [1], with erase_arg being set to what the card
is capable of [2]
Now I'm trying to figure out if this behavior is intended. I guess
one can reduce it to "blkdiscard /dev/mmcblk0p2". Should this
actually fall back to normal erasing or should it return -EOPNOTSUPP?
-michael
[1] https://elixir.bootlin.com/linux/v5.9.12/source/drivers/mmc/core/block.c#L1063
[2] https://elixir.bootlin.com/linux/v5.9.12/source/drivers/mmc/core/mmc.c#L1751
From: "Theodore Y. Ts'o" <tytso@mit.edu> Date: 2020-12-07 18:36:25
On Mon, Dec 07, 2020 at 04:10:27PM +0100, Michael Walle wrote:
Hi,
The problem I'm having is that I'm trying to install debian on
an embedded system onto an sdcard. During installation it will
format the target filesystem, but the "mkfs.ext4 -F /dev/mmcblk0p2"
takes ages.
What I've found out so far:
- mkfs.ext4 tries to discard all blocks on the target device
- with my target device being an sdcard it seems to fallback
to normal erase [1], with erase_arg being set to what the card
is capable of [2]
Now I'm trying to figure out if this behavior is intended. I guess
one can reduce it to "blkdiscard /dev/mmcblk0p2". Should this
actually fall back to normal erasing or should it return -EOPNOTSUPP?
There are three different MMC commands which are defined:
1) DISCARD
2) ERASE
3) SECURE ERASE
The first two are expected to be fast, since it only involves clearing
some metadata fields in the Flash Translation Layer (FTL), so that the
LBA's in the specified range are no longer mapped to a flash page.
The difference between "discard" and "erase" is that "discard" is a
hint, so the device is allowed to ignore it whenever it wants (in
practice, if it's busy doing a GC, or if it's busy writing back blocks
in its writeback cache). "Erase" is guaranteed to work, in that after
an erase, a read from a specified sector MUST return all zeros, but
that can easily be done by redirecting a point in the FTL metadata.
"Secure Erase" is the one which can be slow, since it requires
physically zeroing all of the flash pages (although if the device is
self-encrypting, this in theory could also be fast if you're doing a
secure erase at the granularity of the device's encryption keys, so
all it needs to do is to regenerate the crypto key).
It sounds like your SD card is implementing the "erase" command in a
particularly non-optimal way. If it's common, perhaps we need some
kind of blacklist for drivers with badly implemented erase commands.
As a workaround, you can run mke2fs with the command-line option "-E
discard=0".
Cheers,
- Ted
P.S. If your SD card got "erase" wrong, I'd be a little worried about
what else the FTL implementation may have screwed up. So you want to
under simply getting a different SD card --- especially if this is
something that you plan to distribute as a product to downstream
customers. In general, low-end flash needs to be very carefully
qualified to make sure they are competently implemented if you plan to
deploy in large quantities. An example of what happen if this
qualification process is not done:
https://insideevs.com/news/376037/tesla-mcu-emmc-memory-issue/
Tesla is currently under investigation by the National Highway Traffic
Safety Administration due to cheaping out on their eMMC flash
(probably just a few pennies per unit). Given that customers are
having to pay $1500 to replace their engine controller out of warranty
(and the NHTSA is considering whether or not to force Tesla to eat the
costs, as opposed to forcing their customers to pay $$$), that's an
example of false economy....
From: Michael Walle <hidden> Date: 2020-12-07 20:40:17
Hi Ted,
Am 2020-12-07 19:35, schrieb Theodore Y. Ts'o:
On Mon, Dec 07, 2020 at 04:10:27PM +0100, Michael Walle wrote:
quoted
Hi,
The problem I'm having is that I'm trying to install debian on
an embedded system onto an sdcard. During installation it will
format the target filesystem, but the "mkfs.ext4 -F /dev/mmcblk0p2"
takes ages.
What I've found out so far:
- mkfs.ext4 tries to discard all blocks on the target device
- with my target device being an sdcard it seems to fallback
to normal erase [1], with erase_arg being set to what the card
is capable of [2]
Now I'm trying to figure out if this behavior is intended. I guess
one can reduce it to "blkdiscard /dev/mmcblk0p2". Should this
actually fall back to normal erasing or should it return -EOPNOTSUPP?
There are three different MMC commands which are defined:
1) DISCARD
2) ERASE
3) SECURE ERASE
The first two are expected to be fast, since it only involves clearing
some metadata fields in the Flash Translation Layer (FTL), so that the
LBA's in the specified range are no longer mapped to a flash page.
Mh, where is it specified that the erase command is fast? According
to the Physical Layer Simplified Specification Version 8.00:
The actual erase time may be quite long, and the host may issue CMD7
to deselect the card or perform card disconnection, as described in
the Block Write section, above.
Honest question. Also reading "4.14 Erase Timeout Calculation" doesn't
sound that it is fast.
Also there is this comment:
https://elixir.bootlin.com/linux/v5.9.12/source/drivers/mmc/core/core.c#L1495
The difference between "discard" and "erase" is that "discard" is a
hint, so the device is allowed to ignore it whenever it wants (in
practice, if it's busy doing a GC, or if it's busy writing back blocks
in its writeback cache). "Erase" is guaranteed to work, in that after
an erase, a read from a specified sector MUST return all zeros, but
that can easily be done by redirecting a point in the FTL metadata.
"Secure Erase" is the one which can be slow, since it requires
physically zeroing all of the flash pages (although if the device is
self-encrypting, this in theory could also be fast if you're doing a
secure erase at the granularity of the device's encryption keys, so
all it needs to do is to regenerate the crypto key).
It sounds like your SD card is implementing the "erase" command in a
particularly non-optimal way. If it's common, perhaps we need some
kind of blacklist for drivers with badly implemented erase commands.
As a workaround, you can run mke2fs with the command-line option "-E
discard=0".
I've already tested that "mkfs.ext4 -E nodiscard" is fast (or works in
the same way as before the pre-discard feature).
But I wouldn't say it is a cheapo card (Toshiba Exceria). Although I
cannot guarantee that it might be a china clone, but it looks authentic
;)
P.S. If your SD card got "erase" wrong, I'd be a little worried about
what else the FTL implementation may have screwed up. So you want to
under simply getting a different SD card --- especially if this is
something that you plan to distribute as a product to downstream
customers. In general, low-end flash needs to be very carefully
qualified to make sure they are competently implemented if you plan to
deploy in large quantities. An example of what happen if this
qualification process is not done:
https://insideevs.com/news/376037/tesla-mcu-emmc-memory-issue/
Tesla is currently under investigation by the National Highway Traffic
Safety Administration due to cheaping out on their eMMC flash
(probably just a few pennies per unit). Given that customers are
having to pay $1500 to replace their engine controller out of warranty
(and the NHTSA is considering whether or not to force Tesla to eat the
costs, as opposed to forcing their customers to pay $$$), that's an
example of false economy....
Yeah I'm aware of the Tesla eMMC wear-out problem. But I've seen this
esp. from a user point of view. Like take our product, where the user
can freely choose its sdcard just to then notice that the installation
of its distribution is painfully slow. So I'm interested in understanding
the implications. Like is it really the case that the erase command can
be assumed fast.
-michael
From: "Theodore Y. Ts'o" <tytso@mit.edu> Date: 2020-12-08 02:41:48
On Mon, Dec 07, 2020 at 09:39:32PM +0100, Michael Walle wrote:
quoted
There are three different MMC commands which are defined:
1) DISCARD
2) ERASE
3) SECURE ERASE
The first two are expected to be fast, since it only involves clearing
some metadata fields in the Flash Translation Layer (FTL), so that the
LBA's in the specified range are no longer mapped to a flash page.
Mh, where is it specified that the erase command is fast? According
to the Physical Layer Simplified Specification Version 8.00:
The actual erase time may be quite long, and the host may issue CMD7
to deselect thhe card or perform card disconnection, as described in
the Block Write section, above.
I looked at the eMMC specification from JEDEC (JESD84-A44) and there,
both the "erase" and "trim" are specified that the work is to be
queued to be done at a time which is convenient to the controller
(read: FTL). This is in contrast to the "secure erase" and "secure
trim" commands, where the erasing has to be done NOW NOW NOW for "high
security applications".
The only difference between "erase" and "trim" seems to be that erahse
has to be done in units of the "erase groups" which is typically
larger than the "write pages" which is the granularity required by the
trim command. There is also a comment that when you are erasing the
entire partition, "erase" is preferred over "trim". (Presumably
because it is more convenient? The spec is not clear.)
Unfortunately, the SD Card spec and the eMMC spec both read like they
were written by a standards committee stacked by hardware engineers.
It doesn't look like they had file system engineers in the room,
because the distinctions between "erase" and "trim" are pretty silly,
and not well defined. Aside from what I wrote, the spec is remarkably
silent about what the host OS can depend upon.
From the fs perspective, what we care about is whether or not the
command is a hint or a reliable way to zero a range of sectors. A
command could be a hint if the device is allowed to ignore it, or if
the values of the sector are indeterminate, or if the sectors are
zero'ed or not could change after a power cycle. (I've seen an
implementation where discard would result in the LBA's being read as
zero --- but after a power cycle, reading from the same LBA would
return the old data again. This is standards complaint, but it's not
terribly useful.)
Assuming that the command is reliable, the next question is whether
the erase operation is logical or physical --- which is to say, if an
attacker has physical access to the die, with the ability to bypass
the FTL and directly read the flash cells, could the attack retrieve
the data, even if it required a distructive, physical attack on the
hardware? A logical erase would not require that the data be erased
or otherwise made inaccessible against an attacker who bypasses the
FTL; a physical erase would provide security guarantees that even if
your phone has handed over to state-sponsored attacker, that nothing
could be extracted after a physical erase.
So if I were king, those would be the three levels of discard: "hint",
"reliable logical", and "reliable physical", as those map to real use
cases that are of actual use to a Host. The challenge is mapping what
we *actually* are given by different specs, which were written by
hardware engineers and make distinctions that are not well defined so
that multiple implementations can be "standard compliant", but have
completely different performance profiles, thus making life easy for
the marketing types, and hard for the file system engineers. :-)
All I can tell you is that I know a bunch of Android system team
members at $WORK, and the current assumptions seem to work just fine
for the sorts of devices that are used on mobile handsets --- even
really cheap ones that are sold in India. At least, there are bunch
of "cost optimized" (as well as high end) Android devices running
ext4, and no one has complained to me about mke2fs taking a long time.
I definitely agree with you that the SD Card spec seems to imply that
other standards-compliant implementations could have the erase command
taking minutes, and this seems to be allowable by the spec. I would
consider this to be a flaw in the spec, myself. But I don't sit on
the standards committess, and I don't write the specs. I (and
everyone else) just have to live with them. Sigh....
- Ted
Hi Ted, Michael,
On Tue, 8 Dec 2020 at 03:41, Theodore Y. Ts'o [off-list ref] wrote:
On Mon, Dec 07, 2020 at 09:39:32PM +0100, Michael Walle wrote:
quoted
quoted
There are three different MMC commands which are defined:
1) DISCARD
2) ERASE
3) SECURE ERASE
The first two are expected to be fast, since it only involves clearing
some metadata fields in the Flash Translation Layer (FTL), so that the
LBA's in the specified range are no longer mapped to a flash page.
Mh, where is it specified that the erase command is fast? According
to the Physical Layer Simplified Specification Version 8.00:
The actual erase time may be quite long, and the host may issue CMD7
to deselect thhe card or perform card disconnection, as described in
the Block Write section, above.
Before I go into some more detail, of course I fully agree that
dealing with erase/discard from the eMMC/SD specifications (and other
types of devices) point of view isn't entirely easy. :-)
But I also think we can do better than currently, at least for eMMC/SD.
I looked at the eMMC specification from JEDEC (JESD84-A44) and there,
both the "erase" and "trim" are specified that the work is to be
queued to be done at a time which is convenient to the controller
(read: FTL). This is in contrast to the "secure erase" and "secure
trim" commands, where the erasing has to be done NOW NOW NOW for "high
security applications".
The only difference between "erase" and "trim" seems to be that erahse
has to be done in units of the "erase groups" which is typically
larger than the "write pages" which is the granularity required by the
trim command. There is also a comment that when you are erasing the
entire partition, "erase" is preferred over "trim". (Presumably
because it is more convenient? The spec is not clear.)
Unfortunately, the SD Card spec and the eMMC spec both read like they
were written by a standards committee stacked by hardware engineers.
It doesn't look like they had file system engineers in the room,
because the distinctions between "erase" and "trim" are pretty silly,
and not well defined. Aside from what I wrote, the spec is remarkably
silent about what the host OS can depend upon.
Moreover, the specs have evolved over the years. Somehow, we need to
map a REQ_OP_DISCARD and REQ_OP_SECURE_ERASE to the best matching
operation that the currently inserted eMMC/SD card supports...
Long time time ago, both the SD and eMMC spec introduced support for
real discards commands, as being hints to the card without any
guarantees of what will happen to the data from a logical or a
physical point of view. If the card supports that, we should use it as
the first option for REQ_OP_DISCARD. Although, what should we pick as
the second best option, when the card doesn't support discard - that's
when it becomes more tricky. And the similar applies for
REQ_OP_SECURE_ERASE, or course.
If you have any suggestions for how we can improve in the above
decisions, feel free to suggest something.
Another issue that most likely is causing poor performance for
REQ_OP_DISCARD/REQ_OP_SECURE_ERASE for eMMC/SD, is that in
mmc_queue_setup_discard() we set up the maximum discard sectors
allowed per request and the discard granularity.
To find performance bottlenecks, I would start looking at what actual
eMMC/SD commands/args we end up mapping towards the
REQ_OP_DISCARD/REQ_OP_SECURE_ERASE requests. Then definitely, I would
also look at the values we end up picking as max discard sectors and
the discard granularity.
From the fs perspective, what we care about is whether or not the
command is a hint or a reliable way to zero a range of sectors. A
command could be a hint if the device is allowed to ignore it, or if
the values of the sector are indeterminate, or if the sectors are
zero'ed or not could change after a power cycle. (I've seen an
implementation where discard would result in the LBA's being read as
zero --- but after a power cycle, reading from the same LBA would
return the old data again. This is standards complaint, but it's not
terribly useful.)
:-)
Assuming that the command is reliable, the next question is whether
the erase operation is logical or physical --- which is to say, if an
attacker has physical access to the die, with the ability to bypass
the FTL and directly read the flash cells, could the attack retrieve
the data, even if it required a distructive, physical attack on the
hardware? A logical erase would not require that the data be erased
or otherwise made inaccessible against an attacker who bypasses the
FTL; a physical erase would provide security guarantees that even if
your phone has handed over to state-sponsored attacker, that nothing
could be extracted after a physical erase.
So if I were king, those would be the three levels of discard: "hint",
"reliable logical", and "reliable physical", as those map to real use
cases that are of actual use to a Host. The challenge is mapping what
we *actually* are given by different specs, which were written by
hardware engineers and make distinctions that are not well defined so
that multiple implementations can be "standard compliant", but have
completely different performance profiles, thus making life easy for
the marketing types, and hard for the file system engineers. :-)
I agree, these are the three levels that make sense to support.
Honestly I haven't been paying enough attention to discussions for the
generic block layer around discards. However, considering what you
just stated above, we seem to be missing one request operation, don't
we?
[...]
Kind regards
Uffe
From: Michael Walle <hidden> Date: 2020-12-08 11:27:22
Hi Ulf, Hi Ted,
Am 2020-12-08 10:49, schrieb Ulf Hansson:
On Tue, 8 Dec 2020 at 03:41, Theodore Y. Ts'o [off-list ref] wrote:
quoted
On Mon, Dec 07, 2020 at 09:39:32PM +0100, Michael Walle wrote:
quoted
quoted
There are three different MMC commands which are defined:
1) DISCARD
2) ERASE
3) SECURE ERASE
The first two are expected to be fast, since it only involves clearing
some metadata fields in the Flash Translation Layer (FTL), so that the
LBA's in the specified range are no longer mapped to a flash page.
Mh, where is it specified that the erase command is fast? According
to the Physical Layer Simplified Specification Version 8.00:
The actual erase time may be quite long, and the host may issue CMD7
to deselect thhe card or perform card disconnection, as described in
the Block Write section, above.
Before I go into some more detail, of course I fully agree that
dealing with erase/discard from the eMMC/SD specifications (and other
types of devices) point of view isn't entirely easy. :-)
But I also think we can do better than currently, at least for eMMC/SD.
quoted
I looked at the eMMC specification from JEDEC (JESD84-A44) and there,
both the "erase" and "trim" are specified that the work is to be
queued to be done at a time which is convenient to the controller
(read: FTL). This is in contrast to the "secure erase" and "secure
trim" commands, where the erasing has to be done NOW NOW NOW for "high
security applications".
Oh this might also be because I've cited from the wrong place, namely the
mmc_init_card() function. But what I really meant was the sd card equivalent
which should be mmc_read_ssr(). Sorry.
discard_support = UNSTUFF_BITS(resp, 313 - 288, 1);
card->erase_arg = (card->scr.sda_specx && discard_support) ?
SD_DISCARD_ARG : SD_ERASE_ARG;
quoted
The only difference between "erase" and "trim" seems to be that erahse
has to be done in units of the "erase groups" which is typically
larger than the "write pages" which is the granularity required by the
trim command. There is also a comment that when you are erasing the
entire partition, "erase" is preferred over "trim". (Presumably
because it is more convenient? The spec is not clear.)
Unfortunately, the SD Card spec and the eMMC spec both read like they
were written by a standards committee stacked by hardware engineers.
It doesn't look like they had file system engineers in the room,
because the distinctions between "erase" and "trim" are pretty silly,
and not well defined. Aside from what I wrote, the spec is remarkably
silent about what the host OS can depend upon.
Moreover, the specs have evolved over the years. Somehow, we need to
map a REQ_OP_DISCARD and REQ_OP_SECURE_ERASE to the best matching
operation that the currently inserted eMMC/SD card supports...
Do we really need to map these functions? What if we don't have an
actual discard, but just a slow erase (I'm now assuming that erase
will likely be slow on sdcards)? Can't we just tell the user space
there is no discard? Like on a normal HDD? I really don't know the
implications, seems like mmc_erase() is just there for the linux
discard feature.
Coming from the user space side. Does mkfs.ext4 assumes its pre-discard
is fast? I'd think so, right? I'd presume it was intented to tell the
FTL of the block device, "hey these blocks are unused, you can do some
wear leveling with them".
Long time time ago, both the SD and eMMC spec introduced support for
real discards commands, as being hints to the card without any
guarantees of what will happen to the data from a logical or a
physical point of view. If the card supports that, we should use it as
the first option for REQ_OP_DISCARD. Although, what should we pick as
the second best option, when the card doesn't support discard - that's
when it becomes more tricky. And the similar applies for
REQ_OP_SECURE_ERASE, or course.
If you have any suggestions for how we can improve in the above
decisions, feel free to suggest something.
Another issue that most likely is causing poor performance for
REQ_OP_DISCARD/REQ_OP_SECURE_ERASE for eMMC/SD, is that in
mmc_queue_setup_discard() we set up the maximum discard sectors
allowed per request and the discard granularity.
To find performance bottlenecks, I would start looking at what actual
eMMC/SD commands/args we end up mapping towards the
REQ_OP_DISCARD/REQ_OP_SECURE_ERASE requests. Then definitely, I would
also look at the values we end up picking as max discard sectors and
the discard granularity.
I'm just about finding some SD cards and looking how they behave timing
wise and what they report they support (ie. erase or discard). Looks
like other cards are doing better. But I'd have to find out if they
support the discard (mine doesn't) and if they are slow too if I force
them to use the normal erase.
quoted
From the fs perspective, what we care about is whether or not the
command is a hint or a reliable way to zero a range of sectors. A
command could be a hint if the device is allowed to ignore it, or if
the values of the sector are indeterminate, or if the sectors are
zero'ed or not could change after a power cycle. (I've seen an
implementation where discard would result in the LBA's being read as
zero --- but after a power cycle, reading from the same LBA would
return the old data again. This is standards complaint, but it's not
terribly useful.)
:-)
quoted
Assuming that the command is reliable, the next question is whether
the erase operation is logical or physical --- which is to say, if an
attacker has physical access to the die, with the ability to bypass
the FTL and directly read the flash cells, could the attack retrieve
the data, even if it required a distructive, physical attack on the
hardware? A logical erase would not require that the data be erased
or otherwise made inaccessible against an attacker who bypasses the
FTL; a physical erase would provide security guarantees that even if
your phone has handed over to state-sponsored attacker, that nothing
could be extracted after a physical erase.
So if I were king, those would be the three levels of discard: "hint",
"reliable logical", and "reliable physical", as those map to real use
cases that are of actual use to a Host. The challenge is mapping what
we *actually* are given by different specs, which were written by
hardware engineers and make distinctions that are not well defined so
that multiple implementations can be "standard compliant", but have
completely different performance profiles, thus making life easy for
the marketing types, and hard for the file system engineers. :-)
I agree, these are the three levels that make sense to support.
Honestly I haven't been paying enough attention to discussions for the
generic block layer around discards. However, considering what you
just stated above, we seem to be missing one request operation, don't
we?
On Tue, 8 Dec 2020 at 12:26, Michael Walle [off-list ref] wrote:
Hi Ulf, Hi Ted,
Am 2020-12-08 10:49, schrieb Ulf Hansson:
quoted
On Tue, 8 Dec 2020 at 03:41, Theodore Y. Ts'o [off-list ref] wrote:
quoted
On Mon, Dec 07, 2020 at 09:39:32PM +0100, Michael Walle wrote:
quoted
quoted
There are three different MMC commands which are defined:
1) DISCARD
2) ERASE
3) SECURE ERASE
The first two are expected to be fast, since it only involves clearing
some metadata fields in the Flash Translation Layer (FTL), so that the
LBA's in the specified range are no longer mapped to a flash page.
Mh, where is it specified that the erase command is fast? According
to the Physical Layer Simplified Specification Version 8.00:
The actual erase time may be quite long, and the host may issue CMD7
to deselect thhe card or perform card disconnection, as described in
the Block Write section, above.
Before I go into some more detail, of course I fully agree that
dealing with erase/discard from the eMMC/SD specifications (and other
types of devices) point of view isn't entirely easy. :-)
But I also think we can do better than currently, at least for eMMC/SD.
quoted
I looked at the eMMC specification from JEDEC (JESD84-A44) and there,
both the "erase" and "trim" are specified that the work is to be
queued to be done at a time which is convenient to the controller
(read: FTL). This is in contrast to the "secure erase" and "secure
trim" commands, where the erasing has to be done NOW NOW NOW for "high
security applications".
Oh this might also be because I've cited from the wrong place, namely
the
mmc_init_card() function. But what I really meant was the sd card
equivalent
which should be mmc_read_ssr(). Sorry.
discard_support = UNSTUFF_BITS(resp, 313 - 288, 1);
card->erase_arg = (card->scr.sda_specx && discard_support) ?
SD_DISCARD_ARG : SD_ERASE_ARG;
I assumed you were referring to this, but good that you pointed this
out, for clarity.
quoted
quoted
The only difference between "erase" and "trim" seems to be that erahse
has to be done in units of the "erase groups" which is typically
larger than the "write pages" which is the granularity required by the
trim command. There is also a comment that when you are erasing the
entire partition, "erase" is preferred over "trim". (Presumably
because it is more convenient? The spec is not clear.)
Unfortunately, the SD Card spec and the eMMC spec both read like they
were written by a standards committee stacked by hardware engineers.
It doesn't look like they had file system engineers in the room,
because the distinctions between "erase" and "trim" are pretty silly,
and not well defined. Aside from what I wrote, the spec is remarkably
silent about what the host OS can depend upon.
Moreover, the specs have evolved over the years. Somehow, we need to
map a REQ_OP_DISCARD and to the best matching
operation that the currently inserted eMMC/SD card supports...
Do we really need to map these functions? What if we don't have an
actual discard, but just a slow erase (I'm now assuming that erase
will likely be slow on sdcards)? Can't we just tell the user space
there is no discard? Like on a normal HDD?
I have considered that, but not sure what would be the best option.
I really don't know the
implications, seems like mmc_erase() is just there for the linux
discard feature.
mmc_erase() is used for both REQ_OP_DISCARD and REQ_OP_SECURE_ERASE,
but that's an implementation detail that we can change, of course.
Honestly, the hole erase/discard support in the mmc core deserves a
cleanup and I am looking at that (occasionally).
Coming from the user space side. Does mkfs.ext4 assumes its pre-discard
is fast? I'd think so, right? I'd presume it was intented to tell the
FTL of the block device, "hey these blocks are unused, you can do some
wear leveling with them".
I would assume that too.
On the other hand, I guess there are situations when user space could
live with slow formatting times. In particular if the goal is to let
card clean up its internal garbage, as a way to improve "performance"
for later I/O writes.
quoted
Long time time ago, both the SD and eMMC spec introduced support for
real discards commands, as being hints to the card without any
guarantees of what will happen to the data from a logical or a
physical point of view. If the card supports that, we should use it as
the first option for REQ_OP_DISCARD. Although, what should we pick as
the second best option, when the card doesn't support discard - that's
when it becomes more tricky. And the similar applies for
REQ_OP_SECURE_ERASE, or course.
If you have any suggestions for how we can improve in the above
decisions, feel free to suggest something.
Another issue that most likely is causing poor performance for
REQ_OP_DISCARD/REQ_OP_SECURE_ERASE for eMMC/SD, is that in
mmc_queue_setup_discard() we set up the maximum discard sectors
allowed per request and the discard granularity.
To find performance bottlenecks, I would start looking at what actual
eMMC/SD commands/args we end up mapping towards the
REQ_OP_DISCARD/REQ_OP_SECURE_ERASE requests. Then definitely, I would
also look at the values we end up picking as max discard sectors and
the discard granularity.
I'm just about finding some SD cards and looking how they behave timing
wise and what they report they support (ie. erase or discard). Looks
like other cards are doing better. But I'd have to find out if they
support the discard (mine doesn't) and if they are slow too if I force
them to use the normal erase.
Sounds great, looking forward to hear more about your findings.
[...]
Kind regards
Uffe
From: "Theodore Y. Ts'o" <tytso@mit.edu> Date: 2020-12-08 16:53:05
On Tue, Dec 08, 2020 at 12:26:22PM +0100, Michael Walle wrote:
Do we really need to map these functions? What if we don't have an
actual discard, but just a slow erase (I'm now assuming that erase
will likely be slow on sdcards)? Can't we just tell the user space
there is no discard? Like on a normal HDD? I really don't know the
implications, seems like mmc_erase() is just there for the linux
discard feature.
So the potential gotcha here is that "discard" is important for
reducing write amplification, and thus improving the lifespan of
devices. (See my reference to the Tesla engine controller story
earlier.) So if a device doesn't have "discard" but has "erase", and
"erase" is fast, then skipping the discard could end up significantly
reducing the lifespan of your product, and we're back to the NHTSA
investigating whether they should stick Tesla for the $1500 engine
controller replacement when cards die early.
I guess the JEDEC spec does specify a way to query the card for how
long an erase takes, but I don't have the knowledge about how the
actual real-world implementations of these specs (and their many
variants over the years) actually behave. Can the erase times that
they advertise actually be trusted to be accurate? How many of them
actually supply erase times at all, no matter what the spec says?
Coming from the user space side. Does mkfs.ext4 assumes its pre-discard
is fast? I'd think so, right? I'd presume it was intented to tell the
FTL of the block device, "hey these blocks are unused, you can do some
wear leveling with them".
Yes, the assumption is that discard is fast. Exactly how fast seems
to vary; this is one of the reasons why there are three different ways
to do discards on a file system after files are deleted. One way is
to do them after the deleted definitely won't be unwound (i.e., after
the ext4 journal commit). But on some devices, the discard command,
while fast, is slow enough that this will compete with the I/O
completion times of other read commands, thus degrading system
performance. So you can also execute the trim commands out of cron,
using the fstrim command, which will run the discards in the
background, and the system administrator can adjust when fstrim is
executed during times wheno performance isn't critical. (e.g., when
the phone is on a charger in the middle of the night, or at 4am local
time, etc.) Finally, you can configure e2fsck to run the discards
after the file system consistency check is done.
The reason why we have to leave this up to the system administrators
is that we have essentially no guidance from the device how slow the
discard command might be, how it intereferes with other device
operations, and whether the discard might be more likely ignored if
the device is busy. So it might be that the discard will more likely
improve write endurance when it is done when the device is idle. All
of the speccs (SCSI, SATA, UFS, eMMC, SD) are remarkable unhelpful
because performance considerations is generally consider "out of
scope" of standards committees. They want to leave that up to market
forces; which is why big companies (at handset vendors, hyperscale
cloud providers, systems integrators, etc.) have to spend as much
money doing certification testing before deciding which products to
buy; think of it as a full-employment act for storage engineers. :-)
But yes, mke2fs assumes that discard is sufficiently fast that it
doing it at file system format time is extremely reasonable. The
bigger concern is that we can't necessarily count on discard zero'ing
the inode table, and there are robustness reasons (especially if
before we had metadata checksums) where it makes file system repairs
much more robust if the inode table is zero'ed ahead of time.
I'm just about finding some SD cards and looking how they behave timing
wise and what they report they support (ie. erase or discard). Looks
like other cards are doing better. But I'd have to find out if they
support the discard (mine doesn't) and if they are slow too if I force
them to use the normal erase.
The challenge is that this sort of thing gets rapidly out of date, and
it's not just SD cards but also eMMC devices which are built into
various embedded devices, high-end SDHC cards, etc., etc. So doing
this gets very expensive.
That being said, both ext4 and f2fs do pre-discards as part of the
format step, since improving write endurance is important; customers
get cranky when their $1000 smart phones die an early death. So an SD
card that behaves the way yours does would probably get disqualified
very early in the certification step if it were ever intended to be
used in an Android handset, since pretty much all Android devices, or
embedded devices for that matter, use either f2fs or ext4. That's one
of the reasons why I was a bit surprised that your device had such an
"interesting" performance profile. Maybe it was intended for use in
digital cameras, and digital camerase don't issue discards? I don't
know....
quoted
I agree, these are the three levels that make sense to support.
Honestly I haven't been paying enough attention to discussions for the
generic block layer around discards. However, considering what you
just stated above, we seem to be missing one request operation, don't
we?
Yes, that's true. We only have "discard" and "secure discard". Part
of that is because that's the only levels which are available for
SSD's, for which I have the same general complaint vis-a-vis standards
committees and the general lack of usefulness for file system
engineers.
For example, pretty much everyone in the enterprise and hyperscale
cloud world assume that low-numbered LBA's have better performance
profiles, and are located physically at the outer diameter of HDD's,
compared to high-number'ed LBA's. But that's nothing which is
specified by the standards committees, because "performance
considerations are out of scope". Yet we still have to engineer
storage systems which assume this to be true, even though nothing in
the formal specs guarantees this. We just have to trust that anyone
who tries to sell a HDD for which this isn't true, even if it is
"standards complaint", is going to have a bad time, and trust that
this is enough. (Perhaps this is why when a certain HDD manufacturer
tried to sell HDD's containing drive-managed SMR for the NAS market,
without disclosing this fact to consumers, this generated a massive
backlash.... Simply being standards compliant is not enough.)
- Ted
From: Michael Walle <hidden> Date: 2020-12-08 20:57:55
Hi Ulf, Ted,
Am 2020-12-08 17:17, schrieb Ulf Hansson:
On Tue, 8 Dec 2020 at 12:26, Michael Walle [off-list ref] wrote:
quoted
quoted
To find performance bottlenecks, I would start looking at what actual
eMMC/SD commands/args we end up mapping towards the
REQ_OP_DISCARD/REQ_OP_SECURE_ERASE requests. Then definitely, I would
also look at the values we end up picking as max discard sectors and
the discard granularity.
I'm just about finding some SD cards and looking how they behave timing
wise and what they report they support (ie. erase or discard). Looks
like other cards are doing better. But I'd have to find out if they
support the discard (mine doesn't) and if they are slow too if I force
them to use the normal erase.
Sounds great, looking forward to hear more about your findings.
@@ -1675,6 +1675,8 @@ static int mmc_do_erase(struct mmc_card *card, unsigned int from,to<<=9;}+trace_printk("from=0x%x to=0x%x\n",from,to);+if(mmc_card_sd(card))cmd.opcode=SD_ERASE_WR_BLK_START;else
@@ -1747,6 +1749,7 @@ static int mmc_do_erase(struct mmc_card *card, unsigned int from,/* Let's poll to find out when the erase operation completes. */err=mmc_poll_for_busy(card,busy_timeout,MMC_BUSY_ERASE);+trace_printk("mmc_poll_for_busy() done\n");out:mmc_retune_release(card->host);returnerr;
On Tue, 8 Dec 2020 at 17:52, Theodore Y. Ts'o [off-list ref] wrote:
On Tue, Dec 08, 2020 at 12:26:22PM +0100, Michael Walle wrote:
quoted
Do we really need to map these functions? What if we don't have an
actual discard, but just a slow erase (I'm now assuming that erase
will likely be slow on sdcards)? Can't we just tell the user space
there is no discard? Like on a normal HDD? I really don't know the
implications, seems like mmc_erase() is just there for the linux
discard feature.
So the potential gotcha here is that "discard" is important for
reducing write amplification, and thus improving the lifespan of
devices. (See my reference to the Tesla engine controller story
earlier.) So if a device doesn't have "discard" but has "erase", and
"erase" is fast, then skipping the discard could end up significantly
reducing the lifespan of your product, and we're back to the NHTSA
investigating whether they should stick Tesla for the $1500 engine
controller replacement when cards die early.
Yes, exactly. The point about wear leveling and the lifespan of the
device are critical.
That said, we should continue to map discard requests to legacy erase
commands for SD cards, unless the card supports the new discard, of
course.
One thing I realized though, is that we should probably announce and
implement support for secure erase (QUEUE_FLAG_SECERASE) for SD cards,
as that seems to map well towards with the erase command.
An erase is specified in the SD spec as, after the erase the data is
either "0" or "1", which I guess is what is expected from a
REQ_OP_SECURE_ERASE operation?
I guess the JEDEC spec does specify a way to query the card for how
long an erase takes, but I don't have the knowledge about how the
actual real-world implementations of these specs (and their many
variants over the years) actually behave. Can the erase times that
they advertise actually be trusted to be accurate? How many of them
actually supply erase times at all, no matter what the spec says?
For eMMC discard commands are fast, but probably also trim commands.
For erase I don't know.
Then, whether the corresponding "erase times" that are be specified in
the eMMC registers, I guess those always refer to the worst case
scenario. I don't know how useful they really are in the end.
In any case, we may end up with poor erase/discard performance,
because of internal FW implementations.
Although, what I think we may be able to improve, both from eMMC and
SD point of view, is to allow more blocks per discard/erase operation.
But honestly, I don't know how big of a problem this is, even if just
staring at the code, gives me some ideas.
quoted
Coming from the user space side. Does mkfs.ext4 assumes its pre-discard
is fast? I'd think so, right? I'd presume it was intented to tell the
FTL of the block device, "hey these blocks are unused, you can do some
wear leveling with them".
Yes, the assumption is that discard is fast. Exactly how fast seems
to vary; this is one of the reasons why there are three different ways
to do discards on a file system after files are deleted. One way is
to do them after the deleted definitely won't be unwound (i.e., after
the ext4 journal commit). But on some devices, the discard command,
while fast, is slow enough that this will compete with the I/O
completion times of other read commands, thus degrading system
performance. So you can also execute the trim commands out of cron,
using the fstrim command, which will run the discards in the
background, and the system administrator can adjust when fstrim is
executed during times wheno performance isn't critical. (e.g., when
the phone is on a charger in the middle of the night, or at 4am local
time, etc.) Finally, you can configure e2fsck to run the discards
after the file system consistency check is done.
The reason why we have to leave this up to the system administrators
is that we have essentially no guidance from the device how slow the
discard command might be, how it intereferes with other device
operations, and whether the discard might be more likely ignored if
the device is busy. So it might be that the discard will more likely
improve write endurance when it is done when the device is idle. All
of the speccs (SCSI, SATA, UFS, eMMC, SD) are remarkable unhelpful
because performance considerations is generally consider "out of
scope" of standards committees. They want to leave that up to market
forces; which is why big companies (at handset vendors, hyperscale
cloud providers, systems integrators, etc.) have to spend as much
money doing certification testing before deciding which products to
buy; think of it as a full-employment act for storage engineers. :-)
A few comments related to the above.
Even if the discarded blocks are flushed at some wisely selected
point, when the device is idle, that doesn't guarantee that the
internal garbage collection runs inside the device. In the end that
depends on the FW implementation of the card - and I assume it's
likely triggered based on some internal idle time and the amount of
"garbage" there is to deal with.
For both eMMC and SD cards, the specs define commands for how to
manually control the background operations inside the cards. In
principle, this allows us to tell the card when it's a good time to
run garbage collection (and when not to).
Both for eMMC and SD, we are not using this, yet. However, I have been
playing with a couple of different ideas to explore this:
*) Use the runtime PM framework to detect an idle period and then
trigger background operations. The problem is, that we don't really
know how long we will be idle, meaning that we don't know if it's
really a wise decision to trigger the background operations in the
end.
**) Invent a new type of generic block request, as to let userspace
trigger this.
Of course, another option is also to leave this as is, thus relying on
the internal FW of the card to act the best it can.
Do you have any thoughts around this?
[...]
Kind regards
Uffe
From: "Theodore Y. Ts'o" <tytso@mit.edu> Date: 2020-12-09 16:36:56
On Wed, Dec 09, 2020 at 03:51:24PM +0100, Ulf Hansson wrote:
Even if the discarded blocks are flushed at some wisely selected
point, when the device is idle, that doesn't guarantee that the
internal garbage collection runs inside the device. In the end that
depends on the FW implementation of the card - and I assume it's
likely triggered based on some internal idle time and the amount of
"garbage" there is to deal with.
At least from a file system perspective, I don't care when the
internal garbage collection actually runs inside the device. What I
do care is that (a) a read to a discarded sector returns zero's after
it has been discard (or the storage device needs to tell me I can't
count on that), and (b) that eventually, for write endurance reasons,
the garbage collection will *eventually* happen.
If the list of erase blocks or flash pages that are not in use are
tracked in such a way that they are actually garbage collected before
the device actually needs free blocks, it really doesn't matter if it
happens right away, or hours later. (If the device is 90% free,
because it was just formatted and we did a pre-discard at format time,
then it could happen hours or days later.)
But if the device's FTL is too incompetent such that it loses track of
which erase blocks / flash pages do need to be GC'ed, such that it
impacts device lifetime... well then, that's sad, and it would be nice
to find out about this without having to do an expensive,
time-consuming certification process. (OTOH, all the big companies
are doing hardware certifications anyway, because you can't fully
trust the storage vendors, and how many storage vendors are really
going to admit, or make it easy to determine, "the FTL is so
cost-optimized that it's cr*p"? :-)
Having a way to tell the storage device that it would be better to
suspend GC, or to accelerate GC, because we know the device is about
to become much less likely to perform writes, would certainly be a
good and useful thing to do, although I see that as mostly being
useful for improving I/O performance, especially for low-end flash ---
I suspect that for high-end SSD's, which are designed so that they can
handle continuous write streams without much performance degradation,
they have enough oomph in their internal CPU that they can do GC's in
real-time while the device is under a continuous random write workload
with only minimal performance impacts.
*) Use the runtime PM framework to detect an idle period and then
trigger background operations. The problem is, that we don't really
know how long we will be idle, meaning that we don't know if it's
really a wise decision to trigger the background operations in the
end.
**) Invent a new type of generic block request, as to let userspace
trigger this.
I think you really want to give userspace the ability to trigger this.
Whether it's via a generic block request, or an ioctl, I'll leave that
to the people maintain the driver and/or block layer. That's because
userspace will have knowledge to things like, "the screen is off", or
"the phone is on the wireless charger and/or the user has said, "OK,
Google, goodnight" to trigger the night-time home automation commands.
We can of course try to make some automatic determinations based on
the runtime PM framework, but that doesn't necessarily tell us the
likelihood that the system will become busy in the future; OTOH, maybe
that doesn't matter, if the storage needs only a very tiny amount of
time after it's told, "stop GC", to finish up what it's doing so it
can respond to I/O request at full speed?
- Ted