Re: discard and data=writeback

On Tue, Dec 22, 2020 at 11:53 PM Andreas Dilger [off-list ref] wrote:

On Dec 22, 2020, at 9:34 AM, Theodore Y. Ts'o [off-list ref] wrote:

quoted

On Tue, Dec 22, 2020 at 03:59:29PM +0100, Matteo Croce wrote:

quoted

I'm issuing sync + sleep(10) after the extraction, so the writes
should all be flushed.
Also, I repeated the test three times, with very similar results:

So that means the problem is not due to page cache writeback
interfering with the discards.  So it's most likely that the problem
is due to how the blocks are allocated and laid out when using
data=ordered vs data=writeback.

Some experiments to try next.  After extracting the files with
data=ordered and data=writeback on a freshly formatted file system,
use "e2freefrag" to see how the free space is fragmented.  This will
tell us how the file system is doing from a holistic perspective, in
terms of blocks allocated to the extracted files.  (E2freefrag is
showing you the blocks *not* allocated, of course, but that's a mirror
image dual of the blocks that *are* allocated, especially if you start
from an identical known state; hence the use of a freshly formatted
file system.)

Next, we can see how individual files look like with respect to
fragmentation.  This can be done via using filefrag on all of the
files, e.g:

      find . -type f -print0  | xargs -0 filefrag

Another way to get similar (although not identical) information is via
running "e2fsck -E fragcheck" on a file system.  How they differ is
especially more of a big deal on ext3 file systems without extents and
flex_bg, since filefrag tries to take into account metadata blocks
such as indirect blocks and extent tree blocks, and e2fsck -E
fragcheck does not; but it's good enough for getting a good gestalt
for the files' overall fragmentation --- and note that as long as the
average fragment size is at least a megabyte or two, some
fragmentation really isn't that much of a problem from a real-world
performance perspective.  People can get way too invested in trying to
get to perfection with 100% fragmentation-free files.  The problem
with doing this at the expense of all else is that you can end up
making the overall free space fragmentation worse as the file system
ages, at which point the file system performance really dives through
the floor as the file system approaches 100%, or even 80-90% full,
especially on HDD's.  For SSD's fragmentation doesn't matter quite so
much, unless the average fragment size is *really* small, and when you
are discarded freed blocks.

Even if the files are showing no substantial difference in
fragmentation, and the free space is equally A-OK with respect to
fragmentation, the other possibility is the *layout* of the blocks are
such that the order in which they are deleted using rm -rf ends up
being less friendly from a discard perspective.  This can happen if
the directory hierarchy is big enough, and/or the journal size is
small enough, that the rm -rf requires multiple journal transactions
to complete.  That's because with mount -o discard, we do the discards
after each transaction commit, and it might be that even though the
used blocks are perfectly contiguous, because of the order in which
the files end up getting deleted, we end up needing to discard them in
smaller chunks.

For example, one could imagine a case where you have a million 4k
files, and they are allocated contiguously, but if you get
super-unlucky, such that in the first transaction you delete all of
the odd-numbered files, and in second transaction you delete all of
the even-numbered files, you might need to do a million 4k discards

--- but if all of the deletes could fit into a single transaction, you

would only need to do a single million block discard operation.

Finally, you may want to consider whether or not mount -o discard
really makes sense or not.  For most SSD's, especially high-end SSD's,
it probably doesn't make that much difference.  That's because when
you overwrite a sector, the SSD knows (or should know; this might not
be some really cheap, crappy low-end flash devices; but on those
devices, discard might not be making uch of a difference anyway), that
the old contents of the sector is no longer needed.  Hence an
overwrite effectively is an "implied discard".  So long as there is a
sufficient number of free erase blocks, the SSD might be able to keep
up doing the GC for those "implied discards", and so accelerating the
process by sending explicit discards after every journal transaction
might not be necessary.  Or, maybe it's sufficient to run "fstrim"
every week at Sunday 3am local time; or maybe even fstrim once a night
or fstrim once a month --- your mileage may vary.

It's going to vary from SSD to SSD and from workload to workload, but
you might find that mount -o discard isn't buying you all that much

--- if you run a random write workload, and you don't notice any

performance degradation, and you don't notice an increase in the SSD's
write amplification numbers (if they are provided by your SSD), then
you might very well find that it's not worth it to use mount -o
discard.

I personally don't bother using mount -o discard, and instead
periodically run fstrim, on my personal machines.  Part of that is
because I'm mostly just reading and replying to emails, building
kernels and editing text files, and that is not nearly as stressful on
the FTL as a full-blown random write workload (for example, if you
were running a database supporting a transaction processing workload).

The problem (IMHO) with "-o discard" is that if it is only trimming
*blocks* that were deleted, these may be too small to effectively be
processed by the underlying device (e.g. the "super-unlucky" example
above where interleaved 4KB file deletes result in 1M separate 4KB
trim requests to the device, even when the *space* that is freed by
the unlinks could be handled with far fewer large trim requests.

There was a discussion previously ("introduce EXT4_BG_WAS_TRIMMED ...")

https://patchwork.ozlabs.org/project/linux-ext4/patch/1592831677-13945-1-git-send-email-wangshilong1991@gmail.com/

about leveraging the persistent EXT4_BG_WAS_TRIMMED flag in the group
descriptors, and having "-o discard" only track trim on a per-group
basis rather than its current mode of doing trim on a per-block basis,
and then use the same code internally as fstrim to do a trim of free
blocks in that block group.

Using EXT4_BG_WAS_TRIMMED and tracking *groups* to be trimmed would be
a bit more lazy than the current "-o discard" implementation, but would
be more memory efficient, and also more efficient for the device (fewer,
larger trim requests submitted).  It would only need to track groups
that have at least a reasonable amount of free space to be trimmed.  If
the group doesn't have enough free blocks to trim now, it will be checked
again in the future when more blocks are freed.

Hi,

I gave it a quick run refreshing it for 5.10, but it doesn't seem to help.
Are there actions needed other than the patch itself?

Regards,
-- 
per aspera ad upstream