From: David Brown <hidden> Date: 2012-10-31 14:11:59
On 31/10/2012 14:12, Alexander Haase wrote:
Has anyone considered handling TRIM via an idle IO queue? You'd have to
purge queue items that conflicted with incoming writes, but it does get
around the performance complaint. If the idle period never comes, old
TRIMs can be silently dropped to lessen queue bloat.
I am sure it has been considered - but is it worth the effort and the
complications? TRIM has been implemented in several filesystems (ext4
and, I believe, btrfs) - but is disabled by default because it typically
slows down the system. You are certainly correct that putting TRIM at
the back of the queue will avoid the delays it causes - but it still
will not give any significant benefit (except for old SSDs with limited
garbage collection and small over-provisioning ), and you have a lot of
extra complexity to ensure that a TRIM is never pushed back until after
a new write to the same logical sectors.
It would be much easier and safer, and give much better effect, to make
sure the block allocation procedure for filesystems emphasised
re-writing old blocks as soon as possible (when on an SSD). Then there
is no need for TRIM at all. This would have the added benefit of
working well for compressed (or sparse) hard disk image files used by
virtual machines - such image files only take up real disk space for
blocks that are written, so re-writes would save real-world disk space.
As far as parity consistency, bitmaps could track which stripes( and
blocks within those stripes) are expected to be out of parity( also
useful for lazy device init ). Maybe a bit-per-stripe map at the logical
device level and a bit-per-LBA bitmap at the stripe level?
Tracking "no-sync" areas of a raid array is already high on the md raid
things-to-do list (perhaps it is already implemented - I lose track of
which features are planned and which are implemented). And yes, such
no-sync tracking would be useful here. But it is complicated,
especially for raid5/6 (raid1 is not too bad) - should TRIMs that cover
part of a stripe be dropped? Should the md layer remember them and
coalesce them when it can TRIM a whole stripe? Should it try to track
partial synchronisation within a stripe?
Or should the md developers simply say that since supporting TRIM is not
going to have any measurable benefits (certainly not with the sort of
SSD's people use in raid arrays), and since TRIM slows down some
operations, it is better to keep things simple and ignore TRIM entirely?
Even if there are occasional benefits to having TRIM, is it worth it
in the face of added complication in the code and the risk of errors?
There /have/ been developers working on TRIM support on raid5. It seems
to have been a complicated process. But some people like a challenge!
On the other hand, does it hurt if empty blocks are out of parity( due
to TRIM or lazy device init)? The parity recovery of garbage is still
garbage, which is what any sane FS expects from unused blocks. If and
when you do a parity scrub, you will spend a lot of time recovering
garbage and undo any good TRIM might have done, but usual drive
operation should quickly balance that out in a write-intensive
environment where idle TRIM might help.
Yes, it "hurts" if empty blocks are out of sync. On obvious issue is
that you will get errors when scrubbing - the md layer has no way of
knowing that these are unimportant (assuming there is no no-sync
tracking), so any real problems will be hidden by the unimportant ones.
Another issue is for RMW cycles on raid5. Small writes are done by
reading the old data, reading the old parity, writing the new data and
the new parity - but that only works if the parity was correct across
the whole stripe. Even if raid5 TRIM is restricted to whole stripes, a
later small write to that stripe will be a disaster if it is not in sync.
Has anyone considered handling TRIM via an idle IO queue? You'd have to
purge queue items that conflicted with incoming writes, but it does get
around the performance complaint. If the idle period never comes, old
TRIMs can be silently dropped to lessen queue bloat.
I am sure it has been considered - but is it worth the effort and the
complications? TRIM has been implemented in several filesystems (ext4 and, I
believe, btrfs) - but is disabled by default because it typically slows down
the system. You are certainly correct that putting TRIM at the back of the
queue will avoid the delays it causes - but it still will not give any
significant benefit (except for old SSDs with limited garbage collection and
small over-provisioning ), and you have a lot of extra complexity to ensure
that a TRIM is never pushed back until after a new write to the same logical
sectors.
I think that you are vastly understating the need for discard support or what
your first hand experience is, so let me inject some facts into this thread
from working on this for several years (with vendors) :)
Overview:
* In Linux, we have "discard" support which vectors down into the device
appropriate method (TRIM for S-ATA, UNMAP/WRITE_SAME+UNMAP for SCSI, just
discard for various SW only block devices)
* There is support for inline discard in many file systems (ext4, xfs, btrfs,
gfs2, ...)
* There is support for "batched" discard (still online) via tools like fstrim
Every SSD device benefits from TRIM and the SSD companies test this code with
the upstream community.
In our testing with various devices, the inline (mount -o discard) can have a
performance impact so typically using the batched method is better.
For SCSI arrays (less an issue here on this list), the discard allows for
over-provisioning of LUN's.
Device mapper has support (newly added) for dm-thinp targets which can do the
same without hardware support.
It would be much easier and safer, and give much better effect, to make sure
the block allocation procedure for filesystems emphasised re-writing old
blocks as soon as possible (when on an SSD). Then there is no need for TRIM
at all. This would have the added benefit of working well for compressed (or
sparse) hard disk image files used by virtual machines - such image files only
take up real disk space for blocks that are written, so re-writes would save
real-world disk space.
Above you are mixing the need for TRIM (which allows devices like SSD's to do
wear levelling and performance tuning on physical blocks) with the virtual block
layout of SSD devices. Please keep in mind that the block space advertised out
to a file system is contiguous, but SSD's internally remapped the physical
blocks aggressively. Think of physical DRAM and your virtual memory layout.
Doing a naive always allocate and reuse the lowest block would have horrendous
performance impact on certain devices. Even on SSD's where seek is negligible,
having to do lots of small IO's instead of larger, contiguous IO's is much slower.
Regards,
Ric
quoted
As far as parity consistency, bitmaps could track which stripes( and
blocks within those stripes) are expected to be out of parity( also
useful for lazy device init ). Maybe a bit-per-stripe map at the logical
device level and a bit-per-LBA bitmap at the stripe level?
Tracking "no-sync" areas of a raid array is already high on the md raid
things-to-do list (perhaps it is already implemented - I lose track of which
features are planned and which are implemented). And yes, such no-sync
tracking would be useful here. But it is complicated, especially for raid5/6
(raid1 is not too bad) - should TRIMs that cover part of a stripe be dropped?
Should the md layer remember them and coalesce them when it can TRIM a whole
stripe? Should it try to track partial synchronisation within a stripe?
Or should the md developers simply say that since supporting TRIM is not going
to have any measurable benefits (certainly not with the sort of SSD's people
use in raid arrays), and since TRIM slows down some operations, it is better
to keep things simple and ignore TRIM entirely? Even if there are occasional
benefits to having TRIM, is it worth it in the face of added complication in
the code and the risk of errors?
There /have/ been developers working on TRIM support on raid5. It seems to
have been a complicated process. But some people like a challenge!
quoted
On the other hand, does it hurt if empty blocks are out of parity( due
to TRIM or lazy device init)? The parity recovery of garbage is still
garbage, which is what any sane FS expects from unused blocks. If and
when you do a parity scrub, you will spend a lot of time recovering
garbage and undo any good TRIM might have done, but usual drive
operation should quickly balance that out in a write-intensive
environment where idle TRIM might help.
Yes, it "hurts" if empty blocks are out of sync. On obvious issue is that you
will get errors when scrubbing - the md layer has no way of knowing that these
are unimportant (assuming there is no no-sync tracking), so any real problems
will be hidden by the unimportant ones.
Another issue is for RMW cycles on raid5. Small writes are done by reading
the old data, reading the old parity, writing the new data and the new parity
- but that only works if the parity was correct across the whole stripe. Even
if raid5 TRIM is restricted to whole stripes, a later small write to that
stripe will be a disaster if it is not in sync.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
From: David Brown <hidden> Date: 2012-11-13 15:13:36
On 13/11/2012 14:39, Ric Wheeler wrote:
On 10/31/2012 10:11 AM, David Brown wrote:
quoted
On 31/10/2012 14:12, Alexander Haase wrote:
quoted
Has anyone considered handling TRIM via an idle IO queue? You'd have to
purge queue items that conflicted with incoming writes, but it does get
around the performance complaint. If the idle period never comes, old
TRIMs can be silently dropped to lessen queue bloat.
I am sure it has been considered - but is it worth the effort and the
complications? TRIM has been implemented in several filesystems (ext4
and, I believe, btrfs) - but is disabled by default because it
typically slows down the system. You are certainly correct that
putting TRIM at the back of the queue will avoid the delays it causes
- but it still will not give any significant benefit (except for old
SSDs with limited garbage collection and small over-provisioning ),
and you have a lot of extra complexity to ensure that a TRIM is never
pushed back until after a new write to the same logical sectors.
I think that you are vastly understating the need for discard support or
what your first hand experience is, so let me inject some facts into
this thread from working on this for several years (with vendors) :)
That is quite possible - my experience is limited. My aim in this
discussion is not to say that TRIM should be ignored completely, but to
ask if it really is necessary, and if its benefits outweigh its
disadvantages and the added complexity. I am trying to dispel the
widely held myths that TRIM is essential, that SSDs are painfully slow
without it, that SSDs do not work with RAID because RAID does not
support TRIM, and that you must always enable TRIM (and "discard" mount
options) to get the best from your SSDs.
Nothing makes me happier here than seeing someone with strong experience
from multiple vendors bringing in some facts - so thank you for your
comments and help here.
Overview:
* In Linux, we have "discard" support which vectors down into the device
appropriate method (TRIM for S-ATA, UNMAP/WRITE_SAME+UNMAP for SCSI,
just discard for various SW only block devices)
* There is support for inline discard in many file systems (ext4, xfs,
btrfs, gfs2, ...)
* There is support for "batched" discard (still online) via tools like
fstrim
OK.
Every SSD device benefits from TRIM and the SSD companies test this code
with the upstream community.
In our testing with various devices, the inline (mount -o discard) can
have a performance impact so typically using the batched method is better.
I am happy to see you confirm this. I think fstrim is a much more
practical choice than inline trim for many uses (with SATA SSD's at
least - SCSI/SAS SSD's have better "trim" equivalents with less
performance impact, since they can be queued). I also think fstrim will
work better along with RAID and other layered systems, since it will
have fewer, larger TRIMs and allow the RAID system to trim whole stripes
at a time (and just drop any leftovers).
For SCSI arrays (less an issue here on this list), the discard allows
for over-provisioning of LUN's.
Device mapper has support (newly added) for dm-thinp targets which can
do the same without hardware support.
quoted
It would be much easier and safer, and give much better effect, to
make sure the block allocation procedure for filesystems emphasised
re-writing old blocks as soon as possible (when on an SSD). Then
there is no need for TRIM at all. This would have the added benefit
of working well for compressed (or sparse) hard disk image files used
by virtual machines - such image files only take up real disk space
for blocks that are written, so re-writes would save real-world disk
space.
Above you are mixing the need for TRIM (which allows devices like SSD's
to do wear levelling and performance tuning on physical blocks) with the
virtual block layout of SSD devices. Please keep in mind that the block
space advertised out to a file system is contiguous, but SSD's
internally remapped the physical blocks aggressively. Think of physical
DRAM and your virtual memory layout.
I don't think I am mixing these concepts - but I might well be
expressing myself badly.
Suppose the disk has logical blocks log000 to log499, and physical
blocks phy000 to phy599. The filesystem sees 500 blocks, which the
SSD's firmware maps onto the 600 physical blocks as needed (20%
overprovisioning). We start off with a blank SSD.
The filesystem writes out a file to blocks log000 through log009. The
SSD has to map these to physical blocks, and picks phy000 through phy009.
Then the filesystem deletes that file. Logical blocks log000 to log009
are now free for re-use by the filesystem. But without TRIM, the SSD
does not know that - so it must preserve phy000 to phy009.
Then the filesystem writes a new 10-block file. If it picks log010 to
log019 for the logical blocks, then the SSD will write them to phy010
through phy019. Everything works fine, but the SSD is carrying around
these extra physical blocks that it believes are important, because they
are still mapped to logical blocks log000 to log009, and the SSD does
not know they are now unused.
But if instead the filesystem wrote the new file to log000 to log009, we
would have a different case. The SSD would again allocate phy010 to
phy019, since it needs to use blank blocks. But now the SSD has changed
the mapping for log000 to phy010 instead of phy000, and knows that
physical blocks phy000 to phy009 are not needed - without a logical
block mapping, they cannot be accessed by the file system. So these
physical blocks can be re-cycled in exactly the same manner as if they
were TRIM'ed.
In this way, if the filesystem is careful about re-using free logical
blocks (rather than aiming for low fragmentation and contiguous block
allocation, as done for hard disk speed), there is no need for TRIM.
The only benefit of TRIM is to move the recycling process to a slightly
earlier stage - but I believe that effect would be negligible with
appropriate overprovisioning.
That's my theory, anyway.
Doing a naive always allocate and reuse the lowest block would have
horrendous performance impact on certain devices. Even on SSD's where
seek is negligible, having to do lots of small IO's instead of larger,
contiguous IO's is much slower.
Clearly the allocation algorithms would have to be different for SSDs
and hard disks (and I realise this complicates matters - an aim with the
block device system is to keep things device independent when possible.
There is always someone who wants to make a three-way raid1 mirror
from an SSD, a hard disk partition, and a block of memory exported by
iSCSI from a remote server - and it is great that they can do so). And
clearly having lots of small IOs will increase overheads and reduce any
performance benefits. But somewhere here is the possibility to bias the
filesystems' allocation schemes towards reuse, giving most of the
benefits of TRIM "for free".
It may also be the case that filesystems already do this, and I am
recommending a re-invention of a wheel that is already optimised -
obviously you will know that far better than me. I am just trying to
come up with helpful ideas.
mvh.,
David
Regards,
Ric
quoted
quoted
As far as parity consistency, bitmaps could track which stripes( and
blocks within those stripes) are expected to be out of parity( also
useful for lazy device init ). Maybe a bit-per-stripe map at the logical
device level and a bit-per-LBA bitmap at the stripe level?
Tracking "no-sync" areas of a raid array is already high on the md
raid things-to-do list (perhaps it is already implemented - I lose
track of which features are planned and which are implemented). And
yes, such no-sync tracking would be useful here. But it is
complicated, especially for raid5/6 (raid1 is not too bad) - should
TRIMs that cover part of a stripe be dropped? Should the md layer
remember them and coalesce them when it can TRIM a whole stripe?
Should it try to track partial synchronisation within a stripe?
Or should the md developers simply say that since supporting TRIM is
not going to have any measurable benefits (certainly not with the sort
of SSD's people use in raid arrays), and since TRIM slows down some
operations, it is better to keep things simple and ignore TRIM
entirely? Even if there are occasional benefits to having TRIM, is it
worth it in the face of added complication in the code and the risk of
errors?
There /have/ been developers working on TRIM support on raid5. It
seems to have been a complicated process. But some people like a
challenge!
quoted
On the other hand, does it hurt if empty blocks are out of parity( due
to TRIM or lazy device init)? The parity recovery of garbage is still
garbage, which is what any sane FS expects from unused blocks. If and
when you do a parity scrub, you will spend a lot of time recovering
garbage and undo any good TRIM might have done, but usual drive
operation should quickly balance that out in a write-intensive
environment where idle TRIM might help.
Yes, it "hurts" if empty blocks are out of sync. On obvious issue is
that you will get errors when scrubbing - the md layer has no way of
knowing that these are unimportant (assuming there is no no-sync
tracking), so any real problems will be hidden by the unimportant ones.
Another issue is for RMW cycles on raid5. Small writes are done by
reading the old data, reading the old parity, writing the new data and
the new parity - but that only works if the parity was correct across
the whole stripe. Even if raid5 TRIM is restricted to whole stripes,
a later small write to that stripe will be a disaster if it is not in
sync.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Has anyone considered handling TRIM via an idle IO queue? You'd have to
purge queue items that conflicted with incoming writes, but it does get
around the performance complaint. If the idle period never comes, old
TRIMs can be silently dropped to lessen queue bloat.
I am sure it has been considered - but is it worth the effort and the
complications? TRIM has been implemented in several filesystems (ext4
and, I believe, btrfs) - but is disabled by default because it
typically slows down the system. You are certainly correct that
putting TRIM at the back of the queue will avoid the delays it causes
- but it still will not give any significant benefit (except for old
SSDs with limited garbage collection and small over-provisioning ),
and you have a lot of extra complexity to ensure that a TRIM is never
pushed back until after a new write to the same logical sectors.
I think that you are vastly understating the need for discard support or
what your first hand experience is, so let me inject some facts into
this thread from working on this for several years (with vendors) :)
That is quite possible - my experience is limited. My aim in this discussion
is not to say that TRIM should be ignored completely, but to ask if it really
is necessary, and if its benefits outweigh its disadvantages and the added
complexity. I am trying to dispel the widely held myths that TRIM is
essential, that SSDs are painfully slow without it, that SSDs do not work with
RAID because RAID does not support TRIM, and that you must always enable TRIM
(and "discard" mount options) to get the best from your SSDs.
It really is required, the question and challenge is how to use it correctly and
how to use the right technique on the right device.
If you have an extremely light workload on any device (an SSD in a laptop used
for web browsing?), this probably won't matter for a long time but also would
not impact your performance much since you are not pushing a lot of IO :)
Nothing makes me happier here than seeing someone with strong experience from
multiple vendors bringing in some facts - so thank you for your comments and
help here.
quoted
Overview:
* In Linux, we have "discard" support which vectors down into the device
appropriate method (TRIM for S-ATA, UNMAP/WRITE_SAME+UNMAP for SCSI,
just discard for various SW only block devices)
* There is support for inline discard in many file systems (ext4, xfs,
btrfs, gfs2, ...)
* There is support for "batched" discard (still online) via tools like
fstrim
OK.
quoted
Every SSD device benefits from TRIM and the SSD companies test this code
with the upstream community.
In our testing with various devices, the inline (mount -o discard) can
have a performance impact so typically using the batched method is better.
I am happy to see you confirm this. I think fstrim is a much more practical
choice than inline trim for many uses (with SATA SSD's at least - SCSI/SAS
SSD's have better "trim" equivalents with less performance impact, since they
can be queued). I also think fstrim will work better along with RAID and
other layered systems, since it will have fewer, larger TRIMs and allow the
RAID system to trim whole stripes at a time (and just drop any leftovers).
The basic observation - again important to note that this is for S-ATA devices,
not all discard enabled devices - is that an ATA_TRIM command takes about the
same time regardless of the size being trimmed. It is also currently a
non-queued command, so we shut down NCQ (draining the queue for a S-ATA device
on each command).
Basically a good idea for S-ATA to use fewer commands to minimize that impact.
The standards body T13 is thinking about fixing the non-queueable issue so this
might improve.
Note again, there a loads of other device types where this is not such an impact.
As a footnote, if you want to see the various bits of capability we scrape out
of devices, we put a lot of information into /sys/block/sda/queue (discard
support, etc).
quoted
For SCSI arrays (less an issue here on this list), the discard allows
for over-provisioning of LUN's.
Device mapper has support (newly added) for dm-thinp targets which can
do the same without hardware support.
quoted
It would be much easier and safer, and give much better effect, to
make sure the block allocation procedure for filesystems emphasised
re-writing old blocks as soon as possible (when on an SSD). Then
there is no need for TRIM at all. This would have the added benefit
of working well for compressed (or sparse) hard disk image files used
by virtual machines - such image files only take up real disk space
for blocks that are written, so re-writes would save real-world disk
space.
Above you are mixing the need for TRIM (which allows devices like SSD's
to do wear levelling and performance tuning on physical blocks) with the
virtual block layout of SSD devices. Please keep in mind that the block
space advertised out to a file system is contiguous, but SSD's
internally remapped the physical blocks aggressively. Think of physical
DRAM and your virtual memory layout.
I don't think I am mixing these concepts - but I might well be expressing
myself badly.
Suppose the disk has logical blocks log000 to log499, and physical blocks
phy000 to phy599. The filesystem sees 500 blocks, which the SSD's firmware
maps onto the 600 physical blocks as needed (20% overprovisioning). We start
off with a blank SSD.
The filesystem writes out a file to blocks log000 through log009. The SSD has
to map these to physical blocks, and picks phy000 through phy009.
Then the filesystem deletes that file. Logical blocks log000 to log009 are
now free for re-use by the filesystem. But without TRIM, the SSD does not
know that - so it must preserve phy000 to phy009.
Then the filesystem writes a new 10-block file. If it picks log010 to log019
for the logical blocks, then the SSD will write them to phy010 through
phy019. Everything works fine, but the SSD is carrying around these extra
physical blocks that it believes are important, because they are still mapped
to logical blocks log000 to log009, and the SSD does not know they are now
unused.
But if instead the filesystem wrote the new file to log000 to log009, we would
have a different case. The SSD would again allocate phy010 to phy019, since
it needs to use blank blocks. But now the SSD has changed the mapping for
log000 to phy010 instead of phy000, and knows that physical blocks phy000 to
phy009 are not needed - without a logical block mapping, they cannot be
accessed by the file system. So these physical blocks can be re-cycled in
exactly the same manner as if they were TRIM'ed.
In this way, if the filesystem is careful about re-using free logical blocks
(rather than aiming for low fragmentation and contiguous block allocation, as
done for hard disk speed), there is no need for TRIM. The only benefit of TRIM
is to move the recycling process to a slightly earlier stage - but I believe
that effect would be negligible with appropriate overprovisioning.
That's my theory, anyway.
I think that any assumptions about the logical layout for an SSD mapping into
the same physical layout is optimistic.
Still not clear to me why you are trying to combine the two concepts.
Putting things together (contiguous allocation) in your virtual address space
(block space) is good since you can allocate larger IO's to get your file read
from the device into DRAM.
Letting the target storage device know what is used/unused (discard) is totally
unrelated. It allows the device to optimize/garbage collect/wear level/etc.
Not that simple allocation schemes are a bad idea for SSD's (why work harder
than you need to, avoid wasting CPU cycles, etc), but it is not tied into
discard or not.
If you want to see a lot of hard data on SSD's, there is a fairly solid body of
work published at USENIX FAST conferences (www.usenix.org) including work on
various firmware ideas, testing, etc.
quoted
Doing a naive always allocate and reuse the lowest block would have
horrendous performance impact on certain devices. Even on SSD's where
seek is negligible, having to do lots of small IO's instead of larger,
contiguous IO's is much slower.
Clearly the allocation algorithms would have to be different for SSDs and hard
disks (and I realise this complicates matters - an aim with the block device
system is to keep things device independent when possible. There is always
someone who wants to make a three-way raid1 mirror from an SSD, a hard disk
partition, and a block of memory exported by iSCSI from a remote server - and
it is great that they can do so). And clearly having lots of small IOs will
increase overheads and reduce any performance benefits. But somewhere here is
the possibility to bias the filesystems' allocation schemes towards reuse,
giving most of the benefits of TRIM "for free".
It may also be the case that filesystems already do this, and I am
recommending a re-invention of a wheel that is already optimised - obviously
you will know that far better than me. I am just trying to come up with
helpful ideas.
mvh.,
David
It is unfortunately not just SSD's versus S-ATA spindles. We have SAS SSD's,
PCI-e SSD's, enterprise arrays (SCSI luns), consumer S-ATA SSDs and software
only discard enabled devices.
We do work hard to deduce the generic type of the device (again, see the
/sys/block information) but we need to be careful not to spin into a ton of
device specific algorithms :)
Ric