On Thu, Aug 13, 2009 at 12:33 PM, [off-list ref] wrote:
On Thu, 13 Aug 2009, Markus Trippelsdorf wrote:
quoted
On Thu, Aug 13, 2009 at 08:13:12AM -0700, Matthew Wilcox wrote:
quoted
I am planning a complete overhaul of the discard work. Users can send
down discard requests as frequently as they like. The block layer will
cache them, and invalidate them if writes come through. Periodically,
the block layer will send down a TRIM or an UNMAP (depending on the
underlying device) and get rid of the blocks that have remained unwanted
in the interim.
That is a very good idea. I've tested your original TRIM implementation on
my Vertex yesterday and it was awful ;-). The SSD needs hundreds of
milliseconds to digest a single TRIM command. And since your
implementation
sends a TRIM for each extent of each deleted file, the whole system is
unusable after a short while.
An optimal solution would be to consolidate the discard requests, bundle
them and send them to the drive as infrequent as possible.
or queue them up and send them when the drive is idle (you would need to
keep track to make sure the space isn't re-used)
as an example, if you would consider spinning down a drive you don't hurt
performance by sending accumulated trim commands.
David Lang
An alternate approach is the block layer maintain its own bitmap of
used unused sectors / blocks. Unmap commands from the filesystem just
cause the bitmap to be updated. No other effect.
(Big unknown: Where will the bitmap live between reboots? Require DM
volumes so we can have a dedicated bitmap volume in the mix to store
the bitmap to? Maybe on mount, the filesystem has to be scanned to
initially populate the bitmap? Other options?)
Assuming we have a persistent bitmap in place, have a background
scanner that kicks in when the cpu / disk is idle. It just
continuously scans the bitmap looking for contiguous blocks of unused
sectors. Each time it finds one, it sends the largest possible unmap
down the block stack and eventually to the device.
When normal cpu / disk activity kicks in, this process goes to sleep.
That way much of the smarts are concentrated in the block layer, not
in the filesystem code. And it is being done when the disk is
otherwise idle, so you don't have the ncq interference.
Even laptop users should have enough idle cpu available to manage
this. Enterprise would get the large discards it wants, and
unmentioned in the previous discussion, mdraid gets the large discards
it also wants.
ie. If a mdraid raid5/raid6 volume is built of SSDs, it will only be
able to discard a full stripe at a time. Otherwise the P=D1 ^ D2 logic
is lost.
Another benefit of the above is the code should be extremely safe and testable.
Greg
--
Greg Freemyer
Head of EDD Tape Extraction and Processing team
Litigation Triage Solutions Specialist
http://www.linkedin.com/in/gregfreemyer
Preservation and Forensic processing of Exchange Repositories White Paper -
<http://www.norcrossgroup.com/forms/whitepapers/tng_whitepaper_fpe.html>
The Norcross Group
The Intersection of Evidence & Technology
http://www.norcrossgroup.com
From: James Bottomley <James.Bottomley@HansenPartnership.com> Date: 2009-08-13 19:18:21
On Thu, 2009-08-13 at 14:15 -0400, Greg Freemyer wrote:
On Thu, Aug 13, 2009 at 12:33 PM, [off-list ref] wrote:
quoted
On Thu, 13 Aug 2009, Markus Trippelsdorf wrote:
quoted
On Thu, Aug 13, 2009 at 08:13:12AM -0700, Matthew Wilcox wrote:
quoted
I am planning a complete overhaul of the discard work. Users can send
down discard requests as frequently as they like. The block layer will
cache them, and invalidate them if writes come through. Periodically,
the block layer will send down a TRIM or an UNMAP (depending on the
underlying device) and get rid of the blocks that have remained unwanted
in the interim.
That is a very good idea. I've tested your original TRIM implementation on
my Vertex yesterday and it was awful ;-). The SSD needs hundreds of
milliseconds to digest a single TRIM command. And since your
implementation
sends a TRIM for each extent of each deleted file, the whole system is
unusable after a short while.
An optimal solution would be to consolidate the discard requests, bundle
them and send them to the drive as infrequent as possible.
or queue them up and send them when the drive is idle (you would need to
keep track to make sure the space isn't re-used)
as an example, if you would consider spinning down a drive you don't hurt
performance by sending accumulated trim commands.
David Lang
An alternate approach is the block layer maintain its own bitmap of
used unused sectors / blocks. Unmap commands from the filesystem just
cause the bitmap to be updated. No other effect.
(Big unknown: Where will the bitmap live between reboots? Require DM
volumes so we can have a dedicated bitmap volume in the mix to store
the bitmap to? Maybe on mount, the filesystem has to be scanned to
initially populate the bitmap? Other options?)
I wouldn't really have it live anywhere. Discard is best effort; it's
not required for fs integrity. As long as we don't discard an in-use
block we're free to do anything else (including forget to discard,
rediscard a discarded block etc).
It is theoretically possible to run all of this from user space using
the fs mappings, a bit like a defrag command.
One other option would just be to scan on mount, discard everything
empty and redo on next mount ... this might be just the thing for
laptops.
Assuming we have a persistent bitmap in place, have a background
scanner that kicks in when the cpu / disk is idle. It just
continuously scans the bitmap looking for contiguous blocks of unused
sectors. Each time it finds one, it sends the largest possible unmap
down the block stack and eventually to the device.
When normal cpu / disk activity kicks in, this process goes to sleep.
That way much of the smarts are concentrated in the block layer, not
in the filesystem code. And it is being done when the disk is
otherwise idle, so you don't have the ncq interference.
Even laptop users should have enough idle cpu available to manage
this. Enterprise would get the large discards it wants, and
unmentioned in the previous discussion, mdraid gets the large discards
it also wants.
ie. If a mdraid raid5/raid6 volume is built of SSDs, it will only be
able to discard a full stripe at a time. Otherwise the P=D1 ^ D2 logic
is lost.
Another benefit of the above is the code should be extremely safe and testable.
Actually, I think, if we go in-kernel, the discard might be better tied
into the block plugging mechanism. The real test might be no
outstanding commands and queue plugged, keep plugged and begin
discarding.
James
From: Richard Sharpe <hidden> Date: 2009-08-13 20:31:20
On Thu, Aug 13, 2009 at 12:18 PM, James Bottomley
[off-list ref] wrote:
Actually, I think, if we go in-kernel, the discard might be better tied
into the block plugging mechanism. The real test might be no
outstanding commands and queue plugged, keep plugged and begin
discarding.
I am very interested in this topic, as I have implemented UNMAP
support in SCST and scst_local.c and ib_srp.c for one SSD vendor, as
well as the block layer changes to have it work correctly (they were
minor changes and based on Matthew's original TRIM or UNMAP patch from
long ago). I believe that the performance was acceptable for them (I
will have to check).
I am also working on other, non-SSD, devices that are in a lower price
range than the large storage arrays where both DISCARD/UNMAP (and
WRITE SAME) would be useful in Linux. It also seems that Microsoft
supports TRIM in Windows 7 if you switch it on, although that really
only implies we should implement UNMAP support in our firmware and
hook it up to existing mechanisms.
I have logged internal enhancement bugs in bugzilla asking for both
TRIM and UNMAP/WRITE SAME support, and although one environment is
iSCSI in userland, and thus can be dealt with without support in the
Linux kernel, there are use cases where DISCARD/UNMAP support in the
Linux kernel would be useful.
I would be very willing to make the firmware changes needed in our
device to support UNMAP/WRITE SAME and to test changes to the Linux
kernel to support same.
I will go through this thread in more detail when I get back from my
trip to Australia, but if there are any GIT trees around with nascent
support in them I would love to know about them, as it will help my
internal efforts to get UNMAP/WRITE SAME support implemented as well.
--
Regards,
Richard Sharpe
On Thu, Aug 13, 2009 at 12:33 PM, [off-list ref] wrote:
quoted
On Thu, 13 Aug 2009, Markus Trippelsdorf wrote:
quoted
On Thu, Aug 13, 2009 at 08:13:12AM -0700, Matthew Wilcox wrote:
quoted
I am planning a complete overhaul of the discard work. Users can send
down discard requests as frequently as they like. The block layer will
cache them, and invalidate them if writes come through. Periodically,
the block layer will send down a TRIM or an UNMAP (depending on the
underlying device) and get rid of the blocks that have remained unwanted
in the interim.
That is a very good idea. I've tested your original TRIM implementation on
my Vertex yesterday and it was awful ;-). The SSD needs hundreds of
milliseconds to digest a single TRIM command. And since your
implementation
sends a TRIM for each extent of each deleted file, the whole system is
unusable after a short while.
An optimal solution would be to consolidate the discard requests, bundle
them and send them to the drive as infrequent as possible.
or queue them up and send them when the drive is idle (you would need to
keep track to make sure the space isn't re-used)
as an example, if you would consider spinning down a drive you don't hurt
performance by sending accumulated trim commands.
David Lang
An alternate approach is the block layer maintain its own bitmap of
used unused sectors / blocks. Unmap commands from the filesystem just
cause the bitmap to be updated. No other effect.
how does the block layer know what blocks are unused by the filesystem?
or would it be a case of the filesystem generating discard/trim requests
to the block layer so that it can maintain it's bitmap, and then the block
layer generating the requests to the drive below it?
David Lang
(Big unknown: Where will the bitmap live between reboots? Require DM
volumes so we can have a dedicated bitmap volume in the mix to store
the bitmap to? Maybe on mount, the filesystem has to be scanned to
initially populate the bitmap? Other options?)
Assuming we have a persistent bitmap in place, have a background
scanner that kicks in when the cpu / disk is idle. It just
continuously scans the bitmap looking for contiguous blocks of unused
sectors. Each time it finds one, it sends the largest possible unmap
down the block stack and eventually to the device.
When normal cpu / disk activity kicks in, this process goes to sleep.
That way much of the smarts are concentrated in the block layer, not
in the filesystem code. And it is being done when the disk is
otherwise idle, so you don't have the ncq interference.
Even laptop users should have enough idle cpu available to manage
this. Enterprise would get the large discards it wants, and
unmentioned in the previous discussion, mdraid gets the large discards
it also wants.
ie. If a mdraid raid5/raid6 volume is built of SSDs, it will only be
able to discard a full stripe at a time. Otherwise the P=D1 ^ D2 logic
is lost.
Another benefit of the above is the code should be extremely safe and testable.
Greg
On Thu, Aug 13, 2009 at 4:44 PM, [off-list ref] wrote:
On Thu, 13 Aug 2009, Greg Freemyer wrote:
quoted
On Thu, Aug 13, 2009 at 12:33 PM, [off-list ref] wrote:
quoted
On Thu, 13 Aug 2009, Markus Trippelsdorf wrote:
quoted
On Thu, Aug 13, 2009 at 08:13:12AM -0700, Matthew Wilcox wrote:
quoted
I am planning a complete overhaul of the discard work. Users can send
down discard requests as frequently as they like. The block layer will
cache them, and invalidate them if writes come through. Periodically,
the block layer will send down a TRIM or an UNMAP (depending on the
underlying device) and get rid of the blocks that have remained
unwanted
in the interim.
That is a very good idea. I've tested your original TRIM implementation
on
my Vertex yesterday and it was awful ;-). The SSD needs hundreds of
milliseconds to digest a single TRIM command. And since your
implementation
sends a TRIM for each extent of each deleted file, the whole system is
unusable after a short while.
An optimal solution would be to consolidate the discard requests, bundle
them and send them to the drive as infrequent as possible.
or queue them up and send them when the drive is idle (you would need to
keep track to make sure the space isn't re-used)
as an example, if you would consider spinning down a drive you don't hurt
performance by sending accumulated trim commands.
David Lang
An alternate approach is the block layer maintain its own bitmap of
used unused sectors / blocks. Unmap commands from the filesystem just
cause the bitmap to be updated. No other effect.
how does the block layer know what blocks are unused by the filesystem?
or would it be a case of the filesystem generating discard/trim requests to
the block layer so that it can maintain it's bitmap, and then the block
layer generating the requests to the drive below it?
Perhaps an interface (ioctl, etc) can be added to ask a filesystem to
discard all unused blocks in a certain range? (That is, have the
filesystem validate the request under any necessary locks before
passing it to the block IO layer)
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
On Thu, Aug 13, 2009 at 4:44 PM, [off-list ref] wrote:
On Thu, 13 Aug 2009, Greg Freemyer wrote:
quoted
On Thu, Aug 13, 2009 at 12:33 PM, [off-list ref] wrote:
quoted
On Thu, 13 Aug 2009, Markus Trippelsdorf wrote:
quoted
On Thu, Aug 13, 2009 at 08:13:12AM -0700, Matthew Wilcox wrote:
quoted
I am planning a complete overhaul of the discard work. Users can send
down discard requests as frequently as they like. The block layer will
cache them, and invalidate them if writes come through. Periodically,
the block layer will send down a TRIM or an UNMAP (depending on the
underlying device) and get rid of the blocks that have remained
unwanted
in the interim.
That is a very good idea. I've tested your original TRIM implementation
on
my Vertex yesterday and it was awful ;-). The SSD needs hundreds of
milliseconds to digest a single TRIM command. And since your
implementation
sends a TRIM for each extent of each deleted file, the whole system is
unusable after a short while.
An optimal solution would be to consolidate the discard requests, bundle
them and send them to the drive as infrequent as possible.
or queue them up and send them when the drive is idle (you would need to
keep track to make sure the space isn't re-used)
as an example, if you would consider spinning down a drive you don't hurt
performance by sending accumulated trim commands.
David Lang
An alternate approach is the block layer maintain its own bitmap of
used unused sectors / blocks. Unmap commands from the filesystem just
cause the bitmap to be updated. No other effect.
how does the block layer know what blocks are unused by the filesystem?
or would it be a case of the filesystem generating discard/trim requests to
the block layer so that it can maintain it's bitmap, and then the block
layer generating the requests to the drive below it?
David Lang
Yes, my thought.was that block layer would consume the discard/trim
requests from the filesystem in realtime to maintain the bitmap, then
at some later point in time when the system has extra resources it
would generate the calls down to the lower layers and eventually the
drive.
I highlight the lower layers because mdraid is also going to have to
be in the mix if raid5/6 is in use. ie. At a minimum it will have to
adjust the block range to align with the stripe boundaries.
Greg
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Richard Sharpe <hidden> Date: 2009-08-13 22:20:42
On Thu, Aug 13, 2009 at 2:28 PM, Greg Freemyer[off-list ref] wrote:
On Thu, Aug 13, 2009 at 4:44 PM, [off-list ref] wrote:
quoted
On Thu, 13 Aug 2009, Greg Freemyer wrote:
quoted
On Thu, Aug 13, 2009 at 12:33 PM, [off-list ref] wrote:
quoted
On Thu, 13 Aug 2009, Markus Trippelsdorf wrote:
quoted
On Thu, Aug 13, 2009 at 08:13:12AM -0700, Matthew Wilcox wrote:
quoted
I am planning a complete overhaul of the discard work. Users can send
down discard requests as frequently as they like. The block layer will
cache them, and invalidate them if writes come through. Periodically,
the block layer will send down a TRIM or an UNMAP (depending on the
underlying device) and get rid of the blocks that have remained
unwanted
in the interim.
That is a very good idea. I've tested your original TRIM implementation
on
my Vertex yesterday and it was awful ;-). The SSD needs hundreds of
milliseconds to digest a single TRIM command. And since your
implementation
sends a TRIM for each extent of each deleted file, the whole system is
unusable after a short while.
An optimal solution would be to consolidate the discard requests, bundle
them and send them to the drive as infrequent as possible.
or queue them up and send them when the drive is idle (you would need to
keep track to make sure the space isn't re-used)
as an example, if you would consider spinning down a drive you don't hurt
performance by sending accumulated trim commands.
David Lang
An alternate approach is the block layer maintain its own bitmap of
used unused sectors / blocks. Unmap commands from the filesystem just
cause the bitmap to be updated. No other effect.
how does the block layer know what blocks are unused by the filesystem?
or would it be a case of the filesystem generating discard/trim requests to
the block layer so that it can maintain it's bitmap, and then the block
layer generating the requests to the drive below it?
David Lang
Yes, my thought.was that block layer would consume the discard/trim
requests from the filesystem in realtime to maintain the bitmap, then
at some later point in time when the system has extra resources it
would generate the calls down to the lower layers and eventually the
drive.
Why should the block layer be forced to maintain something that is
probably of use for only a limited number of cases? For example, the
devices I work on already maintain their own mapping of HOST-visible
LBAs to underlying storage, and I suspect that most such devices do.
So, you are duplicating something that we already do, and there is no
way that I am aware of to synchronise the two.
All we really need, I believe is for the UNMAP requests to come down
to us with writes barriered until we respond, and it is a relatively
cheap operation, although writes that are already in the cache and
uncommitted to disk present some issues if an UNMAP request comes down
for recently written blocks.
I highlight the lower layers because mdraid is also going to have to
be in the mix if raid5/6 is in use. ie. At a minimum it will have to
adjust the block range to align with the stripe boundaries.
Greg
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu, Aug 13, 2009 at 6:20 PM, Richard
Sharpe[off-list ref] wrote:
On Thu, Aug 13, 2009 at 2:28 PM, Greg Freemyer[off-list ref] wrote:
quoted
On Thu, Aug 13, 2009 at 4:44 PM, [off-list ref] wrote:
quoted
On Thu, 13 Aug 2009, Greg Freemyer wrote:
quoted
On Thu, Aug 13, 2009 at 12:33 PM, [off-list ref] wrote:
quoted
On Thu, 13 Aug 2009, Markus Trippelsdorf wrote:
quoted
On Thu, Aug 13, 2009 at 08:13:12AM -0700, Matthew Wilcox wrote:
quoted
I am planning a complete overhaul of the discard work. Users can send
down discard requests as frequently as they like. The block layer will
cache them, and invalidate them if writes come through. Periodically,
the block layer will send down a TRIM or an UNMAP (depending on the
underlying device) and get rid of the blocks that have remained
unwanted
in the interim.
That is a very good idea. I've tested your original TRIM implementation
on
my Vertex yesterday and it was awful ;-). The SSD needs hundreds of
milliseconds to digest a single TRIM command. And since your
implementation
sends a TRIM for each extent of each deleted file, the whole system is
unusable after a short while.
An optimal solution would be to consolidate the discard requests, bundle
them and send them to the drive as infrequent as possible.
or queue them up and send them when the drive is idle (you would need to
keep track to make sure the space isn't re-used)
as an example, if you would consider spinning down a drive you don't hurt
performance by sending accumulated trim commands.
David Lang
An alternate approach is the block layer maintain its own bitmap of
used unused sectors / blocks. Unmap commands from the filesystem just
cause the bitmap to be updated. No other effect.
how does the block layer know what blocks are unused by the filesystem?
or would it be a case of the filesystem generating discard/trim requests to
the block layer so that it can maintain it's bitmap, and then the block
layer generating the requests to the drive below it?
David Lang
Yes, my thought.was that block layer would consume the discard/trim
requests from the filesystem in realtime to maintain the bitmap, then
at some later point in time when the system has extra resources it
would generate the calls down to the lower layers and eventually the
drive.
Why should the block layer be forced to maintain something that is
probably of use for only a limited number of cases? For example, the
devices I work on already maintain their own mapping of HOST-visible
LBAs to underlying storage, and I suspect that most such devices do.
So, you are duplicating something that we already do, and there is no
way that I am aware of to synchronise the two.
All we really need, I believe is for the UNMAP requests to come down
to us with writes barriered until we respond, and it is a relatively
cheap operation, although writes that are already in the cache and
uncommitted to disk present some issues if an UNMAP request comes down
for recently written blocks.
Richard,
Quoting the original email I saw in this thread:
The unfortunate thing about the TRIM command is that it's not NCQ, so
all NCQ commands have to finish, then we can send the TRIM command and
wait for it to finish, then we can send NCQ commands again.
So TRIM isn't free, and there's a better way for the drive to find
out that the contents of a block no longer matter -- write some new
data to it. So if we just swapped a page in, and we're going to swap
something else back out again soon, just write it to the same location
instead of to a fresh location. You've saved a command, and you've
saved the drive some work, plus you've allowed other users to continue
accessing the drive in the meantime.
I am planning a complete overhaul of the discard work. Users can send
down discard requests as frequently as they like. The block layer will
cache them, and invalidate them if writes come through. Periodically,
the block layer will send down a TRIM or an UNMAP (depending on the
underlying device) and get rid of the blocks that have remained unwanted
in the interim.
Thoughts on that are welcome.
quoted
My thought was that a bitmap was a better solution than a cache of
discard commands.
One of the biggest reasons is that a bitmap can coalesce the unused
areas into much larger discard ranges than a queue that will only have
a limited number of discards to coalesce.
And both Enterprise scsi and mdraid are desirous of larger discard ranges.
Greg
--
Greg Freemyer
Head of EDD Tape Extraction and Processing team
Litigation Triage Solutions Specialist
http://www.linkedin.com/in/gregfreemyer
Preservation and Forensic processing of Exchange Repositories White Paper -
<http://www.norcrossgroup.com/forms/whitepapers/tng_whitepaper_fpe.html>
The Norcross Group
The Intersection of Evidence & Technology
http://www.norcrossgroup.com
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Mark Lord <hidden> Date: 2009-08-14 22:03:14
James Bottomley wrote:
On Thu, 2009-08-13 at 14:15 -0400, Greg Freemyer wrote:
quoted
On Thu, Aug 13, 2009 at 12:33 PM, [off-list ref] wrote:
quoted
On Thu, 13 Aug 2009, Markus Trippelsdorf wrote:
quoted
On Thu, Aug 13, 2009 at 08:13:12AM -0700, Matthew Wilcox wrote:
quoted
I am planning a complete overhaul of the discard work. Users can send
down discard requests as frequently as they like. The block layer will
cache them, and invalidate them if writes come through. Periodically,
the block layer will send down a TRIM or an UNMAP (depending on the
underlying device) and get rid of the blocks that have remained unwanted
in the interim.
That is a very good idea. I've tested your original TRIM implementation on
my Vertex yesterday and it was awful ;-). The SSD needs hundreds of
milliseconds to digest a single TRIM command. And since your
implementation
sends a TRIM for each extent of each deleted file, the whole system is
unusable after a short while.
An optimal solution would be to consolidate the discard requests, bundle
them and send them to the drive as infrequent as possible.
or queue them up and send them when the drive is idle (you would need to
keep track to make sure the space isn't re-used)
as an example, if you would consider spinning down a drive you don't hurt
performance by sending accumulated trim commands.
David Lang
An alternate approach is the block layer maintain its own bitmap of
used unused sectors / blocks. Unmap commands from the filesystem just
cause the bitmap to be updated. No other effect.
(Big unknown: Where will the bitmap live between reboots? Require DM
volumes so we can have a dedicated bitmap volume in the mix to store
the bitmap to? Maybe on mount, the filesystem has to be scanned to
initially populate the bitmap? Other options?)
I wouldn't really have it live anywhere. Discard is best effort; it's
not required for fs integrity. As long as we don't discard an in-use
block we're free to do anything else (including forget to discard,
rediscard a discarded block etc).
It is theoretically possible to run all of this from user space using
the fs mappings, a bit like a defrag command.
..
Already a work-in-progress -- see my wiper.sh script on the hdparm page
at sourceforge. Trimming 50+GB of free space on a 120GB Vertex
(over 100 million sectors) takes a *single* TRIM command,
and completes in only a couple of seconds.
Cheers
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Mark Lord <hidden> Date: 2009-08-14 22:10:39
Bryan Donlan wrote:
..
Perhaps an interface (ioctl, etc) can be added to ask a filesystem to
discard all unused blocks in a certain range? (That is, have the
filesystem validate the request under any necessary locks before
passing it to the block IO layer)
..
While possibly TRIM-specific, this approach has the lowest overhead
and probably the greatest gain-for-pain ratio.
But it may not be as nice for enterprise (?).
On the Indilinx-based SSDs (eg. OCZ Vertex), TRIM seems to trigger an
internal garbage-collection/erase cycle. As such, the drive really prefers
a few LARGE trim lists, rather than many smaller ones.
Here's some information that a vendor has observed from the Win7 use of TRIM:
TRIM command is sent:
- About 2/3 of partition is filled up, when file is deleted.
(I am not talking about send file to trash bin.)
- In the above case, when trash bin gets emptied.
- In the above case, when partition is deleted.
TRIM command is not sent:-
- When file is moved to trash bin
- When partition is formatted. (Both quick and full format)
- When empty partition is deleted
- When file is deleted while there is big remaining free space
..
His words, not mine. But the idea seems to be to batch them in large chunks.
My wiper.sh "trim script" is packaged with the latest hdparm (currently 9.24)
on sourceforge, for those who want to try this stuff for real. No special
kernel support is required to use it.
Cheers
Mark
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
On Fri, Aug 14, 2009 at 6:03 PM, Mark Lord[off-list ref] wrote:
James Bottomley wrote:
quoted
On Thu, 2009-08-13 at 14:15 -0400, Greg Freemyer wrote:
quoted
On Thu, Aug 13, 2009 at 12:33 PM, [off-list ref] wrote:
quoted
On Thu, 13 Aug 2009, Markus Trippelsdorf wrote:
quoted
On Thu, Aug 13, 2009 at 08:13:12AM -0700, Matthew Wilcox wrote:
quoted
I am planning a complete overhaul of the discard work. Users can send
down discard requests as frequently as they like. The block layer
will
cache them, and invalidate them if writes come through. Periodically,
the block layer will send down a TRIM or an UNMAP (depending on the
underlying device) and get rid of the blocks that have remained
unwanted
in the interim.
That is a very good idea. I've tested your original TRIM implementation
on
my Vertex yesterday and it was awful ;-). The SSD needs hundreds of
milliseconds to digest a single TRIM command. And since your
implementation
sends a TRIM for each extent of each deleted file, the whole system is
unusable after a short while.
An optimal solution would be to consolidate the discard requests,
bundle
them and send them to the drive as infrequent as possible.
or queue them up and send them when the drive is idle (you would need to
keep track to make sure the space isn't re-used)
as an example, if you would consider spinning down a drive you don't
hurt
performance by sending accumulated trim commands.
David Lang
An alternate approach is the block layer maintain its own bitmap of
used unused sectors / blocks. Unmap commands from the filesystem just
cause the bitmap to be updated. No other effect.
(Big unknown: Where will the bitmap live between reboots? Require DM
volumes so we can have a dedicated bitmap volume in the mix to store
the bitmap to? Maybe on mount, the filesystem has to be scanned to
initially populate the bitmap? Other options?)
I wouldn't really have it live anywhere. Discard is best effort; it's
not required for fs integrity. As long as we don't discard an in-use
block we're free to do anything else (including forget to discard,
rediscard a discarded block etc).
It is theoretically possible to run all of this from user space using
the fs mappings, a bit like a defrag command.
..
Already a work-in-progress -- see my wiper.sh script on the hdparm page
at sourceforge. Trimming 50+GB of free space on a 120GB Vertex
(over 100 million sectors) takes a *single* TRIM command,
and completes in only a couple of seconds.
Cheers
Mark,
What filesystems does your script support? Running a tool like this
in the middle of the night makes a lot of since to me even from the
perspective of many / most enterprise users.
How do prevent a race where a block becomes used between userspace
asking status and it sending the discard request?
ps: I tried to pull wiper.sh straight from sourceforge, but I'm
getting some crazy page asking all sorts of questions and not letting
me bypass it. I hope sourceforge is broken. The other option is they
meant to do this. :(
Greg
--
Greg Freemyer
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Chris Worley <hidden> Date: 2009-08-14 23:21:34
On Fri, Aug 14, 2009 at 4:10 PM, Mark Lord[off-list ref] wrote:
Bryan Donlan wrote:
..
quoted
Perhaps an interface (ioctl, etc) can be added to ask a filesystem to
discard all unused blocks in a certain range? (That is, have the
filesystem validate the request under any necessary locks before
passing it to the block IO layer)
..
While possibly TRIM-specific, this approach has the lowest overhead
and probably the greatest gain-for-pain ratio.
But it may not be as nice for enterprise (?).
On the Indilinx-based SSDs (eg. OCZ Vertex), TRIM seems to trigger an
internal garbage-collection/erase cycle. As such, the drive really prefers
a few LARGE trim lists, rather than many smaller ones.
Here's some information that a vendor has observed from the Win7 use of
TRIM:
quoted
TRIM command is sent:
- About 2/3 of partition is filled up, when file is deleted.
(I am not talking about send file to trash bin.)
- In the above case, when trash bin gets emptied.
- In the above case, when partition is deleted.
TRIM command is not sent:-
- When file is moved to trash bin
- When partition is formatted. (Both quick and full format)
- When empty partition is deleted
- When file is deleted while there is big remaining free space
..
His words, not mine. But the idea seems to be to batch them in large
chunks.
Sooner is better than waiting to coalesce. The longer an LBA is
inactive, the better for any management scheme. If you wait until
it's reused, you might as well forgo the advantages of TRIM/UNMAP. If
a the controller wants to coalesce, let it coalesce.
Chris
My wiper.sh "trim script" is packaged with the latest hdparm (currently
9.24)
on sourceforge, for those who want to try this stuff for real. No special
kernel support is required to use it.
Cheers
Mark
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Matthew Wilcox <hidden> Date: 2009-08-14 23:45:41
On Fri, Aug 14, 2009 at 05:21:32PM -0600, Chris Worley wrote:
Sooner is better than waiting to coalesce. The longer an LBA is
inactive, the better for any management scheme. If you wait until
it's reused, you might as well forgo the advantages of TRIM/UNMAP. If
a the controller wants to coalesce, let it coalesce.
I'm sorry, you're wrong. There is a tradeoff point, and it's different
for each drive model. Sending down a steady stream of tiny TRIMs is
going to give terrible performance.
--
Matthew Wilcox Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours. We can't possibly take such
a retrograde step."
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Chris Worley <hidden> Date: 2009-08-15 00:19:41
On Fri, Aug 14, 2009 at 5:45 PM, Matthew Wilcox[off-list ref] wrote:
On Fri, Aug 14, 2009 at 05:21:32PM -0600, Chris Worley wrote:
quoted
Sooner is better than waiting to coalesce. The longer an LBA is
inactive, the better for any management scheme. If you wait until
it's reused, you might as well forgo the advantages of TRIM/UNMAP. If
a the controller wants to coalesce, let it coalesce.
I'm sorry, you're wrong. There is a tradeoff point, and it's different
for each drive model. Sending down a steady stream of tiny TRIMs is
going to give terrible performance.
Sounds like you might be using junk for a device?
For junk, a little coalescing may be warranted... like in the I/O
schedular, but no more than 100usecs wait before posting, or then you
effect high performing devices too.
Chris
--
Matthew Wilcox Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours. We can't possibly take such
a retrograde step."
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
On Fri, Aug 14, 2009 at 8:19 PM, Chris Worley[off-list ref] wrote:
On Fri, Aug 14, 2009 at 5:45 PM, Matthew Wilcox[off-list ref] wrote:
quoted
On Fri, Aug 14, 2009 at 05:21:32PM -0600, Chris Worley wrote:
quoted
Sooner is better than waiting to coalesce. The longer an LBA is
inactive, the better for any management scheme. If you wait until
it's reused, you might as well forgo the advantages of TRIM/UNMAP. If
a the controller wants to coalesce, let it coalesce.
I'm sorry, you're wrong. There is a tradeoff point, and it's different
for each drive model. Sending down a steady stream of tiny TRIMs is
going to give terrible performance.
Sounds like you might be using junk for a device?
For junk, a little coalescing may be warranted... like in the I/O
schedular, but no more than 100usecs wait before posting, or then you
effect high performing devices too.
Chris
Why?
AIUI, on every write a high performing device allocates a new erase
block from its free lists, writes to it, and puts the now unused erase
block on the free list. That erase block becomes available for reuse
some milliseconds later.
As long as the SSD has enough free erase blocks to work with I see no
disadvantage in delaying a discard by minutes, hours or days in most
cases. The exception is when the filesystem is almost full and the
SSD is short of erase blocks to work with.
In that case it will want to get as many free erase blocks as it can
as fast as it can get them.
Greg
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Chris Worley <hidden> Date: 2009-08-15 00:38:49
On Fri, Aug 14, 2009 at 6:30 PM, Greg Freemyer[off-list ref] wrote:
On Fri, Aug 14, 2009 at 8:19 PM, Chris Worley[off-list ref] wrote:
quoted
On Fri, Aug 14, 2009 at 5:45 PM, Matthew Wilcox[off-list ref] wrote:
quoted
On Fri, Aug 14, 2009 at 05:21:32PM -0600, Chris Worley wrote:
quoted
Sooner is better than waiting to coalesce. The longer an LBA is
inactive, the better for any management scheme. If you wait until
it's reused, you might as well forgo the advantages of TRIM/UNMAP. If
a the controller wants to coalesce, let it coalesce.
I'm sorry, you're wrong. There is a tradeoff point, and it's different
for each drive model. Sending down a steady stream of tiny TRIMs is
going to give terrible performance.
Sounds like you might be using junk for a device?
For junk, a little coalescing may be warranted... like in the I/O
schedular, but no more than 100usecs wait before posting, or then you
effect high performing devices too.
Chris
Why?
AIUI, on every write a high performing device allocates a new erase
block from its free lists, writes to it, and puts the now unused erase
block on the free list.
So erase blocks are 512 bytes (if I write 512 bytes, an erase block is
now freed)? Not true.
That erase block becomes available for reuse
some milliseconds later.
As long as the SSD has enough free erase blocks to work with I see no
disadvantage in delaying a discard by minutes, hours or days in most
cases. The exception is when the filesystem is almost full and the
SSD is short of erase blocks to work with.
That "exception..." is another good reason why.
In that case it will want to get as many free erase blocks as it can
as fast as it can get them.
On Fri, Aug 14, 2009 at 8:38 PM, Chris Worley[off-list ref] wrote:
On Fri, Aug 14, 2009 at 6:30 PM, Greg Freemyer[off-list ref] wrote:
quoted
On Fri, Aug 14, 2009 at 8:19 PM, Chris Worley[off-list ref] wrote:
quoted
On Fri, Aug 14, 2009 at 5:45 PM, Matthew Wilcox[off-list ref] wrote:
quoted
On Fri, Aug 14, 2009 at 05:21:32PM -0600, Chris Worley wrote:
quoted
Sooner is better than waiting to coalesce. The longer an LBA is
inactive, the better for any management scheme. If you wait until
it's reused, you might as well forgo the advantages of TRIM/UNMAP. If
a the controller wants to coalesce, let it coalesce.
I'm sorry, you're wrong. There is a tradeoff point, and it's different
for each drive model. Sending down a steady stream of tiny TRIMs is
going to give terrible performance.
Sounds like you might be using junk for a device?
For junk, a little coalescing may be warranted... like in the I/O
schedular, but no more than 100usecs wait before posting, or then you
effect high performing devices too.
Chris
Why?
AIUI, on every write a high performing device allocates a new erase
block from its free lists, writes to it, and puts the now unused erase
block on the free list.
So erase blocks are 512 bytes (if I write 512 bytes, an erase block is
now freed)? Not true.
Seriously, how do you know? Are you under NDA?
The write paper I read about typical SSD design described a partial
erase block write as:
Internal logic/micro-controller performs:
Read erase block, modify erase block, allocate new erase block, write
new erase block, free now unused old erase block, old erase block
added to a hardware erase queue the performs the actual erase in the
background at the relatively slow speed of multiple milliseconds..
The purpose of the trim/discard command being to allow the ssd to have
enough free erase blocks ready to go that the writes don't have to
stall while they wait for a erase block to pop out of the erase queue.
Greg
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
From: James Bottomley <hidden> Date: 2009-08-15 12:59:56
On Fri, 2009-08-14 at 18:19 -0600, Chris Worley wrote:
On Fri, Aug 14, 2009 at 5:45 PM, Matthew Wilcox[off-list ref] wrote:
quoted
On Fri, Aug 14, 2009 at 05:21:32PM -0600, Chris Worley wrote:
quoted
Sooner is better than waiting to coalesce. The longer an LBA is
inactive, the better for any management scheme. If you wait until
it's reused, you might as well forgo the advantages of TRIM/UNMAP. If
a the controller wants to coalesce, let it coalesce.
I'm sorry, you're wrong. There is a tradeoff point, and it's different
for each drive model. Sending down a steady stream of tiny TRIMs is
going to give terrible performance.
Sounds like you might be using junk for a device?
For junk, a little coalescing may be warranted... like in the I/O
schedular, but no more than 100usecs wait before posting, or then you
effect high performing devices too.
Um, I think you missed the original point in all of this at the
beginning of the thread: On ATA TRIM commands cannot be tagged. This
means you have to drain the outstanding NCQ commands (stalling the
device) before you can send a TRIM. If we do this for every discard,
the performance impact will be pretty devastating, hence the need to
coalesce. It's nothing really to do with device characteristics, it's
an ATA protocol problem.
James
From: Mark Lord <hidden> Date: 2009-08-15 13:12:18
Greg Freemyer wrote:
What filesystems does your script support? Running a tool like this
in the middle of the night makes a lot of since to me even from the
perspective of many / most enterprise users.
..
It is designed to work on any *mounted* filesystem that supports
the fallocate() system call. It uses fallocate() to reserve the
free space in a temporary file without any I/O, and then FIEMAP/FIBMAP
to get the block lists from the fallocated file, and then SGIO/ATA_16:TRIM
to discard the space, before deleting the fallocated file.
Tested by me on ext4 and xfs. btrfs has a bug that prevents the fallocate
from succeeding at present, but CM say's they're trying to fix that.
It will also work on *unmounted" ext2/ext3/ext4 filesystems,
using dumpe2fs to get the free lists, and on xfs using xfs_db there.
HFS(+) support is coming as well.
Not currently compatible with LVM 1/2, or with some distros that use
imaginary device names in /proc/mounts --> I'm working on those issues.
ps: I tried to pull wiper.sh straight from sourceforge, but I'm
getting some crazy page asking all sorts of questions and not letting
me bypass it. I hope sourceforge is broken. The other option is they
meant to do this. :(
..
That's weird. It should just be a simple click/download,
though you will need to also upgrade hdparm to the latest version.
Cheers
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Mark Lord <hidden> Date: 2009-08-15 13:20:08
Chris Worley wrote:
..
So erase blocks are 512 bytes (if I write 512 bytes, an erase block is
now freed)? Not true.
..
No, erase blocks are typically 512 KILO-bytes, or 1024 sectors.
Logical write blocks are only 512 bytes, but most drives out there
now actually use 4096 bytes as the native internal write size.
Lots of issues there.
The only existing "in the wild" TRIM-capable SSDs today all incur
large overheads from TRIM --> they seem to run a garbage-collection
and erase cycle for each TRIM command, typically taking 100s of milliseconds
regardless of the amount being trimmed.
So it makes send to gather small TRIMs into single larger TRIMs.
But I think, even better, is to just not bother with the bookkeeping,
and instead have the filesystem periodically just issue a TRIM for all
free blocks within a block group, cycling through the block groups
one by one over time.
That's how I'd like it to work on my own machine here.
Server/enterprise users very likely want something different.
Pluggable architecture, anyone? :)
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Mark Lord <hidden> Date: 2009-08-15 13:22:38
James Bottomley wrote:
This means you have to drain the outstanding NCQ commands (stalling the
device) before you can send a TRIM. If we do this for every discard,
the performance impact will be pretty devastating, hence the need to
coalesce. It's nothing really to do with device characteristics, it's
an ATA protocol problem.
..
I don't think that's really much of an issue -- we already have to do
that for cache-flushes whenever barriers are enabled. Yes it costs,
but not too much.
The current problem is that the only existing SSDs in the wild with TRIM,
take 100s of milliseconds per TRIM, mostly regardless of the amount being
TRIMmed. Sure, some TRIMs take only 10-20ms, and very large ones (millions
of sectors) can take 1-2 seconds, but most are in the 100ms range.
Cheers
From: James Bottomley <hidden> Date: 2009-08-15 13:55:30
On Sat, 2009-08-15 at 09:22 -0400, Mark Lord wrote:
James Bottomley wrote:
quoted
This means you have to drain the outstanding NCQ commands (stalling the
device) before you can send a TRIM. If we do this for every discard,
the performance impact will be pretty devastating, hence the need to
coalesce. It's nothing really to do with device characteristics, it's
an ATA protocol problem.
..
I don't think that's really much of an issue -- we already have to do
that for cache-flushes whenever barriers are enabled. Yes it costs,
but not too much.
That's not really what the enterprise is saying about flush barriers.
True, not all the performance problems are NCQ queue drain, but for a
steady workload they are significant.
James
From: jim owens <hidden> Date: 2009-08-15 17:39:53
James Bottomley wrote:
That's not really what the enterprise is saying about flush barriers.
True, not all the performance problems are NCQ queue drain, but for a
steady workload they are significant.
OK, we now know that SSDs designed only to the letter of the ATA
spec will suck doing discards if we send them down as we are
doing today.
Having finally caught up with this thread, I'm going to add some
comments that James already knows but were not stated that some
of the others apparently don't know :
- The current filesystem/blockdev behavior with discard TRIM was
argued and added quickly because this design was what the
Intel SSD architect told us was "the right thing" in Sept 08.
- In the same workshop, Linus said "I'm tired of hardware
vendors telling me to fix it because they are cheap and lazy",
or something close to that, my memory gets bit-errors :)
- We decided not to track and coalesce the discards in the block
or filesystem layer because of the high memory/performance cost.
There is no cheap way to do this, all of the space management
in filesystems is accepting some cost for some user benefit.
- Many people who live in filesystems (like me) are unconvinced
that discard to SSD or an array will help in real world use,
but the current discard design didn't seem to hurt us either.
***begin rant***
I have not seen any analysis of the benefit and cost to the
end user of the TRIM or array UNMAP. We now see that TRIM
as implemented by some (all?) SSDs will come at high cost.
The cost is all born by the host. Do we get any benefit, or
is it all for the device vendor. And when we subtract the cost
from the benefit, does the user actually benefit and how?
I'm tired of working around shit storage products and broken
device protocols from the "T" committees. I suggest we just
add a "white list" of devices that handle the discard fast
and without us needing NCQ queue drain. Then only send TRIM
to devices that are on the white list and throw the others
away in the block device layer.
I do enterprise systems and the cost of RAM in those systems
is awful. And the databases and applications are always big
memory pigs. Our customers always complain about the kernel
using too much memory and they will go ballistic if we take
1GB from their 512GB system unless we can really show them
significant benefit in their production. And so far all
we have is "this is all good stuff" from array vendors.
[and yes, our hardware guys always give me the most pain]
If continuous discard is going to be a PITA for us, then
I say don't do it. Just let a user-space tool do it when
the admin wants. IMO is no different than defragment,
where my experience with a kernel continuous defragment
was that it made a great sales gimmick, but in real production
most people saw no benefit and some had to shut it off
because it actually hurt them. It is all about workload.
jim
P.S. Matthew, that SSD architect told me personally
that the trim of each 512 byte block before rewrite
will be a performance benefit, so if Intel SSDs are
not on the white list, please slap him for me.
On Sat, 15 Aug 2009 08:55:17 -0500
James Bottomley [off-list ref] wrote:
On Sat, 2009-08-15 at 09:22 -0400, Mark Lord wrote:
quoted
James Bottomley wrote:
quoted
This means you have to drain the outstanding NCQ commands (stalling the
device) before you can send a TRIM. If we do this for every discard,
the performance impact will be pretty devastating, hence the need to
coalesce. It's nothing really to do with device characteristics, it's
an ATA protocol problem.
..
I don't think that's really much of an issue -- we already have to do
that for cache-flushes whenever barriers are enabled. Yes it costs,
but not too much.
That's not really what the enterprise is saying about flush barriers.
True, not all the performance problems are NCQ queue drain, but for a
steady workload they are significant.
Flush barriers are nightmare for more than enterprise. You drive
basically goes for a hike for a bit which trashes interactivity as well.
If the device can't do trim and the like without a drain I don't see much
point doing it at all, except maybe to wait for idle devices and run a
filesystem managed background 'strimmer' thread to just weed out now idle
blocks that have stayed idle - eg by adding an inode of all the deleted
untrimmed blocks and giving it an irregular empty ?
Alan
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Mark Lord <hidden> Date: 2009-08-16 14:16:04
Alan Cox wrote:
Flush barriers are nightmare for more than enterprise. You drive
basically goes for a hike for a bit which trashes interactivity as well.
If the device can't do trim and the like without a drain I don't see much
point doing it at all, except maybe to wait for idle devices and run a
filesystem managed background 'strimmer' thread to just weed out now idle
blocks that have stayed idle - eg by adding an inode of all the deleted
untrimmed blocks and giving it an irregular empty ?
..
Agreed. And I believe Matthew also said something similar already.
TRIM for the current (only!) SSDs needs to be a "batched, once in a while"
operation, rather than something done continuously.
And ideally, "once in a while" might be once a day, or once we have more
than a significant percentage of the drive capacity ready for a TRIM.
It needs to batch a lot of stuff into a single TRIM,
and not do it very often at all.
Cheers
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Arjan van de Ven <hidden> Date: 2009-08-16 15:32:21
On Sun, 16 Aug 2009 15:05:30 +0100
Alan Cox [off-list ref] wrote:
On Sat, 15 Aug 2009 08:55:17 -0500
James Bottomley [off-list ref] wrote:
quoted
On Sat, 2009-08-15 at 09:22 -0400, Mark Lord wrote:
quoted
James Bottomley wrote:
quoted
This means you have to drain the outstanding NCQ commands
(stalling the device) before you can send a TRIM. If we do
this for every discard, the performance impact will be pretty
devastating, hence the need to coalesce. It's nothing really
to do with device characteristics, it's an ATA protocol problem.
..
I don't think that's really much of an issue -- we already have
to do that for cache-flushes whenever barriers are enabled. Yes
it costs, but not too much.
That's not really what the enterprise is saying about flush
barriers. True, not all the performance problems are NCQ queue
drain, but for a steady workload they are significant.
Flush barriers are nightmare for more than enterprise. You drive
basically goes for a hike for a bit which trashes interactivity as
well. If the device can't do trim and the like without a drain I
don't see much point doing it at all, except maybe to wait for idle
devices and run a filesystem managed background 'strimmer' thread to
just weed out now idle blocks that have stayed idle - eg by adding an
inode of all the deleted untrimmed blocks and giving it an irregular
empty ?
trim is mostly for ssd's though, and those tend to not have the "goes
for a hike" behavior as much......
I wonder if it's worse to batch stuff up, because then the trim itself
gets bigger and might take longer.....
--
Arjan van de Ven Intel Open Source Technology Centre
For development, discussion and tips for power savings,
visit http://www.lesswatts.org
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
On Sun, Aug 16, 2009 at 08:34:34AM -0700, Arjan van de Ven wrote:
trim is mostly for ssd's though, and those tend to not have the "goes
for a hike" behavior as much......
Mark Lord has claimed that the currently shipping SSD's take "hundreds
of milliseconds" for a TRIM, command. Compared to the usual latency
of an SSD, that could very well be considered "takes a coffee break"
behaviour; maybe not "goes for a hike", but enough that you wouldn't
want to be doing one all the time.
The story that I've heard which worries me is that those of us which
were silly enough to spend $400 and $800 dollars on the first
generation X25-M drives may never get TRIM support, and that TRIM
support might only be offered on the second generation X25-M drives.
I certainly _hope_ that is not true, but in any case, I don't have any
TRIM capable drives at the moment, so it's not something which I'm set
up to test....
- Ted
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: James Bottomley <hidden> Date: 2009-08-16 15:52:23
On Sun, 2009-08-16 at 08:34 -0700, Arjan van de Ven wrote:
On Sun, 16 Aug 2009 15:05:30 +0100
Alan Cox [off-list ref] wrote:
quoted
On Sat, 15 Aug 2009 08:55:17 -0500
James Bottomley [off-list ref] wrote:
quoted
On Sat, 2009-08-15 at 09:22 -0400, Mark Lord wrote:
quoted
James Bottomley wrote:
quoted
This means you have to drain the outstanding NCQ commands
(stalling the device) before you can send a TRIM. If we do
this for every discard, the performance impact will be pretty
devastating, hence the need to coalesce. It's nothing really
to do with device characteristics, it's an ATA protocol problem.
..
I don't think that's really much of an issue -- we already have
to do that for cache-flushes whenever barriers are enabled. Yes
it costs, but not too much.
That's not really what the enterprise is saying about flush
barriers. True, not all the performance problems are NCQ queue
drain, but for a steady workload they are significant.
Flush barriers are nightmare for more than enterprise. You drive
basically goes for a hike for a bit which trashes interactivity as
well. If the device can't do trim and the like without a drain I
don't see much point doing it at all, except maybe to wait for idle
devices and run a filesystem managed background 'strimmer' thread to
just weed out now idle blocks that have stayed idle - eg by adding an
inode of all the deleted untrimmed blocks and giving it an irregular
empty ?
trim is mostly for ssd's though, and those tend to not have the "goes
for a hike" behavior as much......
Well, yes and no ... a lot of SSDs don't actually implement NCQ, so the
impact to them will be less ... although I think enterprise class SSDs
do implement NCQ.
I wonder if it's worse to batch stuff up, because then the trim itself
gets bigger and might take longer.....
So this is where we're getting into the realms of speculation. There
really are only about a couple of people out there with trim
implementing SSDs, so that's not really enough to make any judgement.
However, the enterprise has been doing UNMAP for a while, so we can draw
inferences from them since the SSD FTL will operate similarly. For
them, UNMAP is the same cost in terms of time regardless of the number
of extents. The reason is that it's moving the blocks from the global
in use list to the global free list. Part of the problem is that this
involves locking and quiescing, so UNMAP ends up being quite expensive
to the array but constant in terms of cost (hence they want as few
unmaps for as many sectors as possible).
For SSDs, the FTL has to have a separate operation: erase. Now, one
could see the correct implementation simply moving the sectors from the
in-use list to the to be cleaned list and still do the cleaning in the
background: that would be constant cost (but, again, likely expensive).
Of course, if SSD vendors decided to erase on the spot when seeing TRIM,
this wouldn't be true ...
James
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Mark Lord <hidden> Date: 2009-08-16 16:32:56
James Bottomley wrote:
For SSDs, the FTL has to have a separate operation: erase. Now, one
could see the correct implementation simply moving the sectors from the
in-use list to the to be cleaned list and still do the cleaning in the
background: that would be constant cost (but, again, likely expensive).
Of course, if SSD vendors decided to erase on the spot when seeing TRIM,
this wouldn't be true ...
..
The SSDs based upon the Indilinx Barefoot controller appear to do
the erase on the spot, along with a fair amount of garbage collection.
The overhead does vary by size of the TRIM operation (number of sectors
and extents), but even a single-sector TRIM has very high overhead.
Samsung also now has SSDs at retail with TRIM.
I don't have one of those here.
From: Christoph Hellwig <hch@infradead.org> Date: 2009-08-16 17:00:09
On Sun, Aug 16, 2009 at 10:52:07AM -0500, James Bottomley wrote:
However, the enterprise has been doing UNMAP for a while, so we can draw
inferences from them since the SSD FTL will operate similarly. For
them, UNMAP is the same cost in terms of time regardless of the number
of extents. The reason is that it's moving the blocks from the global
in use list to the global free list. Part of the problem is that this
involves locking and quiescing, so UNMAP ends up being quite expensive
to the array but constant in terms of cost (hence they want as few
unmaps for as many sectors as possible).
How are they doing the unmaps? Using something similar to Mark's wiper
script and using SG_IO? Because right now we do not actually implement
UNMAP support in the kernel. I'd really love to test the XFS batched
discard support with a real UNMAP implementation.
From: Robert Hancock <hidden> Date: 2009-08-16 17:08:58
On 08/15/2009 11:39 AM, jim owens wrote:
***begin rant***
I have not seen any analysis of the benefit and cost to the
end user of the TRIM or array UNMAP. We now see that TRIM
as implemented by some (all?) SSDs will come at high cost.
The cost is all born by the host. Do we get any benefit, or
is it all for the device vendor. And when we subtract the cost
from the benefit, does the user actually benefit and how?
I'm tired of working around shit storage products and broken
device protocols from the "T" committees. I suggest we just
add a "white list" of devices that handle the discard fast
and without us needing NCQ queue drain. Then only send TRIM
to devices that are on the white list and throw the others
away in the block device layer.
They all will require NCQ queue drain. It's an inherent requirement of
the protocol that you can't overlap NCQ and non-NCQ commands, and the
trim command is not NCQ.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: James Bottomley <hidden> Date: 2009-08-16 18:07:39
On Sun, 2009-08-16 at 12:32 -0400, Mark Lord wrote:
James Bottomley wrote:
quoted
For SSDs, the FTL has to have a separate operation: erase. Now, one
could see the correct implementation simply moving the sectors from the
in-use list to the to be cleaned list and still do the cleaning in the
background: that would be constant cost (but, again, likely expensive).
Of course, if SSD vendors decided to erase on the spot when seeing TRIM,
this wouldn't be true ...
..
The SSDs based upon the Indilinx Barefoot controller appear to do
the erase on the spot, along with a fair amount of garbage collection.
Groan. I'm with Jim on this one: If trim is going to cost us in terms
of current fs performance, it's likely not worth it. The whole point of
a TRIM/UNMAP is that we're just passing hints about storage use. If the
drives make us pay the penalty of acting on the hints as we pass them
in, we may as well improve performance just by not hinting. Or at least
it's detrimental hinting in real time.
So I think we've iterated to the conclusion that it has to be a user
space process which tries to identify idle periods and begin trimming.
The overhead does vary by size of the TRIM operation (number of sectors
and extents), but even a single-sector TRIM has very high overhead.
So it's something like X + nY (n == number of sectors). If X is large,
it still argues for batching .. it's just there's likely an upper bound
to the batch where the benefit is no longer worth the cost.
Samsung also now has SSDs at retail with TRIM.
I don't have one of those here.
Heh, OS writers not having access to the devices is about par for the
current course.
James
From: Mark Lord <hidden> Date: 2009-08-16 18:19:12
James Bottomley wrote:
Heh, OS writers not having access to the devices is about par for the
current course.
..
Pity the Linux Foundation doesn't simply step in and supply hardware
to us for new tech like this. Cheap for them, expensive for folks like me.
Cheers
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: James Bottomley <hidden> Date: 2009-08-16 18:24:24
On Sun, 2009-08-16 at 14:19 -0400, Mark Lord wrote:
James Bottomley wrote:
quoted
Heh, OS writers not having access to the devices is about par for the
current course.
..
Pity the Linux Foundation doesn't simply step in and supply hardware
to us for new tech like this. Cheap for them, expensive for folks like me.
Um, to give a developer a selection of manufacturers' SSDs at retail
prices, you're talking several thousand dollars ... in these lean
times, that would be two or three developers not getting travel
sponsorship per chosen SSD recipient. It's not a worthwhile tradeoff.
The best the LF can likely do is try to explain to the manufacturers
that handing out samples at linux conferences (like plumbers) is in
their own interests. It can also manage the handout if necessary
through its HW lending library.
James
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
trim is mostly for ssd's though, and those tend to not have the "goes
for a hike" behavior as much......
Bench one.
I wonder if it's worse to batch stuff up, because then the trim itself
gets bigger and might take longer.....
They seem to implement a sort of async single threaded trim, which can
only have one outstanding trim at a time.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Chris Worley <hidden> Date: 2009-08-16 22:59:27
On Sat, Aug 15, 2009 at 7:20 AM, Mark Lord[off-list ref] wrote:
Chris Worley wrote:
..
quoted
So erase blocks are 512 bytes (if I write 512 bytes, an erase block is
now freed)? Not true.
..
No, erase blocks are typically 512 KILO-bytes, or 1024 sectors.
Logical write blocks are only 512 bytes,
That was my point.
The OS should not make assumptions of the size.
but most drives out there
now actually use 4096 bytes as the native internal write size.
Lots of issues there.
The only existing "in the wild" TRIM-capable SSDs today all incur
large overheads from TRIM
SSD's yes, SSS no.
--> they seem to run a garbage-collection
and erase cycle for each TRIM command, typically taking 100s of milliseconds
regardless of the amount being trimmed.
The OS should not assume a dumb algorithm on the part of the drive.
So it makes send to gather small TRIMs into single larger TRIMs.
If Linux is only to support slow legacy SAS/SATA.
But I think, even better, is to just not bother with the bookkeeping,
and instead have the filesystem periodically just issue a TRIM for all
free blocks within a block group, cycling through the block groups
one by one over time.
That's how I'd like it to work on my own machine here.
Server/enterprise users very likely want something different.
Yes. I didn't realize this was a laptop-only fix.
Thanks,
Chris
Pluggable architecture, anyone? :)
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: John Robinson <hidden> Date: 2009-08-16 23:27:59
On 16/08/2009 20:29, Alan Cox wrote:
quoted
trim is mostly for ssd's though, and those tend to not have the "goes
for a hike" behavior as much......
Bench one.
quoted
I wonder if it's worse to batch stuff up, because then the trim itself
gets bigger and might take longer.....
They seem to implement a sort of async single threaded trim, which can
only have one outstanding trim at a time.
I'm slightly out of my depth here, but: if a single TRIM is issued,
which apparently returns quickly, can one then revert to issuing
ordinary commands like reads and writes and have them complete as
quickly as they normally do, or does any following command have to wait
until the trim completes? This could be useful if it turned out we won't
stall these devices as long as we don't issue more than one TRIM every
few seconds; we could keep a TRIM coalesce queue down to being (say) 5
seconds long (or at least, a configurable small number of seconds).
Cheers,
John.
From: Mark Lord <hidden> Date: 2009-08-17 02:03:27
Chris Worley wrote:
..
The OS should not assume a dumb algorithm on the part of the drive.
..
Welcome to the Real World.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Mark Lord <hidden> Date: 2009-08-17 02:05:43
John Robinson wrote:
..
I'm slightly out of my depth here, but: if a single TRIM is issued,
which apparently returns quickly, can one then revert to issuing
ordinary commands like reads and writes and have them complete as
quickly as they normally do, or does any following command have to wait
until the trim completes? This could be useful if it turned out we won't
stall these devices as long as we don't issue more than one TRIM every
few seconds; we could keep a TRIM coalesce queue down to being (say) 5
seconds long (or at least, a configurable small number of seconds).
..
I have not attempted to instrument that, but I suspect that any
command after the TRIM has to wait. Don't know for sure until
somebody measures it though.
One thing I do know, is that Matthew's first cut of TRIM support
means it takes half an hour to do "rm -r" on a kernel source tree.
-ml
From: Douglas Gilbert <dgilbert@interlog.com> Date: 2009-08-17 04:24:45
Christoph Hellwig wrote:
On Sun, Aug 16, 2009 at 10:52:07AM -0500, James Bottomley wrote:
quoted
However, the enterprise has been doing UNMAP for a while, so we can draw
inferences from them since the SSD FTL will operate similarly. For
them, UNMAP is the same cost in terms of time regardless of the number
of extents. The reason is that it's moving the blocks from the global
in use list to the global free list. Part of the problem is that this
involves locking and quiescing, so UNMAP ends up being quite expensive
to the array but constant in terms of cost (hence they want as few
unmaps for as many sectors as possible).
How are they doing the unmaps? Using something similar to Mark's wiper
script and using SG_IO? Because right now we do not actually implement
UNMAP support in the kernel. I'd really love to test the XFS batched
discard support with a real UNMAP implementation.
The sg3_utils version 1.28 beta at http://sg.danny.cz/sg/
has a new sg_unmap utility and the previous release
included sg_write_same with Unmap bit support.
sg_readcap has been updated to show the TPE and TPRZ bits.
There is a new SCSI GET LBA STATUS command coming
(approved at the last t10 meeting, awaiting the next
SBC-3 draft). That will show the mapped/unmapped
status of logical blocks in a range of LBAs. I can
add a utility for that as well.
Doug Gilbert
From: James Bottomley <hidden> Date: 2009-08-17 13:56:29
On Sun, 2009-08-16 at 12:59 -0400, Christoph Hellwig wrote:
On Sun, Aug 16, 2009 at 10:52:07AM -0500, James Bottomley wrote:
quoted
However, the enterprise has been doing UNMAP for a while, so we can draw
inferences from them since the SSD FTL will operate similarly. For
them, UNMAP is the same cost in terms of time regardless of the number
of extents. The reason is that it's moving the blocks from the global
in use list to the global free list. Part of the problem is that this
involves locking and quiescing, so UNMAP ends up being quite expensive
to the array but constant in terms of cost (hence they want as few
unmaps for as many sectors as possible).
How are they doing the unmaps? Using something similar to Mark's wiper
script and using SG_IO? Because right now we do not actually implement
UNMAP support in the kernel. I'd really love to test the XFS batched
discard support with a real UNMAP implementation.
You mean how is the array vendor testing their implementation? Using
SG_IO ... without any filesystem, I believe.
The testing was initially done to see if the initial maximal discard
proposal from LSF09 was a viable approach (which it wasn't given the
time taken to UNMAP).
James
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Matthew Wilcox <hidden> Date: 2009-08-17 14:10:41
On Mon, Aug 17, 2009 at 08:56:12AM -0500, James Bottomley wrote:
The testing was initially done to see if the initial maximal discard
proposal from LSF09 was a viable approach (which it wasn't given the
time taken to UNMAP).
It would be nice if that feedback could be made public instead of it
leaking out in dribs and drabs like this.
--
Matthew Wilcox Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours. We can't possibly take such
a retrograde step."
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Bill Davidsen <hidden> Date: 2009-08-17 16:38:55
James Bottomley wrote:
On Sun, 2009-08-16 at 14:19 -0400, Mark Lord wrote:
quoted
James Bottomley wrote:
quoted
Heh, OS writers not having access to the devices is about par for the
current course.
..
Pity the Linux Foundation doesn't simply step in and supply hardware
to us for new tech like this. Cheap for them, expensive for folks like me.
Um, to give a developer a selection of manufacturers' SSDs at retail
prices, you're talking several thousand dollars ... in these lean
times, that would be two or three developers not getting travel
sponsorship per chosen SSD recipient. It's not a worthwhile tradeoff.
The best the LF can likely do is try to explain to the manufacturers
that handing out samples at linux conferences (like plumbers) is in
their own interests. It can also manage the handout if necessary
through its HW lending library.
Of install the hardware on a machine and give people access to the
machine in time slots. Faster than FedEx-ing the hardware, and
relatively fast to reinstall the OS from scratch. Testing of this type
doesn't need huge bandwidth.
--
bill davidsen [off-list ref]
CTO TMR Associates, Inc
"You are disgraced professional losers. And by the way, give us our money back."
- Representative Earl Pomeroy, Democrat of North Dakota
on the A.I.G. executives who were paid bonuses after a federal bailout.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
All,
Seems like the high-level wrap-up of all this is:
There are hopes that highly efficient SSDs will appear on the market
that can leverage a passthru non-coalescing discard feature. And that
a whitelist should be created to allow those SSDs to see discards
intermixed with the rest of the data i/o.
For the other known cases:
SSDs that meet the ata-8 spec, but don't exceed it
Enterprise SCSI
mdraid with SSD storage used to build raid5 / raid6 arrays
Non-coalescing is believed detrimental, but a regular flushing of the
unused blocks/sectors via a tool like Mark Lord has written should be
acceptable.
Mark, I don't believe your tool really addresses the mdraid situation,
do you agree. ie. Since your bypassing most of the block stack,
mdraid has no way of snooping on / adjusting the discards you are
sending out.
Thus the 2 solutions that have been worked on already seem to address
the needs of everything but mdraid.
Also, there has been no discussion of dm based volumes. (ie LVM2 based volumes)
For mdraid or dm it seems we need to enhance Mark's script to pass the
trim commands through the full block stack. Mark, please cmiiw
Greg
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: James Bottomley <hidden> Date: 2009-08-17 17:19:49
On Mon, 2009-08-17 at 13:08 -0400, Greg Freemyer wrote:
All,
Seems like the high-level wrap-up of all this is:
There are hopes that highly efficient SSDs will appear on the market
that can leverage a passthru non-coalescing discard feature. And that
a whitelist should be created to allow those SSDs to see discards
intermixed with the rest of the data i/o.
That's not my conclusion. Mine was the NCQ drain would still be
detremental to interleaved trim even if the drive could do it for zero
cost.
For the other known cases:
SSDs that meet the ata-8 spec, but don't exceed it
Enterprise SCSI
No, SCSI will do WRITE_SAME/UNMAP as currently drafted in SBC3
mdraid with SSD storage used to build raid5 / raid6 arrays
Non-coalescing is believed detrimental,
It is? Why?
but a regular flushing of the
unused blocks/sectors via a tool like Mark Lord has written should be
acceptable.
Mark, I don't believe your tool really addresses the mdraid situation,
do you agree. ie. Since your bypassing most of the block stack,
mdraid has no way of snooping on / adjusting the discards you are
sending out.
Thus the 2 solutions that have been worked on already seem to address
the needs of everything but mdraid.
I count three: Mark Lord script via SG_IO. hch enhanced script via
XFS_TRIM and willy current discard inline which he's considering
coalescing for.
James
Also, there has been no discussion of dm based volumes. (ie LVM2 based volumes)
For mdraid or dm it seems we need to enhance Mark's script to pass the
trim commands through the full block stack. Mark, please cmiiw
Greg
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
Chiming in here a bit late, but coalescing requests is also a good way
to prevent read-modify-write cycles.
Specifically, if I remember the concern correctly, for the WRITE_SAME
with unmap bit set, when the IO is not evenly aligned on the "erase
chunk" (whatever they call it) boundary the device can be forced to do a
read-modify-write (of zeroes) to the end or beginning of that region.
For a disk array, the WRITE_SAME with unmap bit when done cleanly on an
aligned boundary can be done entirely in the array's cache. The
read-modify-write can generate several reads to the back end disks which
are significantly slower....
ric
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
On Mon, Aug 17, 2009 at 1:19 PM, James Bottomley[off-list ref] wrote:
On Mon, 2009-08-17 at 13:08 -0400, Greg Freemyer wrote:
quoted
All,
Seems like the high-level wrap-up of all this is:
There are hopes that highly efficient SSDs will appear on the market
that can leverage a passthru non-coalescing discard feature. And that
a whitelist should be created to allow those SSDs to see discards
intermixed with the rest of the data i/o.
That's not my conclusion. Mine was the NCQ drain would still be
detremental to interleaved trim even if the drive could do it for zero
cost.
Maybe I misunderstood Jim Owens previous comment that designing for
devices that only meet the spec. was not his / Linus'es preference.
Instead they want to have a whitelist enabled list of drives that
support trim / ncq without having to drain the queue.
I just re-read his post and he did not explicitly say that, so maybe
I'm mis-representing it.
quoted
For the other known cases:
SSDs that meet the ata-8 spec, but don't exceed it
Enterprise SCSI
No, SCSI will do WRITE_SAME/UNMAP as currently drafted in SBC3
quoted
mdraid with SSD storage used to build raid5 / raid6 arrays
Non-coalescing is believed detrimental,
It is? Why?
For the only compliant SSD in the wild, Mark has shown it to be true
via testing.
For Enterprise SCSI, I thought you said a coalescing solution is
preferred. (I took that to mean non-coalescing is detremental. Not
true?).
For mdraid, if the trims are not coalesced mdraid will have to either
ignore them, or coalesce them themselves. Having them come in bigger
discard ranges is clearly better. (ie. At least the size of a stripe,
so it can adjust the start / end sector to a stripe boundary.)
quoted
but a regular flushing of the
unused blocks/sectors via a tool like Mark Lord has written should be
acceptable.
Mark, I don't believe your tool really addresses the mdraid situation,
do you agree. ie. Since your bypassing most of the block stack,
mdraid has no way of snooping on / adjusting the discards you are
sending out.
Thus the 2 solutions that have been worked on already seem to address
the needs of everything but mdraid.
I count three: Mark Lord script via SG_IO. hch enhanced script via
XFS_TRIM and willy current discard inline which he's considering
coalescing for.
I missed XFS_TRIM somehow. What benefit does XFS_TRIM provide at a
high level? Is it part of the realtime delete file process, or an
after the fact scanner?
James
Greg
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
From: Christoph Hellwig <hch@infradead.org> Date: 2009-08-17 19:13:18
On Mon, Aug 17, 2009 at 08:10:38AM -0600, Matthew Wilcox wrote:
On Mon, Aug 17, 2009 at 08:56:12AM -0500, James Bottomley wrote:
quoted
The testing was initially done to see if the initial maximal discard
proposal from LSF09 was a viable approach (which it wasn't given the
time taken to UNMAP).
It would be nice if that feedback could be made public instead of it
leaking out in dribs and drabs like this.
Yeah, I don't remember hearing about anything like this either.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: James Bottomley <hidden> Date: 2009-08-17 19:18:44
On Mon, 2009-08-17 at 14:21 -0400, Greg Freemyer wrote:
On Mon, Aug 17, 2009 at 1:19 PM, James Bottomley[off-list ref] wrote:
quoted
On Mon, 2009-08-17 at 13:08 -0400, Greg Freemyer wrote:
quoted
All,
Seems like the high-level wrap-up of all this is:
There are hopes that highly efficient SSDs will appear on the market
that can leverage a passthru non-coalescing discard feature. And that
a whitelist should be created to allow those SSDs to see discards
intermixed with the rest of the data i/o.
That's not my conclusion. Mine was the NCQ drain would still be
detremental to interleaved trim even if the drive could do it for zero
cost.
Maybe I misunderstood Jim Owens previous comment that designing for
devices that only meet the spec. was not his / Linus'es preference.
Instead they want to have a whitelist enabled list of drives that
support trim / ncq without having to drain the queue.
There's no way to do this. The spec explicitly requires that you not
overlap tagged and untagged commands. The reason is fairly obvious:
you wouldn't be able to separate the completions.
I just re-read his post and he did not explicitly say that, so maybe
I'm mis-representing it.
quoted
quoted
For the other known cases:
SSDs that meet the ata-8 spec, but don't exceed it
Enterprise SCSI
No, SCSI will do WRITE_SAME/UNMAP as currently drafted in SBC3
quoted
mdraid with SSD storage used to build raid5 / raid6 arrays
Non-coalescing is believed detrimental,
It is? Why?
For the only compliant SSD in the wild, Mark has shown it to be true
via testing.
He only said larger trims take longer. As I said previously, if it's a
X+nY relationship, then we still benefit from accumulation up to some
value of n.
For Enterprise SCSI, I thought you said a coalescing solution is
preferred. (I took that to mean non-coalescing is detremental. Not
true?).
I'm trying to persuade the array vendors to speak for themselves, but it
seems that UNMAP takes time. Of course, in SCSI, this is a taggable
command so we don't have the drain overhead ... but then we can't do
anything that would produce an undetermined state based on out of order
tag execution either.
For mdraid, if the trims are not coalesced mdraid will have to either
ignore them, or coalesce them themselves. Having them come in bigger
discard ranges is clearly better. (ie. At least the size of a stripe,
so it can adjust the start / end sector to a stripe boundary.)
If we did discard accumulation in-kernel (a big if), it would likely be
at the request level; thus md and dm would automatically inherit it.
dm/md are a problem for a userspace accumulation solution, though
(although I suspect the request elevator can fix that).
quoted
quoted
but a regular flushing of the
unused blocks/sectors via a tool like Mark Lord has written should be
acceptable.
Mark, I don't believe your tool really addresses the mdraid situation,
do you agree. ie. Since your bypassing most of the block stack,
mdraid has no way of snooping on / adjusting the discards you are
sending out.
Thus the 2 solutions that have been worked on already seem to address
the needs of everything but mdraid.
I count three: Mark Lord script via SG_IO. hch enhanced script via
XFS_TRIM and willy current discard inline which he's considering
coalescing for.
I missed XFS_TRIM somehow. What benefit does XFS_TRIM provide at a
high level? Is it part of the realtime delete file process, or an
after the fact scanner?
It guarantees that trim does not overlap allocations and writes on a
running system, so it gives us safety of execution.
James
From: James Bottomley <hidden> Date: 2009-08-17 19:24:14
On Mon, 2009-08-17 at 15:12 -0400, Christoph Hellwig wrote:
On Mon, Aug 17, 2009 at 08:10:38AM -0600, Matthew Wilcox wrote:
quoted
On Mon, Aug 17, 2009 at 08:56:12AM -0500, James Bottomley wrote:
quoted
The testing was initially done to see if the initial maximal discard
proposal from LSF09 was a viable approach (which it wasn't given the
time taken to UNMAP).
It would be nice if that feedback could be made public instead of it
leaking out in dribs and drabs like this.
Yeah, I don't remember hearing about anything like this either.
Well you were both in #storage when it got discussed on 4 May. However,
I'm hopeful that the array vendors will make a more direct statement of
the capabilities shortly.
James
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Mark Lord <hidden> Date: 2009-08-17 20:19:42
James Bottomley wrote:
On Mon, 2009-08-17 at 14:21 -0400, Greg Freemyer wrote:
quoted
On Mon, Aug 17, 2009 at 1:19 PM, James Bottomley[off-list ref] wrote:
quoted
On Mon, 2009-08-17 at 13:08 -0400, Greg Freemyer wrote:
..
quoted
quoted
quoted
Non-coalescing is believed detrimental,
It is? Why?
For the only compliant SSD in the wild, Mark has shown it to be true
via testing.
He only said larger trims take longer. As I said previously, if it's a
X+nY relationship, then we still benefit from accumulation up to some
value of n.
..
Err, what I said was, "rm -rf /usr/src/linux" takes over half an hour
with uncoalesced TRIM, and only a scant few seconds in total *with*
coalesced TRIM.
From: Mark Lord <hidden> Date: 2009-08-17 20:28:21
Greg Freemyer wrote:
..
Mark, I don't believe your tool really addresses the mdraid situation,
do you agree. ie. Since your bypassing most of the block stack,
mdraid has no way of snooping on / adjusting the discards you are
sending out.
..
Taking care of mounted RAID / LVM filesystems requires in-kernel TRIM
support, possibly exported via an ioctl().
Taking care of unmounted RAID / LVM filesystems is possible in userland,
but would also benefit from in-kernel support, where layouts are defined
and known better than in userland.
The XFS_TRIM was an idea that Cristoph floated, as a concept for examination.
I think something along those lines would be best, but perhaps with an
interface at the VFS layer. Something that permits a userland tool
to work like this (below) might be nearly ideal:
main() {
int fd = open(filesystem_device);
while (1) {
int g, ngroups = ioctl(fd, GET_NUMBER_OF_BLOCK_GROUPS);
for (g = 0; g < ngroups; ++g) {
ioctl(fd, TRIM_ALL_FREE_EXTENTS_OF_GROUP, g);
}
sleep(3600);
}
}
Not all filesystems have a "block group", or "allocation group" structure,
but I suspect that it's an easy mapping in most cases.
With this scheme, the kernel is absolved of the need to track/coallesce
TRIM requests entirely.
Something like that, perhaps.
From: James Bottomley <hidden> Date: 2009-08-17 20:28:29
On Mon, 2009-08-17 at 16:19 -0400, Mark Lord wrote:
James Bottomley wrote:
quoted
On Mon, 2009-08-17 at 14:21 -0400, Greg Freemyer wrote:
quoted
On Mon, Aug 17, 2009 at 1:19 PM, James Bottomley[off-list ref] wrote:
quoted
On Mon, 2009-08-17 at 13:08 -0400, Greg Freemyer wrote:
..
quoted
quoted
quoted
quoted
Non-coalescing is believed detrimental,
It is? Why?
For the only compliant SSD in the wild, Mark has shown it to be true
via testing.
He only said larger trims take longer. As I said previously, if it's a
X+nY relationship, then we still benefit from accumulation up to some
value of n.
..
Err, what I said was, "rm -rf /usr/src/linux" takes over half an hour
with uncoalesced TRIM, and only a scant few seconds in total *with*
coalesced TRIM.
Yes, sorry, missed the Non- when I read that sentence.
James
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>