From: Mike Snitzer <hidden> Date: 2010-08-25 15:28:32
On Wed, Aug 25 2010 at 4:00am -0400,
Kiyoshi Ueda [off-list ref] wrote:
Hi Tejun,
On 08/25/2010 01:59 AM +0900, Tejun Heo wrote:
quoted
On 08/24/2010 12:24 PM, Kiyoshi Ueda wrote:
quoted
Anyway, only reporting errors for REQ_FLUSH to upper layer without
such a solution would make dm-multipath almost unusable in real world,
although it's better than implicit data loss.
I see.
quoted
quoted
Maybe just turn off barrier support in mpath for now?
If it's possible, it could be a workaround for a short term.
But how can you do that?
I think it's not enough to just drop REQ_FLUSH flag from q->flush_flags.
Underlying devices of a mpath device may have write-back cache and
it may be enabled.
So if a mpath device doesn't set REQ_FLUSH flag in q->flush_flags, it
becomes a device which has write-back cache but doesn't support flush.
Then, upper layer can do nothing to ensure cache flush?
Yeah, I was basically suggesting to forget about cache flush w/ mpath
until it can be fixed. You're saying that if mpath just passes
REQ_FLUSH upwards without retrying, it will be almost unuseable,
right?
Right.
If the error is safe/needed to retry using other paths, mpath should
retry even if REQ_FLUSH. Otherwise, only one path failure may result
in system down.
Just passing any REQ_FLUSH error upwards regardless the error type
will make such situations, and users will feel the behavior as
unstable/unusable.
Right, there are hardware configurations that lend themselves to FLUSH
retries mattering, namely:
1) a SAS drive with 2 ports and a writeback cache
2) theoretically possible: SCSI array that is mpath capable but
advertises cache as writeback (WCE=1)
The SAS case is obviously a more concrete example of why FLUSH retries
are worthwhile in mpath.
But I understand (and agree) that we'd be better off if mpath could
differentiate between failures rather than blindly retrying on failures
like it does today (fails path and retries if additional paths
available).
Anyway, as you said, the flush error handling of dm-mpath is already
broken if data loss really happens on any storage used by dm-mpath.
Although it's a serious issue and quick fix is required, I think
you may leave the old behavior in your patch-set, since it's
a separate issue.
I'm not seeing where anything is broken with current mpath. If a
multipathed LUN is WCE=1 then it should be fair to assume the cache is
mirrored or shared across ports. Therefore retrying the SYNCHRONIZE
CACHE is needed.
Do we still have fear that SYNCHRONIZE CACHE can silently drop data?
Seems unlikely especially given what Tejun shared from SBC.
It seems that at worst, with current mpath, we retry when it doesn't
make sense (e.g. target failure).
Mike
From: Mike Snitzer <hidden> Date: 2010-08-25 15:59:18
On Wed, Aug 25 2010 at 4:00am -0400,
Kiyoshi Ueda [off-list ref] wrote:
quoted
I'm not sure how to proceed here. How much work would
discerning between transport and IO errors take? If it can't be done
quickly enough the retry logic can be kept around to keep the old
behavior but that already was a broken behavior, so... :-(
From: Mike Christie <hidden> Date: 2010-08-25 19:14:28
On 08/25/2010 10:59 AM, Mike Snitzer wrote:
On Wed, Aug 25 2010 at 4:00am -0400,
Kiyoshi Ueda[off-list ref] wrote:
quoted
quoted
I'm not sure how to proceed here. How much work would
discerning between transport and IO errors take? If it can't be done
quickly enough the retry logic can be kept around to keep the old
behavior but that already was a broken behavior, so... :-(
I think it is more disruptive, but is the cleaner approach in the end.
#2 looks hacky. In upper layers, we will have checks for dasd and other
AOE and other drivers. And then #2 does not even work for filesystems
(ext said they need this).
2)
Another option is Hannes' approach of having DM consume req->errors and
SCSI sense more directly.
Hi Mike,
On 08/26/2010 12:28 AM +0900, Mike Snitzer wrote:
Kiyoshi Ueda [off-list ref] wrote:
quoted
Anyway, as you said, the flush error handling of dm-mpath is already
broken if data loss really happens on any storage used by dm-mpath.
Although it's a serious issue and quick fix is required, I think
you may leave the old behavior in your patch-set, since it's
a separate issue.
I'm not seeing where anything is broken with current mpath. If a
multipathed LUN is WCE=1 then it should be fair to assume the cache is
mirrored or shared across ports. Therefore retrying the SYNCHRONIZE
CACHE is needed.
Do we still have fear that SYNCHRONIZE CACHE can silently drop data?
Seems unlikely especially given what Tejun shared from SBC.
Do we have any proof to wipe that fear?
If retrying on flush failure is safe on all storages used with multipath
(e.g. SCSI, CCISS, DASD, etc), then current dm-mpath should be fine in
the real world.
But I'm afraid if there is a storage where something like below can happen:
- a flush command is returned as error to mpath because a part of
cache has physically broken at the time or so, then that part of
data loses and the size of the cache is shrunk by the storage.
- mpath retries the flush command using other path.
- the flush command is returned as success to mpath.
- mpath passes the result, success, to upper layer, but some of
the data already lost.
Thanks,
Kiyoshi Ueda
From: Mike Snitzer <hidden> Date: 2010-08-27 13:49:40
On Fri, Aug 27 2010 at 5:47am -0400,
Kiyoshi Ueda [off-list ref] wrote:
Hi Mike,
On 08/26/2010 12:28 AM +0900, Mike Snitzer wrote:
quoted
Kiyoshi Ueda [off-list ref] wrote:
quoted
Anyway, as you said, the flush error handling of dm-mpath is already
broken if data loss really happens on any storage used by dm-mpath.
Although it's a serious issue and quick fix is required, I think
you may leave the old behavior in your patch-set, since it's
a separate issue.
I'm not seeing where anything is broken with current mpath. If a
multipathed LUN is WCE=1 then it should be fair to assume the cache is
mirrored or shared across ports. Therefore retrying the SYNCHRONIZE
CACHE is needed.
Do we still have fear that SYNCHRONIZE CACHE can silently drop data?
Seems unlikely especially given what Tejun shared from SBC.
Do we have any proof to wipe that fear?
If retrying on flush failure is safe on all storages used with multipath
(e.g. SCSI, CCISS, DASD, etc), then current dm-mpath should be fine in
the real world.
But I'm afraid if there is a storage where something like below can happen:
- a flush command is returned as error to mpath because a part of
cache has physically broken at the time or so, then that part of
data loses and the size of the cache is shrunk by the storage.
- mpath retries the flush command using other path.
- the flush command is returned as success to mpath.
- mpath passes the result, success, to upper layer, but some of
the data already lost.
That does seem like a valid concern. But I'm not seeing why its unique
to SYNCHRONIZE CACHE. Any IO that fails on the target side should be
passed up once the error gets to DM.
Mike
Hi Mike,
On 08/27/2010 10:49 PM +0900, Mike Snitzer wrote:
Kiyoshi Ueda [off-list ref] wrote:
quoted
On 08/26/2010 12:28 AM +0900, Mike Snitzer wrote:
quoted
Kiyoshi Ueda [off-list ref] wrote:
quoted
Anyway, as you said, the flush error handling of dm-mpath is already
broken if data loss really happens on any storage used by dm-mpath.
Although it's a serious issue and quick fix is required, I think
you may leave the old behavior in your patch-set, since it's
a separate issue.
I'm not seeing where anything is broken with current mpath. If a
multipathed LUN is WCE=1 then it should be fair to assume the cache is
mirrored or shared across ports. Therefore retrying the SYNCHRONIZE
CACHE is needed.
Do we still have fear that SYNCHRONIZE CACHE can silently drop data?
Seems unlikely especially given what Tejun shared from SBC.
Do we have any proof to wipe that fear?
If retrying on flush failure is safe on all storages used with multipath
(e.g. SCSI, CCISS, DASD, etc), then current dm-mpath should be fine in
the real world.
But I'm afraid if there is a storage where something like below can happen:
- a flush command is returned as error to mpath because a part of
cache has physically broken at the time or so, then that part of
data loses and the size of the cache is shrunk by the storage.
- mpath retries the flush command using other path.
- the flush command is returned as success to mpath.
- mpath passes the result, success, to upper layer, but some of
the data already lost.
That does seem like a valid concern. But I'm not seeing why its unique
to SYNCHRONIZE CACHE. Any IO that fails on the target side should be
passed up once the error gets to DM.
See the Tejun's explanation again:
http://marc.info/?l=linux-kernel&m=128267361813859&w=2
What I'm concerning is whether the same thing as Tejun explained
for ATA can happen on other types of devices.
Normal write command has data and no data loss happens on error.
So it can be retried cleanly, and if the result of the retry is
success, it's really success, no implicit data loss.
Normal read command has a sector to read. If the sector is broken,
all retries will fail and the error will be reported upwards.
So it can be retried cleanly as well.
Thanks,
Kiyoshi Ueda
From: Hannes Reinecke <hare@suse.de> Date: 2010-08-30 09:54:11
Tejun Heo wrote:
Hello,
On 08/18/2010 09:30 PM, Vladislav Bolkhovitin wrote:
quoted
Basically, I measured how iSCSI link utilization depends from amount
of queued commands and queued data size. This is why I made it as a
table. From it you can see which improvement you will have removing
queue draining after 1, 2, 4, etc. commands depending of commands
sizes.
For instance, on my previous XFS rm example, where rm of 4 files
took 3.5 minutes with nobarrier option, I could see that XFS was
sending 1-3 32K commands in a row. From my table you can see that if
it sent all them at once without draining, it would have about
150-200% speed increase.
You compared barrier off/on. Of course, it will make a big
difference. I think good part of that gain should be realized by the
currently proposed patchset which removes draining. What's needed to
be demonstrated is the difference between ordered-by-waiting and
ordered-by-tag. We've never had code to do that properly.
The original ordered-by-tag we had only applied tag ordering to two or
three command sequences inside a barrier, which doesn't amount to much
(and could even be harmful as it imposes draining of all simple
commands inside the device only to reduce issue latencies for a few
commands). You'll need to hook into filesystem and somehow export the
ordering information down to the driver so that whatever needs
ordering is sent out as ordered commands.
As I've wrote multiple times, I'm pretty skeptical it will bring much.
Ordered tag mandates draining inside the device just like the original
barrier implementation. Sure, it's done at a lower layer and command
issue latencies will be reduced thanks to that but ordered-by-waiting
doesn't require _any_ draining at all. The whole pipeline can be kept
full all the time. I'm often wrong tho, so please feel free to go
ahead and prove me wrong. :-)
Actually, I thought about ordered tag writes, too.
But eventually I had to give up on this for a simple reason:
Ordered tag controls the ordering on the SCSI _TARGET_. But for a
meaningful implementation we need to control the ordering all the way
down from ->queuecommand(). Which means we have three areas we need
to cover here:
- driver (ie between ->queuecommand() and passing it off to the firmware)
- firmware
- fabric
Sadly, the latter two are really hard to influence. And, what's more,
with the new/modern CNAs with multiple queues and possible multiple
routes to the target it becomes impossible to guarantee ordering.
So using ordered tags for FibreChannel is not going to work, which
makes implementing it a bit of a pointless exercise for me.
Cheers,
Hannes
--
Dr. Hannes Reinecke zSeries & Storage
hare@suse.de +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Markus Rex, HRB 16746 (AG Nürnberg)
From: Hannes Reinecke <hare@suse.de> Date: 2010-08-30 10:04:56
Christoph Hellwig wrote:
On Mon, Aug 23, 2010 at 04:01:15PM +0200, Jens Axboe wrote:
quoted
The problem purely exists on arrays that report write back cache enabled
AND don't implement SYNC_CACHE as a noop. Do any of them exist, or are
they purely urban legend?
I haven't seen it. I don't care particularly about this case, but once
it a while people want to disable flushing for testing or because they
really don't care.
aacraid for one falls into this category.
SYNC_CACHE is no-oped in the driver. Otherwise you get a _HUGE_
performance loss.
Cheers,
Hannes
--
Dr. Hannes Reinecke zSeries & Storage
hare@suse.de +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Markus Rex, HRB 16746 (AG Nürnberg)
From: Hannes Reinecke <hare@suse.de> Date: 2010-08-30 11:38:54
Mike Snitzer wrote:
On Wed, Aug 25 2010 at 4:00am -0400,
Kiyoshi Ueda [off-list ref] wrote:
quoted
quoted
I'm not sure how to proceed here. How much work would
discerning between transport and IO errors take? If it can't be done
quickly enough the retry logic can be kept around to keep the old
behavior but that already was a broken behavior, so... :-(
Actually, I think we have two separate issues here:
1) The need of having more detailed I/O errors even in the fs layer. This
we've already discussed at the LSF, consensus here is to allow other
errors than just 'EIO'.
Instead of Mike's approach I would rather use existing error codes here;
this will make the transition somewhat easier.
Initially I would propose to return 'ENOLINK' for a transport failure,
'EIO' for a non-retryable failure on the target, and 'ENODEV' for a
retryable failure on the target.
2) The need to differentiate the various error conditions on the multipath
layer. Multipath needs to distinguish the three error types as specified
in 1)
Mike has been trying to solve 1) and 2) by introducing separate/new error
codes, and I have been trying to use 2) by parsing the sense codes directly
from multipathing.
Given that the fs people have expressed their desire to know about these
error classes, too, it makes sense to have them exposed to the fs layer.
I see if I can come up with a patch.
Cheers,
Hannes
--
Dr. Hannes Reinecke zSeries & Storage
hare@suse.de +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Markus Rex, HRB 16746 (AG Nürnberg)
From: Sergei Shtylyov <hidden> Date: 2010-08-30 12:07:48
Hello.
Hannes Reinecke wrote:
Actually, I think we have two separate issues here:
1) The need of having more detailed I/O errors even in the fs layer. This
we've already discussed at the LSF, consensus here is to allow other
errors than just 'EIO'.
Instead of Mike's approach I would rather use existing error codes here;
this will make the transition somewhat easier.
Initially I would propose to return 'ENOLINK' for a transport failure,
'EIO' for a non-retryable failure on the target, and 'ENODEV' for a
retryable failure on the target.
Are you sure it's not vice versa: EIO for retryable and ENODEV for
non-retryable failures. ENODEV looks more like permanent condition to me.
WBR, Sergei
From: Hannes Reinecke <hare@suse.de> Date: 2010-08-30 12:39:12
Sergei Shtylyov wrote:
Hello.
Hannes Reinecke wrote:
quoted
Actually, I think we have two separate issues here:
1) The need of having more detailed I/O errors even in the fs layer. This
we've already discussed at the LSF, consensus here is to allow other
errors than just 'EIO'.
Instead of Mike's approach I would rather use existing error codes
here;
this will make the transition somewhat easier.
Initially I would propose to return 'ENOLINK' for a transport failure,
'EIO' for a non-retryable failure on the target, and 'ENODEV' for a
retryable failure on the target.
Are you sure it's not vice versa: EIO for retryable and ENODEV for
non-retryable failures. ENODEV looks more like permanent condition to me.
Ok, can do.
And looking a the error numbers again, maybe we should be using 'EREMOTEIO'
for non-retryable failures.
So we would be ending with:
ENOLINK: transport failure
EIO: retryable remote failure
EREMOTEIO: non-retryable remote failure
Does that look okay?
Cheers,
Hannes
--
Dr. Hannes Reinecke zSeries & Storage
hare@suse.de +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Markus Rex, HRB 16746 (AG Nürnberg)
From: Hannes Reinecke <hare@suse.de> Date: 2010-08-30 14:52:36
Hannes Reinecke wrote:
Sergei Shtylyov wrote:
quoted
Hello.
Hannes Reinecke wrote:
quoted
Actually, I think we have two separate issues here:
1) The need of having more detailed I/O errors even in the fs layer. This
we've already discussed at the LSF, consensus here is to allow other
errors than just 'EIO'.
Instead of Mike's approach I would rather use existing error codes
here;
this will make the transition somewhat easier.
Initially I would propose to return 'ENOLINK' for a transport failure,
'EIO' for a non-retryable failure on the target, and 'ENODEV' for a
retryable failure on the target.
Are you sure it's not vice versa: EIO for retryable and ENODEV for
non-retryable failures. ENODEV looks more like permanent condition to me.
Ok, can do.
And looking a the error numbers again, maybe we should be using 'EREMOTEIO'
for non-retryable failures.
So we would be ending with:
ENOLINK: transport failure
EIO: retryable remote failure
EREMOTEIO: non-retryable remote failure
And here is the corresponding patch.
Compile tested only; just to give an idea of the possible implementation.
I have decided to pass the I/O failure information in-line:
- scsi_check_sense() might now return 'TARGET_ERROR' to signal
a permanent error
- scsi_decide_disposition() sets the driver byte of the result
field to 'DID_TARGET_FAILURE' if a return code of 'TARGET_ERROR'
is encountered.
- scsi_io_completion() sets the error to ENOLINK for DID_TRANSPORT_FAILFAST,
EREMOTEIO for DID_TARGET_FAILURE, and EIO for any other error. It also
resets DID_TARGET_FAILURE back to DID_OK once the error code is set.
I'm not 100% happy with this patch; DID_TARGET_FAILURE is really just
a communication vehicle to signal the permanent target failure.
I looked at passing this information directly via an explicit argument
to scsi_finish_command(), but this would include changing
scsi_io_completion(), too. As both of them are exported / public
interfaces I didn't like modifying them.
Another possibility would be to re-use / redefine the 'DRIVER_'
bits; they don't seem to be used a the moment. Eg 'DRIVER_HARD'
for permanent errors, DRIVER_SOFT for link failures.
Opinions welcome.
Cheers,
Hannes
--
Dr. Hannes Reinecke zSeries & Storage
hare@suse.de +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Markus Rex, HRB 16746 (AG Nürnberg)
As I've wrote multiple times, I'm pretty skeptical it will bring much.
Ordered tag mandates draining inside the device just like the original
barrier implementation. Sure, it's done at a lower layer and command
issue latencies will be reduced thanks to that but ordered-by-waiting
doesn't require _any_ draining at all. The whole pipeline can be kept
full all the time. I'm often wrong tho, so please feel free to go
ahead and prove me wrong. :-)
Actually, I thought about ordered tag writes, too.
But eventually I had to give up on this for a simple reason:
Ordered tag controls the ordering on the SCSI _TARGET_. But for a
meaningful implementation we need to control the ordering all the way
down from ->queuecommand(). Which means we have three areas we need
to cover here:
- driver (ie between ->queuecommand() and passing it off to the firmware)
- firmware
- fabric
Sadly, the latter two are really hard to influence. And, what's more,
with the new/modern CNAs with multiple queues and possible multiple
routes to the target it becomes impossible to guarantee ordering.
So using ordered tags for FibreChannel is not going to work, which
makes implementing it a bit of a pointless exercise for me.
The situation is, actually, much better than you think. An SCSI
transport should provide an in-order delivery of commands. In some
transports it is required (e.g. iSCSI), in some - optional (e.g. FC).
For FC "an application client may determine if a device server supports
the precise delivery function by using the MODE SENSE and MODE SELECT
commands to examine and set the enable precise delivery checking (EPDC)
bit in the Fibre Channel Logical Unit Control page" (Fibre Channel
Protocol for SCSI (FCP)). You can find more details in FCP section
"Precise delivery of SCSI commands".
Regarding multiple queues, in case of a multipath access to a device
SCSI requires either each path be a separate I_T nexus, where order of
commands is maintained, or a transport required to maintain in-order
commands delivery among multiple paths in a single I_T nexus (session)
as it is done in iSCSI's MC/S and, most likely, wide SAS ports.
So, everything is in the specs. We only need to use it properly. How it
can be done on the drivers level as well as how errors recovery can be
done using ACA and UA_INTLCK facilities I wrote few weeks ago in the
"[RFC] relaxed barrier semantics" thread.
Vlad
From: Mike Snitzer <hidden> Date: 2010-09-01 00:55:37
Hi Kiyoshi,
On Mon, Aug 30 2010 at 2:13am -0400,
Kiyoshi Ueda [off-list ref] wrote:
quoted
That does seem like a valid concern. But I'm not seeing why its unique
to SYNCHRONIZE CACHE. Any IO that fails on the target side should be
passed up once the error gets to DM.
See the Tejun's explanation again:
http://marc.info/?l=linux-kernel&m=128267361813859&w=2
What I'm concerning is whether the same thing as Tejun explained
for ATA can happen on other types of devices.
Normal write command has data and no data loss happens on error.
So it can be retried cleanly, and if the result of the retry is
success, it's really success, no implicit data loss.
Normal read command has a sector to read. If the sector is broken,
all retries will fail and the error will be reported upwards.
So it can be retried cleanly as well.
I reached out to Fred Knight on this, to get a more insight from a pure
SCSI SBC perspective, and he shared the following:
----- Forwarded message from "Knight, Frederick" [off-list ref] -----
Date: Tue, 31 Aug 2010 13:24:15 -0400
From: "Knight, Frederick" <redacted>
To: Mike Snitzer <redacted>
Subject: RE: safety of retrying SYNCHRONIZE CACHE?
There are requirements in SBC to maintain data integrity. If you WRITE
a block and READ that block, you must get the data you sent in the
WRITE. This will be synchronized around the completion of the WRITE.
Before the WRITE completes, who knows what a READ will return. Maybe
all the old data, maybe all the new data, maybe some mix of old and new
data. Once the WRITE ends successful, all READs of those LBAs (from any
port) will always get the same data.
As for errors, SBC describes how the deferred errors are reported (like
when a CACHE tries to flush but fails). So if a write from cache to
media does have problems, the device would tell you via a CHECK
CONDITION (with the first byte of the sense data set to 71h or 73h. SBC
clause 4.12 and 4.13 cover a lot of this information. It is these error
codes that prevent silent loss of data. And, in this case, when the
CHECK CONDITION is delivered, it will have nothing to do with the
command that was issued (the victim command). If you look into the
sense data, you will see the deferred error flag, and all the additional
information fields will relate to the original I/O
SYNCHRONIZE CACHE is not substantially different than a WRITE (it puts
data on the media). So issuing it multiple times wouldn't be any
different than issuing multiple WRITES (it might put a temporary dent in
performance as everything flushes out to media). If it or any other
commands fail with 71h/73h, then you have to dig down into the sense
data buffer to find out what happened. For example, if you issue a
WRITE command, and it completes into write back cache but later (before
being written to the media), some of the cache breaks and looses data,
then the device must signal a deferred error to tell the host, and cause
a forced error on the LBA in question.
Does that help?
Fred
----- End forwarded message -----
Seems like verifying/improving the handling of CHECK CONDITION is a more
pressing concern than silent data loss purely due to SYNCHRONIZE CACHE
retries. Without proper handling we could completely miss these
deferred errors.
But how to effectively report such errors to upper layers is unclear to
me given that a particular SCSI command can carry error information for
IO that was already acknowledged successful (e.g. to the FS).
drivers/scsi/scsi_error.c's various calls to scsi_check_sense()
illustrate Linux's current CHECK CONDITION handling. I need to look
closer at how deferred errors propagate to upper layers. After an
initial look it seems scsi_error.c does handle retrying commands where
appropriate.
I believe Hannes has concerns/insight here.
Mike
From: Hannes Reinecke <hare@suse.de> Date: 2010-09-01 07:32:29
Mike Snitzer wrote:
Hi Kiyoshi,
On Mon, Aug 30 2010 at 2:13am -0400,
Kiyoshi Ueda [off-list ref] wrote:
quoted
quoted
That does seem like a valid concern. But I'm not seeing why its unique
to SYNCHRONIZE CACHE. Any IO that fails on the target side should be
passed up once the error gets to DM.
See the Tejun's explanation again:
http://marc.info/?l=linux-kernel&m=128267361813859&w=2
What I'm concerning is whether the same thing as Tejun explained
for ATA can happen on other types of devices.
Normal write command has data and no data loss happens on error.
So it can be retried cleanly, and if the result of the retry is
success, it's really success, no implicit data loss.
Normal read command has a sector to read. If the sector is broken,
all retries will fail and the error will be reported upwards.
So it can be retried cleanly as well.
I reached out to Fred Knight on this, to get a more insight from a pure
SCSI SBC perspective, and he shared the following:
----- Forwarded message from "Knight, Frederick" [off-list ref] -----
quoted
Date: Tue, 31 Aug 2010 13:24:15 -0400
From: "Knight, Frederick" <redacted>
To: Mike Snitzer <redacted>
Subject: RE: safety of retrying SYNCHRONIZE CACHE?
There are requirements in SBC to maintain data integrity. If you WRITE
a block and READ that block, you must get the data you sent in the
WRITE. This will be synchronized around the completion of the WRITE.
Before the WRITE completes, who knows what a READ will return. Maybe
all the old data, maybe all the new data, maybe some mix of old and new
data. Once the WRITE ends successful, all READs of those LBAs (from any
port) will always get the same data.
As for errors, SBC describes how the deferred errors are reported (like
when a CACHE tries to flush but fails). So if a write from cache to
media does have problems, the device would tell you via a CHECK
CONDITION (with the first byte of the sense data set to 71h or 73h. SBC
clause 4.12 and 4.13 cover a lot of this information. It is these error
codes that prevent silent loss of data. And, in this case, when the
CHECK CONDITION is delivered, it will have nothing to do with the
command that was issued (the victim command). If you look into the
sense data, you will see the deferred error flag, and all the additional
information fields will relate to the original I/O
SYNCHRONIZE CACHE is not substantially different than a WRITE (it puts
data on the media). So issuing it multiple times wouldn't be any
different than issuing multiple WRITES (it might put a temporary dent in
performance as everything flushes out to media). If it or any other
commands fail with 71h/73h, then you have to dig down into the sense
data buffer to find out what happened. For example, if you issue a
WRITE command, and it completes into write back cache but later (before
being written to the media), some of the cache breaks and looses data,
then the device must signal a deferred error to tell the host, and cause
a forced error on the LBA in question.
Does that help?
Fred
----- End forwarded message -----
Seems like verifying/improving the handling of CHECK CONDITION is a more
pressing concern than silent data loss purely due to SYNCHRONIZE CACHE
retries. Without proper handling we could completely miss these
deferred errors.
Yes.
But how to effectively report such errors to upper layers is unclear to
me given that a particular SCSI command can carry error information for
IO that was already acknowledged successful (e.g. to the FS).
drivers/scsi/scsi_error.c's various calls to scsi_check_sense()
illustrate Linux's current CHECK CONDITION handling. I need to look
closer at how deferred errors propagate to upper layers. After an
initial look it seems scsi_error.c does handle retrying commands where
appropriate.
I believe Hannes has concerns/insight here.
Quite. We _should_ be handling deferred errors correctly;
if you check drivers/scsi/scsi_lib.c:scsi_io_completion()
you'll find this:
if (host_byte(result) == DID_RESET) {
/* Third party bus reset or reset for error recovery
* reasons. Just retry the command and see what
* happens.
*/
action = ACTION_RETRY;
} else if (sense_valid && !sense_deferred) {
...
} else {
description = "Unhandled error code";
action = ACTION_FAIL;
}
ie for deferred errors we're already aborting the command. Not sure
if I agree with this bit in drivers/scsi/scsi_lib.c:
static int scsi_check_sense(struct scsi_cmnd *scmd)
{
struct scsi_device *sdev = scmd->device;
struct scsi_sense_hdr sshdr;
if (! scsi_command_normalize_sense(scmd, &sshdr))
return FAILED; /* no valid sense data */
if (scsi_sense_is_deferred(&sshdr))
return NEEDS_RETRY;
I doubt we can resolve the situation by retrying the command, which
will be the wrong command to retry anyway. I would rather
have those retry 'SUCCESS' and add another case in scsi_io_completion()
to notify us about the deferred error.
I'll be sending a patch.
Cheers,
Hannes
--
Dr. Hannes Reinecke zSeries & Storage
hare@suse.de +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Markus Rex, HRB 16746 (AG Nürnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
From: Hannes Reinecke <hare@suse.de> Date: 2010-09-01 07:38:26
Hannes Reinecke wrote:
Mike Snitzer wrote:
quoted
Hi Kiyoshi,
On Mon, Aug 30 2010 at 2:13am -0400,
Kiyoshi Ueda [off-list ref] wrote:
quoted
quoted
That does seem like a valid concern. But I'm not seeing why its unique
to SYNCHRONIZE CACHE. Any IO that fails on the target side should be
passed up once the error gets to DM.
See the Tejun's explanation again:
http://marc.info/?l=linux-kernel&m=128267361813859&w=2
What I'm concerning is whether the same thing as Tejun explained
for ATA can happen on other types of devices.
Normal write command has data and no data loss happens on error.
So it can be retried cleanly, and if the result of the retry is
success, it's really success, no implicit data loss.
Normal read command has a sector to read. If the sector is broken,
all retries will fail and the error will be reported upwards.
So it can be retried cleanly as well.
I reached out to Fred Knight on this, to get a more insight from a pure
SCSI SBC perspective, and he shared the following:
----- Forwarded message from "Knight, Frederick" [off-list ref] -----
quoted
Date: Tue, 31 Aug 2010 13:24:15 -0400
From: "Knight, Frederick" <redacted>
To: Mike Snitzer <redacted>
Subject: RE: safety of retrying SYNCHRONIZE CACHE?
There are requirements in SBC to maintain data integrity. If you WRITE
a block and READ that block, you must get the data you sent in the
WRITE. This will be synchronized around the completion of the WRITE.
Before the WRITE completes, who knows what a READ will return. Maybe
all the old data, maybe all the new data, maybe some mix of old and new
data. Once the WRITE ends successful, all READs of those LBAs (from any
port) will always get the same data.
As for errors, SBC describes how the deferred errors are reported (like
when a CACHE tries to flush but fails). So if a write from cache to
media does have problems, the device would tell you via a CHECK
CONDITION (with the first byte of the sense data set to 71h or 73h. SBC
clause 4.12 and 4.13 cover a lot of this information. It is these error
codes that prevent silent loss of data. And, in this case, when the
CHECK CONDITION is delivered, it will have nothing to do with the
command that was issued (the victim command). If you look into the
sense data, you will see the deferred error flag, and all the additional
information fields will relate to the original I/O
SYNCHRONIZE CACHE is not substantially different than a WRITE (it puts
data on the media). So issuing it multiple times wouldn't be any
different than issuing multiple WRITES (it might put a temporary dent in
performance as everything flushes out to media). If it or any other
commands fail with 71h/73h, then you have to dig down into the sense
data buffer to find out what happened. For example, if you issue a
WRITE command, and it completes into write back cache but later (before
being written to the media), some of the cache breaks and looses data,
then the device must signal a deferred error to tell the host, and cause
a forced error on the LBA in question.
Does that help?
Fred
----- End forwarded message -----
Seems like verifying/improving the handling of CHECK CONDITION is a more
pressing concern than silent data loss purely due to SYNCHRONIZE CACHE
retries. Without proper handling we could completely miss these
deferred errors.
Yes.
quoted
But how to effectively report such errors to upper layers is unclear to
me given that a particular SCSI command can carry error information for
IO that was already acknowledged successful (e.g. to the FS).
drivers/scsi/scsi_error.c's various calls to scsi_check_sense()
illustrate Linux's current CHECK CONDITION handling. I need to look
closer at how deferred errors propagate to upper layers. After an
initial look it seems scsi_error.c does handle retrying commands where
appropriate.
I believe Hannes has concerns/insight here.
Quite. We _should_ be handling deferred errors correctly;
if you check drivers/scsi/scsi_lib.c:scsi_io_completion()
you'll find this:
if (host_byte(result) == DID_RESET) {
/* Third party bus reset or reset for error recovery
* reasons. Just retry the command and see what
* happens.
*/
action = ACTION_RETRY;
} else if (sense_valid && !sense_deferred) {
...
} else {
description = "Unhandled error code";
action = ACTION_FAIL;
}
ie for deferred errors we're already aborting the command. Not sure
if I agree with this bit in drivers/scsi/scsi_lib.c:
static int scsi_check_sense(struct scsi_cmnd *scmd)
{
struct scsi_device *sdev = scmd->device;
struct scsi_sense_hdr sshdr;
if (! scsi_command_normalize_sense(scmd, &sshdr))
return FAILED; /* no valid sense data */
if (scsi_sense_is_deferred(&sshdr))
return NEEDS_RETRY;
I doubt we can resolve the situation by retrying the command, which
will be the wrong command to retry anyway. I would rather
have those retry 'SUCCESS' and add another case in scsi_io_completion()
to notify us about the deferred error.
Ah. No. That is actually correct. SPC-3 states:
If the task terminates with CHECK CONDITION status and the sense data
describes a deferred error, the command for the terminated task shall
not have been processed.
So we're good after all and I would just add this patch:
to make the whole situation more transparent.
Cheers,
Hannes
--
Dr. Hannes Reinecke zSeries & Storage
hare@suse.de +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Markus Rex, HRB 16746 (AG Nürnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Hi Hannes,
Thank you for working on this issue and sorry for very late reply...
(08/30/10 23:52), Hannes Reinecke wrote:
From: Hannes Reinecke <hare@suse.de>
Date: Mon, 30 Aug 2010 16:21:10 +0200
Subject: [RFC][PATCH] scsi: Detailed I/O errors
Instead of just passing 'EIO' for any I/O errors we should be
notifying the upper layers with some more details about the cause
of this error.
This patch updates the possible I/O errors to:
- ENOLINK: Link failure between host and target
- EIO: Retryable I/O error
- EREMOTEIO: Non-retryable I/O error
'Retryable' in this context means that an I/O error _might_ be
restricted to the I_T_L nexus (vulgo: path), so retrying on another
nexus / path might succeed.
Does 'retryable' of EIO mean retryable in multipath layer?
If so, what is the difference between EIO and ENOLINK?
I've heard of a case where just retrying within path-group is
preferred to (relatively costly) switching group.
So, if EIO (or other error code) can be used to indicate such type
of errors, it's nice.
Also (although this might be a bit off topic from your patch),
can we expand such a distinction to what should be logged?
Currently, it's difficult to distinguish important SCSI/block errors
and less important ones in kernel log.
For example, when I get a link failure on sda, kernel prints something
like below, regardless of whether the I/O is recovered by multipathing or not:
end_request: I/O error, dev sda, sector XXXXX
Setting REQ_QUIET in dm-multipath could mask the message
but also other important ones in SCSI.
Thanks,
--
Jun'ichi Nomura, NEC Corporation
From: Hannes Reinecke <hare@suse.de> Date: 2010-10-18 11:55:26
On 10/18/2010 10:09 AM, Jun'ichi Nomura wrote:
Hi Hannes,
Thank you for working on this issue and sorry for very late reply...
(08/30/10 23:52), Hannes Reinecke wrote:
quoted
From: Hannes Reinecke <hare@suse.de>
Date: Mon, 30 Aug 2010 16:21:10 +0200
Subject: [RFC][PATCH] scsi: Detailed I/O errors
Instead of just passing 'EIO' for any I/O errors we should be
notifying the upper layers with some more details about the cause
of this error.
This patch updates the possible I/O errors to:
- ENOLINK: Link failure between host and target
- EIO: Retryable I/O error
- EREMOTEIO: Non-retryable I/O error
'Retryable' in this context means that an I/O error _might_ be
restricted to the I_T_L nexus (vulgo: path), so retrying on another
nexus / path might succeed.
Does 'retryable' of EIO mean retryable in multipath layer?
If so, what is the difference between EIO and ENOLINK?
Yes, EIO is intended for errors which should be retried at the
multipath layer. This does _not_ include transport errors, which are
signalled by ENOLINK.
Basically, ENOLINK is a transport error, and EIO just means
something is wrong and we weren't able to classify it properly.
If we were, it'd be either ENOLINK or EREMOTEIO.
I've heard of a case where just retrying within path-group is
preferred to (relatively costly) switching group.
So, if EIO (or other error code) can be used to indicate such type
of errors, it's nice.
Yes, that was one of the intention.
Also (although this might be a bit off topic from your patch),
can we expand such a distinction to what should be logged?
Currently, it's difficult to distinguish important SCSI/block errors
and less important ones in kernel log.
For example, when I get a link failure on sda, kernel prints something
like below, regardless of whether the I/O is recovered by multipathing or not:
end_request: I/O error, dev sda, sector XXXXX
Indeed, when using the above we could be modifying the above
message, eg by
end_request: transport error, dev sda, sector XXXXX
or
end_request: target error, dev sda, sector XXXXX
which would improve the output noticeable.
Setting REQ_QUIET in dm-multipath could mask the message
but also other important ones in SCSI.
Hmm. Not sure about that, but I think the above modifications will
be useful already.
I'll be sending an updated patch.
Cheers,
Hannes
--
Dr. Hannes Reinecke zSeries & Storage
hare@suse.de +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Markus Rex, HRB 16746 (AG Nürnberg)
Hi Hannes,
(10/18/10 20:55), Hannes Reinecke wrote:
quoted
Does 'retryable' of EIO mean retryable in multipath layer?
If so, what is the difference between EIO and ENOLINK?
Yes, EIO is intended for errors which should be retried at the
multipath layer. This does _not_ include transport errors, which are
signalled by ENOLINK.
Basically, ENOLINK is a transport error, and EIO just means
something is wrong and we weren't able to classify it properly.
If we were, it'd be either ENOLINK or EREMOTEIO.
quoted
I've heard of a case where just retrying within path-group is
preferred to (relatively costly) switching group.
So, if EIO (or other error code) can be used to indicate such type
of errors, it's nice.
Yes, that was one of the intention.
Great to hear that.
And when it comes to retrying, the next problem is who controls it.
I don't think it's good to duplicate retry logic in multipath and
underlying device like SCSI (i.e. sd retries 5 times).
So perhaps we need a way to disable (or limit) retries in underlying
device at least.
quoted
Also (although this might be a bit off topic from your patch),
can we expand such a distinction to what should be logged?
Currently, it's difficult to distinguish important SCSI/block errors
and less important ones in kernel log.
For example, when I get a link failure on sda, kernel prints something
like below, regardless of whether the I/O is recovered by multipathing or not:
end_request: I/O error, dev sda, sector XXXXX
Indeed, when using the above we could be modifying the above
message, eg by
end_request: transport error, dev sda, sector XXXXX
or
end_request: target error, dev sda, sector XXXXX
which would improve the output noticeable.
It improves but still they look like critical errors
even if multipath saves them.
When I see this:
end_request: target error, dev sda, sector XXXXX
I can't tell whether it's a real error visible to user space
or it's just recoverred by multipath retry/failover afterwards.
quoted
Setting REQ_QUIET in dm-multipath could mask the message
but also other important ones in SCSI.
Hmm. Not sure about that, but I think the above modifications will
be useful already.
I'll be sending an updated patch.
Thank you. I'm looking for that.
--
Jun'ichi Nomura, NEC Corporation
Also (although this might be a bit off topic from your patch),
can we expand such a distinction to what should be logged?
Currently, it's difficult to distinguish important SCSI/block errors
and less important ones in kernel log.
For example, when I get a link failure on sda, kernel prints something
like below, regardless of whether the I/O is recovered by multipathing or not:
end_request: I/O error, dev sda, sector XXXXX
Indeed, when using the above we could be modifying the above
message, eg by
end_request: transport error, dev sda, sector XXXXX
or
end_request: target error, dev sda, sector XXXXX
which would improve the output noticeable.
quoted
Setting REQ_QUIET in dm-multipath could mask the message
but also other important ones in SCSI.
Hmm. Not sure about that, but I think the above modifications will
be useful already.
I'll be sending an updated patch.
Hannes, is there an updated version of this patch? It applied fine with
Linus git tree with a minor reject! I would like to test an updated
version if you have one (the update seems to refer to better logging
only, right?).
Thanks, Malahal.
From: Mike Snitzer <hidden> Date: 2010-11-30 22:59:56
On Thu, Nov 18 2010 at 10:11pm -0500,
Malahal Naineni [off-list ref] wrote:
Hannes Reinecke [hare@suse.de] wrote:
quoted
quoted
Also (although this might be a bit off topic from your patch),
can we expand such a distinction to what should be logged?
Currently, it's difficult to distinguish important SCSI/block errors
and less important ones in kernel log.
For example, when I get a link failure on sda, kernel prints something
like below, regardless of whether the I/O is recovered by multipathing or not:
end_request: I/O error, dev sda, sector XXXXX
Indeed, when using the above we could be modifying the above
message, eg by
end_request: transport error, dev sda, sector XXXXX
or
end_request: target error, dev sda, sector XXXXX
which would improve the output noticeable.
quoted
Setting REQ_QUIET in dm-multipath could mask the message
but also other important ones in SCSI.
Hmm. Not sure about that, but I think the above modifications will
be useful already.
I'll be sending an updated patch.
Hannes, is there an updated version of this patch? It applied fine with
Linus git tree with a minor reject! I would like to test an updated
version if you have one (the update seems to refer to better logging
only, right?).
Hannes,
Any chance you've had time to fold your proposed logging changes in and
rebase this patch? Could you post that updated patch?
I'd like to help see this patch through to inclussion when 2.6.38 merge
window opens. I can help with further review, testing and development.
Please advise, thanks.
Mike
From: Mike Snitzer <hidden> Date: 2010-12-07 23:16:39
Refreshed Hannes' initial "scsi: Detailed I/O errors" patch against
v2.6.37-rc5. v2 introduces __scsi_error_from_host_byte to avoid
the duplicate switch statement. Also a few whitespace and comment
changes.
Split DM mpath change out to separate v2 patch; failed discard is now
retryable in the face of a non-target IO error.
Added improved block layer's I/O error message (based on the finer
grained I/O error returns afforded by SCSI).
Comments/suggestions are welcome.
Thanks,
Mike
Hannes Reinecke (1):
scsi: Detailed I/O errors
Mike Snitzer (2):
dm mpath: propagate target I/O errors immediately
block: improve detail in I/O error messages
block/blk-core.c | 12 +++++++++---
drivers/md/dm-mpath.c | 11 +----------
drivers/scsi/scsi_error.c | 24 +++++++++++++++++-------
drivers/scsi/scsi_lib.c | 24 ++++++++++++++++++++++--
include/scsi/scsi.h | 3 +++
5 files changed, 52 insertions(+), 22 deletions(-)
--
1.7.2.3
From: Mike Snitzer <hidden> Date: 2010-12-07 23:16:40
From: Hannes Reinecke <hare@suse.de>
Instead of just passing 'EIO' for any I/O error we should be
notifying the upper layers with more details about the cause
of this error.
Update the possible I/O errors to:
- ENOLINK: Link failure between host and target
- EIO: Retryable I/O error
- EREMOTEIO: Non-retryable I/O error
'Retryable' in this context means that an I/O error _might_ be
restricted to the I_T_L nexus (vulgo: path), so retrying on another
nexus / path might succeed.
'Non-retryable' means target failure or reservation conflict.
Signed-off-by: Hannes Reinecke <hare@suse.de>
Signed-off-by: Mike Snitzer <redacted>
---
drivers/scsi/scsi_error.c | 24 +++++++++++++++++-------
drivers/scsi/scsi_lib.c | 24 ++++++++++++++++++++++--
include/scsi/scsi.h | 3 +++
3 files changed, 42 insertions(+), 9 deletions(-)
@@ -326,17 +326,19 @@ static int scsi_check_sense(struct scsi_cmnd *scmd)*/returnSUCCESS;-/* these three are not supported */+/* these are not supported */caseCOPY_ABORTED:caseVOLUME_OVERFLOW:caseMISCOMPARE:-returnSUCCESS;+caseBLANK_CHECK:+caseDATA_PROTECT:+returnTARGET_ERROR;caseMEDIUM_ERROR:if(sshdr.asc==0x11||/* UNRECOVERED READ ERR */sshdr.asc==0x13||/* AMNF DATA FIELD */sshdr.asc==0x14){/* RECORD NOT FOUND */-returnSUCCESS;+returnTARGET_ERROR;}returnNEEDS_RETRY;
@@ -344,11 +346,9 @@ static int scsi_check_sense(struct scsi_cmnd *scmd)if(scmd->device->retry_hwerror)returnADD_TO_MLQUEUE;else-returnSUCCESS;+returnTARGET_ERROR;caseILLEGAL_REQUEST:-caseBLANK_CHECK:-caseDATA_PROTECT:default:returnSUCCESS;}
@@ -1502,6 +1503,14 @@ int scsi_decide_disposition(struct scsi_cmnd *scmd)rtn=scsi_check_sense(scmd);if(rtn==NEEDS_RETRY)gotomaybe_retry;+elseif(rtn==TARGET_ERROR){+/*+*Needtomodifyhostbytetosignala+*permanenttargetfailure+*/+scmd->result|=(DID_TARGET_FAILURE<<16);+rtn=SUCCESS;+}/* if rtn == FAILED, we have no sense information;*returningFAILEDwillwaketheerrorhandlerthread*tocollectthesenseandredothedecide
@@ -796,7 +816,7 @@ void scsi_io_completion(struct scsi_cmnd *cmd, unsigned int good_bytes)if(scsi_end_request(cmd,error,good_bytes,result==0)==NULL)return;-error=-EIO;+error=__scsi_error_from_host_byte(cmd,result);if(host_byte(result)==DID_RESET){/* Third party bus reset or reset for error recovery
@@ -405,6 +405,8 @@ static inline int scsi_is_wlun(unsigned int lun)*recoverthelink.Transportclasswill*retryorfailIO*/#define DID_TRANSPORT_FAILFAST 0x0f /* Transport class fastfailed the io */+#define DID_TARGET_FAILURE 0x10 /* Permanent target failure, do not retry on+*otherpaths*/#define DRIVER_OK 0x00 /* Driver status *//*
@@ -434,6 +436,7 @@ static inline int scsi_is_wlun(unsigned int lun)#define TIMEOUT_ERROR 0x2007#define SCSI_RETURN_NOT_HANDLED 0x2008#define FAST_IO_FAIL 0x2009+#define TARGET_ERROR 0x200A/**Midlevelqueuereturnvalues.
From: Mike Snitzer <hidden> Date: 2010-12-07 23:16:41
DM now has more information about the nature of the underlying storage
failure. Path failure is avoided if a request failed due to a target
error. Instead the target error is immediately passed up the stack.
Discard requests that fail due to non-target errors may now be retried.
Signed-off-by: Mike Snitzer <redacted>
---
drivers/md/dm-mpath.c | 11 +----------
1 files changed, 1 insertions(+), 10 deletions(-)
Refreshed Hannes' initial "scsi: Detailed I/O errors" patch against
v2.6.37-rc5. v2 introduces __scsi_error_from_host_byte to avoid
the duplicate switch statement. Also a few whitespace and comment
changes.
Split DM mpath change out to separate v2 patch; failed discard is now
retryable in the face of a non-target IO error.
Added improved block layer's I/O error message (based on the finer
grained I/O error returns afforded by SCSI).
Comments/suggestions are welcome.
I did test the Hannes original patch with the latest Linus' git tree! I
used scsi_debug to simulate path failures as well as 'Media' failures
and it did work as expected. I will test your patches soon.
Thanks, Malahal.
From: Hannes Reinecke <hare@suse.de> Date: 2010-12-17 09:47:33
On 11/30/2010 11:59 PM, Mike Snitzer wrote:
On Thu, Nov 18 2010 at 10:11pm -0500,
Malahal Naineni [off-list ref] wrote:
quoted
Hannes Reinecke [hare@suse.de] wrote:
quoted
quoted
Also (although this might be a bit off topic from your patch),
can we expand such a distinction to what should be logged?
Currently, it's difficult to distinguish important SCSI/block errors
and less important ones in kernel log.
For example, when I get a link failure on sda, kernel prints something
like below, regardless of whether the I/O is recovered by multipathing or not:
end_request: I/O error, dev sda, sector XXXXX
Indeed, when using the above we could be modifying the above
message, eg by
end_request: transport error, dev sda, sector XXXXX
or
end_request: target error, dev sda, sector XXXXX
which would improve the output noticeable.
quoted
Setting REQ_QUIET in dm-multipath could mask the message
but also other important ones in SCSI.
Hmm. Not sure about that, but I think the above modifications will
be useful already.
I'll be sending an updated patch.
Hannes, is there an updated version of this patch? It applied fine with
Linus git tree with a minor reject! I would like to test an updated
version if you have one (the update seems to refer to better logging
only, right?).
Hannes,
Any chance you've had time to fold your proposed logging changes in and
rebase this patch? Could you post that updated patch?
yes, will be following shortly.
I'd like to help see this patch through to inclussion when 2.6.38 merge
window opens. I can help with further review, testing and development.
Ok, thanks.
Cheers,
Hannes
--
Dr. Hannes Reinecke zSeries & Storage
hare@suse.de +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Markus Rex, HRB 16746 (AG Nürnberg)
Hannes,
Any update? I'd really like to see this work get upstream ASAP. I'm
doubtful that is possible for 2.6.38 given the merge window is likely to
close shortly.
Regardless, if we could get consensus on this work now and then stage it
with James that would be great.
Thanks,
Mike
From: Mike Snitzer <hidden> Date: 2011-01-14 01:16:21
On Fri, Dec 10 2010 at 6:40pm -0500,
Malahal Naineni [off-list ref] wrote:
Mike Snitzer [snitzer@redhat.com] wrote:
quoted
Refreshed Hannes' initial "scsi: Detailed I/O errors" patch against
v2.6.37-rc5. v2 introduces __scsi_error_from_host_byte to avoid
the duplicate switch statement. Also a few whitespace and comment
changes.
Split DM mpath change out to separate v2 patch; failed discard is now
retryable in the face of a non-target IO error.
Added improved block layer's I/O error message (based on the finer
grained I/O error returns afforded by SCSI).
Comments/suggestions are welcome.
I did test the Hannes original patch with the latest Linus' git tree! I
used scsi_debug to simulate path failures as well as 'Media' failures
and it did work as expected. I will test your patches soon.
Hannes,
Any update? I'd really like to see this work get upstream ASAP. I'm
doubtful that is possible for 2.6.38 given the merge window is likely to
close shortly.
Sorry for the late answer; the above patches somehow got lost in my
various mail folders :-(
But yes, the patchset looks okay.
Feel free to add my Acked-by: to last two.
Regardless, if we could get consensus on this work now and then stage it
with James that would be great.
Indeed. Will you resend it to linux-scsi or shall I do it?
Cheers,
Hannes
--
Dr. Hannes Reinecke zSeries & Storage
hare@suse.de +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Markus Rex, HRB 16746 (AG Nürnberg)
Hannes,
Any update? I'd really like to see this work get upstream ASAP. I'm
doubtful that is possible for 2.6.38 given the merge window is likely to
close shortly.
Sorry for the late answer; the above patches somehow got lost in my
various mail folders :-(
But yes, the patchset looks okay.
Feel free to add my Acked-by: to last two.
OK.
quoted
Regardless, if we could get consensus on this work now and then stage it
with James that would be great.
Indeed. Will you resend it to linux-scsi or shall I do it?