SSD based sw RAID: is ERC/TLER really important?

Hi, 

nowadays we all know that the ERC/TLER capability is very important for the 
use of spinning drives in RAID systems because, especially for recent hard 
disks, the recovery time in case of media errors could exceed kernel timeouts 
and possibly kick off the entire drive from the RAID set and, in turn, lead to 
a fault of a RAID5 system upon a subsequent error in a second drive. 

But in the case of SSD drives (where, possibly, the error recovery activities 
performed by the drive firmware are very fast) does the presence of the 
ERC/TLER capability really matter? Is the same scenario from the spinning 
drives case actually even probable or only theorical? 

Thank you for the considerations and evaluations you want to express. 

Gianluca 

Hi Gianluca,

On Sat, Jul 24, 2021 at 08:41:06PM +0200, Gianluca Frustagli wrote:

But in the case of SSD drives (where, possibly, the error recovery activities 
performed by the drive firmware are very fast) does the presence of the 
ERC/TLER capability really matter?

If the setting is there, why wouldn't you use it? If the error
recovery is always very fast, as you hypothesise, then the low
timeout you set with smartctl -l scterc will never be reached
anyway.

Is the same scenario from the spinning 
drives case actually even probable or only theorical? 

I don't know what the typical error recovery behaviour of an SSD is
because in years and years I haven't seen such problems with SSDs,
but where they offer the scterc setting I do use it anyway on the
basis of it's not going to hurt anything.

I see it available on enterprise SSD like Samsung SM883, Intel
D3-S4610. I don't see it on Supermicro DOM modules.

I also don't see it on NVMe drives at all (e.g. Samsung PM983), and
NVMe seems to be the future of flash so maybe this setting dies
soon…

Cheers,
Andy

the recovery time in case of media errors could exceed kernel
timeouts and possibly kick off the entire drive from the RAID
set and, in turn, lead to a fault of a RAID5 system upon a
subsequent error in a second drive.

My understanding seems different:

* The purpose of having a short device error retry period is the
  opposite, it is to fail a drive as fast as possible, in
  workloads where latency matters ( or there is also the risk of
  bus/link resets hitting multiple drives). In those cases error
  retry periods of 1-2 seconds (at most) are common, rather than
  the mid-way "7 seconds" from copy-and-paste from web pages..

* The purpose of having a long device error retry is to instead
  to minimize the chances of declaring a drive failed, hoping
  that many retries succeed. (but note the difference between
  reads and writes).

* It is possible to set the kernel timeouts higher than device
  retry periods, if one does not care about latency, to minimize
  the chances of declaring a drive failed (not the difference
  between Linux command timeouts and retry timeouts, the latter
  can also be long).

But in the case of SSD drives (where, possibly, the error
recovery activities performed by the drive firmware are very
fast) [...]

I guess that depends on the firmware: On one hand MLC cells can
become quite unreliable, especially at higher temperatures,
requiring many retries and lots of ECC, on the other on "write"
allocating a new erase-block is easy, as unlike for most HDDs
with a FTL, SDD sector logical and physical sector locations are
independent. Unfortunately most flash SSD drive makers don't
supply technical information on details like error recovery
strategies.

On 7/24/21 4:19 PM, Peter Grandi wrote:

quoted

the recovery time in case of media errors could exceed kernel 
timeouts and possibly kick off the entire drive from the RAID set
and, in turn, lead to a fault of a RAID5 system upon a subsequent
error in a second drive.

My understanding seems different:

You understanding is incorrect.

* The purpose of having a short device error retry period is the 
opposite, it is to fail a drive as fast as possible, in workloads
where latency matters ( or there is also the risk of bus/link resets
hitting multiple drives). In those cases error retry periods of 1-2
seconds (at most) are common, rather than the mid-way "7 seconds"
from copy-and-paste from web pages..

Yes, the short ERC setting helps latency, but the primary purpose is to
be shorter than the kernel timeout.

* The purpose of having a long device error retry is to instead to
minimize the chances of declaring a drive failed, hoping that many
retries succeed. (but note the difference between reads and writes).

Read errors do *not* kick drives out.  It takes several read errors in a
short time to fail a drive out of an array.

A drive not responding before the kernel timeout *will* get it kicked, 
though.  Because the kernel giving up propagates to the raid as a
read error (while the drive is off in la-la land) which then causes
the raid to *reconstruct* the missing sector and *write* it.  Along
with passing the reconstructed data up the chain.

That write will fail because the drive is still in la-la land.  Any
write failure *does* kick the drive out.

* It is possible to set the kernel timeouts higher than device retry
periods, if one does not care about latency, to minimize the chances
of declaring a drive failed (not the difference between Linux command
timeouts and retry timeouts, the latter can also be long).

quoted

But in the case of SSD drives (where, possibly, the error recovery
activities performed by the drive firmware are very fast) [...]

I guess that depends on the firmware: On one hand MLC cells can 
become quite unreliable, especially at higher temperatures, requiring
many retries and lots of ECC, on the other on "write" allocating a
new erase-block is easy, as unlike for most HDDs with a FTL, SDD
sector logical and physical sector locations are independent.
Unfortunately most flash SSD drive makers don't supply technical
information on details like error recovery strategies.

I don't have data on SSD behavior without ERC.  If their retry cycle is 
exhausted within the kernel default 30 seconds, the timeout mismatch 
issue will *not* apply.

Phil

On 24/07/21 22:45, Phil Turmel wrote:

I don't have data on SSD behavior without ERC.  If their retry cycle is
exhausted within the kernel default 30 seconds, the timeout mismatch
issue will *not* apply.

I've also seen stuff that implies (with spinning rust) that the retry
cycle can hang - a read times out, then the next attempt to read the
same data works fine.

It's *possible* the same applies to SSDs, in which case shortening the
timeout could be worthwhile.

And as Phil says, the critical fact is that the drive MUST come back
from la-la-land BEFORE the kernel times out.

Cheers,
Wol

quoted

* The purpose of having a long device error retry is to instead to
minimize the chances of declaring a drive failed, hoping that many
retries succeed. (but note the difference between reads and writes).
* It is possible to set the kernel timeouts higher than device retry
periods, if one does not care about latency, to minimize the
chances of declaring a drive failed (not[e] the difference
between Linux command timeouts and retry timeouts, the latter
can also be long).

You understanding is incorrect.
Read errors do *not* kick drives out. It takes several read
errors in a short time to fail a drive out of an array.

I am sorry that I was not clear enough and therefore:

* You failed to understand the relevance of "note the difference
  between reads and writes" which I added precisely because I
  guessed that someone unfamiliar with storage device would need
  that terse qualifier.

* You failed to understand the relevance of the "to minimize the
  chances of declaring a drive failed".

* You failed to realize that I was addressing tersely the
  original poster's case of a drive being declared failed
  because of a drive timeout longer than the kernel command
  timeout, without going in detail about all other possible
  cases.

quoted

* It is possible to set the kernel timeouts higher than device retry
periods, if one does not care about latency, to minimize the chances
of declaring a drive failed (not the difference between Linux command
timeouts and retry timeouts, the latter can also be long).

I don't have data on SSD behavior without ERC. If their retry
cycle is exhausted within the kernel default 30 seconds, the
timeout mismatch issue will *not* apply.

That as written may confuse readers as to the difference between
the Linux command timeout and the the Linux retry timeout:

  # grep -H . /sys/module/scsi_mod/parameters/eh_deadline 
  /sys/module/scsi_mod/parameters/eh_deadline:-1

  # grep -H . /sys/block/sda/device/*timeout*
  /sys/block/sda/device/eh_timeout:10
  /sys/block/sda/device/timeout:30

Things are different again with the NVME subsystem:

  # grep -H . /sys/module/nvme_core/parameters/*{timeout,retries,latency}*
  /sys/module/nvme_core/parameters/admin_timeout:60
  /sys/module/nvme_core/parameters/io_timeout:30
  /sys/module/nvme_core/parameters/shutdown_timeout:5
  /sys/module/nvme_core/parameters/max_retries:5
  /sys/module/nvme_core/parameters/default_ps_max_latency_us:100000

Some relevant links:

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/managing_storage_devices/configuring-maximum-time-for-storage-error-recovery-with-eh_deadline_managing-storage-devices
https://unix.stackexchange.com/questions/541463/how-to-prevent-disk-i-o-timeouts-which-cause-disks-to-disconnect-and-data-corrup
https://elixir.bootlin.com/linux/v5.13.4/source/drivers/scsi/scsi_error.c

Hi Peter,

On 7/25/21 6:28 AM, Peter Grandi wrote:

quoted

quoted

* The purpose of having a long device error retry is to instead to
minimize the chances of declaring a drive failed, hoping that many
retries succeed. (but note the difference between reads and writes).
* It is possible to set the kernel timeouts higher than device retry
periods, if one does not care about latency, to minimize the
chances of declaring a drive failed (not[e] the difference
between Linux command timeouts and retry timeouts, the latter
can also be long).

quoted

You understanding is incorrect.
Read errors do *not* kick drives out. It takes several read
errors in a short time to fail a drive out of an array.

I am sorry that I was not clear enough and therefore:

* You failed to understand the relevance of "note the difference
   between reads and writes" which I added precisely because I
   guessed that someone unfamiliar with storage device would need
   that terse qualifier.

* You failed to understand the relevance of the "to minimize the
   chances of declaring a drive failed".

* You failed to realize that I was addressing tersely the
   original poster's case of a drive being declared failed
   because of a drive timeout longer than the kernel command
   timeout, without going in detail about all other possible
   cases.

You have reminded me that your mail should have been blackholed--a rule 
I put in many years agao.  I have updated the rule to be more inclusive. 
  I must not be the only one ignoring you, causing you to use multiple 
subdomains.

No need to reply.  I won't see it.

Phil

[...]

I must not be the only one ignoring you, causing you to use
multiple subdomains.

It seems sad that my changing address occasionally when they get
spammed as they get "harvested" may tickle someone's foolish
arrogance of thinking themselves so important that it is done
just to get their worthless attention.
http://www.sabi.co.uk/blog/0705may.html?070527c#070527c

Unfortunately in some cases my technical comments therefore get
replies without any technical content. As to this though:

without going in detail about all other possible cases.

There are indeed many twists and turns and "legacy" situations,
as in several places timeouts and retries are hardcoded, and
IIRC the "default" for retries is 5. But I have spent a bit of
time looking at some of the weirdness and it turns out that
nowadays the 'sd' module defines a 'max_retries' setting in the
device attributes (rather than a module parameter as in
'nvme_core'):

  https://elixir.bootlin.com/linux/latest/source/drivers/scsi/sd.c#L598

It is only available from 5.10 and would be for example at:

  /sys/class/scsi_disk/0:0:0:0/max_retries

I have also noticed that XFS bizarrely has its own layer of
recovery on top of that of the Linux IO subsystems and of the
device itself:

  https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/managing_file_systems/configuring-xfs-error-behavior_managing-file-systems
  https://elixir.bootlin.com/linux/v5.13.5/source/fs/xfs/xfs_buf.c#L1264

[...]

I have also noticed that XFS bizarrely has its own layer of
recovery on top of that of the Linux IO subsystems and of the
device itself:

  https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/managing_file_systems/configuring-xfs-error-behavior_managing-file-systems
  https://elixir.bootlin.com/linux/v5.13.5/source/fs/xfs/xfs_buf.c#L1264

Some people have been proposing to do the same with MD RAID, and
some similar layer sometimes is embedded in the firmware of
hardware RAID host adapters.

Consider this scenario (include the relevant error handling and
timeouts):

  * The device firmware does physical operation retries.
  * The Linux IO subsystem does device operation retries.
  * MD RAID does Linux IO operation retries.
  * XFS does MD RAID IO operation retries.

Is there something weird with this?

BTW there is something similar with read ahead or write behind:
sometimes several software layers assume that lower software
layers are doing too little of it, and do their own additional
read ahead or write behind.