I wonder if this problem can be replicated on loop devices?  Once
there's a way to cause it reliably, we can then start doing a
bisection of the kernel to try and find out where this is happening.

So far, it looks like it happens sometimes on bare RAID6 systems
without lv-thin in place, which is both good and bad.  And without
using VMs on top of the storage either.  So this helps narrow down the
cause.  

Is there any info on the work load on these systems?  Lots of small
fils which are added/removed?  Large files which are just written to
and not touched again?

I assume finding a bad file with corruption and then doing a cp of the
file keeps the same corruption?  

quoted

quoted

quoted

quoted

"Chris" == Chris Dunlop [off-list ref] writes:

Chris> Hi,

Chris> On Thu, May 07, 2020 at 07:30:19PM +0200, Michal Soltys wrote:

quoted

Note: this is just general question - if anyone experienced something 
similar or could suggest how to pinpoint / verify the actual cause.

Thanks to btrfs's checksumming we discovered somewhat (even if quite 
rare) nasty silent corruption going on on one of our hosts. Or perhaps 
"corruption" is not the correct word - the files simply have precise 4kb 
(1 page) of incorrect data. The incorrect pieces of data look on their 
own fine - as something that was previously in the place, or written 
from wrong source.

Chris> "Me too!"

Chris> We are seeing 256-byte corruptions which are always the last 256b of a 4K 
Chris> block. The 256b is very often a copy of a "last 256b of 4k block" from 
Chris> earlier on the file. We sometimes see multiple corruptions in the same 
Chris> file, with each of the corruptions being a copy of a different 256b from 
Chris> earlier on the file. The original 256b and the copied 256b aren't 
Chris> identifiably at a regular offset from each other. Where the 256b isn't a 
Chris> copy from earlier in the file

Chris> I'd be really interested to hear if your problem is just in the last 256b 
Chris> of the 4k block also!

Chris> We haven't been able to track down any the origin of any of the copies 
Chris> where it's not a 256b block earlier in the file. I tried some extensive 
Chris> analysis of some of these occurrences, including looking at files being 
Chris> written around the same time, but wasn't able to identify where the data 
Chris> came from. It could be the "last 256b of 4k block" from some other file 
Chris> being written at the same time, or a non-256b aligned chunk, or indeed not 
Chris> a copy of other file data at all.

Chris> See Also: https://lore.kernel.org/linux-xfs/20180322150226.GA31029@onthe.net.au/ (local)

Chris> We've been able to detect these corruptions via an md5sum calculated as 
Chris> the files are generated, where a later md5sum doesn't match the original.  
Chris> We regularly see the md5sum match soon after the file is written (seconds 
Chris> to minutes), and then go "bad" after doing a "vmtouch -e" to evict the 
Chris> file from memory. I.e. it looks like the problem is occurring somewhere on 
Chris> the write path to disk. We can move the corrupt file out of the way and 
Chris> regenerate the file, then use 'cmp -l' to see where the corruption[s] are, 
Chris> and calculate md5 sums for each 256b block in the file to identify where 
Chris> the 256b was copied from.

Chris> The corruptions are far more likely to occur during a scrub, although we 
Chris> have seen a few of them when not scrubbing. We're currently working around 
Chris> the issue by scrubbing infrequently, and trying to schedule scrubs during 
Chris> periods of low write load.

quoted

The hardware is (can provide more detailed info of course):

- Supermicro X9DR7-LN4F
- onboard LSI SAS2308 controller (2 sff-8087 connectors, 1 connected to 
backplane)
- 96 gb ram (ecc)
- 24 disk backplane

- 1 array connected directly to lsi controller (4 disks, mdraid5, 
internal bitmap, 512kb chunk)
- 1 array on the backplane (4 disks, mdraid5, journaled)
- journal for the above array is: mdraid1, 2 ssd disks (micron 5300 pro 
disks)
- 1 btrfs raid1 boot array on motherboard's sata ports (older but still 
fine intel ssds from DC 3500 series)

Chris> Ours is on similar hardware:

Chris> - Supermicro X8DTH-IF
Chris> - LSI SAS 9211-8i  (LSI SAS2008, PCI-e 2.0, multiple firmware versions)
Chris> - 192GB ECC RAM
Chris> - A mix of 12 and 24-bay expanders (some daisy chained: lsi-expander-expander)

Chris> We swapped the LSI HBA for another of the same model, the problem 
Chris> persisted. We have a SAS9300 card on the way for testing.

quoted

Raid 5 arrays are in lvm volume group, and the logical volumes are used 
by VMs. Some of the volumes are linear, some are using thin-pools (with 
metadata on the aforementioned intel ssds, in mirrored config). LVM uses 
large extent sizes (120m) and the chunk-size of thin-pools is set to 
1.5m to match underlying raid stripe. Everything is cleanly aligned as 
well.

Chris> We're not using VMs nor lvm thin on this storage.

Chris> Our main filesystem is xfs + lvm + raid6 and this is where we've seen all 
Chris> but one of these corruptions (70-100 since Mar 2018).

Chris> The problem has occurred on all md arrays under the lvm, on disks from 
Chris> multiple vendors and models, and on disks attached to all expanders.

Chris> We've seen one of these corruptions with xfs directly on a hdd partition.  
Chris> I.e. no mdraid or lvm involved. This fs an order of magnitude or more less 
Chris> utilised than the main fs in terms of data being written.

quoted

We did not manage to rule out (though somewhat _highly_ unlikely):

- lvm thin (issue always - so far - occured on lvm thin pools)
- mdraid (issue always - so far - on mdraid managed arrays)
- kernel (tested with - in this case - debian's 5.2 and 5.4 kernels, 
happened with both - so it would imply rather already longstanding bug 
somewhere)

Chris> - we're not using lvm thin
Chris> - problem has occurred once on non-mdraid (xfs directly on a hdd partition)
Chris> - problem NOT seen on kernel 3.18.25
Chris> - problem seen on, so far, kernels 4.4.153 - 5.4.2

quoted

And finally - so far - the issue never occured:

- directly on a disk
- directly on mdraid
- on linear lvm volume on top of mdraid

Chris> - seen once directly on disk (partition)
Chris> - we don't use mdraid directly
Chris> - our problem arises on linear lvm on top of mdraid (raid6)

quoted

As far as the issue goes it's:

- always a 4kb chunk that is incorrect - in a ~1 tb file it can be from 
a few to few dozens of such chunks
- we also found (or rather btrfs scrub did) a few small damaged files as 
well
- the chunks look like a correct piece of different or previous data

The 4kb is well, weird ? Doesn't really matter any chunk/stripes sizes 
anywhere across the stack (lvm - 120m extents, 1.5m chunks on thin 
pools; mdraid - default 512kb chunks). It does nicely fit a page though 
...

Anyway, if anyone has any ideas or suggestions what could be happening 
(perhaps with this particular motherboard or vendor) or how to pinpoint 
the cause - I'll be grateful for any.

Chris> Likewise!

Chris> Cheers,

Chris> Chris