On Wed, May 13, 2020 at 01:49:10PM -0400, John Stoffel wrote:

I wonder if this problem can be replicated on loop devices?  Once
there's a way to cause it reliably, we can then start doing a
bisection of the kernel to try and find out where this is happening.

I ran a week or so of attempting to replicate the problem in a VM on loop 
devices replicating the lvm/raid config, without success. Basically just 
having a random bunch of 1-25 concurrent writers banging out middling to 
largish files.

The fact it wasn't replicable in that environment could be pointing 
towards the LSI driver or hardware - or I simply wasn't able to match  
the conditions well enough.

So far, it looks like it happens sometimes on bare RAID6 systems
without lv-thin in place, which is both good and bad.  And without
using VMs on top of the storage either.  So this helps narrow down the
cause.

Note: We don't have any bare RAID6 so I haven't seen it there: our main fs 
is xfs on sequential LVM on raid6 (6 x 11-disk sets), and we saw it once 
on xfs directly on HDD partition.

Is there any info on the work load on these systems?  Lots of small
fils which are added/removed?  Large files which are just written to
and not touched again?

Large files written and not touched again. Most of the time 2-5 concurrent 
writers but regularly (daily) up to 20-25 concurrent.

I assume finding a bad file with corruption and then doing a cp of the
file keeps the same corruption?

Yep.