On Wed, Feb 21, 2018 at 05:55:40PM +0100, Emmanuel Florac wrote:

Le Wed, 21 Feb 2018 16:23:43 +0100
Andrea Mazzocchi [off-list ref] écrivait:

quoted

quoted

Also, you are running a very old kernel, so, please make sure you
try to run a newer xfs_repair.  

We installed yesterday 3.10.0-693.17.1.el7. I know that CentOS and
RedHat keep old stable kernel version and backport important stuff:
do you think that upgrading to a more recent kernel (4 and above)
would be better, even if less stable?

Yes, there are several things backported over Enterprise distributed kernels,
but, hte upstream community can't offer support on that, once we can't easily
keep track of what has been backported or not, so, that's why we usually tell
people reporting bugs here, using an enterprise distributed kernel, to report
the bug directly to their support representatives.

Actually the 3.10 from CentOS 7 has a lot of things backported, for
instance it supports XFS with crc metadata which was introduced in 3.16.

However in the recent year I've never met any reason NOT to use a
recent LTS kernel (like a 4.9 or 4.14). I don't really understand why
RedHat sticks to these absurdly old kernel releases as a basis (and
taking the pain of backporting stuff for years and years).

There are several reasons for that, and it is not only Red Hat who does it. One
of the reasons is ABI stability.

quoted

quoted

Also, this is more a guess than anything. If you see this happening
often (even after xfs_repair), you might want to double-check your
storage stack and see if this is not corrupting anything, bad
configured storage stacks in virtual environments are very usual
culprits on filesystem corruption cases.  

How could we check our storage stack and see if it is the one to
blame?

Hard to say, what KVM disk format are you using? Raw, qcow2, LVM
volumes? If these are files (raw or qcow2), what kind of filesystem and
hardware stack are they living on? Are there any error on the hosting
system?

At tne VM level, do you see any IO error? Are you using the virtio disk
driver or something else? 

^ Good start, also, what kind of caching policies is a good information,
particularly, using any cache policies for VM disks other than 'None', is a good
way to start having problems per my experience.

Also:

http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F

Don't forget xfs_repair :)

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	[off-list ref]
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

-- 
Carlos