Bcache file corruption

Hi,

I'm trying to run bcache on a Hyper-V VM, following setup:

caching device 50GB fixed disk on a Samsung 850 Pro 1 TB Raid 1

backing device 700GB dynamic disk on a 8x 4 TB RAID6


The bcache device is formatted as LVM with 1 volume XFS. The writeback mode is activated.

When I try to restore a 600 GB Postgresql backup, I get a file corruption error.

I have tried 3 times, and the file corruption is each time at a different (file) location,
additionally the corruption is not present when restoring the same backup on a non-bcache disk.

Based on that info, I assume that bcache has got some issue.

Are there any known issues with maybe XFS, or file corruption? Does anyone have an idea on how to figure out what the issue is?


Kernel:

Linux host 3.10.0-514.6.1.el7.x86_64 #1 SMP Wed Jan 18 13:06:36 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

I compiled bcache as a module according to: http://10sa.com/sql_stories/?p=1052

BR
Thomas

Hi Thomas,

Zitat von Thomas Rosenstein [off-list ref]:

Hi,

I'm trying to run bcache on a Hyper-V VM, following setup:
[...]
Linux host 3.10.0-514.6.1.el7.x86_64 #1 SMP Wed Jan 18 13:06:36 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

I compiled bcache as a module according to: http://10sa.com/sql_stories/?p=1052

what catches the eye is the old kernel version. While many fixes may have been back-ported by the distro developers, I have no idea to what extend this was actually done. I remember having seen quite a number of severe bugs, i.e. in the disk subsystem, that may well have lead to the problems you face.

Are you in a position to boot a more recent (mainline) kernel for a test, if just to compare results?

Regards,
Jens

Hi Jens,

guess I can do that, elrepo has 4.4 as far as I know, so I guess this could be worth a try!

Thanks, will report back after another 600GB restore

Thomas

On 7 Feb 2017, at 15:59, Jens-U. Mozdzen wrote:

Hi Thomas,

Zitat von Thomas Rosenstein [off-list ref]:

quoted

Hi,

I'm trying to run bcache on a Hyper-V VM, following setup:
[...]
Linux host 3.10.0-514.6.1.el7.x86_64 #1 SMP Wed Jan 18 13:06:36 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

I compiled bcache as a module according to: http://10sa.com/sql_stories/?p=1052

what catches the eye is the old kernel version. While many fixes may have been back-ported by the distro developers, I have no idea to what extend this was actually done. I remember having seen quite a number of severe bugs, i.e. in the disk subsystem, that may well have lead to the problems you face.

Are you in a position to boot a more recent (mainline) kernel for a test, if just to compare results?

Regards,
Jens

Hi Jens,

Linux host 4.9.8 #1 SMP Tue Feb 7 19:15:34 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Fixed the issue, could restore the whole DB and it could catch up.

Thanks, maybe someone could add a warning? 😊

Regards
Thomas

On 7 Feb 2017, at 15:59, Jens-U. Mozdzen wrote:

Hi Thomas,

Zitat von Thomas Rosenstein [off-list ref]:

quoted

Hi,

I'm trying to run bcache on a Hyper-V VM, following setup:
[...]
Linux host 3.10.0-514.6.1.el7.x86_64 #1 SMP Wed Jan 18 13:06:36 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

I compiled bcache as a module according to: http://10sa.com/sql_stories/?p=1052

what catches the eye is the old kernel version. While many fixes may have been back-ported by the distro developers, I have no idea to what extend this was actually done. I remember having seen quite a number of severe bugs, i.e. in the disk subsystem, that may well have lead to the problems you face.

Are you in a position to boot a more recent (mainline) kernel for a test, if just to compare results?

Regards,
Jens