Bcache file corruption

Hi,

I'm trying to run bcache on a Hyper-V VM, following setup:

caching device 50GB fixed disk on a Samsung 850 Pro 1 TB Raid 1

backing device 700GB dynamic disk on a 8x 4 TB RAID6


The bcache device is formatted as LVM with 1 volume XFS. The writeback 
mode is activated.

When I try to restore a 600 GB Postgresql backup, I get a file 
corruption error.

I have tried 3 times, and the file corruption is each time at a 
different (file) location,
additionally the corruption is not present when restoring the same 
backup on a non-bcache disk.

Based on that info, I assume that bcache has got some issue.

Are there any known issues with maybe XFS, or file corruption? Does 
anyone have an idea on how to figure out what the issue is?


Kernel:

Linux host 3.10.0-514.6.1.el7.x86_64 #1 SMP Wed Jan 18 13:06:36 UTC 2017 
x86_64 x86_64 x86_64 GNU/Linux

I compiled bcache as a module according to: 
http://10sa.com/sql_stories/?p=1052

BR
Thomas

Hi Thomas,

Zitat von Thomas Rosenstein [off-list ref]:

Hi,

I'm trying to run bcache on a Hyper-V VM, following setup:
[...]
Linux host 3.10.0-514.6.1.el7.x86_64 #1 SMP Wed Jan 18 13:06:36 UTC  
2017 x86_64 x86_64 x86_64 GNU/Linux

I compiled bcache as a module according to:  
http://10sa.com/sql_stories/?p=1052

what catches the eye is the old kernel version. While many fixes may  
have been back-ported by the distro developers, I have no idea to what  
extend this was actually done. I remember having seen quite a number  
of severe bugs, i.e. in the disk subsystem, that may well have lead to  
the problems you face.

Are you in a position to boot a more recent (mainline) kernel for a  
test, if just to compare results?

Regards,
Jens

Hi Jens,

guess I can do that, elrepo has 4.4 as far as I know, so I guess this 
could be worth a try!

Thanks, will report back after another 600GB restore

Thomas

On 7 Feb 2017, at 15:59, Jens-U. Mozdzen wrote:

Hi Thomas,

Zitat von Thomas Rosenstein [off-list ref]:

quoted

Hi,

I'm trying to run bcache on a Hyper-V VM, following setup:
[...]
Linux host 3.10.0-514.6.1.el7.x86_64 #1 SMP Wed Jan 18 13:06:36 UTC 
2017 x86_64 x86_64 x86_64 GNU/Linux

I compiled bcache as a module according to: 
http://10sa.com/sql_stories/?p=1052

what catches the eye is the old kernel version. While many fixes may 
have been back-ported by the distro developers, I have no idea to what 
extend this was actually done. I remember having seen quite a number 
of severe bugs, i.e. in the disk subsystem, that may well have lead to 
the problems you face.

Are you in a position to boot a more recent (mainline) kernel for a 
test, if just to compare results?

Regards,
Jens

Hi Jens,

Linux host 4.9.8 #1 SMP Tue Feb 7 19:15:34 UTC 2017 x86_64 x86_64 x86_64 
GNU/Linux

Fixed the issue, could restore the whole DB and it could catch up.

Thanks, maybe someone could add a warning? 😊

Regards
Thomas

On 7 Feb 2017, at 15:59, Jens-U. Mozdzen wrote:

Hi Thomas,

Zitat von Thomas Rosenstein [off-list ref]:

quoted

Hi,

I'm trying to run bcache on a Hyper-V VM, following setup:
[...]
Linux host 3.10.0-514.6.1.el7.x86_64 #1 SMP Wed Jan 18 13:06:36 UTC 
2017 x86_64 x86_64 x86_64 GNU/Linux

I compiled bcache as a module according to: 
http://10sa.com/sql_stories/?p=1052

what catches the eye is the old kernel version. While many fixes may 
have been back-ported by the distro developers, I have no idea to what 
extend this was actually done. I remember having seen quite a number 
of severe bugs, i.e. in the disk subsystem, that may well have lead to 
the problems you face.

Are you in a position to boot a more recent (mainline) kernel for a 
test, if just to compare results?

Regards,
Jens