Re: BTRFS FS corruption

On Wed, Dec 29, 2021 at 03:30:46PM -0500, Mark Murawski wrote:

Kernel: Debian 5.10.70-1~bpo10+1 (2021-10-10)

I've been trying to reproduce the "ghost dirents" bug on 5.10 without
success.  It's much easier to hit on 5.14 and 5.15, but apparently not
impossible on 5.10.

# ls -al /mnt/var/lib/postgresql/12/main/pg_stat_tmp/
ls: cannot access '/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp':
No such file or directory
ls: cannot access '/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp':
No such file or directory
ls: cannot access '/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp':
No such file or directory
ls: cannot access '/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp':
No such file or directory
ls: cannot access '/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp':
No such file or directory
ls: cannot access '/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp':
No such file or directory
ls: cannot access '/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp':
No such file or directory
ls: cannot access '/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp':
No such file or directory
ls: cannot access '/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp':
No such file or directory
total 80
drwx------ 1 1001 1001   94 Dec 29 13:52 .
drwx------ 1 1001 1001  552 Dec 29 13:51 ..
-rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat
-rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat
-rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat
-rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat
-rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat
-rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat
-rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat
-rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat
-rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat
-rw------- 1 1001 1001    0 Dec 29 13:52 db_106964.stat
-rw------- 1 1001 1001    0 Dec 29 13:52 db_106964.stat
-rw------- 1 1001 1001    0 Dec 29 13:52 db_106964.stat
-rw------- 1 1001 1001    0 Dec 29 13:52 db_106964.stat
-rw------- 1 1001 1001    0 Dec 29 13:52 db_106964.stat
-rw------- 1 1001 1001    0 Dec 29 13:52 db_106964.stat
-rw------- 1 1001 1001    0 Dec 29 13:52 db_106964.stat
-rw------- 1 1001 1001    0 Dec 29 13:52 db_106964.stat
-rw------- 1 1001 1001    0 Dec 29 13:52 db_106964.stat
-rw------- 1 1001 1001 1864 Dec 29 13:52 db_16391.stat
-rw------- 1 1001 1001  840 Dec 29 13:52 global.stat
-rw------- 1 1001 1001  840 Dec 29 13:52 global.stat
-rw------- 1 1001 1001  840 Dec 29 13:52 global.stat
-rw------- 1 1001 1001  840 Dec 29 13:52 global.stat
-rw------- 1 1001 1001  840 Dec 29 13:52 global.stat
-rw------- 1 1001 1001  840 Dec 29 13:52 global.stat
-rw------- 1 1001 1001  840 Dec 29 13:52 global.stat
-rw------- 1 1001 1001  840 Dec 29 13:52 global.stat
-rw------- 1 1001 1001  840 Dec 29 13:52 global.stat
-rw------- 1 1001 1001  840 Dec 29 13:52 global.stat
-????????? ? ?    ?       ?            ? global.tmp
-????????? ? ?    ?       ?            ? global.tmp
-????????? ? ?    ?       ?            ? global.tmp
-????????? ? ?    ?       ?            ? global.tmp
-????????? ? ?    ?       ?            ? global.tmp
-????????? ? ?    ?       ?            ? global.tmp
-????????? ? ?    ?       ?            ? global.tmp
-????????? ? ?    ?       ?            ? global.tmp
-????????? ? ?    ?       ?            ? global.tmp

This looks like the "ghost dirents" bug I've been chasing for a while.

Some prior discussion at

	https://lore.kernel.org/linux-btrfs/CAL3q7H4CYtaW_aEQSEZ_KxZ_ba3u=FmPT8VtXH+OE6FTR8oxOQ@mail.gmail.com/ (local)

This is a dev/test vm so we can do intrusive operations on it...

If you can reproduce this reliably, can you apply a kernel patch:

	v5.16-rc1: 9a35fc9542fa btrfs: change error handling for btrfs_delete_*_in_log

There's a link to the commit in the kernel.org thread.

Should I run btrfsck?

Ghost dirents can be removed by the normal 'rm -rf' tool (though it will
complain with errors).  You can run btrfs check to confirm that this is
the only problem on the filesystem.

btrfs scrub did not show any errors/fixes and it did not fix the issue.

Scrub verifies metadata block csums, and there will be no errors in
csums from the ghost dirent bug case, so scrub will not detect or report
ghost dirents.