On Mon, Mar 29, 2021 at 5:12 AM Hendrik Friedel [off-list ref] wrote:

Hello,

I have a filesystem which is sometimes very slow, or even currently
hangs deleting a file (plain and simple rm in bash).

Label: 'DataPool1'  uuid: c4a6a2c9-5cf0-49b8-812a-0784953f9ba3
         Total devices 2 FS bytes used 5.65TiB
         devid    1 size 7.28TiB used 6.71TiB path /dev/sda1
         devid    2 size 7.28TiB used 6.71TiB path /dev/sdh1

I did run a scrub without errors.

Checking the logs, I find:
dmesg -T |grep -i btrfs
[Mo Mär 29 09:29:16 2021] Btrfs loaded, crc32c=crc32c-intel
[Mo Mär 29 09:29:16 2021] BTRFS: device label DataPool1 devid 2 transid
649014 /dev/sdh1 scanned by btrfs (213)
[Mo Mär 29 09:29:16 2021] BTRFS: device label DataPool1 devid 1 transid
649014 /dev/sda1 scanned by btrfs (213)
[Mo Mär 29 09:29:16 2021] BTRFS: device label Daten devid 1 transid
254377 /dev/sdd2 scanned by btrfs (213)
[Mo Mär 29 09:29:16 2021] BTRFS: device label DockerImages devid 1
transid 209067 /dev/sdc2 scanned by btrfs (213)
[Mo Mär 29 09:29:21 2021] BTRFS info (device sda1): disk space caching
is enabled
[Mo Mär 29 09:29:21 2021] BTRFS info (device sda1): has skinny extents
[Mo Mär 29 09:29:21 2021] BTRFS info (device sdd2): enabling ssd
optimizations
[Mo Mär 29 09:29:21 2021] BTRFS info (device sdd2): disk space caching
is enabled
[Mo Mär 29 09:29:21 2021] BTRFS info (device sdd2): has skinny extents
[Mo Mär 29 09:29:21 2021] BTRFS info (device sdc2): turning on sync
discard
[Mo Mär 29 09:29:21 2021] BTRFS info (device sdc2): enabling ssd
optimizations
[Mo Mär 29 09:29:21 2021] BTRFS info (device sdc2): disk space caching
is enabled
[Mo Mär 29 09:29:21 2021] BTRFS info (device sdc2): has skinny extents
[Mo Mär 29 09:29:22 2021] BTRFS info (device sda1): bdev /dev/sda1 errs:
wr 133, rd 133, flush 0, corrupt 0, gen 1

Maybe, the last line is concerning?

Yes. Do a 'btrfs scrub' and check dmesg for detailed errors. Next
'btrfs check --readonly' (must be done offline ie booted from usb
stick). And if it all comes up without errors or problems, you can
zero the statistics with 'btrfs dev stats -z'. But otherwise we need
to see the errors to know what's going wrong. It's not normal to have
either read or write errors. It could be related to the problem, or an
additional problem.

Syslog tells me:
Mar 28 20:22:19 homeserver kernel: [1297978.357508] task:btrfs-cleaner
state:D stack:    0 pid:20078 ppid:     2 flags:0x00004000
Mar 28 20:22:19 homeserver kernel: [1297978.357547]
wait_current_trans+0xc2/0x120 [btrfs]
Mar 28 20:22:19 homeserver kernel: [1297978.357564]
start_transaction+0x46d/0x540 [btrfs]
Mar 28 20:22:19 homeserver kernel: [1297978.357577]
btrfs_drop_snapshot+0x90/0x7f0 [btrfs]
Mar 28 20:22:19 homeserver kernel: [1297978.357594]  ?
btrfs_delete_unused_bgs+0x3e/0x850 [btrfs]
Mar 28 20:22:19 homeserver kernel: [1297978.357609]
btrfs_clean_one_deleted_snapshot+0xd7/0x130 [btrfs]
Mar 28 20:22:19 homeserver kernel: [1297978.357622]
cleaner_kthread+0xfa/0x120 [btrfs]
Mar 28 20:22:19 homeserver kernel: [1297978.357636]  ?
btrfs_alloc_root+0x3d0/0x3d0 [btrfs]
Mar 28 20:22:19 homeserver kernel: [1297978.360473]
wait_current_trans+0xc2/0x120 [btrfs]
Mar 28 20:22:19 homeserver kernel: [1297978.360488]
start_transaction+0x46d/0x540 [btrfs]
Mar 28 20:22:19 homeserver kernel: [1297978.360503]
btrfs_create+0x58/0x1f0 [btrfs]
Mar 28 20:22:19 homeserver kernel: [1297978.363057]
wait_current_trans+0xc2/0x120 [btrfs]
Mar 28 20:22:19 homeserver kernel: [1297978.363072]
start_transaction+0x46d/0x540 [btrfs]
Mar 28 20:22:19 homeserver kernel: [1297978.363086]
btrfs_rmdir+0x5c/0x180 [btrfs]
Mar 28 20:26:20 homeserver kernel: [1298220.024321] task:btrfs-cleaner
state:D stack:    0 pid:20078 ppid:     2 flags:0x00004000
Mar 28 20:26:20 homeserver kernel: [1298220.024382]
wait_current_trans+0xc2/0x120 [btrfs]
Mar 28 20:26:20 homeserver kernel: [1298220.024419]
start_transaction+0x46d/0x540 [btrfs]
Mar 28 20:26:20 homeserver kernel: [1298220.024442]
btrfs_drop_snapshot+0x90/0x7f0 [btrfs]
Mar 28 20:26:20 homeserver kernel: [1298220.024476]  ?
btrfs_delete_unused_bgs+0x3e/0x850 [btrfs]
Mar 28 20:26:20 homeserver kernel: [1298220.024504]
btrfs_clean_one_deleted_snapshot+0xd7/0x130 [btrfs]
Mar 28 20:26:20 homeserver kernel: [1298220.024531]
cleaner_kthread+0xfa/0x120 [btrfs]
Mar 28 20:26:20 homeserver kernel: [1298220.024558]  ?
btrfs_alloc_root+0x3d0/0x3d0 [btrfs]
Mar 28 20:26:20 homeserver kernel: [1298220.030300]
wait_current_trans+0xc2/0x120 [btrfs]
Mar 28 20:26:20 homeserver kernel: [1298220.030331]
start_transaction+0x46d/0x540 [btrfs]
Mar 28 20:26:20 homeserver kernel: [1298220.030361]
btrfs_create+0x58/0x1f0 [btrfs]
Mar 28 20:28:21 homeserver kernel: [1298340.854109] task:btrfs-cleaner
state:D stack:    0 pid:20078 ppid:     2 flags:0x00004000
Mar 28 20:28:21 homeserver kernel: [1298340.854151]
wait_current_trans+0xc2/0x120 [btrfs]
Mar 28 20:28:21 homeserver kernel: [1298340.854169]
start_transaction+0x46d/0x540 [btrfs]
Mar 28 20:28:21 homeserver kernel: [1298340.854183]
btrfs_drop_snapshot+0x90/0x7f0 [btrfs]
Mar 28 20:28:21 homeserver kernel: [1298340.854202]  ?
btrfs_delete_unused_bgs+0x3e/0x850 [btrfs]
Mar 28 20:28:21 homeserver kernel: [1298340.854218]
btrfs_clean_one_deleted_snapshot+0xd7/0x130 [btrfs]
Mar 28 20:28:21 homeserver kernel: [1298340.854232]
cleaner_kthread+0xfa/0x120 [btrfs]
Mar 28 20:28:21 homeserver kernel: [1298340.854247]  ?
btrfs_alloc_root+0x3d0/0x3d0 [btrfs]
Mar 28 20:28:21 homeserver kernel: [1298340.857610]
wait_current_trans+0xc2/0x120 [btrfs]
Mar 28 20:28:21 homeserver kernel: [1298340.857627]
start_transaction+0x46d/0x540 [btrfs]
Mar 28 20:28:21 homeserver kernel: [1298340.857643]
btrfs_create+0x58/0x1f0 [btrfs]
Mar 28 20:58:34 homeserver kernel: [1300153.336160] task:btrfs-transacti
state:D stack:    0 pid:20080 ppid:     2 flags:0x00004000
Mar 28 20:58:34 homeserver kernel: [1300153.336215]
btrfs_commit_transaction+0x92b/0xa50 [btrfs]
Mar 28 20:58:34 homeserver kernel: [1300153.336246]
transaction_kthread+0x15d/0x180 [btrfs]
Mar 28 20:58:34 homeserver kernel: [1300153.336273]  ?
btrfs_cleanup_transaction+0x590/0x590 [btrfs]


What could I do to find the cause?

What kernel version?

-- 
Chris Murphy