Re: BTRFS constantly reports "No space left on device" even with a huge unallocated space

On Tue, Aug 30, 2016 at 6:50 AM, Ronan Arraes Jardim Chagas
[off-list ref] wrote:

Hi!

Em Ter, 2016-08-30 às 10:12 +0800, Wang Xiaoguang escreveu:

quoted

For metadata, "bytes_may_use" is about 80GB, it's very big,
I think this value is very abnormal.

So this explains why you have huge unallocated space, you still
get ENOSPC error. In kernel btrfs, there is a function
should_alloc_chunk()
to determine whether to allocate new chunks(new device space)
  num_bytes = total_bytes - bytes_readonly; it's 2147483648
  num_allocated = bytes_used + bytes_reserved; it's 977354752

if num_allocated < num_bytes * 0.8, it will not allocate new device
space :) even you
have huge unallocated space.

I think the root reason is that bytes_may_use has some computation
error and
is not be converted to bytes_used or bytes_reserved.

I just explain why you get ENOSPC error even with huge unallocated
space
from
codes :)

Thanks! At least we known why ENOSPC is happening.

quoted

Can you work out a reproducer for this ENOSPC error, then I can
dig into codes to figure out the true reason.

Unfortunately I failed in every attempt to trigger the problem. It
happens randomly and I could not figure out yet what was triggering it.
First, I though it was related to a build process inside a chroot jail,
but then I see the problem happening after the computer being idle for
a long time (+- 1h). So, no clues yet :(

Is there any workaround I can do?

It sounds related to read-only snapshots to me. I wonder if this
system has something busy that's writing to a file, database, even
maybe something just spamming journald, and then there's a read-only
snapshot during the write, which then triggers the enospc.

Ronan, if you're given a work around, then it's even less likely the
bug gets fixed. But if you can disable snapper snapshots entirely and
the problem doesn't happen; or if you can increase the frequency of
snapper snapshots and the problem happens more often, that might help
narrow it down to a point where it's more easily reproduced. If it's
not related, that's still useful to know.

-- 
Chris Murphy