On 2021/12/7 11:44, Christoph Anton Mitterer wrote:

On Tue, 2021-12-07 at 11:29 +0800, Qu Wenruo wrote:

quoted

For other regular operations, you either got ENOSPC just like all
other
fses which runs out of space, or do it without problem.

Furthermore, balance in this case is not really the preferred way to
free up space, really freeing up data is the correct way to go.

Well but to be honest... that makes btrfs kinda broke for that
particular purpose.


The software which runs on the storage and provides the data to the
experiments does in fact make sure that the space isn't fully used (per
default, it leave a gap of 4GB).

While this gap is configurable it seems a bit odd if one would have to
set it to ~1TB per fs... just to make sure that btrfs doesn't run out
of space for metadata.


And btrfs *does* show that plenty of space is left (always around 700-
800 GB)... so the application thinks it can happily continue to write,
while in fact it fails (and the cannot even start anymore as it fails
to create lock files).

That's the problem with dynamic chunk allocation, and to be honest, I
don't have any better idea how to make it work just like traditional fses.

You could consider it as something like thin-provisioned device, which
would have the same problem (reporting tons of free space, but will hang
if underlying space is used up).

My understanding was the when not using --mixed, btrfs has block groups
for data and metadata.

And it seems here that the data block groups have several 100 GB still
free, while - AFAIU you - the metadata block groups are already full.



I also wouldn't want to regularly balance (which doesn't really seem to
help that much so far)... cause it puts quite some IO load on the
systems.


So if csum data needs so much space... why can't it simply reserve e.g.
60 GB for metadata instead of just 17 GB?

Because all chunks are allocated on demand, if 1) your workload has
every unbalanced data/metadata usage, like this case (almost 1000:1).
2) You run out of space, then you will hit this particular problem.

If I really had to reserve ~ 1TB of storage to be unused (per 16TB fs)
just to get that working... I would need to move stuff back to ext4,
cause that's such a big loss we couldn't justify to our funding
agencies.

It won't matter if you reserve 1T or not for the data.

It can still go the same problem even if there are tons of unused data
space.
Fragmented data space can still cause the same problem.

And we haven't had that issue with e.g. ext4 ... that seems to reserve
just enough for meta, so that we could basically fill up the fs close
to the end.

Ext4/XFS has a similar problem but much harder to hit, inode limits.

They use pre-determined inode limits (determined at mkfs time), thus you
can ran out of inodes before free space is used up.

Tools like "df" has ways to report such limits, but unfortunately for
btrfs there is no such way, but using btrfs specific tool to do that.

Thanks,
Qu

Cheers,
Chris.