Re: Out of space on small-ish partition, clobber and other methods haven't worked

Thanks, Duncan for the thorough and thoughtful reply.  We appreciate
that btrfs is stabilizing (emphasis understood).  In our case, we
think the benefits for cow for running on compact flash will outweigh
the potential for a btrfs-specific hiccup.  In this particular case,
these systems do not contain unique data.  The background you give
strengthens our resolve to have in place contingency plans in case of
a hiccup.

We've moved to 4.1 in our production release, and before asking people
here for support in future, we'll try our symptom against the latest
kernel rev available.

 - Jerry




On Thu, Jan 21, 2016 at 4:41 AM, Duncan [off-list ref] wrote:

Chris Murphy posted on Wed, 20 Jan 2016 19:28:35 -0700 as excerpted:

quoted

On Wed, Jan 20, 2016 at 2:22 PM, Jerry Steinhauer
[off-list ref] wrote:

quoted

% rm a.file
rm: cannot remove 'a.file': No space left on device
% cat /dev/null > a.file
-sh: a.file: No space left on device

quoted

quoted

% btrfs fi df /data
System, single: total=32.00MiB, used=4.00KiB
Data+Metadata, single: total=506.00MiB, used=500.39MiB
GlobalReserve, single: total=12.00MiB, used=6.45MiB

I see somewhere between 6MiB and 12MiB that should be available for file
removal.

I don't.  See that global reserve?  6.45 MiB into its emergency reserve,
so effectively -6.45 MiB of space available for file removal.

First of all, any time global reserve is used at all the filesystem is in
very dire straits, and he's 6.45 MiB into the 12.00 MiB global reserve,
so that alone tells us "we're not in Kansas any more!" =8^0

Second, the btrfs fi show (which you didn't quote) says 540 MiB capacity.

System           32 MiB total, can't be used for anything else
Data+Metadata   506 MiB total, shared data/metadata as it's a small
filesystem  (See why I didn't list global reserve here, below.)


Total           538 MiB chunked out.  While that's 2 MiB from the
reported 540 capacity, I don't believe system includes the reserved space
(for boot loader, etc) at the beginning of the partition.  Between that
and the limits of the chunk-allocator, he's likely all chunked-out, no
possibility of allocating further chunks.

Global reserve is normally reserved from metadata, which of course is
shared data/metadata here, due to the size of the filesystem (which makes
shared a practical necessity, the problems would be much worse if data
and metadata chunks were separate!).

So of the 506 MiB in data/metadata, 12 MiB are global reserve.  Which
means there's only 494 MiB of normal data/metadata space, plus 12 MiB of
global reserve.

But the DF shows 500.39 MiB of data/metadata used, which means we're
roughly 6.4 MiB past normal data/metadata usage into the emergency use
only global reserve, which is indeed (roughly) what global reserve shows,
6.45 MiB used.

So as I said, that btrfs is in pretty severely dire straits!  Not only is
all the available data/metadata space used, but we're well past half way
into the emergency global reserve as well.  No WONDER there's no space
left even to delete a file (which because btrfs is COW, copy-on-write,
requires metadata space even to delete a file, as the metadata block
containing the original data cannot be rewritten in place and must be
written elsewhere... thus answering the question of why btrfs needs space
even on the unlink).


As for solutions, there's still a couple things (plus one already tried)
to try to get out of the situation:

0) Try clobbering the file, reducing it to zero size, but you did and
that didn't work.  It might have if the btrfs wasn't already so far into
global reserve.

1) As CMurphy says (with two Chris Ms on the list that isn't clear, so
CMurphy it is), try a later kernel, either 4.1.x or 4.4.  AFAIK there
were a few patches having to do with ENOSPC errors and allowing file
deletes to take from global reserve, as the result should be more room
afterward and that's exactly the sort of thing global reserve is supposed
to be there for.  Tho it's just a try, no guarantees.

2) This could be difficult on embedded, but the other option is
temporarily adding a second device (btrfs device add), to give the
filesystem a bit of work with.  That takes space as well, but luckily, I
believe it's system-chunk space, and there's plenty of room there, so it
should be possible.

The idea is to get enough metadata space to work with to get out of the
fix by deleting a file or the like (normally, a balance could help as
well, but that primarily helps to reclaim empty chunks from say data, so
they can be reassigned to metadata, and since this is shared data/
metadata, that's unlikely to help).

Then when the filesystem is back to usable and enough has been deleted so
what's on the temporary second device will fit back on the first device
again, btrfs device delete the second one.

I'm unfamiliar with how small an added device can be and still be useful
at that level, or more precisely, how the system chunk shrinks with total
device size, but the one small data point I have here is a 256 MiB /boot,
which has a 16 MiB system chunk, so I'm guessing it should shrink at
least that far.

So let's say 16 MiB system, and it's into global reserve by ~6.5 MiB, so
we want to give it at least that much more, plus something to work with.

So I'd suggest a 24 MiB or if it's available, 32 MiB, second device, at
minimum.  Smaller can be tried, with the hope that the system chunk
shrinks to say 8 MiB or smaller if the device is small enough, but I'm
not sure it will.

As for actually making available a device on embedded, if there's no USB
port available and thus the "simple" solution of plugging in a thumb
drive is out of the question... maybe there's enough memory to create a
tmpfs and do a loopback file on it, then add that loopback file as the
temporary second device.  Of course if the power dies or the system
otherwise crashes when part of the filesystem's on that tmpfs... not good
news.  And obviously in that case it /better/ be temporary, because you
can't reboot without losing that tmpfs and with it the loopback.  But if
there's no other way to get access to a suitable device and the system
and power is stable enough...


So that answers the what to do to exit that state question, and in a
parenthetical  I answered the question of why it's requiring space to
unlink -- btrfs is cow, copy-on-write, so even unlinking a file requires
space to copy the metadata block containing the information about that
file for the write.  And it makes the third question moot, as we have the
root cause already -- the cow nature of btrfs.


Meanwhile, one more thing to address.  Despite what various distros may
claim, here on this list, btrfs is considered "stablizING, but not yet
fully stable or mature."  Production usage, particularly without backups,
isn't recommended, occasional bugs can be expected, and the standard
recommendation is using no older than the last two of either current
kernels or LTS kernels.   With the just released 4.4 being an LTS kernel,
that makes 4.1 the previous one back and the oldest recommended kernel,
tho with 4.4 being so new, still being on the LTS before that, 3.18,
would still be somewhat acceptable if you're already working on updating
to 4.1.  But before that, while we'll try to support as best we can,
chances are very good among the first requests is going to be to update
to something not so ancient.

Under those conditions, honestly, it may be that btrfs isn't yet stable
enough to be the right choice, particularly for embedded projects that
are supposed to be field-usable without backups and without available
technical maintenance for some years.  As I said, we're stabilizing, and
actually, I'm not sure about the devs (I'm a list regular and btrfs user,
not a dev) and other list regulars, but it may be that with LTS 4.4 we'll
extend the informal support scope to three LTS series and thus support
3.18 awhile longer, but btrfs is definitely not where /I'd/ recommend
using btrfs on designed to be field usable without ready backups or
direct tech supervision embedded, just yet.

OTOH, if it's embedded but with backups and direct tech supervision, then
btrfs may be just fine, if you're willing to put up with the occasional
bug and accept that you must be prepared to actually have to use those
backups, should one of those occasional bugs require it, and if keeping
generally to the last two LTS (or current) kernel series is acceptable.

--
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html