On 2022/6/23 11:32, Song Liu wrote:

On Wed, Jun 22, 2022 at 5:39 PM Qu Wenruo [off-list ref] wrote:

quoted

[...]

quoted

E.g.
btrfs uses 64KiB as stripe size.
O = Old data
N = New writes

         0       32K     64K
D1      |OOOOOOO|NNNNNNN|
D2      |NNNNNNN|OOOOOOO|
P       |NNNNNNN|NNNNNNN|

In above case, no matter if the new write reaches disks, as long as the
crash happens before we update all the metadata and superblock (which
implies a flush for all involved devices), the fs will only try to read
the old data.

I guess we are using "write hole" for different scenarios. I use "write hole"
for the case that we corrupt data that is not being written to. This happens
with the combination of failed drive and power loss. For example, we have
raid5 with 3 drives. Each stripe has two data and one parity. When D1
failed, read to D1 is calculated based on D2 and P; and write to D1
requires updating D2 and P at the same time. Now imagine we lost
power (or crash) while writing to D2 (and P). When the system comes back
after reboot, D2 and P are out of sync. Now we lost both D2 and D1. Note
that D1 is not being written to before the power loss.

For that powerloss + device lose case, journal is the only way to go,
unless we do extra work to avoid partial write.

For btrfs, maybe we can avoid write hole by NOT writing to D2 when D1
contains valid data (and the drive is failed). Instead, we can write a new
version of D1 and D2 to a different stripe. If we loss power during the write,
the old data is not corrupted. Does this make sense? I am not sure
whether it is practical in btrfs though.

That makes sense, but that also means the extent allocator needs extra
info, not just which space is available.

And there would make ENOSPC handling even more challenging, what if we
have no space left but only partially written stripes?

There are some ideas, like extra layer for RAID56 to do extra mapping
between logical address to physical address, but I'm not yet confident
if we will see new (and even more complex) challenges going that path.

quoted

So at this point, our data read on old data is still correct.
But the parity no longer matches, thus degrading our ability to tolerate
device lost.

With write-intent bitmap, we know this full stripe has something out of
sync, so we can re-calculate the parity.

Although, all above condition needs two things:

- The new write is CoWed.
    It's mandatory for btrfs metadata, so no problem. But for btrfs data,
    we can have NODATACOW (also implies NDOATASUM), and in that case,
    corruption will be unavoidable.

- The old data should never be changed
    This means, the device can not disappear during the recovery.
    If powerloss + device missing happens, this will not work at all.

quoted

You must either:
   1/ have a safe duplicate of the blocks being written, so they can be
     recovered and re-written after a crash.  This is what journalling
     does.  Or

Yes, journal would be the next step to handle NODATACOW case and device
missing case.

quoted

   2/ Only write to location which don't contain valid data.  i.e.  always
     write full stripes to locations which are unused on each device.
     This way you cannot lose existing data.  Worst case: that whole
     stripe is ignored.  This is how I would handle RAID5 in a
     copy-on-write filesystem.

That is something we considered in the past, but considering even now we
still have space reservation problems sometimes, I doubt such change
would cause even more problems than it can solve.

quoted

However, I see you wrote:

quoted

Thus as long as no device is missing, a write-intent-bitmap is enough to
address the write hole in btrfs (at least for COW protected data and all
metadata).

That doesn't make sense.  If no device is missing, then there is no
write hole.
If no device is missing, all you need to do is recalculate the parity
blocks on any stripe that was recently written.

That's exactly what we need and want to do.

I guess the goal is to find some files after crash/power loss. Can we
achieve this with file mtime? (Sorry if this is a stupid question...)

There are two problems here:

1. After power loss, we won't see the mtime update at all.
    As the mtime update will be protected by metadata CoW, since the
    powerloss happens when the current transaction is not committed,
    we will only see the old metadata after recovery.

    Thus to btrfs, at next reboot, it can not see the new mtime at all.

    AKA everything CoWed is updated atomically, we can only see
    trnsaction last committed.

    Although there is something special like log tree for fsync(), it has
    its own limitation (can not happen across transaction boundary), thus
    still not suitable for things like random data write.

2. Will not work for metadata, unless we scrub the whole metadata at
    recovery time
    The core problem here is graduality.
    The target file can be in TiB size, or for the whole metadata.

    Scrubbing such large range before allowing user to do any write can
    lead to super unhappy end users.

So for now, as a (kinda) quick solution, I'd like to go write-intent
bitmap first, then journal, just like md-raid.

Thanks,
Qu

Thanks,
Song