On Wed, Jun 22, 2022 at 5:39 PM Qu Wenruo [off-list ref] wrote:

[...]

E.g.
btrfs uses 64KiB as stripe size.
O = Old data
N = New writes

        0       32K     64K
D1      |OOOOOOO|NNNNNNN|
D2      |NNNNNNN|OOOOOOO|
P       |NNNNNNN|NNNNNNN|

In above case, no matter if the new write reaches disks, as long as the
crash happens before we update all the metadata and superblock (which
implies a flush for all involved devices), the fs will only try to read
the old data.

I guess we are using "write hole" for different scenarios. I use "write hole"
for the case that we corrupt data that is not being written to. This happens
with the combination of failed drive and power loss. For example, we have
raid5 with 3 drives. Each stripe has two data and one parity. When D1
failed, read to D1 is calculated based on D2 and P; and write to D1
requires updating D2 and P at the same time. Now imagine we lost
power (or crash) while writing to D2 (and P). When the system comes back
after reboot, D2 and P are out of sync. Now we lost both D2 and D1. Note
that D1 is not being written to before the power loss.

For btrfs, maybe we can avoid write hole by NOT writing to D2 when D1
contains valid data (and the drive is failed). Instead, we can write a new
version of D1 and D2 to a different stripe. If we loss power during the write,
the old data is not corrupted. Does this make sense? I am not sure
whether it is practical in btrfs though.

So at this point, our data read on old data is still correct.
But the parity no longer matches, thus degrading our ability to tolerate
device lost.

With write-intent bitmap, we know this full stripe has something out of
sync, so we can re-calculate the parity.

Although, all above condition needs two things:

- The new write is CoWed.
   It's mandatory for btrfs metadata, so no problem. But for btrfs data,
   we can have NODATACOW (also implies NDOATASUM), and in that case,
   corruption will be unavoidable.

- The old data should never be changed
   This means, the device can not disappear during the recovery.
   If powerloss + device missing happens, this will not work at all.

quoted

You must either:
  1/ have a safe duplicate of the blocks being written, so they can be
    recovered and re-written after a crash.  This is what journalling
    does.  Or

Yes, journal would be the next step to handle NODATACOW case and device
missing case.

quoted

  2/ Only write to location which don't contain valid data.  i.e.  always
    write full stripes to locations which are unused on each device.
    This way you cannot lose existing data.  Worst case: that whole
    stripe is ignored.  This is how I would handle RAID5 in a
    copy-on-write filesystem.

That is something we considered in the past, but considering even now we
still have space reservation problems sometimes, I doubt such change
would cause even more problems than it can solve.

quoted

However, I see you wrote:

quoted

Thus as long as no device is missing, a write-intent-bitmap is enough to
address the write hole in btrfs (at least for COW protected data and all
metadata).

That doesn't make sense.  If no device is missing, then there is no
write hole.
If no device is missing, all you need to do is recalculate the parity
blocks on any stripe that was recently written.

That's exactly what we need and want to do.

I guess the goal is to find some files after crash/power loss. Can we
achieve this with file mtime? (Sorry if this is a stupid question...)

Thanks,
Song