Hi Josef and Amir,

Thank you for the replies! We were aware that Josef had proposed
something like this a few years ago [1], but didn't know it was being
currently used inside Facebook. Glad to hear it!

@Josef: Thanks for the link! I think CrashMonkey does what you have in
log-writes, but our goal is to go a bit further:

- We want to test replaying subsets of writes between flush/fua.
Indeed, this is one of the major focus points of the work. Given W1 W2
W3 Flush, CrashMonkey will generate states, (W1), (W1 W3), (W1 W2),
etc. We believe many interesting bugs lie in this space. The problem
is that there are a large number of possible crash states, so we are
working on techniques to find "interesting" crash states. For now, our
plan is to focus on write requests tagged with the META flag.

- We want to aid the users in testing data consistency after a crash.
The plan is that after each crash state, after running fsck, if the
file system mounts, we allow the user to run a number of custom tests.
To help the user figure out what data should be present in the crash
state, we plan to provide functionality that informs the user at which
point the crash occurred (similar to the "mark" functionality in
log-writes, but instead of indicating a single point in the stream, it
would provide a snapshot of fs state)

@Amir: Given that Josef's code is already in the kernel, do you think
changing CrashMonkey code would be useful? We are always happy to
provide something for upstream, but we want to be sure how much work
would be involved.

[1] https://lwn.net/Articles/637079/

Thanks,
Vijay

On Tue, Aug 15, 2017 at 12:33 PM, Josef Bacik [off-list ref] wrote:

On Mon, Aug 14, 2017 at 11:32:02AM -0500, Vijay Chidambaram wrote:

quoted

Hi,

I'm Vijay Chidambaram, an Assistant Professor at the University of
Texas at Austin. My research group is developing CrashMonkey, a
file-system agnostic framework to test file-system crash consistency
on power failures. We are developing CrashMonkey publicly at Github
[1]. This is very much a work-in-progress, so we welcome feedback.

CrashMonkey works by recording all the IO from running a given
workload, then *constructing* possible crash states (while honoring
FUA and FLUSH flags). A crash state is the state of storage after an
abrupt power failure or crash. For each crash state, CrashMonkey runs
the filesystem-provided fsck on top of the state, and checks if the
file-system recovers correctly. Once the file system mounts correctly,
we can run further tests to check data consistency.  The work was
presented at HotStorage 17. The workshop paper is available at [2] and
the slides at [3].

Our plan was to post on the mailing lists after reproducing an
existing bug. We are not there yet, but I saw some posts where others
were considering building something similar, so I thought I would post
about our work.

[1] https://github.com/utsaslab/crashmonkey
[2] http://www.cs.utexas.edu/~vijay/papers/hotstorage17-crashmonkey.pdf
[3] http://www.cs.utexas.edu/~vijay/papers/hotstorage17-crashmonkey-slides.pdf

I did this same work 3 years ago

https://github.com/torvalds/linux/blob/master/Documentation/device-mapper/log-writes.txt
https://github.com/josefbacik/log-writes

I have xfstests patches I need to get upstreamed at some point that does
fsstress and then replays the logs and verifies, and also one that makes fsx
store state so we can verify fsync() is doing the right thing.  We run this on
our major releases on xfs, ext4, and btrfs to make sure everything is working
right internally at Facebook.  You'll notice a bunch of commits recently because
we thought we found an xfs replay problem (we didn't).  This stuff is actively
used, I'd welcome contributions to it if you have anything to add.  One thing I
haven't done yet and have on my list is to randomly replay writes between
flush/fua, but it hasn't been a pressing priority yet.  Thanks,

Josef