Thread (28 messages) flat view 28 messages, 5 authors, 2021-11-11

Re: [PATCH v2 2/3] wrapper: provide function to sync directories

From: Ævar Arnfjörð Bjarmason <hidden>
Date: 2021-11-10 14:47:35

On Wed, Nov 10 2021, Patrick Steinhardt wrote:
quoted hunk ↗ jump to hunk
[[PGP Signed Part:Undecided]]
In ec983eb5d2 (core.fsyncobjectfiles: batched disk flushes, 2021-10-04),
we have introduced batched syncing of object files. This mode works by
only requesting a writeback of the page cache backing the file on
written files, followed by a single hardware-flush via a temporary file
created in the directory we want to flush. Given modern journaling file
systems, this pattern is expected to be durable.

While it's possible to reuse the `git_fsync()` helper to synchronize the
page cache only, there is no helper which would allow for doing a
hardware flush of a directory by creating a temporary file. Other
callers which want to follow the same pattern would thus have to repeat
this logic.

Extract a new helper `git_fsync_dir()` from the object files code which
neatly encapsulates this logic such that it can be reused.

Signed-off-by: Patrick Steinhardt <redacted>
---
 bulk-checkin.c    | 13 +++----------
 git-compat-util.h |  7 +++++++
 wrapper.c         | 21 +++++++++++++++++++++
 3 files changed, 31 insertions(+), 10 deletions(-)
diff --git a/bulk-checkin.c b/bulk-checkin.c
index 4deee1af46..e6ebdd1db5 100644
--- a/bulk-checkin.c
+++ b/bulk-checkin.c
@@ -98,16 +98,9 @@ static void do_batch_fsync(void)
 	 * hardware.
 	 */
 
-	if (needs_batch_fsync) {
-		struct strbuf temp_path = STRBUF_INIT;
-		struct tempfile *temp;
-
-		strbuf_addf(&temp_path, "%s/bulk_fsync_XXXXXX", get_object_directory());
-		temp = xmks_tempfile(temp_path.buf);
-		fsync_or_die(get_tempfile_fd(temp), get_tempfile_path(temp));
-		delete_tempfile(&temp);
-		strbuf_release(&temp_path);
-	}
+	if (needs_batch_fsync &&
+	    git_fsync_dir(get_object_directory()) < 0)
+		die_errno("fsyncing object directory");
Nit: Similar to 1/3, but this message is new: We say "fsyncing object
directory", but it would be better to pass in some "verbose" flag to
git_fsync_dir() so we can say e.g.:

    error_errno(_("couldn't create core.fsyncRefFiles=batch tempfile '%s' in '%s'"), ...)
    error_errno(_("couldn't fsync() core.fsyncRefFiles=batch tempfile '%s' in '%s'"), ...)

I.e. being able to say specifically why we failed, permission error or
the tempfile? fsync() didn't work etc?

Looking at the underlying APIs maybe they already have a mode to "die"
or "warn" appropriately? Or...
+int git_fsync_dir(const char *path)
+{
+	struct strbuf temp_path = STRBUF_INIT;
+	struct tempfile *temp;
+
+	strbuf_addf(&temp_path, "%s/bulk_fsync_XXXXXX", path);
+
+	temp = mks_tempfile(temp_path.buf);
+	if (!temp)
+		return -1;
+
+	if (git_fsync(get_tempfile_fd(temp), FSYNC_HARDWARE_FLUSH) < 0)
+		return -1;
...if they do maybe we should use their non-fatal mode, because
with/without that these "return -1" need to be "goto cleanup" so we can
attempt to clean up after ourselves here.

I think this whole thing would be better if we generalized tmp-objdir.h
a bit, so it could create and manage an arbitrary file in an arbitrary
directory, and that API should really be generalized to a user of
tempfile.c.

I.e. we'd then create this file, sync it optionally, whine if it does't
work, and be guaranteed to try to clean anything that goes wrong up
atexit().
Keyboard shortcuts
hback out one level
jnext message in thread
kprevious message in thread
ldrill in
Escclose help / fold thread tree
?toggle this help