On May 25, 2021, at 4:31 PM, David Howells [off-list ref] wrote:

Andreas Dilger [off-list ref] wrote:

quoted

As described elsewhere in the thread, allowing concurrent create and unlink
in a directory (rename probably not needed) would be invaluable for scaling
multi-threaded workloads.  Neil Brown posted a prototype patch to add this
to the VFS for NFS:

Actually, one thing I'm looking at is using vfs_tmpfile() to create a new file
(or a replacement file when invalidation is required) and then using
vfs_link() to attach directory entries in the background (possibly using
vfs_link() with AT_LINK_REPLACE[1] instead of unlink+link).

Any thoughts on how that might scale?  vfs_tmpfile() doesn't appear to require
the directory inode lock.  I presume the directory is required for security
purposes in addition to being a way to specify the target filesystem.

I don't see how that would help much?  Yes, the tmpfile allocation would be
out-of-line vs. the directory lock, so this may reduce the lock hold time
by some fraction, but this would still need to hold the directory lock
when linking the tmpfile into the directory, in the same way that create
and unlink are serialized against other threads working in the same dir.

Having the directory locking scale with the size of the directory is what
will get orders of magnitude speedups for large concurrent workloads.
In ext4 this means write locking the directory leaf blocks independently,
with read locks for the interior index blocks unless new leaf blocks are
added (they are currently never removed).

It's the same situation as back with the BKL locking the entire kernel,
before we got fine-grained locking throughout the kernel.

David

[1] https://lore.kernel.org/linux-fsdevel/cover.1580251857.git.osandov@fb.com/ (local)

Cheers, Andreas