[PATCH v7 00/12] vfs: add O_CREAT|O_DIRECTORY to open*(2)

This series implements new semantics for the O_CREAT|O_DIRECTORY flag
combination for open*(2): perform a mkdir and open the resulting
directory; return a pinning fd (which mkdir does not).

The v6 was found by Brauner to contain a bug because ->mkdir
implementations may leave the dentry negative, and do_open() cannot deal
with that (NULL pointer deref.) This behavior occurs in kernfs and
tracefs at least. Since atomic mkdir is wanted for cgroupfs (as per
Brauner at LPC26), kernfs is changed to return the new kernfs_node from
the kernfs_syscall_ops ->mkdir() callbacks and splicing its inode.

Filesystems that do not support O_CREAT|O_DIRECTORY set
SB_I_NO_ATOMIC_MKDIR, and lookup_open() refuses the open with EOPNOTSUPP
for them before doing any lookup. These are all ->atomic_open()
filesystems, which cannot create directories without protocol
extensions, and tracefs, whose ->mkdir() drops the parent's i_rwsem so
that the directory at the name may not be the one created by the call
(or at least it has to be verified that this plays well with the
atomicity guarantee that O_CREAT|O_DIRECTORY attempts to provide.) For
ecryptfs and apparmorfs there are similar concerns and hence they are
also blocked for now.

Most of the work happens in "vfs: add O_CREAT|O_DIRECTORY to open*(2),"
as may be expected. Before that there is some clean-up work and
preparation. The "vfs: short-circuit MAY_WRITE access for O_DIRECTORY
opens" patch is also worth paying extra attention to as it
short-circuits doomed opening of directories as writable. This check (to
prevent one from opening directories as writable) is currently done very
late in do_open(), after an inode has been obtained. However, when
introducing O_CREAT|O_DIRECTORY this is unacceptable as it would create
the directory and then still fail. This patch does however change some
user visible error codes.

Moreover, "vfs: change ->create/->mkdir operations unavailable errno"
also changes the ->create and ->mkdir unavailable errnos to -EOPNOTSUPP.
A previous attempt to do this more widely stranded before.[1] However,
that regression was only for vfat and does seem specific to changing the
return code from vfs_symlink(). We can drop this patch if needed, but it
would be _really_ ugly.

Changes from vn to v(n+1):
v6: https://lore.kernel.org/linux-fsdevel/20260913185016.523376-1-jkoolstra@xs4all.nl/ (local)
   - Change kernfs to instantiate ->mkdir dentry.
   - Fix kernfs locking when initializing a new inode to hold
     kernfs_iattr_rwsem.
   - Add flag SB_I_NO_ATOMIC_MKDIR to disable O_CREAT|O_DIRECTORY on
     unsupported filesystems (instead of only looking at ->atomic_open.)
   - Add tests to verify behavior of O_CREAT|O_DIRECTORY on cgroupfs and
     tracefs.
v5: https://lore.kernel.org/linux-fsdevel/20260823160706.358293-1-jkoolstra@xs4all.nl/ (local)
   - Uniformize ->create and ->mkdir unavailable errnos to -EOPNOTSUPP.
   - Return -EOPNOTSUPP instead of -ENOENT when O_CREAT|O_DIRECTORY is
     not supported by a ->atomic_open filesystem.
   - Added test cases for dangling symlink and sticky directory
     behaviour, and a test that verifies O_TMPFILE|O_CREAT is still
     -EINVAL.
   - Split off "vfs: lookup_open(): lock the parent as I_MUTEX_PARENT"
     as a separate patch.
   - Fixed the issues pointed out by Brauner in v5.
v4: https://lore.kernel.org/linux-fsdevel/20260712175539.1565444-1-jkoolstra@xs4all.nl/ (local)
   - rebased on 7.2, which includes my changes to auditing in
     lookup_open() as well as Neil Brown's changes to the same function
     for the implementation of vfs_lookup_open().
v3: https://lore.kernel.org/linux-fsdevel/20260704164149.3480051-1-jkoolstra@xs4all.nl/ (local)
   - address the inconsistency in calling audit_inode_child() in
     lookup_open() versus vfs_create() in a separate series.
   - fclog.c selftest header fix moved to separate patch.
   - add a "with trailing slash test" success test to the included
     selftests. 
   - pass struct qstr by pointer to trailing_slashes() and add a comment
     on what it does
   - changed commit message of "vfs: add O_CREAT|O_DIRECTORY to
     open*(2)" to address comments of Brauner

[1]: https://lore.kernel.org/linux-fsdevel/90228c39-04e7-41ef-ad91-84a8bb866650@sirena.org.uk/ (local)


Jori Koolstra (12):
  fs/namei.c: use trailing_slashes()
  vfs: prepare vfs_creat|mkdir_no_perm for reuse in lookup_open()
  vfs: lookup_open(): move setting FMODE_CREATED down
  vfs: move ->create check in lookup_open() to before try_break_deleg()
  vfs: lookup_open(): use vfs_create_no_perm()
  vfs: lookup_open(): lock the parent as I_MUTEX_PARENT
  kernfs: hold kernfs_iattr_rwsem when initializing a new inode
  vfs: add O_CREAT|O_DIRECTORY to open*(2)
  vfs: use finish_lookup() helper in __lookup_slow() and lookup_open()
  vfs: short-circuit MAY_WRITE access for O_DIRECTORY opens
  vfs: change ->create/->mkdir operations unavailable errno
  selftest: add tests for open*(O_CREAT|O_DIRECTORY)

 arch/alpha/include/uapi/asm/errno.h           |   2 +-
 arch/mips/include/uapi/asm/errno.h            |   2 +-
 arch/parisc/include/uapi/asm/errno.h          |   2 +-
 arch/sparc/include/uapi/asm/errno.h           |   2 +-
 fs/9p/vfs_super.c                             |   1 +
 fs/ceph/super.c                               |   1 +
 fs/ecryptfs/main.c                            |   1 +
 fs/fuse/inode.c                               |   1 +
 fs/gfs2/ops_fstype.c                          |   1 +
 fs/kernfs/dir.c                               |  30 +-
 fs/kernfs/inode.c                             |  37 +-
 fs/kernfs/kernfs-internal.h                   |   2 +
 fs/kernfs/mount.c                             |   3 -
 fs/namei.c                                    | 275 +++++++++-----
 fs/nfs/super.c                                |   1 +
 fs/open.c                                     |  42 ++-
 fs/resctrl/rdtgroup.c                         |  35 +-
 fs/smb/client/cifsfs.c                        |   1 +
 fs/tracefs/inode.c                            |   1 +
 fs/vboxsf/super.c                             |   1 +
 include/linux/fcntl.h                         |   6 +
 include/linux/fs/super_types.h                |   1 +
 include/linux/kernfs.h                        |   7 +-
 include/uapi/asm-generic/errno.h              |   2 +-
 kernel/cgroup/cgroup-internal.h               |   3 +-
 kernel/cgroup/cgroup.c                        |  16 +-
 security/apparmor/apparmorfs.c                |   1 +
 .../testing/selftests/filesystems/.gitignore  |   1 +
 tools/testing/selftests/filesystems/Makefile  |   2 +-
 .../filesystems/open_o_creat_o_dir.c          | 346 ++++++++++++++++++
 .../testing/selftests/filesystems/wrappers.h  |  11 +
 31 files changed, 691 insertions(+), 146 deletions(-)
 create mode 100644 tools/testing/selftests/filesystems/open_o_creat_o_dir.c


base-commit: 90201351a862182bdabfc1ad1c97b1e1b6ed5931
-- 
2.56.0