[PATCH v2 1/2] iov: Bypass usercopy hardening for copy_to_iter()
From: Chuck Lever <cel@kernel.org>
Date: 2026-03-30 14:36:48
Also in:
linux-block, linux-fsdevel, linux-hardening
Subsystem:
kernel hardening (not covered by other areas), the rest, userspace copyin/copyout (uiovec), virtio core · Maintainers:
Kees Cook, Linus Torvalds, Alexander Viro, "Michael S. Tsirkin", Jason Wang
From: Chuck Lever <redacted> Profiling NFSD under an iozone workload showed that hardened usercopy checks consume roughly 1.3% of CPU in the TCP receive path. The runtime check in check_object_size() validates that copy buffers reside in expected kernel memory regions (slab, stack, and non-text), which is meaningful when data crosses the user/kernel boundary but adds no value when both source and destination are kernel addresses. Split check_copy_size() so that copy_to_iter() can bypass the runtime check_object_size() call for non-user-backed iterators (ITER_KVEC, ITER_BVEC, ITER_FOLIOQ, ITER_XARRAY, and ITER_DISCARD). Existing callers of check_copy_size() are unaffected; user-backed iterators still receive the full usercopy validation. This benefits all kernel consumers of copy_to_iter(), including the TCP receive path used by the NFS client and server, NVMe-TCP, and any other subsystem that uses non-user-backed receive buffers. Signed-off-by: Chuck Lever <redacted> --- include/linux/ucopysize.h | 16 +++++++++++++++- include/linux/uio.h | 9 +++++++-- tools/virtio/linux/ucopysize.h | 6 ++++++ 3 files changed, 28 insertions(+), 3 deletions(-)
diff --git a/include/linux/ucopysize.h b/include/linux/ucopysize.h
index 41c2d9720466..d187108f845a 100644
--- a/include/linux/ucopysize.h
+++ b/include/linux/ucopysize.h@@ -41,8 +41,14 @@ static inline void copy_overflow(int size, unsigned long count) __copy_overflow(size, count); } +/* + * Copy size validation without usercopy hardening. Checks + * compile-time object size and runtime overflow, but skips + * check_object_size(). Use check_copy_size() when @addr + * may point to userspace-accessible memory. + */ static __always_inline __must_check bool -check_copy_size(const void *addr, size_t bytes, bool is_source) +__compiletime_check_copy_size(const void *addr, size_t bytes, bool is_source) { int sz = __builtin_object_size(addr, 0); if (unlikely(sz >= 0 && sz < bytes)) {
@@ -56,6 +62,14 @@ check_copy_size(const void *addr, size_t bytes, bool is_source) } if (WARN_ON_ONCE(bytes > INT_MAX)) return false; + return true; +} + +static __always_inline __must_check bool +check_copy_size(const void *addr, size_t bytes, bool is_source) +{ + if (!__compiletime_check_copy_size(addr, bytes, is_source)) + return false; check_object_size(addr, bytes, is_source); return true; }
diff --git a/include/linux/uio.h b/include/linux/uio.h
index a9bc5b3067e3..45b323e4be97 100644
--- a/include/linux/uio.h
+++ b/include/linux/uio.h@@ -216,8 +216,13 @@ size_t copy_page_to_iter_nofault(struct page *page, unsigned offset, static __always_inline __must_check size_t copy_to_iter(const void *addr, size_t bytes, struct iov_iter *i) { - if (check_copy_size(addr, bytes, true)) - return _copy_to_iter(addr, bytes, i); + if (user_backed_iter(i)) { + if (check_copy_size(addr, bytes, true)) + return _copy_to_iter(addr, bytes, i); + } else { + if (__compiletime_check_copy_size(addr, bytes, true)) + return _copy_to_iter(addr, bytes, i); + } return 0; }
diff --git a/tools/virtio/linux/ucopysize.h b/tools/virtio/linux/ucopysize.h
index 8beb7755d060..a330e14c81c5 100644
--- a/tools/virtio/linux/ucopysize.h
+++ b/tools/virtio/linux/ucopysize.h@@ -12,6 +12,12 @@ static inline void copy_overflow(int size, unsigned long count) { } +static __always_inline __must_check bool +__compiletime_check_copy_size(const void *addr, size_t bytes, bool is_source) +{ + return true; +} + static __always_inline __must_check bool check_copy_size(const void *addr, size_t bytes, bool is_source) {
--
2.53.0