Re: [RFC] splice: add wait-all semantics for socket-to-pipe reads

On Tue, Sep 1, 2026 at 8:53 PM 天狼 [off-list ref] wrote:

Hi,

I would like to ask whether splice(2) could gain wait-all semantics for
blocking socket-to-pipe transfers, analogous to recv(MSG_WAITALL).

The concrete use case is a high-performance S3 FUSE client.  Once an HTTP
response header has been parsed, the response body length and the exact range
requested by FUSE are known.  The body is moved from a TCP socket into a pipe
used for an fd-backed FUSE reply.  No userspace data buffer is needed.

Today userspace has to loop around splice() until the requested length has
been transferred.  On a fast local network this commonly turns one logical
read into several splice syscalls and scheduling transitions.  In one recent
profile, 3,475 S3 GET requests caused 25,463 splice syscalls (7.33 per GET).
The profile contained no poll or setsockopt calls in the hot path, and 80.86%
of the sampled CPU time was in the kernel.

The obvious workarounds all have undesirable properties:

  - looping over splice() in userspace adds syscalls and wakeups for short
    positive returns;
  - poll plus SO_RCVLOWAT waits for data to accumulate in the socket receive
    buffer, adds another syscall, and couples latency to an advisory readiness
    threshold;
  - recv(MSG_WAITALL) requires a userspace buffer and loses the socket-to-pipe
    splice path;
  - increasing SO_RCVBUF does not make splice wait for the requested length.

The current TCP implementation is already close to providing the desired
behavior.  tcp_splice_read() obtains its time budget with sock_rcvtimeo() and
loops while data remains immediately available.  However, after making any
positive progress, a later iteration that finds no data returns immediately
because of:

        else if (!ret) {
                if (spliced)
                        break;
                ...
                ret = sk_wait_data(sk, &timeo, NULL);
        }

Would a new flag such as SPLICE_F_WAITALL be acceptable?

The use case makes sense, but I think it depends on the numbers.

Another aspect would be whether we want to add a new (potentially
buggy) feature for a single user in this AI era.

It would be great if the change is proven to improve performance
of the official client that has a lot more users worldwide.
https://github.com/awslabs/mountpoint-s3/

 For a blocking
socket-to-pipe splice it would mean:

  - keep transferring until len bytes have entered the pipe;
  - after a short positive transfer followed by temporary lack of input,
    wait in the kernel with sk_wait_data() instead of returning;
  - use the socket's existing SO_RCVTIMEO as the total receive wait budget;
  - still return early for EOF, shutdown, a pending socket error, or a signal;
  - on timeout/error/signal after partial progress, return the partial byte
    count, matching the usual stream-receive convention; otherwise return the
    corresponding error;
  - retain the existing behavior when SPLICE_F_WAITALL is absent.

SPLICE_F_NONBLOCK/O_NONBLOCK would remain orthogonal: WAITALL would not cause
sleeping on a nonblocking descriptor.  The destination pipe must either have
enough capacity for len or be drained concurrently, as with blocking splice
today.  For the motivating workload the pipe is sized for the FUSE request.

The important property is that data can be spliced into the pipe immediately
as TCP data arrives.  Unlike poll plus SO_RCVLOWAT, this does not require the
whole range to accumulate in the socket buffer before transfer begins, while
still keeping the wait-and-retry loop inside one syscall.

This appears implementable as a small conditional change in the TCP splice
read loop plus UAPI documentation and selftests.  I am sending this RFC first
to confirm whether a splice flag is the right interface and whether the
SO_RCVTIMEO-based partial-return semantics are acceptable before preparing a
patch.

The userspace implementation and CI profile that motivated this request are
available here:

  https://github.com/topling/ngs3fs/commit/4b8f5a69ceebd9211021a5d11496373d63f2e43d
  https://github.com/topling/ngs3fs/actions/runs/33586457571

Thanks,
leipeng