Eric Dumazet wrote:
When a socket is over its receive buffer limit and its sk_rcvbuf can
not grow, every incoming segment that needs memory goes through
tcp_prune_queue(), which runs tcp_collapse_ofo_queue() and tcp_collapse()
on the receive queue. Both walk their whole queue, even when there is
nothing left to collapse.
A peer can trigger this walk per packet, without refilling memory:
1) The segment lands last in the out-of-order queue. Since commit
b0e01253a764 ("tcp: refine tcp_prune_ofo_queue() logic"),
tcp_prune_ofo_queue() drops nothing, the segment is dropped and
the queues are unchanged. The peer can send it again forever.
2) The segment lands second to last: one skb is pruned, and one small
segment fills that space again.
3) The out-of-order queue is empty and the receive queue alone is over
the limit (application not reading). Collapsing frees nothing,
the out-of-order segment is dropped, and the window is not closed.
4) Collapsing frees just one skb, enough to queue the segment. The next
segment is over the limit again.
Only allow a collapse if at least 1/8 of sk_rcvbuf of new memory was
queued in the receive queues since the previous collapse. This restores
the "12.5 % of the queue capacity" amortization of the 2018 SegmentSmack
fixes (CVE-2018-5390), using the same unit as the sk_rcvbuf limit, so
that peers sending small (bloated) segments are not penalized.
The credit is the net growth of sk_rmem_alloc in tcp_queue_rcv() and
tcp_data_queue_ofo(), so that:
- Coalescing a segment into the tailroom of the previous skb charges no
memory, and earns no credit.
- An out-of-order segment replacing or covering queued skbs only earns
the memory it adds.
- Duplicates earn nothing.
- Data moved from the out-of-order queue to the receive queue
(tcp_ofo_queue()) is not credited twice.
In-order data is credited too, so that a receive queue filled by
bloated in-order segments can still be collapsed when a hole appears.
In-order segments are rate limited as well: a peer could otherwise send
small in-order segments that each trigger a collapse freeing a single
skb, enough to queue the segment, while the out-of-order queue holds
many skbs that can not be collapsed.
The credit is tracked in a new u32 tp->rcv_collapse_credit, placed in a
hole of the tcp_sock_write_rx group: no other field moves, and tcp_sock
size is unchanged. tcp_queue_rcv() now samples sk_rmem_alloc, and
writes the credit until it saturates at sk_rcvbuf, so that the fast path
stops writing it after a full buffer of new memory. Saturating at the
goal would not work: sk_rcvbuf, and the goal with it, can grow later
(for instance in tcp_clamp_window(), just before the goal is checked),
and a socket over its limit can not earn more credit until the
application reads.
When collapsing is not allowed, tcp_prune_queue() goes directly to
tcp_prune_ofo_queue(), which is cheap in the cases above.
A regular flow can see a difference when it goes over its limit before
1/8 of sk_rcvbuf of new memory was queued since the previous collapse:
out-of-order skbs can be pruned (even for an in-order segment filling a
hole), and an in-order segment that still does not fit is dropped,
with a zero window advertised, where a collapse might have made room.
Fixes: b0e01253a764 ("tcp: refine tcp_prune_ofo_queue() logic")
Reported-by: Chris Mason <mason@kernel.org>
Signed-off-by: Eric Dumazet <edumazet@kernel.org>
Reviewed-by: Willem de Bruijn <willemb@google.com>