Re: TCP transmit performance regression

On Tue, Jul 10, 2012 at 12:39 PM, Eric Dumazet [off-list ref] wrote:

Please dont send private messages for discussing general linux stuff.

Next time I wont reply.

On Tue, 2012-07-10 at 12:00 +0800, Ming Lei wrote:

quoted

On Mon, Jul 9, 2012 at 9:54 PM, Eric Dumazet [off-list ref] wrote:

quoted

On Mon, 2012-07-09 at 21:23 +0800, Ming Lei wrote:

quoted

Looks the patch replaces skb_clone with netdev_alloc_skb_ip_align and
introduces extra copies on incoming data, so would you mind explaining
it in a bit detail? And why is skb_clone not OK for the purpose?

Problem with cloning is that some paths will have to make a private copy
of the skb.

Looks you convert some private copy into all copy in rx path, :-)

For small speed device, a copy is probably unnoticed.

The copy still has some effect on low speed device, for example, your recent
patch on asix driver can improve tx performance from ~75M to ~92M.

rtl8169 does that (copybreak) for security issues on Gbps link speed,
and I get Gbps link speed on an old AMD host with no problem.

As you discovered, the slowdown comes from SLAB debug on the 30K huge
skb. To recover from this we must patch usbnet to not constantly
allocate/free such big RX skb but recycle them. Once we do that, you'll
find out that copybreak improves general performance on low ram devices
by an order of magnitude.

Looks your copybreak patch doesn't improve tx performance on smsc95xx.

quoted

quoted

So you dont see the cost here in the driver, but later in upper stacks.

Since this driver defaults to a huge RX area of more than 16Kbytes,
a copy to a much smaller skb (we call this 'copybreak' in our jargon )
is more than welcome to avoid OOM problems anyway.

Looks 'memory compaction' has been implemented already to address
the big buffer allocation problem.

Usually its too late (not enough ram to perform the compaction), and
a collapse having to compact 3MB is very expensive and blows cpu caches.

I noticed that on machines with 1GB or 2GB ram. These machines are
called ChromeBooks and every lost network frame is analyzed in Google.
And we had problems because some wifi adapters use 8KB skbs for incoming
frames.

Kernel stack size is 8KB or more, so could you find process creation failure
in your ChromeBooks machine at the same time?

(Not even 32KB !!! This is just crazy !!)

Relying on TCP collapsing is just very lazy. What about other
protocols ?

I guess that on beagle this can happen very fast.

Previously I only found there was usbnet OOMs triggered by
kmalloc(GFP_ATOMIC), but kmalloc(GFP_KERNEL) can succeed.
Some times later, the problem disappeared.

quoted

Also the allocated huge RX SKB buffer will be freed after all cloned buffers
are consumed, so I still don't know what is the real problem with cloned buffer.

IF they are consumed.

But IF they arent because application is not fast enough to drain, you
end with sockets storing huge amount of data in their receive buffer.

So a single 100 bytes payload holds the 32KB block.

If you allowed your UDP socket to store 130.000 bytes of payload, you
can consume 13.000 * 32KB = ~40 MB

Looks it is one advantage of copybreak.

quoted

quoted

TCP coalescing (skb_try_coalesce) for example wont work for cloned skbs,
so TCP receive window will close pretty fast, and performance sucks in
lossy environments (like the Internet)

I didn't observe the above thing, so could you provide a way to reproduce it?

netstat -s can show you interesting TCP counters. But as driver lies on
skb->truesize, you can also have unexpected crashes with malicious
senders. With a 64 ratio, its easy to consume all ram.

TCP coalescing is great as soon as you have Out Of Order queueing
because of packet losses. You avoid expensive collapses and
dropping/purge of OFO queue. Sender has to resend previously sent data.

quoted

Suppose the above is true, looks skb_clone is useless, isn't it?

cloning has some uses, for example if you dont need to touch packet
content, only mess with skb->data, skb->len, skb->tail.

But if you need to change a single bit in the payload, or play with skb
fragments (struct skb_shared_info), you have to make a full copy of the
30KB buffer, even if the skb contained only 10 bytes of payload.

So the netdev_alloc_skb_ip_align() can be replaced with skb_clone()
in asix driver since not bits are touched in asix_rx_fixup? The default MTU is
1500 and rx_urb_size is 2048.

If so, could we use copybreak only for case of rx_urb_size > 4096?
And for ax88172, the dev->rx_urb_size is always 2048, looks the copy
is not needed at all.

I would just switch off turbo mode by default, I doubt it has any
advantage.

At least for smsc95xx, I think 32K buffer is not worthy of the feature.

Coalescing up to 16K of incoming frames adds latency for no performance
gain, once you do it the right way (that is without OOM risks).
Currently, skb->truesize lie is very bad.

Thanks,
-- 
Ming Lei

On Tue, 2012-07-10 at 15:22 +0800, Ming Lei wrote:

Kernel stack size is 8KB or more, so could you find process creation failure
in your ChromeBooks machine at the same time?

I believe you mix a lot of things.

Have you ever heard of sockets limits ?

All available ram on a machine is not for whoever wants it, thanks God.

No : TCP stack was dropping frames, because of socket limits.

Only because skbs were fat (8KB allocated/truesize, for a single 1500
bytes frame)

If application is fast and read skb as soon as the arrive, no problem is
detected.

But if  application is slow, or a TCP packet is lost on network,
man packets are queued into ofo queue. And eventually not enough room is
avalable -> we drop incoming frames, and sender has to restransmit them.

So instead of loading your web pages as fast as possible, you have to
wait for retransmits.

So you see nothing at all, no kernel logs, no failed memory attempts.

Only its slower than necessary

On Tue, Jul 10, 2012 at 4:28 PM, Eric Dumazet [off-list ref] wrote:

On Tue, 2012-07-10 at 15:22 +0800, Ming Lei wrote:

quoted

Kernel stack size is 8KB or more, so could you find process creation failure
in your ChromeBooks machine at the same time?

I believe you mix a lot of things.

Have you ever heard of sockets limits ?

All available ram on a machine is not for whoever wants it, thanks God.

No : TCP stack was dropping frames, because of socket limits.

Only because skbs were fat (8KB allocated/truesize, for a single 1500
bytes frame)

Could you explain why the truesize of SKB is 8KB for single
1500bytes frame?

I observed it is 2560bytes for RX SKBs inside asix_rx_fixup with
rx_urb_size of 2048 on beagle-xm.

If application is fast and read skb as soon as the arrive, no problem is
detected.

But if  application is slow, or a TCP packet is lost on network,
man packets are queued into ofo queue. And eventually not enough room is
avalable -> we drop incoming frames, and sender has to restransmit them.

So instead of loading your web pages as fast as possible, you have to
wait for retransmits.

So you see nothing at all, no kernel logs, no failed memory attempts.

Only its slower than necessary

Thanks,
-- 
Ming Lei

I am kind of annoyed you sent on netdev a copy of a _private_ mail.

Next time, make sure you dont do that without my consent.

On Tue, 2012-07-10 at 21:37 +0800, Ming Lei wrote:

Could you explain why the truesize of SKB is 8KB for single
1500bytes frame?

Because the driver uses skb_alloc(4096) for example ?

I don't know, you don't tell us the driver.


Goal is to have skb->head points to a 2048 bytes area, so truesize
should be 2048 + sizeof(sk_buff)  (including struct shared_info)

I observed it is 2560bytes for RX SKBs inside asix_rx_fixup with
rx_urb_size of 2048 on beagle-xm.

Thats because using 2048 bytes for the urb buffer (excluding
shared_info) means you need :

2048 + sizeof(struct shared_info) + sizeof(sk_buff) = 2560

In fact 2048 + sizeof(struct shared_info) means a full 4096 area is
used.

You have 2560 on recent kernels because the way netdev_alloc_frag()
works.

Thats why copybreak can actually saves ram. Since it is adding a copy,
we try to use it only on slow devices.

On Tue, Jul 10, 2012 at 10:02 PM, Eric Dumazet [off-list ref] wrote:

I am kind of annoyed you sent on netdev a copy of a _private_ mail.

I am sure that your reply which includes below is not from a private mail:

       Only because skbs were fat (8KB allocated/truesize, for a single
       1500 bytes frame)

Next time, make sure you dont do that without my consent.

OK

On Tue, 2012-07-10 at 21:37 +0800, Ming Lei wrote:

quoted

Could you explain why the truesize of SKB is 8KB for single
1500bytes frame?

Because the driver uses skb_alloc(4096) for example ?

I don't know, you don't tell us the driver.


Goal is to have skb->head points to a 2048 bytes area, so truesize
should be 2048 + sizeof(sk_buff)  (including struct shared_info)

quoted

I observed it is 2560bytes for RX SKBs inside asix_rx_fixup with
rx_urb_size of 2048 on beagle-xm.

Thats because using 2048 bytes for the urb buffer (excluding
shared_info) means you need :

2048 + sizeof(struct shared_info) + sizeof(sk_buff) = 2560

In fact 2048 + sizeof(struct shared_info) means a full 4096 area is
used.

You have 2560 on recent kernels because the way netdev_alloc_frag()
works.

Thats why copybreak can actually saves ram. Since it is adding a copy,
we try to use it only on slow devices.

Looks single page allocation won't put too much pressure on MM, that is
why I suggested to avoid copy if the skb buffer size is less or equal one
page. Anyway, unnecessary copy will increase computation and consume power.


Thanks,
-- 
Ming Lei

On Tue, 2012-07-10 at 22:22 +0800, Ming Lei wrote:

Looks single page allocation won't put too much pressure on MM, that is
why I suggested to avoid copy if the skb buffer size is less or equal one
page. Anyway, unnecessary copy will increase computation and consume power.

AFAIK this long thread started with drivers/net/usb/smsc95xx.c using
32KB buffers, thats order-3 pages, not 'single page'

Definitely very wrong. You can try to claim the contrary, it wont be
wise.