On Mon, 2005-03-28 at 18:12 +0200, Andi Kleen wrote:

Rik van Riel [off-list ref] writes:

quoted

   one for sending and one for receiving
2) have a global emergency mempool available to receive network
   packets when GFP_ATOMIC fails - this is useful since we don't
   know who a packet is for when we get the NIC interrupt, and
   it's easy to have just one pool to check

This does not work because mempools assume you can sleep,
and in most NIC drivers you cant while doing RX refill.
The NIC drivers can be rewritten to do this refilling in
a workqueue. But it is not clear it is useful anyways because
Linux failing to allocate a buffer is no different from
the network overflowing the hardware queue of the network
device, which Linux cannot do anything about. 

Basically a network consists of lots of interconnected
queues, and even if you try to make the Linux specific
side of the queue reliable there are lots of other queues
that can still lose packets.

With TCP that is no problem of course because in case of 
a packet loss the packet is just retransmitted. 

So in short using mempools on receiving is not needed.

Now memory allocation for writing is a different chapter.
You cannot recover from a lost writing.
The kernel currently goes even in endless loops in this
case (e.g. on TCP FIN allocation)

With the exception of routing the allocation fortunately 
usually happens there in:
- socket context: great you can use a per socket mempool
- thread context: you can sleep

With routing that is not the case, but it does not matter
because it typically does not allocate new packets, but
just sends out existing ones.

In short you need mempool only for TX. With some luck you even only
need them for the skbuff head and a small header buffer, since I would
expect iscsi TX to be typically zero copy for data and just passing
struct page *s around.

that's always true for Open-iSCSI and for sfnet in FFP(rcf3720) but not
for other implementations. moreover, if NIC does not support
checksumming on Tx than sendpage() interface will fail to sendmsg()
which will allocate new skb each time.