On Tue, 2005-03-29 at 17:20 +0200, Andi Kleen wrote:

quoted

Actually, not in 2.6 ... we had the same issue in SCSI using mempools
for sglist allocation.  All of the mempool alocation paths now take gfp_
flags, so you can specify GFP_ATOMIC for interrupt context.

Just does not work when you are actually short of memory.

Just think a second on how a mempool works: In the extreme
case when it cannot allocate system memory anymore it has
to wait for someone else to free a memory block into the mempool,
then pass it on to the next allocator etc. Basically 
it is a direct bypass pipeline for memory to pass memory
directly from one high priority user to another. This only
works with sleeping. Otherwise you could not handle an arbitary
number of users with a single mempool.

So to get a reliable mempool you have to sleep on allocation.

But that's not what we use them for.  You are confusing reliability with
forward progress.

In SCSI we use GFP_ATOMIC mempools in order to make forward progress.
All the paths are coded to expect a failure (in which case we requeue).
For forward progress what we need is the knowledge that there are n
resources out there dedicated to us.  When they return they get
reallocated straight to us and we can restart the queue processing
(there's actually a SCSI trigger that does this).

For receive mempools, the situation is much the same; if you have n
reserved buffers, then you have to drop the n+1 th packet.  However, the
resources will free up and go back to your mempool, and eventually you
accept the packet on retransmit.

The killer scenario (and why we require a mempool) is that someone else
gets the memory before you but then becomes blocked on another
allocation, so now you have no more allocations to allow forward
progress.

James

quoted

The object isn't to make the queues *reliable* it's to ensure the system
can make forward progress.  So all we're trying to ensure is that the
sockets used to service storage have some probability of being able to
send and receive packets during low memory.

For that it is enough to make the sender reliable. Retransmit
takes care of the rest.

No ... we cannot get down to the situation where GFP_ATOMIC always
fails.  Now we have no receive capacity at all and the system deadlocks.

quoted

In your scenario, if we're out of memory and the system needs several
ACK's to the swap device for pages to be released to the system, I don't
see how we make forward progress since without a reserved resource to
allocate from how does the ack make it up the stack to the storage
driver layer?

Typically because the RX ring of the driver has some packets left.

Also since TCP is very persistent and there is some memory
activity left you will have at least occasionally a time slot
where a GFP_ATOMIC allocation can succeed.

That's what I think a mempool is required to guarantee.  Without it,
there are scenarios where GFP_ATOMIC always fails.

James