Re: Block device throttling [Re: Distributed storage.]

On Monday 13 August 2007 05:04, Evgeniy Polyakov wrote:

On Mon, Aug 13, 2007 at 04:04:26AM -0700, Daniel Phillips 

(phillips@phunq.net) wrote:

quoted

On Monday 13 August 2007 01:14, Evgeniy Polyakov wrote:

quoted

quoted

Oops, and there is also:

3) The bio throttle, which is supposed to prevent deadlock, can
itself deadlock.  Let me see if I can remember how it goes.

  * generic_make_request puts a bio in flight
  * the bio gets past the throttle and initiates network IO
  * net calls sk_alloc->alloc_pages->shrink_caches
  * shrink_caches submits a bio recursively to our block device
  * this bio blocks on the throttle
  * net may never get the memory it needs, and we are wedged

If system is in such condition, it is already broken - throttle
limit must be lowered (next time) not to allow such situation.

Agreed that the system is broken, however lowering the throttle
limit gives no improvement in this case.

How is it ever possible? The whole idea of throttling is to remove
such situation, and now you say it can not be solved.

It was solved, by not throttling writeout that comes from shrink_caches.
Ugly.

If limit is for 
1gb of pending block io, and system has for example 2gbs of ram (or
any other resonable parameters), then there is no way we can deadlock
in allocation, since it will not force page reclaim mechanism.

The problem is that sk_alloc (called from our block driver via 
socket->write) would recurse into shrink_pages, which recursively 
submits IO to our block driver and blocks on the throttle.  Subtle 
indeed, and yet another demonstration of why vm recursion is a Bad 
Thing.

I will find a traceback for you tomorrow, which makes this deadlock much 
clearer.

Regards