On Wed 11-04-12 11:40:05, Vivek Goyal wrote:

On Wed, Apr 11, 2012 at 12:24:25AM +0200, Jan Kara wrote:

[..]

quoted

quoted

I have implemented and posted patches for per bdi per cgroup congestion
flag. The only problem I see with that is that a group might be congested
for a long time because of lots of other IO happening (say direct IO) and
if you keep on backing off and never submit the metadata IO (transaction),
you get starved. And if you go ahead and submit IO in a congested group,
we are back to serialization issue.

  Clearly, we mustn't throttle metadata IO once it gets to the block layer.
That's why we discuss throttling of processes at transaction start after
all. But I agree starvation is an issue - I originally thought blk-throttle
throttles synchronously which wouldn't have starvation issues. But when
that's not the case things are a bit more tricky. We could treat
transaction start as an IO of some size (since we already have some
estimation how large a transaction will be when we are starting it) and let
the transaction start only when our "virtual" IO would be submitted but
I feel that gets maybe too complicated... Maybe we could just delay the
transaction start by the amount reported from blk-throttle layer? Something
along your callback for throttling you implemented?

I think now I have lost you. It probably stems from the fact that I don't
know much about transactions and filesystem.
 
So all the metadata IO will happen thorough journaling thread and that
will be in root group which should remain unthrottled. So any journal
IO going to disk should remain unthrottled.

  Yes, that is true at least for ext3/ext4 or btrfs. In principle we don't
have to have the journal thread (as is the case of reiserfs where random
writer may end up doing commit) but let's not complicate things
unnecessarily.

Now, IIRC, fsync problem with throttling was that we had opened a
transaction but could not write it back to disk because we had to
wait for all the cached data to go to disk (which is throttled). So
my question is, can't we first wait for all the data to be flushed
to disk and then open a transaction for metadata. metadata will be
unthrottled so filesystem will not have to do any tricks like bdi is
congested or not.

  Actually that's what's happening. We first do filemap_write_and_wait()
which syncs all the data and then we go and force transaction commit to
make sure all metadata got to stable storage. The problem is that writeout
of data may need to allocate new blocks and that starts a transaction and
while the transaction is started we may need to do some reads (e.g. of
bitmaps etc.) which may be throttled and at that moment the whole
filesystem is blocked. I don't remember the stack traces you showed me so
I'm not sure it this is what your observed but it's certainly one possible
scenario. The reason why fsync triggers problems is simply that it's the
only place where process normally does significant amount of writing. In
most cases flusher thread / journal thread do it so this effect is not
visible. And to precede your question, it would be rather hard to avoid IO
while the transaction is started due to locking.

[..]

quoted

quoted

I guess throttling at bdi layer will take care of network filesystem
case too?

  Yes. At least for client side. On sever side Steve wants server to have
insight into how much IO we could push in future so that it can limit
number of outstanding requests if I understand him right. I'm not sure we
really want / are able to provide this amount of knowledge to filesystems
even less userspace...

I am not sure what does it mean but server could simply query the bdi
and read configured rate and then it knows at what rate IO will go to
disk and make predictions about future?

  Yeah, that would work if we had the current bandwidth for current cgroup
exposed in bdi.
 

quoted

quoted

Also per bdi limit mechanism will not solve the issue of global throttling
where in case of btrfs an IO might go to multiple bdi's. So throttling limits
are not total but per bdi.

  Well, btrfs plays tricks with bdi's but there is a special bdi called
"btrfs" which backs the whole filesystem and that is what's put in
sb->s_bdi or in each inode's i_mapping->backing_dev_info. So we have a
global bdi to work with.

Ok, that's good to know. How would we configure this special bdi? I am
assuming there is no backing device visible in /sys/block/<device>/queue/?
Same is true for network file systems.

  Where should be the backing device visible? Now it's me who is lost :)

								Honza
-- 
Jan Kara [off-list ref]
SUSE Labs, CR

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>