Re: Latency writing to an mlocked ext4 mapping

On Wed 19-10-11 22:59:55, Andy Lutomirski wrote:

On Wed, Oct 19, 2011 at 7:17 PM, Andy Lutomirski [off-list ref] wrote:

quoted

On Wed, Oct 19, 2011 at 6:15 PM, Andy Lutomirski [off-list ref] wrote:

quoted

On Wed, Oct 19, 2011 at 6:02 PM, Andreas Dilger [off-list ref] wrote:

quoted

What kernel are you using?  A change to keep pages consistent during writeout was landed not too long ago (maybe Linux 3.0) in order to allow checksumming of the data.

3.0.6, with no relevant patches.  (I have a one-liner added to the tcp
code that I'll submit sometime soon.)  Would this explain the latency
in file_update_time or is that a separate issue?  file_update_time
seems like a good thing to make fully asynchronous (especially if the
file in question is a fifo, but I've already moved my fifos to tmpfs).

On 2.6.39.4, I got one instance of:

call_rwsem_down_read_failed ext4_map_blocks ext4_da_get_block_prep
__block_write_begin ext4_da_write_begin ext4_page_mkwrite do_wp_page
handle_pte_fault handle_mm_fault do_page_fault page_fault

but I'm not seeing the large numbers of the ext4_page_mkwrite trace
that I get on 3.0.6.  file_update_time is now by far the dominant
cause of latency.

The culprit seems to be do_wp_page -> file_update_time ->
mark_inode_dirty_sync.  This surprises me for two reasons:

 - Why the _sync?  Are we worried that data will be written out before
the metadata?  If so, surely there's a better way than adding latency
here.

  _sync just means that inode will become dirty for fsync(2) purposes but
not for fdatasync(2) purposes - i.e. it's just a timestamp update (or
it could be something similar).

 - Why are we calling file_update_time at all?  Presumably we also
update the time when the page is written back (if not, that sounds
like a bug, since the contents may be changed after something saw the
mtime update), and, if so, why bother updating it on the first write?
Anything that relies on this behavior is, I think, unreliable, because
the page could be made writable arbitrarily early by another program
that changes nothing.

  We don't update timestamp when the page is written back. I believe this
is mostly because we don't know whether the data has been changed by a
write syscall, which already updated the timestamp, or by mmap. That is
also the reason why we update the timestamp at page fault time.

  The reason why file_update_time() blocks for you is probably that it
needs to get access to buffer where inode is stored on disk and because a
transaction including this buffer is committing at the moment, your thread
has to wait until the transaction commit finishes. This is mostly a problem
specific to how ext4 works so e.g. xfs shouldn't have it.

  Generally I believe the attempts to achieve any RT-like latencies when
writing to a filesystem are rather hopeless. How much hopeless depends on
the load of the filesystem (e.g., in your case of mostly idle filesystem I
can imagine some tweaks could reduce your latencies to an acceptable level
but once the disk gets loaded you'll be screwed). So I'd suggest that
having RT thread just store log in memory (or write to a pipe) and have
another non-RT thread write the data to disk would be a much more robust
design.

								Honza
-- 
Jan Kara [off-list ref]
SUSE Labs, CR

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>