Re: Observed deadlock in ext4 under 3.2.23-rt37 & 3.2.33-rt50

On Wed, 2013-01-02 at 23:22 -0500, Theodore Ts'o wrote:

On Wed, Jan 02, 2013 at 10:09:43PM -0500, Steven Rostedt wrote:

quoted

-- Steve

quoted

Trace 1:
[<ffffffffa01a085d>] jbd2_log_wait_commit+0xcd/0x150
[<ffffffffa01b74a5>] ext4_sync_file+0x1e5/0x480
[<ffffffff8117a42b>] vfs_fsync_range+0x2b/0x30
[<ffffffff8117a44c>] vfs_fsync+0x1c/0x20
[<ffffffff8117a68a>] do_fsync+0x3a/0x60
[<ffffffff8117a6c3>] sys_fdatasync+0x13/0x20
[<ffffffff814e7feb>] system_call_fastpath+0x16/0x1b

Is this process running at a real-time priority?  If so, it looks like
a classic priority inversion problem.  fsync() triggers a journal
commit, and then waits for the jbd2 process to do the work.  If you
have real-time threads/processes which prevent the jbd2 process from
scheduling, that would explain what's going on.

In general, real-time processes/threads should *not* be doing file
system I/O, but if you must, you need to make sure that you've
adjusted the jbd2 kernel threads to run at the same or slightly higher
priority than the highest priority process which will be writing to
the file system.

With -rt things can get worse too. Even caused by kjournald being upped
in priority.

Anytime you have something that does the following in order to break
lock ordering:

repeat:
	lock(A);
	<do something>
	if (!trylock(B)) {
		unlock(A);
		cpu_relax();
		goto repeat;
	}


We can live lock, because spinlocks in -rt turn into a mutex. Thus, the
holder of lock B may not be on another CPU but actually on the current
CPU and is waiting for the process that is in this loop. If that process
happens to be an RT task, then the system stops.

-- Steve