On Thu, Dec 15, 2011 at 09:42:25AM +0800, Shaohua Li wrote:

On Thu, 2011-12-15 at 09:20 +0800, Darrick J. Wong wrote:

quoted

On Thu, Dec 15, 2011 at 09:02:57AM +0800, Shaohua Li wrote:

quoted

On Wed, 2011-12-14 at 22:30 +0800, Ted Ts'o wrote:

quoted

On Wed, Dec 14, 2011 at 09:34:00PM +0800, Wu Fengguang wrote:

quoted

Hi,

Shaohua recently found that ext4 writeback mode could perform worse
than ordered mode in some cases. It may not be a big problem, however
we'd like to share some information on our findings.

I tested both 3.2 and 3.1 kernels on normal SATA disks and USB key.
The interesting thing is, data=writeback used to run a bit faster
than data=ordered, however situation get inverted presumably by the
IO-less dirty throttling.

Interesting.  What sort of workloads are you using to do these
measurements?  How many writer threads; I assume you are doing
sequential writes which are extending one or more files, etc?

I suspect it's due to the throttling meaning that each thread is
getting to send less data to the disk, and so there is more seeking
going on with data=writeback, where as with data=ordered, at each
journal commit we are forcing all of the dirty pages out to disk, one
inode at a time, and this is resulting in a more efficient writeback
compared to when the writeback code is getting to make its own choices
about how much each inode gets to write out at at time.

It would be interesting to see what would happen if in
ext4_da_writepages(), we completely ignore how many pages are
requested to be written back by the writeback code, and just simply
write back all of the dirty pages, and see if that brings the
performance back.

I saw the issue in a machine with a LSI 1068e HBA card and 12 disks.
there is about 20% performance regression with data=writeback comparing
3.1 and 3.2-rc. with data=order, there is small regression too.
Reverting writeback changes recover the regression for both cases.

My investigation shows the block size writing to disk isn't changed with
data=writeback. The block size is still very big, 256k IIRC, which is
the max block size in the disks. And I just have one thread for each
disk, so seek definitely isn't a problem in my workload.

I found sometimes one disk hasn't any request inflight, but we can't
send request to the disk, because the scsi host's resource (the queue
depth) is used out, looks we send too many requests from other disks and
leave some disks starved. The resource imbalance in scsi isn't a new

I wonder, does the patch in:
http://lkml.indiana.edu/hypermail/linux/kernel/1105.3/02339.html
help with this starvation problem?  I noticed a similar problem and sent a
patch, but LSI folks never responded.  Maybe two complaining users can change
that.  The biggest MaxQ I've seen on LSI SAS is 511, and the driver clamps the
value it passes to the SCSI layer to whatever the controller reports as its
MaxQ (in /proc/mpt/summary).

this should recover the regression too. But I'm afraid it's just a
workaround and will hide some issues. what if I have 120 disks instead
of 12 disks? I observed one disk can burst 20 requests while the total
the scsi host queue depth is 127, leaving other disks starved. I'm
hoping to understand why there is such imbalance.

<shrug> I didn't say it would /fix/ the imbalanced-starvation problem, but we
might as well take full advantage of the hardware.  Even if all it does is
enable the user to plug in more disks before things get whacky, I was hoping
that someone else could at least give it a spin and say "Yes, this does what
it's alleged to do, and without breaking things". :)

afaict SCSI doesn't try to balance requests heading towards the HBA; it's all
FCFS.

--D

Thanks,
Shaohua