Re: working on extent locks for i_mutex

On Mon, Jan 16, 2012 at 10:46:29AM -0700, Allison Henderson wrote:

On 01/15/2012 04:57 PM, Dave Chinner wrote:

quoted

On Fri, Jan 13, 2012 at 01:50:52PM -0700, Allison Henderson wrote:

quoted

On 01/12/2012 09:34 PM, Dave Chinner wrote:

quoted

On Thu, Jan 12, 2012 at 08:01:43PM -0700, Allison Henderson wrote:

quoted

Hi All,

I know this is an old topic, but I am poking it again because I've
had some work items wrap up, and Im planning on picking up on this
one again.  I am thinking about implementing extent locks to replace
i_mutex.  So I just wanted to touch base with folks and see what
people are working on because I know there were some folks out there
that were thing about doing similar solutions.

What locking API are you looking at? If you are looking at an
something like:

read_range_{try}lock(lock, off, len)
read_range_unlock(lock, off, len)
write_range_{try}lock(lock, off, len)
write_range_unlock(lock, off, len)

and implementing with an rbtree or a btree for tracking, then I
definitely have a use for it in XFS - replacing the current rwsem
that is used for the iolock. Range locks like this are the only
thing we need to allow concurrent buffered writes to the same file
to maintain the per-write exclusion that posix requires.

Yes that is generally the idea I was thinking about doing, but at
the time, I was not thinking outside the scope of ext4.  You are
thinking maybe it should be in vfs layer so that it's something that
all the filesystems will use?  That seems to be the impression I'm
getting from folks.  Thx!

Yes, that's what I'm suggesting. Not so much a vfs layer function,
but a library (range locks could be useful outside filesystems) so
locating it in lib/ was what I was thinking....

Cheers,

Dave.

Alrighty, that sounds good to me.  I will aim to keep it as general
purpose as I can.  I am going to start some proto typing and will
post back when I get something working.  Thx for the feedback all!
:)

Hi Allison,

For this project, do you have a schedule? Would you like to share to me? This
lock contention heavily impacts the performance of direct IO in our production
environment. So we hope to improve it ASAP.

I have done some direct IO benchmarks to compare ext4 with xfs using fio
in Intel SSD. The result shows that, in direct IO, xfs outperforms ext4 and
ext4 with dioread_nolock.

To understand the effect of lock contention, I define a new function called 
ext4_file_aio_write() that calls __generic_file_aio_write() without acquiring 
i_mutex lock. Meanwhile, I remove DIO_LOCKING flag when __blockdev_direct_IO() 
is called and do the similar benchmarks. The result shows that the performance 
in ext4 is almost the same to the xfs. Thus, it proves that the i_mutex heavily
impacts the performance. Hopefully the result is useful for you. :-)

I post the result in here.

config file:
[global]
filesize=64G
size=64G
bs=16k
ioengine=psync
direct=1
filename=/mnt/ext4/benchmark
runtime=600
group_reporting
thread

[randrw]
numjobs=32
rw=randrw
rwmixread=90

result:

iops			1 (r/w)		2		3
ext4			5584/622	5726/636	5719/636
ext4+dioread_nolock	7105/789	7117/793	7129/795
ext4+dio_nolock		8920/992	8956/995	8976/997
xfs			8726/971	8962/994	8975/998

bandwidth		1 (r/w)		2		3		KB/s
ext4			89359/9955.3	91621/10186	91519/10185
ext4+dioread_nolock	113691/12635	113882/12692	114066/12728
ext4+dio_nolock		142731/15888	143301/15930	143617/15959
xfs			139627/15537	143400/15914	143603/15980

latency			1 (r/w)		2		3		usec
ext4			5163.28/5048.31	5037.81/4914.82	5041.49/4932.81
ext4+dioread_nolock	1220.04/29510.5 1213.67/29418.9 1208.77/29361.49
ext4+dio_nolock		3226.61/3194.35	3214.59/3178.09	3207.34/3173.78
xfs			3299.87/3266.32	3213.73/3182.20	3208.16/3178.10

Regards,
Zheng

--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html