Re: [RFC] performance regression with "ext4: Allow parallel DIO reads"
From: Joseph Qi <joseph.qi@linux.alibaba.com>
Date: 2019-08-23 08:07:36
On 19/8/23 15:57, Joseph Qi wrote:
Hi Dave, On 19/8/22 13:40, Dave Chinner wrote:quoted
On Wed, Aug 21, 2019 at 09:04:57AM +0800, Joseph Qi wrote:quoted
Hi Ted， On 19/8/21 00:08, Theodore Y. Ts'o wrote:quoted
On Tue, Aug 20, 2019 at 11:00:39AM +0800, Joseph Qi wrote:quoted
I've tested parallel dio reads with dioread_nolock, it doesn't have significant performance improvement and still poor compared with reverting parallel dio reads. IMO, this is because with parallel dio reads, it take inode shared lock at the very beginning in ext4_direct_IO_read().Why is that a problem? It's a shared lock, so parallel threads should be able to issue reads without getting serialized?The above just tells the result that even mounting with dioread_nolock, parallel dio reads still has poor performance than before (w/o parallel dio reads).quoted
Are you using sufficiently fast storage devices that you're worried about cache line bouncing of the shared lock? Or do you have some other concern, such as some other thread taking an exclusive lock?The test case is random read/write described in my first mail. AndRegardless of dioread_nolock, ext4_direct_IO_read() is taking inode_lock_shared() across the direct IO call. And writes in ext4 _always_ take the inode_lock() in ext4_file_write_iter(), even though it gets dropped quite early when overwrite && dioread_nolock is set. But just taking the lock exclusively in write fro a short while is enough to kill all shared locking concurrency...quoted
from my preliminary investigation, shared lock consumes more in such scenario.If the write lock is also shared, then there should not be a scalability issue. The shared dio locking is only half-done in ext4, so perhaps comparing your workload against XFS would be an informative exercise...I've done the same test workload on xfs, it behaves the same as ext4 after reverting parallel dio reads and mounting with dioread_lock.
A typo here, s/dioread_lock/dioread_nolock/
Here is the test result: psync, randrw, direct=1, numofjobs=8 4k: ----------------------------------------- ext4 | READ 123450KB/s | WRITE 123368KB/s ----------------------------------------- xfs | READ 123848KB/s | WRITE 123761KB/s ----------------------------------------- 16k: ----------------------------------------- ext4 | READ 222477KB/s | WRITE 222322KB/s ----------------------------------------- xfs | READ 223261KB/s | WRITE 223106KB/s ----------------------------------------- 64k: ----------------------------------------- ext4 | READ 427406KB/s | WRITE 426197KB/s ----------------------------------------- xfs | READ 403697KB/s | WRITE 402555KB/s ----------------------------------------- 512k: ----------------------------------------- ext4 | READ 618752KB/s | WRITE 619054KB/s ----------------------------------------- xfs | READ 614954KB/s | WRITE 615254KB/s ----------------------------------------- 1M: ----------------------------------------- ext4 | READ 615011KB/s | WRITE 612255KB/s ----------------------------------------- xfs | READ 624087KB/s | WRITE 621290KB/s -----------------------------------------