Re: [PATCH 6/6] btrfs: do not block inode logging for so long during transaction commit
From: Filipe Manana <fdmanana@kernel.org>
Date: 2021-02-03 10:52:15
On Tue, Feb 2, 2021 at 2:15 PM Wang Yugui [off-list ref] wrote:
Hi, Filipe Manana There are some dbench(sync mode) result on the same hardware, but with different linux kernel 4.14.200 Operation Count AvgLat MaxLat ---------------------------------------- WriteX 225281 5.163 82.143 Flush 32161 2.250 62.669 Throughput 236.719 MB/sec (sync open) 32 clients 32 procs max_latency=82.149 ms 4.19.21 Operation Count AvgLat MaxLat ---------------------------------------- WriteX 118842 10.946 116.345 Flush 16506 0.115 44.575 Throughput 125.973 MB/sec (sync open) 32 clients 32 procs max_latency=116.390 ms 4.19.150 Operation Count AvgLat MaxLat ---------------------------------------- WriteX 144509 9.151 117.353 lush 20563 0.128 52.014 Throughput 153.707 MB/sec (sync open) 32 clients 32 procs max_latency=117.379 ms 5.4.91 Operation Count AvgLat MaxLat ---------------------------------------- WriteX 367033 4.377 1908.724 Flush 52037 0.159 39.871 Throughput 384.554 MB/sec (sync open) 32 clients 32 procs max_latency=1908.968 ms
Ok, it seems somewhere between 4.19 and 5.4, something made the latency much worse for you at least. Is it only when using sync open (O_SYNC, dbench's -s flag), what about when not using it? I'll have to look at it, but it will likely take some time. Thanks.
5.10.12+patches Operation Count AvgLat MaxLat ---------------------------------------- WriteX 429696 3.960 2239.973 Flush 60771 0.621 6.794 Throughput 452.385 MB/sec (sync open) 32 clients 32 procs max_latency=1963.312 ms MaxLat / AvgLat of WriteX is increased from 82.143/5.163=15.9 to 2239.973/3.960=565.6. For QoS, can we have an option to tune the value of MaxLat / AvgLat of WriteX to less than 100? Best Regards Wang Yugui (wangyugui@e16-tech.com) 2021/02/02quoted
Hi, Filipe Mananaquoted
On Tue, Feb 2, 2021 at 5:42 AM Wang Yugui [off-list ref] wrote:quoted
Hi, Filipe Manana The dbench result with these patches is very good. thanks a lot. This is the dbench(synchronous mode) result , and then a question. command: dbench -s -t 60 -D /btrfs/ 32 mount option:ssd,space_cache=v2 kernel:5.10.12 + patchset 1 + this patchsetpatchset 1 and "this patchset" are the same, did you mean two different patchsets or just a single patchset?patchset1: btrfs: some performance improvements for dbench alike workloads patchset2: btrfs: more performance improvements for dbench workloads https://patchwork.kernel.org/project/linux-btrfs/list/?series=422801 I'm sorroy that I have replayed to the wrong patchset.quoted
quoted
Question: for synchronous mode, the result type 1 is perfect?What do you mean by perfect? You mean if result 1 is better than result 2?In result 1, the MaxLat of Flush of dbench synchronous mode is fast as expected, the same level as kernel 5.4.91. But in result 2, the MaxLat of Flush of dbench synchronous mode is big as write level, but this is synchronous mode, most job should be done already before flush.quoted
quoted
and there is still some minor place about the flush to do for the result type2?By "minor place" you mean the huge difference I suppose.quoted
result type 1: Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 868942 0.028 3.017 Close 638536 0.003 0.061 Rename 36851 0.663 4.000 Unlink 175182 0.399 5.358 Qpathinfo 789014 0.014 1.846 Qfileinfo 137684 0.002 0.047 Qfsinfo 144241 0.004 0.059 Sfileinfo 70913 0.008 0.046 Find 304554 0.057 1.889 ** WriteX 429696 3.960 2239.973 ReadX 1363356 0.005 0.358 LockX 2836 0.004 0.038 UnlockX 2836 0.002 0.018 ** Flush 60771 0.621 6.794 Throughput 452.385 MB/sec (sync open) 32 clients 32 procs max_latency=1963.312 ms + stat -f -c %T /btrfs/ btrfs + uname -r 5.10.12-4.el7.x86_64 result type 2: Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 888943 0.028 2.679 Close 652765 0.002 0.058 Rename 37705 0.572 3.962 Unlink 179713 0.383 3.983 Qpathinfo 806705 0.014 2.294 Qfileinfo 140752 0.002 0.125 Qfsinfo 147909 0.004 0.049 Sfileinfo 72374 0.008 0.104 Find 311839 0.058 2.305 ** WriteX 439656 3.854 1872.109 ReadX 1396868 0.005 0.324 LockX 2910 0.004 0.026 UnlockX 2910 0.002 0.025 ** Flush 62260 0.750 1659.364 Throughput 461.856 MB/sec (sync open) 32 clients 32 procs max_latency=1872.118 ms + stat -f -c %T /btrfs/ btrfs + uname -r 5.10.12-4.el7.x86_64I'm not sure what your question is exactly. Are both results after applying the same patchset, or are they before and after applying the patchset, respectively?Both result after applying the same patchset. and both on the same server, same SAS SSD disk. but the result is not stable, and the major diff is MaxLat of Flush. Server:Dell T7610 CPU: E5-2660 v2(10core 20threads) x2 SSD:TOSHIBA PX05SMQ040 Memory:192G (with ECC)quoted
If they are both with the patchset applied, and you wonder about the big variation in the "Flush" operations, I am not sure about why it is so. Both throughput and max latency are better in result 2. It's normal to have variations across dbench runs, I get them too, and I do several runs (5 or 6) to check things out. I don't use virtualization (testing on bare metal), I set the cpu governor mode to performance (instead of the "powersave" default) and use a non-debug kernel configuration, because otherwise I get significant variations in latencies and throughput too (though I never got a huge difference such as from 6.794 to 1659.364).This is a bare metal(dell T7610). CPU mode is set to performance by BIOS. and I checked it by 'cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor' Maybe I used a SAS ssd, and the queue depth of SAS SSD is 254. smaller than 1023 of a NVMe SSD,but it is still enough for dbench 32 threads? The huge difference of MaxLat of Flush such as from 6.794 to 1659.364 is a problem. It is not easy to re-product both, mabye easy to reproduce the small one, maybe easy to reproduce the big one. Best Regards Wang Yugui (wangyugui@e16-tech.com) 2021/02/02