On 1/14/21 12:45 PM, Dongdong Tao wrote:

Hi Coly,

I've got the testing data for multiple threads with larger IO depth.

Hi Dongdong,

Thanks for the testing number.

*Here is the testing steps:
*1. make-bcache -B <> -C <> --writeback

2. Open two tabs, start different fio task in them at the same time.
Tab1 run below fio command:
sudo fio --name=random-writers --filename=/dev/bcache0 --ioengine=libaio
--iodepth=32 --rw=randrw --blocksize=64k,8k  --direct=1 --runtime=24000

Tab2 run below fio command:
sudo fio --name=random-writers2 --filename=/dev/bcache0
--ioengine=libaio --iodepth=8 --rw=randwrite --bs=4k --rate_iops=150
--direct=1 --write_lat_log=rw --log_avg_msec=20

Why you limit the iodep to 8 and iops to 150 on cache device?
For cache device the limitation is small. Iosp 150 with 4KB block size,
it means every hour writing (150*4*60*60=2160000KB=) 2GB data. For 35
hours it is only 70GB.


What if the iodeps is 128 or 64, and no iops rate limitation ?

Note
- Tab1 fio will run for 24000 seconds, which is the one to cause the
fragmentation and made the cache_available_percent drops to under 40.
- Tab2 fio is the one that I'm capturing the latency and I have let it
run for about 35 hours, which is long enough to allow the
cache_available_percent drops under 30.
- This testing method utilized fio benchmark with larger read block
size/small write block size to cause the high fragmentation, However in
a real production env, there could be
   various reasons or a combination of various reasons to cause the high
fragmentation,  but I believe it should be ok to use any method to cause
the fragmentation to verify if
   bcache with this patch is responding better than the master in this
situation. 

*Below is the testing result:*

The total run time is about 35 hours, the latency points in the charts
for each run are 1.5 million

Master:
fio-lat-mater.png

Master + patch:
fio-lat-patch.png
Combine them together:
fio-lat-mix.png

Now we can see the master is even worse when we increase the iodepth,
which makes sense since the backing HDD is being stressed more hardly.

*Below are the cache stats changing during the run:*
Master:
bcache-stats-master.png

Master + the patch:
bcache-stats-patch.png

That's all the testing done with 400GB NVME with 512B block size.

Coly, do you want me to continue the same testing on 1TB nvme with
different block size ?
or is it ok to skip the 1TB testing and continue the test with 400GB
NVME but with different block size? 
feel free to let me know any other test scenarios that we should cover
here. 

Yes please, more testing is desired for performance improvement. So far
I don't see performance number for real high work load yet.

Thanks.

Coly Li