On Tuesday 11 Jun 2013, OS Engineering wrote:

Hi Jens,

In continuation with our previous communication, we have carried out
performance comparison among EnhanceIO, bcache and dm-cache.

We found that EnhanceIO provides better throughput on zipf workload (with
theta=1.2) in comparison to bcache and dm-cache for write through caches.
However, for write back caches, we found that dm-cache had best throughput
followed by EnhanceIO and then bcache. Dm-cache commits on-disk metadata
every time a REQ_SYNC or REQ_FUA bio is written. If no such requests are
made then it commits metadata once every second. If power is lost, it may
lose some recent writes. However, EnhanceIO and bcache do not acknowledge
IO completion until both IO and metadata hits the SSD. Hence, EnhanceIO
and bcache provide higher data integrity at a cost of performance.

So it won't be fair to compare them with dm-cache in write-back mode since 
guarantees are different. I am sure if similar (SYNC/FUA enforcement for 
persistence) guarantees are implemented in bcache/EnhanceIO, they'll offer a 
much better performance. 

The fio config and setup information follows:
HDD              : 100GB
SSD              :  20GB
Mode             : write through / write back
Cache block_size : 4KB for bcache, EnhanceIO and 256KB for dm-cache

The other options have been left to their default values.

Note:
1) In case of dm-cache, we used 2 partitions of same SSD with 1GB partition
as metadata and 20GB partition as caching device. This has been done so as
to ensure a fair comparison as EnhanceIO and bcache do not have a separate
metadata device.

2) In order to ensure proper cache warm up, We have turned off sequential
bypass in bcache. This does not impact our performance results as they are
taken for random workload.

For each test, we first performed a warm up run using the following fio
options: fio --direct=1 --size=90% --filesize=20G --blocksize=4k
--ioengine=libaio --rw=rw --rwmixread=100 --rwmixwrite=0 --iodepth=8 ...

Then, we performed our actual run with the following fio options:
fio --direct=1 --size=100% --filesize=20G --blocksize=4k --ioengine=libaio
--rw=randrw --rwmixread=90 --rwmixwrite=10 --iodepth=8 --numjobs=4
--random_distribution=zipf:1.2 ...

Did you experiment a little with varying iodepth and numjobs? Is this the best 
combination you found out? The numbers below appear low considering a 90% hit 
ratio for EnhanceIO. SSD baseline performance shown below appears low. However 
it should not affect this comparison. All caching solutions are being subjected 
to the same ratio of HDD/SSD performance.

============================= Write Through ===============================
Type      Read Latency(ms)   Write Latency(ms)    Read(MB/s)    Write(MB/s)
===========================================================================
EnhanceIO      1.58              16.53               32.91       3.65
bcache         0.58              31.05               27.17       3.02
dm-cache       0.24              27.45               31.05       3.44

EnhanceIO shows much higher read latency here. Is that because of READFILLs ? 
Write latency, read-write throughputs are good.

============================= Write Back ==================================
Type      Read Latency(ms)    Write Latency(ms)    Read(MB/s)   Write(MB/s)
===========================================================================
EnhanceIO      0.34               4.98               138.72      15.40
bcache         0.95               1.76               106.82      11.85
dm-cache       0.58               0.55               193.76      21.52

Here EnhanceIO's read latency is better compared the other two. Write latency 
is larger than the other two. 

-Amit

============================ Base Line ====================================
Type      Read Latency(ms)    Write Latency(ms)    Read(MB/s)   Write(MB/s)
===========================================================================
HDD            6.22              27.23                13.51       1.49
SSD            0.47               0.42               235.87      26.21

We have written scripts that aid in cache creation, deletion and
performance run for all these caching solutions. These scripts can be
found at:
https://github.com/stec-inc/EnhanceIO/tree/master/performance_test

Thanks and Regards,
sTec Team

PROPRIETARY-CONFIDENTIAL INFORMATION INCLUDED

This electronic transmission, and any documents attached hereto, may
contain confidential, proprietary and/or legally privileged information.
The information is intended only for use by the recipient named above. If
you received this electronic message in error, please notify the sender
and delete the electronic message. Any disclosure, copying, distribution,
or use of the contents of information received in error is strictly
prohibited, and violators will be pursued legally. --
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/