I've been running bcache on my devel machine, and inspired by the new
write-up on the bcache wiki, decided to do some simple benchmarking to
see if I could put some numbers to the performance benefit I see on my
PC.

I decided to test the various cache modes using fio, and since I use
postgresql a bit, pgbench (using both postgresql 9.1 & 9.2). I have no
particular skill with regard to benchmarking, so perhaps I've screwed
something up, but I was a bit surprised by the numbers.

With pgbench (which is primarily writes) writeback mode was the
decisive winner, as expected. I saw little difference between the
other modes, including cache mode "none".

With fio (using the parameters specified on the wiki), cache mode None
produced the highest IOps & throughput.

Does that make sense? Why would cache mode none be comparable to any
of the other modes?

## Environment ##

Linux zapazoid 3.5.0+ #1 SMP Sat Aug 25 12:42:26 EDT 2012 x86_64 GNU/Linux

/dev/mapper/vg0-pg_base on /var/lib/postgresql/9.1/main/base type ext4
(rw,relatime)
/dev/mapper/vg0-pg_xlog on /var/lib/postgresql/9.1/main/pg_xlog type
ext4 (rw,relatime)
/dev/mapper/vg0-home on /home type ext4 (rw,relatime)

Hardware:

Phenom II X6
8GB RAM
7200 RPM 320GB HDD
OCZ Vertex 3 64GB SSD


Benchmark Script:

#!/bin/sh

outfile=bcache_bench-results.txt
pg_bin=/usr/lib/postgresql/9.2/bin/

for cm in writethrough writeback writearound none; do
  echo $cm > /sys/block/sda/sda2/bcache/cache_mode
  cat /sys/block/sda/sda2/bcache/cache_mode
  echo
  # fio benchmark
  echo "Running fio with bcache in mode: $cm"
  cat ./bcache.fio
  fio ./bcache.fio
  echo

  # pgbench benchmark
  # First pg 9.1, then 9.2
  for p in 5432 5433; do
    if [ "$p" = 5432 ]; then
      version=9.1
    else
      version=9.2
    fi
    echo "Running pgbench against postgresql $version, with bcache in mode: $cm"
    su - postgres <<EOF
    "${pg_bin}psql" -p $p -c 'create database pg_bench_db;'
    "${pg_bin}pgbench" -p $p -i -s 10 -c 10 -T 300 pg_bench_db
    "${pg_bin}pgbench" -p $p   -s 10 -c 10 -T 300 pg_bench_db
    "${pg_bin}psql" -p $p -c 'drop database pg_bench_db;'
EOF
  done
done| tee "$outfile"

Fio version: 2.0.8

Fio config:

[test]
randrepeat=1
ioengine=libaio
bs=4k
ba=4k
size=8G
direct=1
gtod_reduce=1
norandommap
iodepth=32



## Results ##


[writethrough] writeback writearound none

Running fio with bcache in mode: writethrough
[test]
randrepeat=1
ioengine=libaio
bs=4k
ba=4k
size=8G
direct=1
gtod_reduce=1
norandommap
iodepth=32
test: (g=0): rw=read, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
2.0.8
Starting 1 process

test: (groupid=0, jobs=1): err= 0: pid=12901
  read : io=8192.0MB, bw=31633KB/s, iops=7908 , runt=265183msec
  cpu          : usr=5.27%, sys=29.96%, ctx=1932374, majf=0, minf=52
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=2097152/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=8192.0MB, aggrb=31633KB/s, minb=31633KB/s, maxb=31633KB/s,
mint=265183msec, maxt=265183msec

Disk stats (read/write):
    dm-3: ios=2095437/238, merge=0/0, ticks=8387259/33094,
in_queue=8420690, util=100.00%, aggrios=0/0, aggrmerge=0/0,
aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
  bcache0: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%

Running pgbench against postgresql 9.1, with bcache in mode: writethrough
CREATE DATABASE
transaction type: TPC-B (sort of)
scaling factor: 10
query mode: simple
number of clients: 10
number of threads: 1
duration: 300 s
number of transactions actually processed: 266049
tps = 884.810200 (including connections establishing)
tps = 884.964389 (excluding connections establishing)
DROP DATABASE
Running pgbench against postgresql 9.2, with bcache in mode: writethrough
CREATE DATABASE
transaction type: TPC-B (sort of)
scaling factor: 10
query mode: simple
number of clients: 10
number of threads: 1
duration: 300 s
number of transactions actually processed: 275057
tps = 916.809784 (including connections establishing)
tps = 916.991406 (excluding connections establishing)
DROP DATABASE
writethrough [writeback] writearound none

Running fio with bcache in mode: writeback
[test]
randrepeat=1
ioengine=libaio
bs=4k
ba=4k
size=8G
direct=1
gtod_reduce=1
norandommap
iodepth=32
test: (g=0): rw=read, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
2.0.8
Starting 1 process

test: (groupid=0, jobs=1): err= 0: pid=13069
  read : io=8192.0MB, bw=35757KB/s, iops=8939 , runt=234599msec
  cpu          : usr=5.34%, sys=29.18%, ctx=2039126, majf=0, minf=51
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=2097152/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=8192.0MB, aggrb=35757KB/s, minb=35757KB/s, maxb=35757KB/s,
mint=234599msec, maxt=234599msec

Disk stats (read/write):
    dm-3: ios=2095690/75, merge=0/0, ticks=7433820/106,
in_queue=7434173, util=100.00%, aggrios=0/0, aggrmerge=0/0,
aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
  bcache0: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%

Running pgbench against postgresql 9.1, with bcache in mode: writeback
CREATE DATABASE
transaction type: TPC-B (sort of)
scaling factor: 10
query mode: simple
number of clients: 10
number of threads: 1
duration: 300 s
number of transactions actually processed: 675318
tps = 2250.938899 (including connections establishing)
tps = 2251.295253 (excluding connections establishing)
DROP DATABASE
Running pgbench against postgresql 9.2, with bcache in mode: writeback
CREATE DATABASE
transaction type: TPC-B (sort of)
scaling factor: 10
query mode: simple
number of clients: 10
number of threads: 1
duration: 300 s
number of transactions actually processed: 823609
tps = 2741.663194 (including connections establishing)
tps = 2742.135127 (excluding connections establishing)
DROP DATABASE
writethrough writeback [writearound] none

Running fio with bcache in mode: writearound
[test]
randrepeat=1
ioengine=libaio
bs=4k
ba=4k
size=8G
direct=1
gtod_reduce=1
norandommap
iodepth=32
test: (g=0): rw=read, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
2.0.8
Starting 1 process

test: (groupid=0, jobs=1): err= 0: pid=13226
  read : io=8192.0MB, bw=33949KB/s, iops=8487 , runt=247091msec
  cpu          : usr=5.47%, sys=32.60%, ctx=1961342, majf=0, minf=51
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=2097152/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=8192.0MB, aggrb=33949KB/s, minb=33949KB/s, maxb=33949KB/s,
mint=247091msec, maxt=247091msec

Disk stats (read/write):
    dm-3: ios=2094874/58, merge=0/0, ticks=7816729/4730,
in_queue=7821618, util=100.00%, aggrios=0/0, aggrmerge=0/0,
aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
  bcache0: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%

Running pgbench against postgresql 9.1, with bcache in mode: writearound
CREATE DATABASE
transaction type: TPC-B (sort of)
scaling factor: 10
query mode: simple
number of clients: 10
number of threads: 1
duration: 300 s
number of transactions actually processed: 264464
tps = 881.501731 (including connections establishing)
tps = 881.650022 (excluding connections establishing)
DROP DATABASE
Running pgbench against postgresql 9.2, with bcache in mode: writearound
CREATE DATABASE
transaction type: TPC-B (sort of)
scaling factor: 10
query mode: simple
number of clients: 10
number of threads: 1
duration: 300 s
number of transactions actually processed: 283804
tps = 945.962582 (including connections establishing)
tps = 946.128733 (excluding connections establishing)
DROP DATABASE
writethrough writeback writearound [none]

Running fio with bcache in mode: none
[test]
randrepeat=1
ioengine=libaio
bs=4k
ba=4k
size=8G
direct=1
gtod_reduce=1
norandommap
iodepth=32
test: (g=0): rw=read, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
2.0.8
Starting 1 process

test: (groupid=0, jobs=1): err= 0: pid=13367
  read : io=8192.0MB, bw=37969KB/s, iops=9492 , runt=220931msec
  cpu          : usr=5.99%, sys=29.32%, ctx=1992824, majf=0, minf=51
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=2097152/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=8192.0MB, aggrb=37969KB/s, minb=37969KB/s, maxb=37969KB/s,
mint=220931msec, maxt=220931msec

Disk stats (read/write):
    dm-3: ios=2096590/41, merge=0/0, ticks=6995926/4349,
in_queue=7000464, util=100.00%, aggrios=0/0, aggrmerge=0/0,
aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
  bcache0: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%

Running pgbench against postgresql 9.1, with bcache in mode: none
CREATE DATABASE
transaction type: TPC-B (sort of)
scaling factor: 10
query mode: simple
number of clients: 10
number of threads: 1
duration: 300 s
number of transactions actually processed: 265415
tps = 884.688970 (including connections establishing)
tps = 884.861368 (excluding connections establishing)
DROP DATABASE
Running pgbench against postgresql 9.2, with bcache in mode: none
CREATE DATABASE
transaction type: TPC-B (sort of)
scaling factor: 10
query mode: simple
number of clients: 10
number of threads: 1
duration: 300 s
number of transactions actually processed: 282727
tps = 942.373702 (including connections establishing)
tps = 942.546713 (excluding connections establishing)
DROP DATABASE



-davidc