Re: Quick bcache benchmark

That's what you'd expect in writethrough mode when you aren't getting
any cache hits - try flipping on writeback and see what happens.

On Fri, Dec 16, 2011 at 10:49 AM, Marcus Sorensen [off-list ref] wrote:

Actually I think this IS user error. I ran a benchmark with FIO, and
the results were practically identical with and without bcache.  I
applied the 3.1.4 kernel patch on top of your 3.1 tree, even though it
applied cleanly I'm guessing that wiped something out. Here are my
stats after running the benchmark on bcache, and also included is the
fio config.

bypassed 32.1G
cache_bypass_hits 5482
cache_bypass_misses 194862
cache_hit_ratio 3
cache_hits 786
cache_miss_collisions 206
cache_misses 19447
cache_readaheads 0

[global]
ioengine=libaio
iodepth=4
invalidate=1 #make sure we're not cached locally
direct=1 #don't use buffers during test (test without local caches)
thread
ramp_time=20
time_based
runtime=180

[8RandomReadWriters]
rw=randrw
numjobs=8
blocksize=4k
size=1G

[2SequentialReadWriters]
rw=rw
numjobs=2
size=4G
blocksize_range=64k-1M


On Thu, Dec 15, 2011 at 9:28 PM, Marcus Sorensen [off-list ref] wrote:

quoted

Thanks! I'll put it through some more tests. I kind of figured that
something more real-world would help.

On Thu, Dec 15, 2011 at 7:17 PM, Kent Overstreet [off-list ref] wrote:

quoted

Sorry, I was thinking about that issue for awhile and then I got distracted...

It's not user error, it's an irritating corner case. Basically, it's
the result of a workaround for a particularly obscure data corruption
bug.

If a write bypasses the cache, it has to invalidate that region of the
cache; the null key it leaves in the cache will block cache misses
from adding that data to the cache until the btree node fills up (and
possibly splits).

It hasn't been an issue for us in normal operation, but when you're
just testing - i.e. you don't have much load - that node split may not
happen for a long time, and so if for some reason a bunch of data
bypassed the cache... well, you see what happens.

Unfortunately a better solution to the original race is not going to
be simple, so it's probably not going to be done in the very near
future. It's a _very_ difficult race to hit, but in the meantime I'd
rather lose performance than corrupt data.

But the good news is if you put normal server-ish load on it the issue
should go away in steady state operation.

On Thu, Dec 15, 2011 at 3:40 PM, Marcus Sorensen [off-list ref] wrote:

quoted

Any ideas on this? Do you think it's a bug, or am I just holding it wrong? :-)

On Sat, Dec 10, 2011 at 8:02 AM, Marcus Sorensen [off-list ref] wrote:

quoted

That keeps the 'bypassed' value from increasing, but it doesn't change
write performance.

BEFORE:
[root@sansrv2-10 stats_day]# cat *
27.6M
83
3500
0
166
24380
40660
0

...benchmarking...

AFTER:

[root@sansrv2-10 stats_day]#  for i in `ls`; do echo -n "$i "; cat $i;

quoted

done 2>/dev/null

bypassed 27.6M
cache_bypass_hits 83
cache_bypass_misses 3500
cache_hit_ratio 0
cache_hits 410
cache_miss_collisions 48879
cache_misses 80545
cache_readaheads 0

/sys/fs/bcache/60da061c-d646-4ebe-931a-d8580add411d

average_key_size 0
block_size 2.0k
btree_cache_size 3.2M
bucket_size 1.0M
cache_available_percent 100
clear_stats congested 0
congested_threshold_us 0
dirty_data 0
io_error_halflife 0
io_error_limit 8
root_usage_percent 0
synchronous 1
tree_depth 1


On Fri, Dec 9, 2011 at 11:33 PM, Kent Overstreet
[off-list ref] wrote:

quoted

On Fri, Dec 09, 2011 at 10:09:55AM -0700, Marcus Sorensen wrote:

quoted

Here's some more info. I'm running kernel 3.1.4. When I do random
writes, the 'bypassed' number increases in stats. Now I'm random
writing direct to /dev/bcache0 and get the same result.

Weird. From what you're describing it sounds like throttling is screwed
up (and it was recently), but I can't reproduce it now.

Can you try echoing 0 to congested_threshold_us in the cache set dir,
and seeing if that fixes it?

quoted

There also seems to be some work needed with clean-up, since I'm
unfamiliar with how bcache works I attempted to make-bcache twice,
thinking I'd start over. That worked, but because my cache device was
already registered I was unable to re-register my newly formatted
cache dev, got "kobject_add_internal failed for bcache with -EEXIST,
don't try to register things with the same name in the same
directory." I was still able to use my cache device via the old uuid,
but this will probably cause problems on reboot. Perhaps an unregister
file in /sys/fs/bcache would help, I also tried rmmod'ing bcache to
see if I could clear /sys/fs/bcache, but no luck. make-bcache should
perhaps check for an existing superblock, ask for confirmation, and
give some sort instruction on how to unregister, or do it for you if
you reformat.

Yeah, I think for some reason bcache isn't opening the devices
exclusively on 3.1. I'll have a look...

--
To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html