On Mon, Oct 01, 2012 at 07:18:40PM +0000, Brad Walker wrote:

Kent Overstreet <koverstreet@...> writes:

quoted

By disk you do mean spinning disk? Or just to the bcache device?

I'm wondering if your storage array just is that fast (which would
explain the 7 ms) or something weird is going on.

Cache hit ratio or iostat would tell you.

The cache_hit_ratio is 99%. And yet, iostat still shows i/o running to the
raid array.

quoted

quoted

Yes, if I run the same test over a 1GB region, runs really fast. 
Pretty close to the max IOPS rate of the SSD.

So I'm thinking there is a problem here or I have a bcache config issue.

Sounds like some sort of bcache problem. hrm.

Most likely cause is something is keeping the cache from warming up, and
some IO is still going to disk. That used to be an issue with the old
synchronization for updating the cache on cache miss, but it shouldn't
be anymore...

Check number of cache misses after a run... if it's going up when all
the data should be in the cache, that's one bug. If there's no cache
misses and you're still seeing 7 ms latency... well, that would be
weird. queueing delays, maybe..

After running my tests when the cache is fully warmed, the cache_hit_ratio
goes to 99%.

Yet, cache misses are stable and not changing. Cache hits are increasing and
still iostat is showing 32K blocks being read from disk.

Any ideas on how to debug this?

What about cache_bypass_hits, cache_bypass_misses?