Re: problem w/ read caching..

On Thu, Sep 27, 2012 at 11:28:20PM +0000, Brad Walker wrote:

Kent Overstreet <koverstreet@...> writes:

quoted

Sounds like a good portion of your IO is bypassing the cache. That
will happen if some of it's sequential, or if the SSD latency goes
over a threshold - sequential_cutoff, congested_read_threshold_us and
congest_write_threshold_us (if I'm remembering the names correctly)
are the settings that control all that. 0 disables all of them.

So I set the sequential_cutoff and congested_read_threshold_us to both be 0.
Since I was only doing reads, I figured there was no need to mess with the write
option.

But, I'm still seeing a problem. 

My hardware is:
1 - Dell PowerEdge R710 w/ 24 x Xeon processors, 96GB of ram
2 - Micron P320H SSD
3 - LSI storage device connected by a SAS interface

What I see is that when I do random reads over a 10GB region, the cache warms up
but hits a read response plateau at about 7ms. I still see a LOT (i.e. 32000
IOPS) of I/O to the disk. 

By disk you do mean spinning disk? Or just to the bcache device?

I'm wondering if your storage array just is that fast (which would
explain the 7 ms) or something weird is going on.

Cache hit ratio or iostat would tell you.

Yes, if I run the same test over a 1GB region, runs really fast. Pretty close to
the max IOPS rate of the SSD.

So I'm thinking there is a problem here or I have a bcache config issue.

Sounds like some sort of bcache problem. hrm.

Most likely cause is something is keeping the cache from warming up, and
some IO is still going to disk. That used to be an issue with the old
synchronization for updating the cache on cache miss, but it shouldn't
be anymore...

Check number of cache misses after a run... if it's going up when all
the data should be in the cache, that's one bug. If there's no cache
misses and you're still seeing 7 ms latency... well, that would be
weird. queueing delays, maybe..

Kent Overstreet <koverstreet@...> writes:

By disk you do mean spinning disk? Or just to the bcache device?

I'm wondering if your storage array just is that fast (which would
explain the 7 ms) or something weird is going on.

Cache hit ratio or iostat would tell you.

The cache_hit_ratio is 99%. And yet, iostat still shows i/o running to the
raid array.

quoted

Yes, if I run the same test over a 1GB region, runs really fast. 
Pretty close to the max IOPS rate of the SSD.

So I'm thinking there is a problem here or I have a bcache config issue.

Sounds like some sort of bcache problem. hrm.

Most likely cause is something is keeping the cache from warming up, and
some IO is still going to disk. That used to be an issue with the old
synchronization for updating the cache on cache miss, but it shouldn't
be anymore...

Check number of cache misses after a run... if it's going up when all
the data should be in the cache, that's one bug. If there's no cache
misses and you're still seeing 7 ms latency... well, that would be
weird. queueing delays, maybe..

After running my tests when the cache is fully warmed, the cache_hit_ratio
goes to 99%.

Yet, cache misses are stable and not changing. Cache hits are increasing and
still iostat is showing 32K blocks being read from disk.

Any ideas on how to debug this?

Thanks for the help.

-brad w.