Andrew Dunn wrote:

New data now, I got this from dmesg when it went down again. Hopefully
there is some significance to you guys.

quoted

[14269.650381] sd 10:0:3:0: rejecting I/O to offline device
[14269.650453] sd 10:0:3:0: rejecting I/O to offline device
[14269.650524] sd 10:0:3:0: rejecting I/O to offline device
[14269.650595] sd 10:0:3:0: rejecting I/O to offline device
[14269.650672] sd 10:0:3:0: [sdh] Unhandled error code
[14269.650675] sd 10:0:3:0: [sdh] Result: hostbyte=DID_NO_CONNECT

driverbyte=DRIVER_OK

quoted

[14269.650680] end_request: I/O error, dev sdh, sector 1435085631
[14269.650749] raid5:md0: read error not correctable (sector

1435085568 on sdh1).

quoted

[14269.650753] raid5: Disk failure on sdh1, disabling device.
[14269.650754] raid5: Operation continuing on 7 devices.
[14269.650886] raid5:md0: read error not correctable (sector

1435085576 on sdh1).

quoted

[14269.650890] raid5:md0: read error not correctable (sector

1435085584 on sdh1).

quoted

[14269.650894] raid5:md0: read error not correctable (sector

1435085592 on sdh1).

[...]

I am not convinced this is a drive failure (yet).  You have 
sdh,sdi,sdj,sdk,sdl,sdm all reporting errors or error recovery.

This sounds like a physical backplane failure (is this on an expander 
system? we have seen this/had this happen before), a cable to the SATA 
card failing (we have seen this/had this happen before), or a power 
supply issue (can't handle all the drives in constant operation, which 
we have seen before as well).

Driver issues are possible, but it is pursuing normal failure code 
paths, so unless the driver is tickling the remove code on its own ...

Smart could be offlining the drive, and having it non-responsive. 
Something else could be doing that as well (vibration, power quality, ...)

What does

	hdparm -I /dev/sdh

tell us?

If nothing, we need to use sdparm to get some information.

Joe

-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics, Inc.
email: landman@scalableinformatics.com
web  : http://scalableinformatics.com
        http://scalableinformatics.com/jackrabbit
phone: +1 734 786 8423 x121
fax  : +1 866 888 3112
cell : +1 734 612 4615