Re: I/O errors without erros from underlying device
From: John Stoffel <hidden>
Date: 2015-12-08 04:02:13
quoted
quoted
quoted
quoted
"Arkadiusz" == Arkadiusz Miskiewicz [off-list ref] writes:
Arkadiusz> On Monday 07 of December 2015, John Stoffel wrote:
quoted
quoted
quoted
quoted
quoted
quoted
"Arkadiusz" == Arkadiusz Miśkiewicz [off-list ref] writes:
Arkadiusz> On Monday 07 of December 2015, John Stoffel wrote:
quoted
Arkadiusz> 4.3.0 kernel, raid6 array:
quoted
quoted
quoted
I think there's a bug in the 4.3.x and 4.4-rc3 and lower with block merges. I ran into these over the weekend, where v4.2.6 was stable, but anything higher would lock up and crash on me.
Arkadiusz> Well, no crashes here.
quoted
That's good. It was hard(er) to hit when I wasn't running KVM VMs at the same time on the server, and I was running strictly RAID1 disks, so it's hard to know.quoted
quoted
So first step would be to make sure you get and test v4.4-rc4.
Arkadiusz> Do you know which commit there?
quoted
Try this, from the master lkml git repository: 2873d32ff493ecbfb7d2c7f56812ab941dda42f4
Arkadiusz> It's merge commit. Don't see any obvious patch in that merge that would help Arkadiusz> my case. The merge from Jens Axboe talking about blk something or other. In my case, it lead to instant lockups. In your case... hard to know. Sorry. Arkadiusz> Anyway I would expect my problem to be related to badblock Arkadiusz> lists which numbers are close to dmesg error message: [ Arkadiusz> 848.988518] Buffer I/O error on dev md7, logical block Arkadiusz> 3907148544, async page read
quoted
quoted
quoted
http://sprunge.us/XSWI
Arkadiusz> But how to repair these if write() also fails and Arkadiusz> http://www.spinics.net/lists/raid/msg49325.html suggests that write should Arkadiusz> "fix" these (by using replacement blocks I guess) ? Arkadiusz> md7 : active raid6 sdg[10] sdad1[9] sdac1[8] sdag1[7] sdaf1[6]
quoted
quoted
quoted
sdae1[5] sdaj1[4] sdai1[3] sdah1[2] sdn1[1] Arkadiusz> 31255089152 blocks super 1.2 level 6, 512k chunk, algorithm 2 [10/10] [UUUUUUUUUU]
Arkadiusz> bitmap: 1/30 pages [4KB], 65536KB chunk
quoted
Arkadiusz> array had weird failure where many disks went into failed state
quoted
quoted
quoted
but Arkadiusz> remove && adding these disks "fixed" it (turns out not really fixed it).
Arkadiusz> Unfortunately now some reads fail:
quoted
Arkadiusz> pread(4, 0x1483a00, 4096, 16003680464896) = -1 EIO (Input/output
quoted
quoted
quoted
error)
Arkadiusz> To reproduce used xfs_io Arkadiusz> xfs_io -d -c "pread 16003680464896 4096" /dev/md7 Arkadiusz> pread64: Input/output error Arkadiusz> which does pread exactly as shown above.
quoted
Arkadiusz> write also fails for that area: Arkadiusz> xfs_io -d -c "pwrite 16003680464896 4096" /dev/md7 Arkadiusz> pwrite64: Input/output error
quoted
Arkadiusz> Note that nothing is written in dmesg when that happens.
quoted
Arkadiusz> I've tried various offsets and sizes of pread and at some point
quoted
quoted
quoted
that was logged: Arkadiusz> [ 848.988518] Buffer I/O error on dev md7, logical block 3907148544, async page read
Arkadiusz> but no error from underlying devices.
quoted
Arkadiusz> List of bad blocks: Arkadiusz> http://sprunge.us/XSWI
quoted
Arkadiusz> What can I do now?
quoted
Arkadiusz> (loosing data from that few sectors is acceptable if the rest
quoted
quoted
quoted
will be readable)
Arkadiusz> Thanks, Arkadiusz> -- Arkadiusz> Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org ) Arkadiusz> -- Arkadiusz> To unsubscribe from this list: send the line "unsubscribe
quoted
quoted
quoted
linux-raid" in Arkadiusz> the body of a message to majordomo@vger.kernel.org
Arkadiusz> More majordomo info at
quoted
quoted
Arkadiusz> -- Arkadiusz> Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org ) Arkadiusz> -- Arkadiusz> Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org ) Arkadiusz> -- Arkadiusz> Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org ) Arkadiusz> -- Arkadiusz> To unsubscribe from this list: send the line "unsubscribe linux-raid" in Arkadiusz> the body of a message to majordomo@vger.kernel.org Arkadiusz> More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html