Re: Failed RAID5 array grow after reboot interruption; mdadm: Failed to restore critical section for reshape, sorry.

From
Neil Brown <hidden>
Date
2008-06-19 04:25:34
On Monday June 16, jmolina@tgen.org wrote:
During the grow process, this system slowly went unresponsive, and I
was forced to reboot it after about 30 hours.  At first I was not
able to run any mdadm commands to see the status of the grow (about
30 minutes after starting), then I was not able to log in with a new
shell, then after about 24 hours I was able to use a previously
opened shell to see that tons of CRON jobs and other work had backed
up, however during all of this time the system was still acting as
an IP router doing NAT.  Finally, after about 30 hours, the dhcpd
daemon stopped giving out leases and then finally traffic stopped
and I could not ping the host any longer (not a lease problem). 
This is a bit of a worry.  It sounds like the system was running out
of memory.  It would seem to suggest that either the reshape process
was leaking memory, or that it was blocking writeout somehow so that
other memory wasn't getting freed.
However I cannot measure it doing either of these things.

If you can reproduce this, I'd love to see the content of
   /proc/meminfo
   /proc/slabinfo
   /proc/slab_allocators

at 5 minutes intervals.   But I don't expect you'll want to try that
experiment :-)


NeilBrown
Thread (1 message) 1 message, 1 author, last reply 2008-06-19
Keyboard shortcuts
hback out one level
jnext message in thread / next series-diff position
kprevious message in thread / previous series-diff position
ldrill in
Escclose help / fold thread tree
?toggle this help