RE: File system corruption during setting new size (native/extarnal metatdat) after expansion

-----Original Message-----
From: linux-raid-owner@vger.kernel.org [mailto:linux-raid-
owner@vger.kernel.org] On Behalf Of NeilBrown
Sent: Thursday, February 17, 2011 5:38 AM
To: Kwolek, Adam
Cc: linux-raid@vger.kernel.org; Williams, Dan J; Ciechanowski, Ed;
Neubauer, Wojciech
Subject: Re: File system corruption during setting new size
(native/extarnal metatdat) after expansion

On Wed, 16 Feb 2011 15:24:53 +0000 "Kwolek, Adam"
[off-list ref]
wrote:

quoted

Hi,

I've met again array_size change problem. A while ago I've indicated

it, during array expansion, file system corruption occurs.

quoted

The previous conclusion was that my tests were wrong (too small

arrays, on bigger arrays it seems to be ok),

quoted

I've got now reproduction in our lab on quite big arrays (250G) and

on my small test arrays also.

quoted

This problem is related to array expansion using native and external

metadata.

quoted

Problem is that after expansion file system on array is corrupted.
Operation that corrupts file system is array size change via sysfs

(and for native metadata during reshape finalization in md).

quoted

In logs I've got information:
	VFS: busy inodes on changed media or resized disks

My investigation in md goes to block_dev.c/inode.c and flush_disk()

(called from check_disk_size_change()).

quoted

I've found above text there (about busy inodes).
My traces shows that in __invalidate_inodes(), inode->i_count == 2

and it is locked. This makes me to look how invalidate_inodes() is
called.

quoted

Before invalidate_inodes() is called, cache is cleared by

shrink_dcache_sb().

quoted

If I've commented shrink_dcache_sb() file system seems to be safe. It

is the same when invalidate_inodes() is commented (shrink_cache_sb()
called only).

quoted

If I've changed functions order (1. invalidate_inodes(), 2.

shrink_dcache_sb()) in __invalidate_device() (block_dev.c) there is no
corruption also.

quoted

When after shrink_dcache_sb() is called invalidate_inodes() on busy

inodes corruption occurs (if it called earlier it doesn't matter).

quoted

This problem can be reproduced on mounted arrays only. If array is

not mounted, file system corruption disappears.

quoted

This means that whole expansion process is correct.
In some cases (rarely) inodes on mounted device are not locked and

then FS corruption doesn't happened.

quoted

I'd like to know your opinion.
Do you think that problem is in releasing cache but not all inodes

(busy case), commands order...

quoted

... or probably changing size on mounted array is a mistake (for

current code)?

quoted

BR
Adam


---------------------------------
My test commands:

#create container
mdadm -C /dev/md/imsm0 -amd -e imsm -n 3 /dev/sdb /dev/sdc /dev/sde -

R

quoted

#create volume
mdadm -C /dev/md/raid5vol_0 -amd -l 5 --chunk 64 --size 104857 -n 3

/dev/sdb /dev/sdc /dev/sde -R

quoted

mkfs /dev/md/raid5vol_0
mount /dev/md/raid5vol_0 /mnt/vol

#copy some files from current directory
cp * /mnt/vol

#add spare
mdadm --add /dev/md/imsm0 /dev/sdd

#start reshape
mdadm --grow /dev/md/imsm0 --raid-devices 4 --backup-file=/backup.bak

Thanks for the script - that makes things a lot easier.

I had to add some "mdadm --wait" commands etc to make it work reliably,
but
then I could get it to generate filesystem corruption quite easily.

I also had "mdmon" crashing because imsm_num_data_members got a 'map'
of NULL,
and then tried to get_imsm_raid_level on it.  Something needs to be
fixed
there (I hacked it so that if it got a NULL, it tried again with
second_map
== 0, but I don't know if that is right).

The problem seems to be in the VFS/VM layer.  I can see two ways to fix
it.
I'll see if I can find some who wants to take responsibility and make a
decision.

1/ in invalidate_inodes (in fs/inodes.c), inside the loop, add a
clause:

		if (inode->i_state & I_DIRTY)
			continue;

  That is what is causing the problem - dirty inodes are being
discarded.

2/ in check_disk_size_change (in fs/block_dev.c), replace

         flush_disk(bdev);
   with

         if (disk_size == 0 || bdev_size == 0)
               flush_disk(bdev);

 because I don't think there is any justification for calling
flush_bdev if
 we are just changing the size of the disk - though I don't really know
why
 that is there at all, so it is hard to say for sure.   Certainly if
 disk_size > bdev_size we shouldn't be calling flush disk.

So I suggest you make one of those changes to continue with your
testing, and
I'll try to get something sorted out.

If you could look into the circumstances that might make
imsm_num_data_members
get a NULL map, I would appreciate that.

Thanks,
NeilBrown

Thank you for workarounds/temporary fixes.

Regarding imsm_num_data_members() setting second_map to 0 cannot help as it is always called with this parameter set to 0.
In this situation, when first map should be always present, we can have some race condition.
I have no reproduction for mdmon crash you are observing, but I'll try some changes in my scripts and I'll carefully watch any signs 
that can indicate reproduction of this problem.

If you could let me know details about changes you made to my scenario, it could help.

Thanks again
Adam



 

--
To unsubscribe from this list: send the line "unsubscribe linux-raid"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html