Re: raid10 make_request failure during iozone benchmark upon btrfs

Hi,

On 03/07/2012 02:39, NeilBrown wrote:

[snip]

quoted

quoted

Could you please double check that you are running a kernel with

commit aba336bd1d46d6b0404b06f6915ed76150739057
Author: NeilBrown[off-list ref]
Date:   Thu May 31 15:39:11 2012 +1000

      md: raid1/raid10: fix problem with merge_bvec_fn

in it?

I am indeed. I searched the list beforehand and noticed the patch in
question. Not sure which -rc it landed in but I checked my source tree
and it's definitely in there.

Cheers,

--Kerin

Thanks.
Looking at it again I see that it is definitely a different bug, that patch
wouldn't affect it.

But I cannot see what could possibly be causing the problem.
You have a 256K chunk size, so requests should be limited to 512 sectors
aligned at a 512-sector boundary.
However all the requests that a causing errors are 512 sectors long, but
aligned on a 256-sector boundary (which is not also 512-sector).  This is
wrong.

I see.

It could be that btrfs is submitting bad requests, but I think it always uses
bio_add_page, and bio_add_page appears to do the right thing.
It could be that dm-linear is causing problem, but it seems to correctly after
the underlying device for alignment, and reports that alignment to
bio_add_page.
It could be that md/raid10 is the problem but I cannot find any fault in
raid10_mergeable_bvec - performs much the same tests that the
raid01 make_request function does.

So it is a mystery.

Is this failure repeatable?

Yes, it's reproducible with 100% consistency. Furthermore, I tried to
use the btrfs volume as a store for the package manager, so as to try
with a 'realistic' workload. Many of these errors were triggered
immediately upon invoking the package manager. In case it matters, the
package manager is portage (in Gentoo Linux) and the directory structure
entails a shallow directory depth with a large number of distributed
small files. I haven't been able to reproduce with xfs, ext4 or reiserfs.

If so, could you please insert
    WARN_ON_ONCE(1);
in drivers/md/raid10.c where it prints out the message: just after the
"bad_map:" label.

Also, in raid10_mergeable_bvec, insert
    WARN_ON_ONCE(max<  0);
just before
		if (max<  0)
			/* bio_add cannot handle a negative return */
			max = 0;

and then see if either of those generate a warning, and post the full stack
trace  if they do.

OK. I ran iozone again on a fresh filesystem, mounted with the default
options. Here's the trace that appears, just before the first
make_request_bug message:

WARNING: at drivers/md/raid10.c:1094 make_request+0xda5/0xe20()
Hardware name: ProLiant MicroServer
Modules linked in: btrfs zlib_deflate lzo_compress kvm_amd kvm sp5100_tco i2c_piix4
Pid: 1031, comm: btrfs-submit-1 Not tainted 3.5.0-rc5 #3
Call Trace:
[<ffffffff81031987>] ? warn_slowpath_common+0x67/0xa0
[<ffffffff81442b45>] ? make_request+0xda5/0xe20
[<ffffffff81460b34>] ? __split_and_process_bio+0x2d4/0x600
[<ffffffff81063429>] ? set_next_entity+0x29/0x60
[<ffffffff810652c3>] ? pick_next_task_fair+0x63/0x140
[<ffffffff81450b7f>] ? md_make_request+0xbf/0x1e0
[<ffffffff8123d12f>] ? generic_make_request+0xaf/0xe0
[<ffffffff8123d1c3>] ? submit_bio+0x63/0xe0
[<ffffffff81040abd>] ? try_to_del_timer_sync+0x7d/0x120
[<ffffffffa016839a>] ? run_scheduled_bios+0x23a/0x520 [btrfs]
[<ffffffffa0170e40>] ? worker_loop+0x120/0x520 [btrfs]
[<ffffffffa0170d20>] ? btrfs_queue_worker+0x2e0/0x2e0 [btrfs]
[<ffffffff810520c5>] ? kthread+0x85/0xa0
[<ffffffff815441f4>] ? kernel_thread_helper+0x4/0x10
[<ffffffff81052040>] ? kthread_freezable_should_stop+0x60/0x60
[<ffffffff815441f0>] ? gs_change+0xb/0xb

Cheers,

--Kerin