On Mon, Jan 22, 2018 at 10:06:26AM +0800, Hongzhi, Song wrote:

Hello,

I create a virtual disk-image using qemu-img.

And then I use /dev/nbd to map the image.

I mount the /dev/nbd to a local dir with ext4-format

Finally, I have some trouble about ext4-filesystem and block device, with
using demand of rsync or dd to write the image.

CCing Eric Blake and Paolo Bonzini (NBD maintainers in QEMU).

Is QEMU running?

Is anything accessing test.img while rsync is running?

Reproduce :

    qemu-img create test.img 2G

    mkfs.ext4 -F test.img

     qemu-nbd -f raw -c /dev/nbd0 test.img

     mount -r ext4 /dev/nbd0 LOCAL_DIR/

    rsync -av META_DATA_DIR/  LOCAL_DIR/

Qemu Version:

    QEMU emulator version 2.10.0
    Copyright (c) 2003-2017 Fabrice Bellard and the QEMU \
    Project developers

Kernel Version:

    4.12+HEAD (I have test 4.15-rc7 and 4.12)

Machine:

    intel-x86-64

CPU:

    xeon

Architecture:

    I86

The problem is not always  appear step by step, but often using lava that is
a auto-testing method.

And it just appears in some special board.

If someone has similar trouble or has resolution or wants more detail,
please connect with me. Thanks.

Detail:


329.11 EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts:
(null)
329.12 block nbd0: Connection timed out
329.13 block nbd0: shutting down sockets
329.14 blk_update_request: I/O error, dev nbd0, sector 304384
329.15 blk_update_request: I/O error, dev nbd0, sector 304640
329.16 blk_update_request: I/O error, dev nbd0, sector 304896
329.17 blk_update_request: I/O error, dev nbd0, sector 305152
329.18 blk_update_request: I/O error, dev nbd0, sector 305408
329.19 blk_update_request: I/O error, dev nbd0, sector 305664
329.20 blk_update_request: I/O error, dev nbd0, sector 305920
329.21 blk_update_request: I/O error, dev nbd0, sector 306176
329.22 blk_update_request: I/O error, dev nbd0, sector 306432
329.23 blk_update_request: I/O error, dev nbd0, sector 306688
329.24 EXT4-fs warning (device nbd0): ext4_end_bio:322: I/O error -5 writing
to inode 160 (offset 8388608 size 8388608 starting block 38400)
329.25 Buffer I/O error on device nbd0, logical block 38144
329.26 Buffer I/O error on device nbd0, logical block 38145
329.27 Buffer I/O error on device nbd0, logical block 38146
329.28 Buffer I/O error on device nbd0, logical block 38147
329.29 Buffer I/O error on device nbd0, logical block 38148
329.30 Buffer I/O error on device nbd0, logical block 38149
329.31 Buffer I/O error on device nbd0, logical block 38150
329.32 Buffer I/O error on device nbd0, logical block 38151
329.33 Buffer I/O error on device nbd0, logical block 38152
329.34 Buffer I/O error on device nbd0, logical block 38153
329.35 EXT4-fs warning (device nbd0): ext4_end_bio:322: I/O error -5 writing
to inode 160 (offset 8388608 size 8388608 starting block 38656)
329.36 EXT4-fs warning (device nbd0): ext4_end_bio:322: I/O error -5 writing
to inode 160 (offset 8388608 size 8388608 starting block 38912)
329.37 EXT4-fs warning (device nbd0): ext4_end_bio:322: I/O error -5 writing
to inode 160 (offset 16777216 size 8388608 starting block 39168)
329.38 EXT4-fs warning (device nbd0): ext4_end_bio:322: I/O error -5 writing
to inode 160 (offset 16777216 size 8388608 starting block 39424)
329.39 EXT4-fs warning (device nbd0): ext4_end_bio:322: I/O error -5 writing
to inode 160 (offset 16777216 size 8388608 starting block 39680)
329.40 EXT4-fs warning (device nbd0): ext4_end_bio:322: I/O error -5 writing
to inode 160 (offset 16777216 size 8388608 starting block 39936)
329.41 EXT4-fs warning (device nbd0): ext4_end_bio:322: I/O error -5 writing
to inode 160 (offset 16777216 size 8388608 starting block 40192)
329.42 EXT4-fs warning (device nbd0): ext4_end_bio:322: I/O error -5 writing
to inode 160 (offset 16777216 size 8388608 starting block 40448)
329.43 EXT4-fs error (device nbd0): __ext4_get_inode_loc:4520: inode #222:
block 174: comm kworker/u113:0: unable to read itable block
329.44 EXT4-fs warning (device nbd0): ext4_end_bio:322: I/O error -5 writing
to inode 160 (offset 16777216 size 8388608 starting block 40704)
329.45 ------------[ cut here ]------------
329.46 kernel BUG at /kernel-source//fs/buffer.c:3091!
329.47 invalid opcode: 0000 [#1] PREEMPT SMP
329.48 Modules linked in: nbd xt_CHECKSUM iptable_mangle ipt_REJECT
nf_reject_ipv4 xt_tcpudp ebtable_filter ebtables ip6_tables ipt_MASQUERADE
nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xfrm_user iptable_nat
xt_addrtype iptable_filter ip_tables xt_conntrack x_tables br_netfilter
bridge stp llc intel_rapl sb_edac intel_powerclamp coretemp crct10dif_pclmul
crct10dif_common aesni_intel aes_x86_64 crypto_simd cryptd glue_helper
iTCO_wdt iTCO_vendor_support lpc_ich i2c_i801 wmi acpi_pad acpi_power_meter
nfsd openvswitch nf_defrag_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4
nf_nat nf_conntrack kvm_intel kvm irqbypass fuse
329.49 CPU: 30 PID: 6 Comm: kworker/u113:0 Not tainted
4.12.18-rt0-yocto-preempt-rt #1
329.50 Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS
GRNDSDP1.86B.0036.R05.1407140519 07/14/2014
329.51 Workqueue: writeback wb_workfn (flush-43:0)
329.52 task: ffff977bec759e00 task.stack: ffffa2930524c000
329.53 RIP: 0010:submit_bh_wbc+0x155/0x160
329.54 RSP: 0018:ffffa2930524f7e0 EFLAGS: 00010246
329.55 RAX: 0000000000620005 RBX: ffff977f05cddc18 RCX: 0000000000000000
329.56 RDX: ffff977f05cddc18 RSI: 0000000000020800 RDI: 0000000000000001
329.57 RBP: ffffa2930524f808 R08: ff00000000000000 R09: 00ffffffffffffff
329.58 R10: ffffa2930524f920 R11: 000000000000058c R12: 000000000000a598
329.59 R13: ffffffffba15c500 R14: ffff977fe1bab400 R15: ffff977fea643000
329.60 FS: 0000000000000000(0000) GS:ffff977befa00000(0000)
knlGS:0000000000000000
329.61 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
329.62 CR2: 00007f7d70000010 CR3: 000000035ce0e000 CR4: 00000000001406e0
329.63 Call Trace:
329.64 __sync_dirty_buffer+0x41/0xa0
329.65 ext4_commit_super+0x1d6/0x2a0
329.66 __ext4_error_inode+0xb2/0x170
329.67 ? wake_up_bit+0x30/0x30
329.68 __ext4_get_inode_loc+0x365/0x460
329.69 ext4_reserve_inode_write+0x51/0xb0
329.70 ? ext4_writepages+0xd23/0x1090
329.71 ext4_mark_inode_dirty+0x48/0x200
329.72 ext4_writepages+0xd23/0x1090
329.73 ? __schedule+0x25b/0x6a0
329.74 do_writepages+0x39/0xc0
329.75 ? do_writepages+0x39/0xc0
329.76 ? unpin_current_cpu+0x5e/0x80
329.77 ? migrate_enable+0x2dd/0x3c0
329.78 __writeback_single_inode+0x4d/0x490
329.79 ? __writeback_single_inode+0x4d/0x490
329.80 ? rt_spin_unlock+0x27/0x40
329.81 writeback_sb_inodes+0x230/0x4d0
329.82 wb_writeback+0xde/0x3d0
329.83 wb_workfn+0xb8/0x420
329.84 ? wb_workfn+0xb8/0x420
329.85 ? unpin_current_cpu+0x5e/0x80
329.86 ? migrate_enable+0x2dd/0x3c0
329.87 ? _raw_spin_lock+0x17/0x40
329.88 process_one_work+0x1cf/0x470
329.89 ? migrate_disable+0x12f/0x200
329.90 worker_thread+0x4b/0x450
329.91 kthread+0x124/0x140
329.92 ? process_one_work+0x470/0x470
329.93 ? kthread_create_on_node+0x40/0x40
329.94 ret_from_fork+0x22/0x30
329.95 Code: 89 ef 44 09 f7 41 89 7c 24 14 4c 89 e7 e8 04 c4 22 00 5b 31 c0
41 5c 41 5d 41 5e 41 5f 5d c3 f0 80 62 01 f7 e9 19 ff ff ff 0f 0b <0f> 0b 0f
0b 0f 0b 0f 0b 0f 0b 90 0f 1f 44 00 00 55 49 89 f9 48
329.96 RIP: submit_bh_wbc+0x155/0x160 RSP: ffffa2930524f7e0
329.97 ---[ end trace 0000000000000002 ]---
329.98 Buffer I/O error on dev nbd0, logical block 262144, lost sync page
write
329.99 JBD2: Error -5 detected when updating journal superblock for nbd0-8.
329.100 Aborting journal on device nbd0-8.
329.101 ------------[ cut here ]------------
329.102 kernel BUG at /kernel-source//fs/buffer.c:3091!
329.103 invalid opcode: 0000 [#2] PREEMPT SMP
329.104 Modules linked in: nbd xt_CHECKSUM iptable_mangle ipt_REJECT
nf_reject_ipv4 xt_tcpudp ebtable_filter ebtables ip6_tables ipt_MASQUERADE
nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xfrm_user iptable_nat
xt_addrtype iptable_filter ip_tables xt_conntrack x_tables br_netfilter
bridge stp llc intel_rapl sb_edac intel_powerclamp coretemp crct10dif_pclmul
crct10dif_common aesni_intel aes_x86_64 crypto_simd cryptd glue_helper
iTCO_wdt iTCO_vendor_support lpc_ich i2c_i801 wmi acpi_pad acpi_power_meter
nfsd openvswitch nf_defrag_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4
nf_nat nf_conntrack kvm_intel kvm irqbypass fuse
329.105 CPU: 36 PID: 7760 Comm: jbd2/nbd0-8 Tainted: G D
4.12.18-rt0-yocto-preempt-rt #1
329.106 Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS
GRNDSDP1.86B.0036.R05.1407140519 07/14/2014
329.107 task: ffff977fdd853c00 task.stack: ffffa29308934000
329.108 RIP: 0010:submit_bh_wbc+0x155/0x160
329.109 RSP: 0018:ffffa29308937bd0 EFLAGS: 00010246
329.110 RAX: 0000000004000005 RBX: ffff977f05cf62e8 RCX: 0000000000000000
329.111 RDX: ffff977f05cf62e8 RSI: 0000000000020800 RDI: 0000000000000001
329.112 RBP: ffffa29308937bf8 R08: 0000000000000100 R09: 0000000000000000
329.113 R10: ffff977fdd853f30 R11: 0000000000000000 R12: ffff977fea6bc800
329.114 R13: 0000000000020800 R14: ffff977fc8a33000 R15: ffff977fea6bc800
329.115 FS: 0000000000000000(0000) GS:ffff977befb80000(0000)
knlGS:0000000000000000
329.116 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
329.117 CR2: 0000000000915d50 CR3: 000000035ce0e000 CR4: 00000000001406e0
329.118 Call Trace:
329.119 submit_bh+0x10/0x20
329.120 jbd2_write_superblock+0xaf/0x1e0
329.121 jbd2_journal_update_sb_errno+0x3f/0x50
329.122 __journal_abort_soft+0x3d/0x60
329.123 jbd2_write_superblock+0x1b4/0x1e0
329.124 jbd2_journal_update_sb_log_tail+0x3a/0x90
329.125 jbd2_journal_commit_transaction+0xb89/0x18d0
329.126 ? __this_cpu_preempt_check+0x13/0x20
329.127 ? __perf_event_task_sched_in+0x66/0x190
329.128 ? _raw_spin_lock_irqsave+0x1d/0x50
329.129 ? lock_timer_base+0x81/0xa0
329.130 ? _raw_spin_unlock_irqrestore+0x18/0x50
329.131 ? try_to_del_timer_sync+0x44/0x60
329.132 ? del_timer_sync+0x47/0xf0
329.133 kjournald2+0xc6/0x250
329.134 ? kjournald2+0xc6/0x250
329.135 ? wake_bit_function+0x60/0x60
329.136 kthread+0x124/0x140
329.137 ? commit_timeout+0x10/0x10
329.138 ? kthread_create_on_node+0x40/0x40
329.139 ret_from_fork+0x22/0x30
329.140 Code: 89 ef 44 09 f7 41 89 7c 24 14 4c 89 e7 e8 04 c4 22 00 5b 31 c0
41 5c 41 5d 41 5e 41 5f 5d c3 f0 80 62 01 f7 e9 19 ff ff ff 0f 0b <0f> 0b 0f
0b 0f 0b 0f 0b 0f 0b 90 0f 1f 44 00 00 55 49 89 f9 48
329.141 RIP: submit_bh_wbc+0x155/0x160 RSP: ffffa29308937bd0
329.142 ---[ end trace 0000000000000003 ]---
329.143 block nbd0: Connection timed out
329.144 block nbd0: Connection timed out
329.145 block nbd0: Connection timed out
329.146 block nbd0: Connection timed out
329.147 block nbd0: Connection timed out
329.148 block nbd0: Connection timed out
329.149 block nbd0: Connection timed out
329.150 block nbd0: Connection timed out
329.151 block nbd0: Connection timed out