Re: mdadm resync causes stable system to crash every 2 or 3 hours
From: Ryan Patterson <hidden>
Date: 2021-09-07 22:55:50
On Tue, Sep 7, 2021 at 5:18 AM Roman Mamedov [off-list ref] wrote:
On Tue, 7 Sep 2021 12:52:01 +0500 Roman Mamedov [off-list ref] wrote:quoted
On Mon, 6 Sep 2021 20:44:31 -0400 Ryan Patterson [off-list ref] wrote:quoted
My file server is usually very stable. The past week I had two mdadm arrays that required recync operations. * newly created raid6 array (14 x 16TB seagate exos) * existing raid 6 array, after a reboot resync on hot spare (14 x 4TB seagate barracuda) During both resync operations (they ran one at a time) the system would routinely experience a major error and require a hard reboot, every two or three hours. I saw several errors, such as: * kernel watchdog soft lockups [md127_raid6:364] * general protection faults (I have a few saved with the full exception stack) * exceptions in iommu routines (again I have the full error with exception stack saved) * full system lockupSo in other words the server is very stable, unless asked to do full-speed reads from all disks at the same time. I'd suggest to check or improve cooling on the HBA cards, and then try a different PSU.Also the motherboard chipset cooling, since that's a lot of PCI-E traffic. Maybe the CPU cooling as well, or at least check the CPU temperatures during this load. And since you have full logs and backtraces, there's no point in waiting to post those, just go ahead. Maybe they will point to something other than suspect hardware, or at least to which part of hardware to suspect. -- With respect, Roman
Thanks for the suggestions. Hardware overheating might be my problem. I have several (loud) case fans blowing away. But the HBA cards and mobo southbridge are only passively cooled. Maybe I could mount fans on each cards' headsink. I'll investigate. The power supply is not an off the shelf job. So I don't convientanly have a replacement to try. I might have to bite the bullet and buy a second. I forgot to put in my original note that I ran memtest86 on this machine for four full cycles with no faults found. Also nothing is overclocked. Here are some of the errors I recorded. Maybe somebody can see a pattern in them... [Thu Sep 2 22:05:25 2021] md: resync of RAID array md127 [Thu Sep 2 23:58:58 2021] perf: interrupt took too long (2570 > 2500), lowering kernel.perf_event_max_sample_rate to 77750 [Fri Sep 3 02:14:38 2021] general protection fault, probably for non-canonical address 0x1000000000000: 0000 [#1] SMP PTI [Fri Sep 3 02:14:38 2021] CPU: 3 PID: 371 Comm: md127_raid6 Not tainted 5.10.0-8-amd64 #1 Debian 5.10.46-4 [Fri Sep 3 02:14:38 2021] Hardware name: Gigabyte Technology Co., Ltd. Z87X-UD4H/Z87X-UD4H-CF, BIOS F9 03/18/2014 [Fri Sep 3 02:14:38 2021] RIP: 0010:bio_associate_blkg+0x17/0x70 [Fri Sep 3 02:14:38 2021] Code: dc fe ff ff 48 89 c3 e9 05 ff ff ff 0f 1f 80 00 00 00 00 0f 1f 44 00 00 48 83 ec 08 48 8b 47 48 48 85 c0 74 17 48 85 ff 74 3d <48> 8b 70 28 e8 d0 fc ff ff 48 83 c4 08 e9 b7 c1 cc ff 48 89 3c 24 [Fri Sep 3 02:14:38 2021] RSP: 0018:ffffbde280747b58 EFLAGS: 00010286 [Fri Sep 3 02:14:38 2021] RAX: 0001000000000000 RBX: ffff95f8334a8000 RCX: 0000000000000000 [Fri Sep 3 02:14:38 2021] RDX: 0000000000000001 RSI: 00000000000000c0 RDI: ffff95f8334a8c60 [Fri Sep 3 02:14:38 2021] RBP: ffff95f700dfb400 R08: 0000000000000001 R09: 00000000fffffffd [Fri Sep 3 02:14:38 2021] R10: 0000000000000000 R11: ffff95f8334a8c60 R12: 0000000000000b80 [Fri Sep 3 02:14:38 2021] R13: 0000000000000000 R14: ffff95f741a09000 R15: ffff95f8334a8000 [Fri Sep 3 02:14:38 2021] FS: 0000000000000000(0000) GS:ffff95fe1f2c0000(0000) knlGS:0000000000000000 [Fri Sep 3 02:14:38 2021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Sep 3 02:14:38 2021] CR2: 00007f3cbf016514 CR3: 00000005c740a001 CR4: 00000000001706e0 [Fri Sep 3 02:14:38 2021] Call Trace: [Fri Sep 3 02:14:38 2021] ops_run_io+0x43d/0xd20 [raid456] [Fri Sep 3 02:14:38 2021] ? irq_init_percpu_irqstack+0x80/0x100 [Fri Sep 3 02:14:38 2021] handle_stripe+0xd9e/0x1fd0 [raid456] [Fri Sep 3 02:14:38 2021] handle_active_stripes.constprop.0+0x3af/0x590 [raid456] [Fri Sep 3 02:14:38 2021] raid5d+0x375/0x5a0 [raid456] [Fri Sep 3 02:14:38 2021] md_thread+0x94/0x160 [md_mod] [Fri Sep 3 02:14:38 2021] ? add_wait_queue_exclusive+0x70/0x70 [Fri Sep 3 02:14:38 2021] ? md_write_inc+0x50/0x50 [md_mod] [Fri Sep 3 02:14:38 2021] kthread+0x11b/0x140 [Fri Sep 3 02:14:38 2021] ? __kthread_bind_mask+0x60/0x60 [Fri Sep 3 02:14:38 2021] ret_from_fork+0x22/0x30 [Fri Sep 3 02:14:38 2021] Modules linked in: ipt_REJECT nf_reject_ipv4 xt_multiport nft_compat nft_counter nf_tables nfnetlink rfkill nls_ascii nls_cp437 vfat fat snd_hda_codec_realtek snd_hda_codec_generic mei_hdcp intel_rapl_msr snd_hda_codec_hdmi led trig_audio snd_hda_intel snd_intel_dspcfg soundwire_intel soundwire_generic_allocation snd_soc_core snd_compress soundwire_cadence snd_hda_codec snd_hda_core snd_hwdep soundwire_bus snd_pcm intel_rapl_common x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel iTCO_wdt snd_timer kvm sg mei_me mei snd intel_pmc_bxt soundcore irqbypass iTCO_vendor_support at24 watchdog evdev pcspkr efi_pstore rapl intel_cstate intel_uncore sunrpc msr fuse configfs efivarfs ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 btrfs blake2b_generic dm_crypt dm_mod raid10 raid1 raid0 multipath linear raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c crc32c_generic md_mod sd_mod t10_pi crc_t10dif crct10dif_generic hid_generic usbhid hid [Fri Sep 3 02:14:38 2021] i915 crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel i2c_algo_bit drm_kms_helper libaes mxm_wmi crypto_simd ahci libahci mpt3sas i2c_i801 cryptd xhci_pci glue_helper i2c_smbus cec xh ci_hcd libata drm raid_class ehci_pci scsi_transport_sas lpc_ich ehci_hcd e1000e scsi_mod usbcore ptp pps_core usb_common fan wmi video button [Fri Sep 3 02:14:38 2021] ---[ end trace 5cc209277d97aabd ]--- [Fri Sep 3 02:14:38 2021] RIP: 0010:bio_associate_blkg+0x17/0x70 [Fri Sep 3 02:14:38 2021] Code: dc fe ff ff 48 89 c3 e9 05 ff ff ff 0f 1f 80 00 00 00 00 0f 1f 44 00 00 48 83 ec 08 48 8b 47 48 48 85 c0 74 17 48 85 ff 74 3d <48> 8b 70 28 e8 d0 fc ff ff 48 83 c4 08 e9 b7 c1 cc ff 48 89 3c 24 [Fri Sep 3 02:14:38 2021] RSP: 0018:ffffbde280747b58 EFLAGS: 00010286 [Fri Sep 3 02:14:38 2021] RAX: 0001000000000000 RBX: ffff95f8334a8000 RCX: 0000000000000000 [Fri Sep 3 02:14:38 2021] RDX: 0000000000000001 RSI: 00000000000000c0 RDI: ffff95f8334a8c60 [Fri Sep 3 02:14:38 2021] RBP: ffff95f700dfb400 R08: 0000000000000001 R09: 00000000fffffffd [Fri Sep 3 02:14:38 2021] R10: 0000000000000000 R11: ffff95f8334a8c60 R12: 0000000000000b80 [Fri Sep 3 02:14:38 2021] R13: 0000000000000000 R14: ffff95f741a09000 R15: ffff95f8334a8000 [Fri Sep 3 02:14:38 2021] FS: 0000000000000000(0000) GS:ffff95fe1f2c0000(0000) knlGS:0000000000000000 [Fri Sep 3 02:14:38 2021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Sep 3 02:14:38 2021] CR2: 00007f3cbf016514 CR3: 000000010828e003 CR4: 00000000001706e0 [Fri Sep 3 02:14:38 2021] ------------[ cut here ]------------ [Fri Sep 3 02:14:38 2021] WARNING: CPU: 3 PID: 371 at kernel/exit.c:725 do_exit+0x47/0xaa0 [Fri Sep 3 02:14:38 2021] Modules linked in: ipt_REJECT nf_reject_ipv4 xt_multiport nft_compat nft_counter nf_tables nfnetlink rfkill nls_ascii nls_cp437 vfat fat snd_hda_codec_realtek snd_hda_codec_generic mei_hdcp intel_rapl_msr snd_hda_codec_hdmi led trig_audio snd_hda_intel snd_intel_dspcfg soundwire_intel soundwire_generic_allocation snd_soc_core snd_compress soundwire_cadence snd_hda_codec snd_hda_core snd_hwdep soundwire_bus snd_pcm intel_rapl_common x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel iTCO_wdt snd_timer kvm sg mei_me mei snd intel_pmc_bxt soundcore irqbypass iTCO_vendor_support at24 watchdog evdev pcspkr efi_pstore rapl intel_cstate intel_uncore sunrpc msr fuse configfs efivarfs ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 btrfs blake2b_generic dm_crypt dm_mod raid10 raid1 raid0 multipath linear raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c crc32c_generic md_mod sd_mod t10_pi crc_t10dif crct10dif_generic hid_generic usbhid hid [Fri Sep 3 02:14:38 2021] i915 crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel i2c_algo_bit drm_kms_helper libaes mxm_wmi crypto_simd ahci libahci mpt3sas i2c_i801 cryptd xhci_pci glue_helper i2c_smbus cec xh ci_hcd libata drm raid_class ehci_pci scsi_transport_sas lpc_ich ehci_hcd e1000e scsi_mod usbcore ptp pps_core usb_common fan wmi video button [Fri Sep 3 02:14:38 2021] CPU: 3 PID: 371 Comm: md127_raid6 Tainted: G D 5.10.0-8-amd64 #1 Debian 5.10.46-4 [Fri Sep 3 02:14:38 2021] Hardware name: Gigabyte Technology Co., Ltd. Z87X-UD4H/Z87X-UD4H-CF, BIOS F9 03/18/2014 [Fri Sep 3 02:14:38 2021] RIP: 0010:do_exit+0x47/0xaa0 [Fri Sep 3 02:14:38 2021] Code: ec 40 65 48 8b 04 25 28 00 00 00 48 89 44 24 38 31 c0 48 8b 83 b8 0b 00 00 48 85 c0 74 0e 48 8b 10 48 39 d0 0f 84 64 04 00 00 < 0f> 0b 65 8b 0d d0 b5 38 68 89 c8 25 00 ff ff 00 89 44 24 0c 0f 85 [Fri Sep 3 02:14:38 2021] RSP: 0018:ffffbde280747ee0 EFLAGS: 00010216 [Fri Sep 3 02:14:38 2021] RAX: ffffbde280747e50 RBX: ffff95f7b8619780 RCX: ffff95fe1f2d8a08 [Fri Sep 3 02:14:38 2021] RDX: ffff95f769981fc8 RSI: 0000000000000027 RDI: 000000000000000b [Fri Sep 3 02:14:38 2021] RBP: 000000000000000b R08: 0000000000000000 R09: ffffbde280747798 [Fri Sep 3 02:14:38 2021] R10: ffffbde280747790 R11: ffffffff992cb3e8 R12: 000000000000000b [Fri Sep 3 02:14:38 2021] R13: 0000000000000000 R14: ffff95f7b8619780 R15: 0000000000000000 [Fri Sep 3 02:14:38 2021] FS: 0000000000000000(0000) GS:ffff95fe1f2c0000(0000) knlGS:0000000000000000 [Fri Sep 3 02:14:38 2021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Sep 3 02:14:38 2021] CR2: 00007f3cbf016514 CR3: 000000010828e003 CR4: 00000000001706e0 [Fri Sep 3 02:14:38 2021] Call Trace: [Fri Sep 3 02:14:38 2021] ? md_write_inc+0x50/0x50 [md_mod] [Fri Sep 3 02:14:38 2021] ? kthread+0x11b/0x140 [Fri Sep 3 02:14:38 2021] rewind_stack_do_exit+0x17/0x20 [Fri Sep 3 02:14:38 2021] RIP: 0000:0x0 [Fri Sep 3 02:14:38 2021] Code: Unable to access opcode bytes at RIP 0xffffffffffffffd6. [Fri Sep 3 02:14:38 2021] RSP: 0000:0000000000000000 EFLAGS: 00000000 ORIG_RAX: 0000000000000000 [Fri Sep 3 02:14:38 2021] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 [Fri Sep 3 02:14:38 2021] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 [Fri Sep 3 02:14:38 2021] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [Fri Sep 3 02:14:38 2021] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [Fri Sep 3 02:14:38 2021] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [Fri Sep 3 02:14:38 2021] ---[ end trace 5cc209277d97aabe ]--- Message from syslogd@wxyz at Sep 3 07:30:26 ... kernel:[ 3191.941392] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [md127_raid6:364] Message from syslogd@wxyz at Sep 3 07:30:26 ... kernel:[ 3191.941392] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [md127_raid6:364] [Fri Sep 3 09:05:56 2021] ------------[ cut here ]------------ [Fri Sep 3 09:05:56 2021] WARNING: CPU: 7 PID: 284 at drivers/iommu/intel/iommu.c:2440 __domain_mapping.cold+0x4e/0x55 [Fri Sep 3 09:05:56 2021] Modules linked in: xfs ipt_REJECT nf_reject_ipv4 xt_multiport nft_compat nft_counter nf_tables nfnetlink rfkill nls_ascii nls_cp437 v fat fat intel_rapl_msr mei_hdcp snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_codec_hdmi snd_hda_intel snd_intel_dspcfg soundwire_intel inte l_rapl_common soundwire_generic_allocation snd_soc_core snd_compress x86_pkg_temp_thermal intel_powerclamp soundwire_cadence snd_hda_codec snd_hda_core coretemp snd_hwdep soundwire_bus snd_pcm snd_timer kvm_intel snd iTCO_wdt intel_pmc_bxt iTCO_vendor_support watchdog soundcore kvm sg mei_me mei at24 irqbypass rapl int el_cstate intel_uncore pcspkr evdev efi_pstore sunrpc msr fuse configfs efivarfs ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 btrfs blake2b_generic dm_cry pt dm_mod raid10 raid1 raid0 multipath linear raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c crc32c_generic md_mod sd _mod t10_pi crc_t10dif crct10dif_generic hid_generic usbhid [Fri Sep 3 09:05:56 2021] hid i915 crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel xhci_pci ahci libahci i2c_algo_ bit mxm_wmi xhci_hcd drm_kms_helper libaes crypto_simd libata cryptd glue_helper mpt3sas cec ehci_pci ehci_hcd drm i2c_i801 i2c_smbus raid_class usbcore scsi_tr ansport_sas e1000e lpc_ich scsi_mod ptp pps_core usb_common fan wmi video button [Fri Sep 3 09:05:56 2021] CPU: 7 PID: 284 Comm: kworker/7:1H Tainted: G W 5.10.0-8-amd64 #1 Debian 5.10.46-4 [Fri Sep 3 09:05:56 2021] Hardware name: Gigabyte Technology Co., Ltd. Z87X-UD4H/Z87X-UD4H-CF, BIOS F9 03/18/2014 [Fri Sep 3 09:05:56 2021] Workqueue: kblockd blk_mq_run_work_fn [Fri Sep 3 09:05:56 2021] RIP: 0010:__domain_mapping.cold+0x4e/0x55 [Fri Sep 3 09:05:56 2021] Code: 68 d6 fd ff 8b 05 bd 97 f2 00 4c 8b 0c 24 4c 8b 5c 24 08 4c 8b 54 24 10 85 c0 4c 8b 44 24 18 74 09 83 e8 01 89 05 9d 97 f2 00 < 0f> 0b e9 4e 29 d6 ff 4c 89 ea 4c 89 f6 48 c7 c7 70 dd b4 b0 e8 29 [Fri Sep 3 09:05:56 2021] RSP: 0018:ffffa62d80787b28 EFLAGS: 00010246 [Fri Sep 3 09:05:56 2021] RAX: 0000000000000000 RBX: 00000001720a0003 RCX: 0000000000000000 [Fri Sep 3 09:05:56 2021] RDX: 0000000000000000 RSI: ffff968f1f3d8a00 RDI: ffff968f1f3d8a00 [Fri Sep 3 09:05:56 2021] RBP: ffff96883feee068 R08: 0000000000000073 R09: ffff96883feee000 [Fri Sep 3 09:05:56 2021] R10: 00000000001720a0 R11: ffff968819832100 R12: 0000000000000001 [Fri Sep 3 09:05:56 2021] R13: ffff9689b51af180 R14: 00000000000df60d R15: 0000000000000003 [Fri Sep 3 09:05:56 2021] FS: 0000000000000000(0000) GS:ffff968f1f3c0000(0000) knlGS:0000000000000000 [Fri Sep 3 09:05:56 2021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Sep 3 09:05:56 2021] CR2: 0000562a47c025b8 CR3: 000000018a00a002 CR4: 00000000001706e0 [Fri Sep 3 09:05:56 2021] Call Trace: [Fri Sep 3 09:05:56 2021] domain_mapping+0x1b/0xa0 [Fri Sep 3 09:05:56 2021] intel_map_sg+0x120/0x220 [Fri Sep 3 09:05:56 2021] dma_map_sg_attrs+0x45/0x50 [Fri Sep 3 09:05:56 2021] scsi_dma_map+0x35/0x40 [scsi_mod] [Fri Sep 3 09:05:56 2021] _base_build_sg_scmd+0x68/0x2d0 [mpt3sas] [Fri Sep 3 09:05:56 2021] scsih_qcmd+0x38c/0x4b0 [mpt3sas] [Fri Sep 3 09:05:56 2021] scsi_queue_rq+0x31f/0xa60 [scsi_mod] [Fri Sep 3 09:05:56 2021] blk_mq_dispatch_rq_list+0x11a/0x7f0 [Fri Sep 3 09:05:56 2021] ? asm_exc_xen_hypervisor_callback+0x20/0x20 [Fri Sep 3 09:05:56 2021] ? elv_rb_del+0x1f/0x30 [Fri Sep 3 09:05:56 2021] ? deadline_remove_request+0x55/0xb0 [Fri Sep 3 09:05:56 2021] __blk_mq_do_dispatch_sched+0xb8/0x2c0 [Fri Sep 3 09:05:56 2021] __blk_mq_sched_dispatch_requests+0x13f/0x190 [Fri Sep 3 09:05:56 2021] blk_mq_sched_dispatch_requests+0x30/0x60 [Fri Sep 3 09:05:56 2021] __blk_mq_run_hw_queue+0x47/0xe0 [Fri Sep 3 09:05:56 2021] process_one_work+0x1b6/0x350 [Fri Sep 3 09:05:56 2021] worker_thread+0x53/0x3e0 [Fri Sep 3 09:05:56 2021] ? process_one_work+0x350/0x350 [Fri Sep 3 09:05:56 2021] kthread+0x11b/0x140 [Fri Sep 3 09:05:56 2021] ? __kthread_bind_mask+0x60/0x60 [Fri Sep 3 09:05:56 2021] ret_from_fork+0x22/0x30 [Fri Sep 3 09:05:56 2021] ---[ end trace 3b8b2da3617ea49e ]--- [Fri Sep 3 09:05:56 2021] DMAR: DRHD: handling fault status reg 3 [Fri Sep 3 09:05:56 2021] DMAR: [DMA Write] Request device [01:00.0] PASID ffffffff fault addr df60d000 [fault reason 05] PTE Write access is not set [Fri Sep 3 09:05:56 2021] DMAR: ERROR: DMA PTE for vPFN 0xdf60d already set (to 1000000000000 not 1a4388003) [Fri Sep 3 09:05:56 2021] ------------[ cut here ]------------ [Fri Sep 3 06:40:40 2021] md: resync of RAID array md127 [Fri Sep 3 07:30:00 2021] rcu: INFO: rcu_sched self-detected stall on CPU [Fri Sep 3 07:30:00 2021] rcu: 2-....: (5249 ticks this GP) idle=0c2/1/0x4000000000000000 softirq=56412/56412 fqs=2624 [Fri Sep 3 07:30:00 2021] (t=5250 jiffies g=91613 q=8034) [Fri Sep 3 07:30:00 2021] NMI backtrace for cpu 2 [Fri Sep 3 07:30:00 2021] CPU: 2 PID: 364 Comm: md127_raid6 Not tainted 5.10.0-8-amd64 #1 Debian 5.10.46-4 [Fri Sep 3 07:30:00 2021] Hardware name: Gigabyte Technology Co., Ltd. Z87X-UD4H/Z87X-UD4H-CF, BIOS F9 03/18/2014 [Fri Sep 3 07:30:00 2021] Call Trace: [Fri Sep 3 07:30:00 2021] <IRQ> [Fri Sep 3 07:30:00 2021] dump_stack+0x6b/0x83 [Fri Sep 3 07:30:00 2021] nmi_cpu_backtrace.cold+0x32/0x69 [Fri Sep 3 07:30:00 2021] ? lapic_can_unplug_cpu+0x80/0x80 [Fri Sep 3 07:30:00 2021] nmi_trigger_cpumask_backtrace+0xd7/0xe0 [Fri Sep 3 07:30:00 2021] rcu_dump_cpu_stacks+0xa2/0xd0 [Fri Sep 3 07:30:00 2021] rcu_sched_clock_irq.cold+0x1ff/0x3d6 [Fri Sep 3 07:30:00 2021] update_process_times+0x8c/0xc0 [Fri Sep 3 07:30:00 2021] tick_sched_handle+0x22/0x60 [Fri Sep 3 07:30:00 2021] tick_sched_timer+0x7c/0xb0 [Fri Sep 3 07:30:00 2021] ? tick_do_update_jiffies64.part.0+0xc0/0xc0 [Fri Sep 3 07:30:00 2021] __hrtimer_run_queues+0x12a/0x270 [Fri Sep 3 07:30:00 2021] hrtimer_interrupt+0x110/0x2c0 [Fri Sep 3 07:30:00 2021] __sysvec_apic_timer_interrupt+0x5f/0xd0 [Fri Sep 3 07:30:00 2021] asm_call_irq_on_stack+0x12/0x20 [Fri Sep 3 07:30:00 2021] </IRQ> [Fri Sep 3 07:30:00 2021] sysvec_apic_timer_interrupt+0x72/0x80 [Fri Sep 3 07:30:00 2021] asm_sysvec_apic_timer_interrupt+0x12/0x20 [Fri Sep 3 07:30:00 2021] RIP: 0010:native_queued_spin_lock_slowpath+0x1a2/0x1d0 [Fri Sep 3 07:30:00 2021] Code: 83 e0 03 83 ee 01 48 c1 e0 05 48 63 f6 48 05 80 c8 02 00 48 03 04 f5 00 89 97 a9 48 89 10 8b 42 08 85 c0 75 09 f3 90 8b 42 08 [Fri Sep 3 07:30:00 2021] RSP: 0018:ffffa90180533c30 EFLAGS: 00000246 [Fri Sep 3 07:30:00 2021] RAX: 0000000000000000 RBX: ffff89241ed21500 RCX: 00000000000c0000 [Fri Sep 3 07:30:00 2021] RDX: ffff892adf2ac880 RSI: 0000000000000003 RDI: ffff89241ed21568 [Fri Sep 3 07:30:00 2021] RBP: ffff8923e22fc800 R08: 00000000000c0000 R09: 00000000fffffffd [Fri Sep 3 07:30:00 2021] R10: 0000000000000000 R11: 0000000000000000 R12: ffff89241ed21568 [Fri Sep 3 07:30:00 2021] R13: ffffa90180533dc0 R14: ffff8923e22fc800 R15: 0000000000000000 [Fri Sep 3 07:30:00 2021] _raw_spin_lock+0x1a/0x20 [Fri Sep 3 07:30:00 2021] handle_stripe+0x57c/0x1fd0 [raid456] [Fri Sep 3 07:30:00 2021] handle_active_stripes.constprop.0+0x3af/0x590 [raid456] [Fri Sep 3 07:30:00 2021] raid5d+0x375/0x5a0 [raid456] [Fri Sep 3 07:30:00 2021] md_thread+0x94/0x160 [md_mod] [Fri Sep 3 07:30:00 2021] ? add_wait_queue_exclusive+0x70/0x70 [Fri Sep 3 07:30:00 2021] ? md_write_inc+0x50/0x50 [md_mod] [Fri Sep 3 07:30:00 2021] kthread+0x11b/0x140 [Fri Sep 3 07:30:00 2021] ? __kthread_bind_mask+0x60/0x60 [Fri Sep 3 07:30:00 2021] ret_from_fork+0x22/0x30 [Fri Sep 3 07:30:25 2021] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [md127_raid6:364] [Fri Sep 3 07:30:25 2021] Modules linked in: ipt_REJECT nf_reject_ipv4 xt_multiport nft_compat nft_counter nf_tables nfnetlink rfkill nls_ascii nls_cp437 vfa _hda_codec_hdmi snd_hda_intel snd_intel_dspcfg soundwire_intel intel_rapl_common soundwire_generic_allocation snd_soc_core x86_pkg_temp_thermal snd_compress s m_intel snd_pcm kvm snd_timer snd irqbypass iTCO_wdt intel_pmc_bxt soundcore mei_me mei rapl sg iTCO_vendor_support watchdog intel_cstate intel_uncore pcspkr jbd2 btrfs blake2b_generic dm_crypt dm_mod raid10 raid1 raid0 multipath linear raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_p hid [Fri Sep 3 07:30:25 2021] i915 crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel ghash_clmulni_intel i2c_algo_bit drm_kms_helper xhci_pci cec xhci bata raid_class cryptd glue_helper usbcore scsi_transport_sas i2c_i801 i2c_smbus scsi_mod ptp lpc_ich pps_core usb_common fan wmi video button [Fri Sep 3 07:30:25 2021] CPU: 2 PID: 364 Comm: md127_raid6 Not tainted 5.10.0-8-amd64 #1 Debian 5.10.46-4 [Fri Sep 3 07:30:25 2021] Hardware name: Gigabyte Technology Co., Ltd. Z87X-UD4H/Z87X-UD4H-CF, BIOS F9 03/18/2014 [Fri Sep 3 07:30:25 2021] RIP: 0010:native_queued_spin_lock_slowpath+0x1a2/0x1d0 [Fri Sep 3 07:30:25 2021] Code: 83 e0 03 83 ee 01 48 c1 e0 05 48 63 f6 48 05 80 c8 02 00 48 03 04 f5 00 89 97 a9 48 89 10 8b 42 08 85 c0 75 09 f3 90 8b 42 08 [Fri Sep 3 07:30:25 2021] RSP: 0018:ffffa90180533c30 EFLAGS: 00000246 [Fri Sep 3 07:30:25 2021] RAX: 0000000000000000 RBX: ffff89241ed21500 RCX: 00000000000c0000 [Fri Sep 3 07:30:25 2021] RDX: ffff892adf2ac880 RSI: 0000000000000003 RDI: ffff89241ed21568 [Fri Sep 3 07:30:25 2021] RBP: ffff8923e22fc800 R08: 00000000000c0000 R09: 00000000fffffffd [Fri Sep 3 07:30:25 2021] R10: 0000000000000000 R11: 0000000000000000 R12: ffff89241ed21568 [Fri Sep 3 07:30:25 2021] R13: ffffa90180533dc0 R14: ffff8923e22fc800 R15: 0000000000000000 [Fri Sep 3 07:30:25 2021] FS: 0000000000000000(0000) GS:ffff892adf280000(0000) knlGS:0000000000000000 [Fri Sep 3 07:30:25 2021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Sep 3 07:30:25 2021] CR2: 000055b5eb31cd98 CR3: 00000002c060a003 CR4: 00000000001706e0 [Fri Sep 3 07:30:25 2021] Call Trace: [Fri Sep 3 07:30:25 2021] _raw_spin_lock+0x1a/0x20 [Fri Sep 3 07:30:25 2021] handle_stripe+0x57c/0x1fd0 [raid456] [Fri Sep 3 07:30:25 2021] handle_active_stripes.constprop.0+0x3af/0x590 [raid456] [Fri Sep 3 07:30:25 2021] raid5d+0x375/0x5a0 [raid456] [Fri Sep 3 07:30:25 2021] md_thread+0x94/0x160 [md_mod] [Fri Sep 3 07:30:25 2021] ? add_wait_queue_exclusive+0x70/0x70 [Fri Sep 3 07:30:25 2021] ? md_write_inc+0x50/0x50 [md_mod] [Fri Sep 3 07:30:25 2021] kthread+0x11b/0x140 [Fri Sep 3 07:30:25 2021] ? __kthread_bind_mask+0x60/0x60 [Fri Sep 3 07:30:25 2021] ret_from_fork+0x22/0x30 [Fri Sep 3 07:30:53 2021] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [md127_raid6:364] [Fri Sep 3 07:30:53 2021] Modules linked in: ipt_REJECT nf_reject_ipv4 xt_multiport nft_compat nft_counter nf_tables nfnetlink rfkill nls_ascii nls_cp437 vfa _hda_codec_hdmi snd_hda_intel snd_intel_dspcfg soundwire_intel intel_rapl_common soundwire_generic_allocation snd_soc_core x86_pkg_temp_thermal snd_compress s m_intel snd_pcm kvm snd_timer snd irqbypass iTCO_wdt intel_pmc_bxt soundcore mei_me mei rapl sg iTCO_vendor_support watchdog intel_cstate intel_uncore pcspkr jbd2 btrfs blake2b_generic dm_crypt dm_mod raid10 raid1 raid0 multipath linear raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_p hid [Fri Sep 3 07:30:53 2021] i915 crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel ghash_clmulni_intel i2c_algo_bit drm_kms_helper xhci_pci cec xhci bata raid_class cryptd glue_helper usbcore scsi_transport_sas i2c_i801 i2c_smbus scsi_mod ptp lpc_ich pps_core usb_common fan wmi video button [Fri Sep 3 07:30:53 2021] CPU: 2 PID: 364 Comm: md127_raid6 Tainted: G L 5.10.0-8-amd64 #1 Debian 5.10.46-4 [Fri Sep 3 07:30:53 2021] Hardware name: Gigabyte Technology Co., Ltd. Z87X-UD4H/Z87X-UD4H-CF, BIOS F9 03/18/2014 [Fri Sep 3 07:30:53 2021] RIP: 0010:native_queued_spin_lock_slowpath+0x1a2/0x1d0 [Fri Sep 3 07:30:53 2021] Code: 83 e0 03 83 ee 01 48 c1 e0 05 48 63 f6 48 05 80 c8 02 00 48 03 04 f5 00 89 97 a9 48 89 10 8b 42 08 85 c0 75 09 f3 90 8b 42 08 [Fri Sep 3 07:30:53 2021] RSP: 0018:ffffa90180533c30 EFLAGS: 00000246 [Fri Sep 3 07:30:53 2021] RAX: 0000000000000000 RBX: ffff89241ed21500 RCX: 00000000000c0000 [Fri Sep 3 07:30:53 2021] RDX: ffff892adf2ac880 RSI: 0000000000000003 RDI: ffff89241ed21568 [Fri Sep 3 07:30:53 2021] RBP: ffff8923e22fc800 R08: 00000000000c0000 R09: 00000000fffffffd [Fri Sep 3 07:30:53 2021] R10: 0000000000000000 R11: 0000000000000000 R12: ffff89241ed21568 [Fri Sep 3 07:30:53 2021] R13: ffffa90180533dc0 R14: ffff8923e22fc800 R15: 0000000000000000 [Fri Sep 3 07:30:53 2021] FS: 0000000000000000(0000) GS:ffff892adf280000(0000) knlGS:0000000000000000 [Fri Sep 3 07:30:53 2021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Sep 3 07:30:53 2021] CR2: 000055b5eb31cd98 CR3: 00000002c060a003 CR4: 00000000001706e0 [Fri Sep 3 07:30:53 2021] Call Trace: [Fri Sep 3 07:30:53 2021] _raw_spin_lock+0x1a/0x20 [Fri Sep 3 07:30:53 2021] handle_stripe+0x57c/0x1fd0 [raid456] [Fri Sep 3 07:30:53 2021] handle_active_stripes.constprop.0+0x3af/0x590 [raid456] [Fri Sep 3 07:30:53 2021] raid5d+0x375/0x5a0 [raid456] [Fri Sep 3 07:30:53 2021] md_thread+0x94/0x160 [md_mod] [Fri Sep 3 07:30:53 2021] ? add_wait_queue_exclusive+0x70/0x70 [Fri Sep 3 07:30:53 2021] ? md_write_inc+0x50/0x50 [md_mod] [Fri Sep 3 07:30:53 2021] kthread+0x11b/0x140 [Fri Sep 3 07:30:53 2021] ? __kthread_bind_mask+0x60/0x60 [Fri Sep 3 07:30:53 2021] ret_from_fork+0x22/0x30 [Fri Sep 3 07:31:03 2021] rcu: INFO: rcu_sched self-detected stall on CPU [Fri Sep 3 07:31:03 2021] rcu: 2-....: (21002 ticks this GP) idle=0c2/1/0x4000000000000000 softirq=56412/56412 fqs=10500 [Fri Sep 3 07:31:03 2021] (t=21003 jiffies g=91613 q=27314) [Fri Sep 3 07:31:03 2021] NMI backtrace for cpu 2 [Fri Sep 3 07:31:03 2021] CPU: 2 PID: 364 Comm: md127_raid6 Tainted: G L 5.10.0-8-amd64 #1 Debian 5.10.46-4 [Fri Sep 3 07:31:03 2021] Hardware name: Gigabyte Technology Co., Ltd. Z87X-UD4H/Z87X-UD4H-CF, BIOS F9 03/18/2014 [Fri Sep 3 07:31:03 2021] Call Trace: [Fri Sep 3 07:31:03 2021] <IRQ> [Fri Sep 3 07:31:03 2021] dump_stack+0x6b/0x83 [Fri Sep 3 07:31:03 2021] nmi_cpu_backtrace.cold+0x32/0x69 [Fri Sep 3 07:31:03 2021] ? lapic_can_unplug_cpu+0x80/0x80 [Fri Sep 3 07:31:03 2021] nmi_trigger_cpumask_backtrace+0xd7/0xe0 [Fri Sep 3 07:31:03 2021] rcu_dump_cpu_stacks+0xa2/0xd0 [Fri Sep 3 07:31:03 2021] rcu_sched_clock_irq.cold+0x1ff/0x3d6 [Fri Sep 3 07:31:03 2021] update_process_times+0x8c/0xc0 [Fri Sep 3 07:31:03 2021] tick_sched_handle+0x22/0x60 [Fri Sep 3 07:31:03 2021] tick_sched_timer+0x7c/0xb0 [Fri Sep 3 07:31:03 2021] ? tick_do_update_jiffies64.part.0+0xc0/0xc0 [Fri Sep 3 07:31:03 2021] __hrtimer_run_queues+0x12a/0x270 [Fri Sep 3 07:31:03 2021] hrtimer_interrupt+0x110/0x2c0 [Fri Sep 3 07:31:03 2021] __sysvec_apic_timer_interrupt+0x5f/0xd0 [Fri Sep 3 07:31:03 2021] asm_call_irq_on_stack+0x12/0x20 [Fri Sep 3 07:31:03 2021] </IRQ> [Fri Sep 3 07:31:03 2021] sysvec_apic_timer_interrupt+0x72/0x80 [Fri Sep 3 07:31:03 2021] asm_sysvec_apic_timer_interrupt+0x12/0x20 [Fri Sep 3 07:31:03 2021] RIP: 0010:native_queued_spin_lock_slowpath+0x1a2/0x1d0 [Fri Sep 3 07:31:03 2021] Code: 83 e0 03 83 ee 01 48 c1 e0 05 48 63 f6 48 05 80 c8 02 00 48 03 04 f5 00 89 97 a9 48 89 10 8b 42 08 85 c0 75 09 f3 90 8b 42 08 [Fri Sep 3 07:31:03 2021] RSP: 0018:ffffa90180533c30 EFLAGS: 00000246 [Fri Sep 3 07:31:03 2021] RAX: 0000000000000000 RBX: ffff89241ed21500 RCX: 00000000000c0000 [Fri Sep 3 07:31:03 2021] RDX: ffff892adf2ac880 RSI: 0000000000000003 RDI: ffff89241ed21568 [Fri Sep 3 07:31:03 2021] RBP: ffff8923e22fc800 R08: 00000000000c0000 R09: 00000000fffffffd [Fri Sep 3 07:31:03 2021] R10: 0000000000000000 R11: 0000000000000000 R12: ffff89241ed21568 [Fri Sep 3 07:31:03 2021] R13: ffffa90180533dc0 R14: ffff8923e22fc800 R15: 0000000000000000 [Fri Sep 3 07:31:03 2021] _raw_spin_lock+0x1a/0x20 [Fri Sep 3 07:31:03 2021] handle_stripe+0x57c/0x1fd0 [raid456] [Fri Sep 3 07:31:03 2021] handle_active_stripes.constprop.0+0x3af/0x590 [raid456] [Fri Sep 3 07:31:03 2021] raid5d+0x375/0x5a0 [raid456] [Fri Sep 3 07:31:03 2021] md_thread+0x94/0x160 [md_mod] [Fri Sep 3 07:31:03 2021] ? add_wait_queue_exclusive+0x70/0x70 [Fri Sep 3 07:31:03 2021] ? md_write_inc+0x50/0x50 [md_mod] [Fri Sep 3 07:31:03 2021] kthread+0x11b/0x140 [Fri Sep 3 07:31:03 2021] ? __kthread_bind_mask+0x60/0x60 [Fri Sep 3 07:31:03 2021] ret_from_fork+0x22/0x30 [Fri Sep 3 07:31:29 2021] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [md127_raid6:364] [Fri Sep 3 07:31:29 2021] Modules linked in: ipt_REJECT nf_reject_ipv4 xt_multiport nft_compat nft_counter nf_tables nfnetlink rfkill nls_ascii nls_cp437 vfa _hda_codec_hdmi snd_hda_intel snd_intel_dspcfg soundwire_intel intel_rapl_common soundwire_generic_allocation snd_soc_core x86_pkg_temp_thermal snd_compress s m_intel snd_pcm kvm snd_timer snd irqbypass iTCO_wdt intel_pmc_bxt soundcore mei_me mei rapl sg iTCO_vendor_support watchdog intel_cstate intel_uncore pcspkr jbd2 btrfs blake2b_generic dm_crypt dm_mod raid10 raid1 raid0 multipath linear raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_p hid [Fri Sep 3 07:31:29 2021] i915 crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel ghash_clmulni_intel i2c_algo_bit drm_kms_helper xhci_pci cec xhci bata raid_class cryptd glue_helper usbcore scsi_transport_sas i2c_i801 i2c_smbus scsi_mod ptp lpc_ich pps_core usb_common fan wmi video button [Fri Sep 3 07:31:29 2021] CPU: 2 PID: 364 Comm: md127_raid6 Tainted: G L 5.10.0-8-amd64 #1 Debian 5.10.46-4 _____________ Ryan Patterson May the wings of liberty never lose a feather.