Latest net-next from GIT panic
From: Paweł Staszewski <hidden>
Date: 2017-09-19 22:38:12
Just tried latest net-next git and found kernel panic. Below link to bugzilla. https://bugzilla.kernel.org/attachment.cgi?id=258499
Latest net-next from GIT panic
29 messages, 4 authors, 2017-09-20 · open the first message on its own page
Re: Latest net-next from GIT panic
From: Paweł Staszewski <hidden>
Date: 2017-09-19 23:47:38
Added few more screenshoots from kernels 4.14-rc1(net-next) and 4.14-rc1(linux-next) https://bugzilla.kernel.org/show_bug.cgi?id=197005 W dniu 2017-09-20 o 00:35, Paweł Staszewski pisze:
Just tried latest net-next git and found kernel panic. Below link to bugzilla. https://bugzilla.kernel.org/attachment.cgi?id=258499
Re: Latest net-next from GIT panic
From: Paweł Staszewski <hidden>
Date: 2017-09-20 00:04:10
Some information about enviroment: Server is acting as a ip router with bgp There are 6x bgp sessions - each with full bgp table ~600k prefixes And it looks like panic is appearing after bgp sessions are connected - not by traffic - cause at time when panic occured there is almost no traffic. Also when I run tris server without turning on BGP and push thru this server traffic by pktgen there is no panic. just after it learn routes it panick W dniu 2017-09-20 o 01:45, Paweł Staszewski pisze:
Added few more screenshoots from kernels 4.14-rc1(net-next) and 4.14-rc1(linux-next) https://bugzilla.kernel.org/show_bug.cgi?id=197005 W dniu 2017-09-20 o 00:35, Paweł Staszewski pisze:quoted
Just tried latest net-next git and found kernel panic. Below link to bugzilla. https://bugzilla.kernel.org/attachment.cgi?id=258499
Re: Latest net-next from GIT panic
From: Paweł Staszewski <hidden>
Date: 2017-09-20 00:08:43
Just checked kernel 4.13.2 and same problem Just after start all 6 bgp sessions - and kernel starts to learn routes it panic. https://bugzilla.kernel.org/attachment.cgi?id=258509 W dniu 2017-09-20 o 02:01, Paweł Staszewski pisze:
Some information about enviroment: Server is acting as a ip router with bgp There are 6x bgp sessions - each with full bgp table ~600k prefixes And it looks like panic is appearing after bgp sessions are connected - not by traffic - cause at time when panic occured there is almost no traffic. Also when I run tris server without turning on BGP and push thru this server traffic by pktgen there is no panic. just after it learn routes it panick W dniu 2017-09-20 o 01:45, Paweł Staszewski pisze:quoted
Added few more screenshoots from kernels 4.14-rc1(net-next) and 4.14-rc1(linux-next) https://bugzilla.kernel.org/show_bug.cgi?id=197005 W dniu 2017-09-20 o 00:35, Paweł Staszewski pisze:quoted
Just tried latest net-next git and found kernel panic. Below link to bugzilla. https://bugzilla.kernel.org/attachment.cgi?id=258499
Re: Latest net-next from GIT panic
From: Paweł Staszewski <hidden>
Date: 2017-09-20 00:29:00
Latest working kernel with same configuration and kernel config 4.12.13 There is no panic after routes from all 6x bgp sessions are learned. ip r | wc -l 653112 W dniu 2017-09-20 o 02:06, Paweł Staszewski pisze:
Just checked kernel 4.13.2 and same problem Just after start all 6 bgp sessions - and kernel starts to learn routes it panic. https://bugzilla.kernel.org/attachment.cgi?id=258509 W dniu 2017-09-20 o 02:01, Paweł Staszewski pisze:quoted
Some information about enviroment: Server is acting as a ip router with bgp There are 6x bgp sessions - each with full bgp table ~600k prefixes And it looks like panic is appearing after bgp sessions are connected - not by traffic - cause at time when panic occured there is almost no traffic. Also when I run tris server without turning on BGP and push thru this server traffic by pktgen there is no panic. just after it learn routes it panick W dniu 2017-09-20 o 01:45, Paweł Staszewski pisze:quoted
Added few more screenshoots from kernels 4.14-rc1(net-next) and 4.14-rc1(linux-next) https://bugzilla.kernel.org/show_bug.cgi?id=197005 W dniu 2017-09-20 o 00:35, Paweł Staszewski pisze:quoted
Just tried latest net-next git and found kernel panic. Below link to bugzilla. https://bugzilla.kernel.org/attachment.cgi?id=258499
Re: Latest net-next from GIT panic
From: Eric Dumazet <hidden>
Date: 2017-09-20 03:24:32
On Wed, 2017-09-20 at 02:06 +0200, Paweł Staszewski wrote:
Just checked kernel 4.13.2 and same problem Just after start all 6 bgp sessions - and kernel starts to learn routes it panic. https://bugzilla.kernel.org/attachment.cgi?id=258509
Unfortunately we have not enough information from these traces. Can you get a full stack trace ? Alternatively, can you bisect ? Thanks.
Re: Latest net-next from GIT panic
From: Paweł Staszewski <hidden>
Date: 2017-09-20 08:01:03
Hi Will try bisecting tonight W dniu 2017-09-20 o 05:24, Eric Dumazet pisze:
On Wed, 2017-09-20 at 02:06 +0200, Paweł Staszewski wrote:quoted
Just checked kernel 4.13.2 and same problem Just after start all 6 bgp sessions - and kernel starts to learn routes it panic. https://bugzilla.kernel.org/attachment.cgi?id=258509Unfortunately we have not enough information from these traces. Can you get a full stack trace ? Alternatively, can you bisect ? Thanks.
Re: Latest net-next from GIT panic
From: Paweł Staszewski <hidden>
Date: 2017-09-20 08:47:01
Trying to make video from ipmi :) with that results: https://bugzilla.kernel.org/attachment.cgi?id=258521 catched two more lines where it starts - panic from 4.13.2. Now will try tro do some bisection W dniu 2017-09-20 o 09:58, Paweł Staszewski pisze:
Hi Will try bisecting tonight W dniu 2017-09-20 o 05:24, Eric Dumazet pisze:quoted
On Wed, 2017-09-20 at 02:06 +0200, Paweł Staszewski wrote:quoted
Just checked kernel 4.13.2 and same problem Just after start all 6 bgp sessions - and kernel starts to learn routes it panic. https://bugzilla.kernel.org/attachment.cgi?id=258509Unfortunately we have not enough information from these traces. Can you get a full stack trace ? Alternatively, can you bisect ? Thanks.
Re: Latest net-next from GIT panic
From: Paweł Staszewski <hidden>
Date: 2017-09-20 09:47:53
Ok looks like ending bisection Latest bisected kernel when there is no kernel panic 4.12.0+ (from next) - but only this warning: [ 309.030019] NETDEV WATCHDOG: enp4s0f0 (ixgbe): transmit queue 0 timed out [ 309.030034] ------------[ cut here ]------------ [ 309.030040] WARNING: CPU: 35 PID: 0 at dev_watchdog+0xcf/0x139 [ 309.030041] Modules linked in: bonding ipmi_si x86_pkg_temp_thermal [ 309.030045] CPU: 35 PID: 0 Comm: swapper/35 Not tainted 4.12.0+ #5 [ 309.030046] task: ffff88086d98a000 task.stack: ffffc90003378000 [ 309.030048] RIP: 0010:dev_watchdog+0xcf/0x139 [ 309.030049] RSP: 0018:ffff88087fbc3ea8 EFLAGS: 00010246 [ 309.030050] RAX: 000000000000003d RBX: ffff88046b680000 RCX: 0000000000000000 [ 309.030050] RDX: ffff88087fbd2f01 RSI: 0000000000000000 RDI: ffff88087fbcda08 [ 309.030051] RBP: ffff88087fbc3eb8 R08: 0000000000000000 R09: ffff88087ff80a04 [ 309.030051] R10: 0000000000000000 R11: ffff88086d98a001 R12: 0000000000000000 [ 309.030052] R13: ffff88087fbc3ef8 R14: ffff88086d98a000 R15: ffffffff81c06008 [ 309.030053] FS: 0000000000000000(0000) GS:ffff88087fbc0000(0000) knlGS:0000000000000000 [ 309.030054] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 309.030054] CR2: 00007fba600f6098 CR3: 000000086b955000 CR4: 00000000001406e0 [ 309.030055] Call Trace: [ 309.030057] <IRQ> [ 309.030059] ? netif_tx_lock+0x79/0x79 [ 309.030062] call_timer_fn.isra.24+0x17/0x77 [ 309.030063] run_timer_softirq+0x118/0x161 [ 309.030065] ? netif_tx_lock+0x79/0x79 [ 309.030066] ? ktime_get+0x2b/0x42 [ 309.030070] ? lapic_next_deadline+0x21/0x27 [ 309.030073] ? clockevents_program_event+0xa8/0xc5 [ 309.030076] __do_softirq+0xa8/0x19d [ 309.030078] irq_exit+0x5d/0x6b [ 309.030079] smp_apic_timer_interrupt+0x2a/0x36 [ 309.030082] apic_timer_interrupt+0x89/0x90 [ 309.030085] RIP: 0010:mwait_idle+0x4e/0x6a [ 309.030086] RSP: 0018:ffffc9000337be98 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff10 [ 309.030087] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 [ 309.030087] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff88086d98a000 [ 309.030088] RBP: ffffc9000337be98 R08: ffff88046f8279a0 R09: ffff88046f827040 [ 309.030089] R10: ffff88086d98a000 R11: ffff88086d98a000 R12: 0000000000000000 [ 309.030089] R13: ffff88086d98a000 R14: ffff88086d98a000 R15: ffff88086d98a000 [ 309.030090] </IRQ> [ 309.030094] arch_cpu_idle+0xa/0xc [ 309.030095] default_idle_call+0x19/0x1b [ 309.030102] do_idle+0xbc/0x196 [ 309.030104] cpu_startup_entry+0x1d/0x20 [ 309.030105] start_secondary+0xd8/0xdc [ 309.030108] secondary_startup_64+0x9f/0x9f [ 309.030109] Code: cc 75 bd eb 35 48 89 df c6 05 c3 dc 74 00 01 e8 3a 62 fe ff 44 89 e1 48 89 de 48 89 c2 48 c7 c7 0f 65 a4 81 31 c0 e8 3d 4c b5 ff <0f> ff 48 8b 83 e0 01 00 00 48 89 df ff 50 78 48 8b 05 a0 bc 6a [ 309.030128] ---[ end trace 9102cb25703ae2d9 ]--- I just marked it as good - cause this problem above is differend - and im going to: git bisect good Bisecting: 1787 revisions left to test after this (roughly 11 steps) W dniu 2017-09-20 o 10:44, Paweł Staszewski pisze:
Trying to make video from ipmi :) with that results: https://bugzilla.kernel.org/attachment.cgi?id=258521 catched two more lines where it starts - panic from 4.13.2. Now will try tro do some bisection W dniu 2017-09-20 o 09:58, Paweł Staszewski pisze:quoted
Hi Will try bisecting tonight W dniu 2017-09-20 o 05:24, Eric Dumazet pisze:quoted
On Wed, 2017-09-20 at 02:06 +0200, Paweł Staszewski wrote:quoted
Just checked kernel 4.13.2 and same problem Just after start all 6 bgp sessions - and kernel starts to learn routes it panic. https://bugzilla.kernel.org/attachment.cgi?id=258509Unfortunately we have not enough information from these traces. Can you get a full stack trace ? Alternatively, can you bisect ? Thanks.
Re: Latest net-next from GIT panic
From: Paweł Staszewski <hidden>
Date: 2017-09-20 10:23:41
Ok kernel crashed with different panic that i didnt catch when i was doing bisect and now my bisection is broken :) git bisect good Bisecting: 1787 revisions left to test after this (roughly 11 steps) error: Your local changes to the following files would be overwritten by checkout: Documentation/00-INDEX Documentation/ABI/stable/sysfs-class-udc Documentation/ABI/testing/configfs-usb-gadget-uac1 Documentation/ABI/testing/ima_policy Documentation/ABI/testing/sysfs-bus-iio Documentation/ABI/testing/sysfs-bus-iio-meas-spec Documentation/ABI/testing/sysfs-bus-iio-timer-stm32 Documentation/ABI/testing/sysfs-class-net Documentation/ABI/testing/sysfs-class-power-twl4030 Documentation/ABI/testing/sysfs-class-typec Documentation/DMA-API.txt Documentation/IRQ-domain.txt Documentation/Makefile Documentation/PCI/MSI-HOWTO.txt Documentation/RCU/00-INDEX Documentation/RCU/Design/Requirements/Requirements.html Documentation/RCU/checklist.txt Documentation/admin-guide/README.rst Documentation/admin-guide/devices.txt Documentation/admin-guide/index.rst Documentation/admin-guide/kernel-parameters.txt Documentation/admin-guide/pm/cpufreq.rst Documentation/admin-guide/pm/intel_pstate.rst Documentation/admin-guide/ras.rst Documentation/arm/Atmel/README Documentation/block/biodoc.txt Documentation/conf.py Documentation/core-api/assoc_array.rst Documentation/core-api/atomic_ops.rst Documentation/core-api/index.rst Documentation/crypto/asymmetric-keys.txt Documentation/dev-tools/index.rst Documentation/dev-tools/sparse.rst Documentation/devicetree/bindings/arm/amlogic.txt Documentation/devicetree/bindings/arm/atmel-at91.txt Documentation/devicetree/bindings/arm/ccn.txt Documentation/devicetree/bindings/arm/cpus.txt Documentation/devicetree/bindings/arm/gemini.txt Documentation/devicetree/bindings/arm/hisilicon/hisilicon.txt Documentation/devicetree/bindings/arm/keystone/keystone.txt Documentation/devicetree/bindings/arm/mediatek.txt Documentation/devicetree/bindings/arm/rockchip.txt Documentation/devicetree/bindings/arm/shmobile.txt Documentation/devicetree/bindings/arm/tegra.txt Documentation/devicetree/bindings/ata/ahci-fsl-qoriq.txt Documentation/devicetree/bindings/bus/brcm,gisb-arb.txt Documentation/devicetree/bindings/clock/brcm,iproc-clocks.txt Documentation/devicetree/bindings/cpufreq/ti-cpufreq.txt Documentation/devicetree/bindings/gpio/gpio_atmel.txt Documentation/devicetree/bindings/iio/adc/amlogic,meson-saradc.txt Documentation/devicetree/bindings/iio/adc/renesas,gyroadc.txt Documentation/devicetree/bindings/iio/adc/st,stm32-adc.txt Documentation/devicetree/bindings/iio/imu/st_lsm6dsx.txt Documentation/devicetree/bindings/interrupt-controller/allwinner,sunxi-nmi.txt Documentation/devicetree/bindings/interrupt-controller/aspeed,ast2400-vic.txt Documentation/devicetree/bindings/interrupt-controller/mediatek,sysirq.txt Documentation/devicetree/bindings/leds/common.txt Documentation/devicetree/bindings/mfd/hi6421.txt Documentation/devicetree/bindings/mfd/tps65910.txt Documentation/devicetree/bindings/mmc/fsl-esdhc.txt Documentation/devicetree/bindings/mmc/k3-dw-mshc.txt Documentation/devicetree/bindings/mmc/rockchip-dw-mshc.txt Documentation/devicetree/bindings/mmc/ti-omap-hsmmc.txt Documentation/devicetree/bindings/mtd/atmel-nand.txt Documentation/devicetree/bindings/net/dsa/b53.txt Documentation/devicetree/bindings/net/ethernet.txt Documentation/devicetree/bindings/net/macb.txt Documentation/devicetree/bindings/net/marvell-orion-mdio.txt Documentation/devicetree/bindings/net/ti,wilink-st.txt Documentation/devicetree/bindings/net/wireless/ti,wlcore.txt Documentation/devicetree/bindings/nvmem/rockchip-efuse.txt Documentation/devicetree/bindings/opp/opp.txt Documentation/devicetree/bindings/phy/bcm-ns-usb3-phy.txt Documentation/devicetree/bindings/phy/brcm-sata-phy.txt Documentation/devicetree/bindings/phy/meson8b-usb2-phy.txt Documentation/devicetree/bindings/phy/phy-rockchip-inno-usb2.txt Documentation/devicetree/bindings/power/rockchip-io-domain.txt Documentation/devicetree/bindings/power/supply/bq27xxx.txt Documentation/devicetree/bindings/property-units.txt Documentation/devicetree/bindings/regulator/regulator.txt Documentation/devicetree/bindings/serial/8 error: The following untracked working tree files would be overwritten by checkout: Documentation/ABI/testing/sysfs-class-net-phydev Documentation/DocBook/.gitignore Documentation/DocBook/Makefile Documentation/DocBook/filesystems.tmpl Documentation/DocBook/kernel-hacking.tmpl Documentation/DocBook/kernel-locking.tmpl Documentation/DocBook/kgdb.tmpl Documentation/DocBook/libata.tmpl Documentation/DocBook/librs.tmpl Documentation/DocBook/lsm.tmpl Documentation/DocBook/mtdnand.tmpl Documentation/DocBook/networking.tmpl Documentation/DocBook/rapidio.tmpl Documentation/DocBook/s390-drivers.tmpl Documentation/DocBook/scsi.tmpl Documentation/DocBook/sh.tmpl Documentation/DocBook/stylesheet.xsl Documentation/DocBook/w1.tmpl Documentation/DocBook/z8530book.tmpl Documentation/Makefile.sphinx Documentation/RCU/trace.txt Documentation/devicetree/bindings/i2c/i2c-mt6577.txt Documentation/devicetree/bindings/misc/allwinner,syscon.txt Documentation/devicetree/bindings/net/cortina.txt Documentation/devicetree/bindings/net/dsa/ksz.txt Documentation/devicetree/bindings/net/dwmac-sun8i.txt Documentation/devicetree/bindings/net/qca,qca7000.txt Documentation/devicetree/bindings/power/max8903-charger.txt Documentation/devicetree/bindings/power_supply/maxim,max14656.txt Documentation/devicetree/bindings/ptp/brcm,ptp-dte.txt Documentation/devicetree/bindings/timer/moxa,moxart-timer.txt Documentation/doc-guide/docbook.rst Documentation/networking/tls.txt Documentation/prctl/no_new_privs.txt Documentation/prctl/seccomp_filter.txt Documentation/security/00-INDEX Documentation/security/IMA-templates.txt Documentation/security/LSM.txt Documentation/security/LoadPin.txt Documentation/security/SELinux.txt Documentation/security/Smack.txt Documentation/security/Yama.txt Documentation/security/apparmor.txt Documentation/security/conf.py Documentation/security/credentials.txt Documentation/security/keys-ecryptfs.txt Documentation/security/keys-request-key.txt Documentation/security/keys-trusted-encrypted.txt Documentation/security/keys.txt Documentation/security/self-protection.txt Documentation/security/tomoyo.txt Documentation/sphinx/convert_template.sed Documentation/sphinx/post_convert.sed Documentation/sphinx/tmplcvt Documentation/usb/typec.rst Documentation/usb/usb3-debug-port.rst arch/arm/boot/dts/rk1108-evb.dts arch/arm/boot/dts/rk1108.dtsi arch/arm/boot/dts/tegra20-whistler.dts arch/arm/mach-omap2/opp.c arch/arm/mach-omap2/pmu.c arch/ia64/include/asm/siginfo.h arch/m32r/include/uapi/asm/siginfo.h arch/microblaze/include/asm/bitops.h arch/microblaze/include/asm/bug.h arch/microblaze/include/asm/bugs.h arch/microblaze/include/asm/div64.h arch/microblaze/include/asm/emergency-restart.h arch/microblaze/include/asm/fb.h arch/microblaze/include/asm/hardirq.h arch/microblaze/include/asm/irq_regs.h arch/microblaze/include/asm/kdebug.h arch/microblaze/include/asm/kmap_types.h arch/microblaze/include/asm/linkage.h arch/microblaze/include/asm/local.h arch/microblaze/include/asm/local64.h arch/microblaze/include/asm/parport.h arch/microblaze/include/asm/percpu.h arch/microblaze/include/asm/serial.h arch/microblaze/include/asm/shmparam.h arch/microblaze/include/asm/topology.h arch/microblaze/include/asm/ucontext.h arch/microblaze/include/asm/vga.h arch/microblaze/include/asm/xor.h arch/microblaze/include/uapi/asm/bitsperlong.h arch/microblaze/include/uapi/asm/errno.h arch/microblaze/include/uapi/asm/fcntl.h arch/microblaze/include/uapi/asm/ioctl.h arch/microblaze/include/uapi/asm/ioctls.h arch/microblaze/include/uapi/asm/ipcbuf.h arch/microblaze/include/uapi/asm/kvm_para.h arch/microblaze/include/uapi/asm/mman.h arch/microblaze/include/uapi/asm/msgbuf.h arch/microblaze/include/uapi/asm/param.h arch/microblaze/include/uapi/asm/poll.h arch/microblaze/include/uapi/asm/resource.h arch/microblaze/include/uapi/asm/sembuf.h arch/microblaze/include/uapi/asm/shmbuf.h arch/microblaze/include/uapi/asm/siginfo.h arch/microblaze/include/uapi/asm/signal.h arch/microblaze/includ Aborting W dniu 2017-09-20 o 11:45, Paweł Staszewski pisze:
Ok looks like ending bisection Latest bisected kernel when there is no kernel panic 4.12.0+ (from next) - but only this warning: [ 309.030019] NETDEV WATCHDOG: enp4s0f0 (ixgbe): transmit queue 0 timed out [ 309.030034] ------------[ cut here ]------------ [ 309.030040] WARNING: CPU: 35 PID: 0 at dev_watchdog+0xcf/0x139 [ 309.030041] Modules linked in: bonding ipmi_si x86_pkg_temp_thermal [ 309.030045] CPU: 35 PID: 0 Comm: swapper/35 Not tainted 4.12.0+ #5 [ 309.030046] task: ffff88086d98a000 task.stack: ffffc90003378000 [ 309.030048] RIP: 0010:dev_watchdog+0xcf/0x139 [ 309.030049] RSP: 0018:ffff88087fbc3ea8 EFLAGS: 00010246 [ 309.030050] RAX: 000000000000003d RBX: ffff88046b680000 RCX: 0000000000000000 [ 309.030050] RDX: ffff88087fbd2f01 RSI: 0000000000000000 RDI: ffff88087fbcda08 [ 309.030051] RBP: ffff88087fbc3eb8 R08: 0000000000000000 R09: ffff88087ff80a04 [ 309.030051] R10: 0000000000000000 R11: ffff88086d98a001 R12: 0000000000000000 [ 309.030052] R13: ffff88087fbc3ef8 R14: ffff88086d98a000 R15: ffffffff81c06008 [ 309.030053] FS: 0000000000000000(0000) GS:ffff88087fbc0000(0000) knlGS:0000000000000000 [ 309.030054] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 309.030054] CR2: 00007fba600f6098 CR3: 000000086b955000 CR4: 00000000001406e0 [ 309.030055] Call Trace: [ 309.030057] <IRQ> [ 309.030059] ? netif_tx_lock+0x79/0x79 [ 309.030062] call_timer_fn.isra.24+0x17/0x77 [ 309.030063] run_timer_softirq+0x118/0x161 [ 309.030065] ? netif_tx_lock+0x79/0x79 [ 309.030066] ? ktime_get+0x2b/0x42 [ 309.030070] ? lapic_next_deadline+0x21/0x27 [ 309.030073] ? clockevents_program_event+0xa8/0xc5 [ 309.030076] __do_softirq+0xa8/0x19d [ 309.030078] irq_exit+0x5d/0x6b [ 309.030079] smp_apic_timer_interrupt+0x2a/0x36 [ 309.030082] apic_timer_interrupt+0x89/0x90 [ 309.030085] RIP: 0010:mwait_idle+0x4e/0x6a [ 309.030086] RSP: 0018:ffffc9000337be98 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff10 [ 309.030087] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 [ 309.030087] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff88086d98a000 [ 309.030088] RBP: ffffc9000337be98 R08: ffff88046f8279a0 R09: ffff88046f827040 [ 309.030089] R10: ffff88086d98a000 R11: ffff88086d98a000 R12: 0000000000000000 [ 309.030089] R13: ffff88086d98a000 R14: ffff88086d98a000 R15: ffff88086d98a000 [ 309.030090] </IRQ> [ 309.030094] arch_cpu_idle+0xa/0xc [ 309.030095] default_idle_call+0x19/0x1b [ 309.030102] do_idle+0xbc/0x196 [ 309.030104] cpu_startup_entry+0x1d/0x20 [ 309.030105] start_secondary+0xd8/0xdc [ 309.030108] secondary_startup_64+0x9f/0x9f [ 309.030109] Code: cc 75 bd eb 35 48 89 df c6 05 c3 dc 74 00 01 e8 3a 62 fe ff 44 89 e1 48 89 de 48 89 c2 48 c7 c7 0f 65 a4 81 31 c0 e8 3d 4c b5 ff <0f> ff 48 8b 83 e0 01 00 00 48 89 df ff 50 78 48 8b 05 a0 bc 6a [ 309.030128] ---[ end trace 9102cb25703ae2d9 ]--- I just marked it as good - cause this problem above is differend - and im going to: git bisect good Bisecting: 1787 revisions left to test after this (roughly 11 steps) W dniu 2017-09-20 o 10:44, Paweł Staszewski pisze:quoted
Trying to make video from ipmi :) with that results: https://bugzilla.kernel.org/attachment.cgi?id=258521 catched two more lines where it starts - panic from 4.13.2. Now will try tro do some bisection W dniu 2017-09-20 o 09:58, Paweł Staszewski pisze:quoted
Hi Will try bisecting tonight W dniu 2017-09-20 o 05:24, Eric Dumazet pisze:quoted
On Wed, 2017-09-20 at 02:06 +0200, Paweł Staszewski wrote:quoted
Just checked kernel 4.13.2 and same problem Just after start all 6 bgp sessions - and kernel starts to learn routes it panic. https://bugzilla.kernel.org/attachment.cgi?id=258509Unfortunately we have not enough information from these traces. Can you get a full stack trace ? Alternatively, can you bisect ? Thanks.
Re: Latest net-next from GIT panic
From: Paweł Staszewski <hidden>
Date: 2017-09-20 10:25:02
Soo far bisected and marked: git bisect start # bad: [07dd6cc1fff160143e82cf5df78c1db0b6e03355] Linux 4.13.2 git bisect bad 07dd6cc1fff160143e82cf5df78c1db0b6e03355 # good: [5d7d2e03e0f01a992e3521b180c3d3e67905f269] Linux 4.12.13 git bisect good 5d7d2e03e0f01a992e3521b180c3d3e67905f269 # good: [6f7da290413ba713f0cdd9ff1a2a9bb129ef4f6c] Linux 4.12 git bisect good 6f7da290413ba713f0cdd9ff1a2a9bb129ef4f6c # bad: [ac7b75966c9c86426b55fe1c50ae148aa4571075] Merge tag 'pinctrl-v4.13-1' of git://git.kernel.org/pub/scm/linux/kernel/git/linusw/linux-pinctrl git bisect bad ac7b75966c9c86426b55fe1c50ae148aa4571075 # good: [e24dd9ee5399747b71c1d982a484fc7601795f31] Merge branch 'next' of git://git.kernel.org/pub/scm/linux/kernel/git/jmorris/linux-security git bisect good e24dd9ee5399747b71c1d982a484fc7601795f31 # good: [e24dd9ee5399747b71c1d982a484fc7601795f31] Merge branch 'next' of git://git.kernel.org/pub/scm/linux/kernel/git/jmorris/linux-security git bisect good e24dd9ee5399747b71c1d982a484fc7601795f31 # good: [e24dd9ee5399747b71c1d982a484fc7601795f31] Merge branch 'next' of git://git.kernel.org/pub/scm/linux/kernel/git/jmorris/linux-security git bisect good e24dd9ee5399747b71c1d982a484fc7601795f31 W dniu 2017-09-20 o 12:21, Paweł Staszewski pisze:
Ok kernel crashed with different panic that i didnt catch when i was doing bisect and now my bisection is broken :) git bisect good Bisecting: 1787 revisions left to test after this (roughly 11 steps) error: Your local changes to the following files would be overwritten by checkout: Documentation/00-INDEX Documentation/ABI/stable/sysfs-class-udc Documentation/ABI/testing/configfs-usb-gadget-uac1 Documentation/ABI/testing/ima_policy Documentation/ABI/testing/sysfs-bus-iio Documentation/ABI/testing/sysfs-bus-iio-meas-spec Documentation/ABI/testing/sysfs-bus-iio-timer-stm32 Documentation/ABI/testing/sysfs-class-net Documentation/ABI/testing/sysfs-class-power-twl4030 Documentation/ABI/testing/sysfs-class-typec Documentation/DMA-API.txt Documentation/IRQ-domain.txt Documentation/Makefile Documentation/PCI/MSI-HOWTO.txt Documentation/RCU/00-INDEX Documentation/RCU/Design/Requirements/Requirements.html Documentation/RCU/checklist.txt Documentation/admin-guide/README.rst Documentation/admin-guide/devices.txt Documentation/admin-guide/index.rst Documentation/admin-guide/kernel-parameters.txt Documentation/admin-guide/pm/cpufreq.rst Documentation/admin-guide/pm/intel_pstate.rst Documentation/admin-guide/ras.rst Documentation/arm/Atmel/README Documentation/block/biodoc.txt Documentation/conf.py Documentation/core-api/assoc_array.rst Documentation/core-api/atomic_ops.rst Documentation/core-api/index.rst Documentation/crypto/asymmetric-keys.txt Documentation/dev-tools/index.rst Documentation/dev-tools/sparse.rst Documentation/devicetree/bindings/arm/amlogic.txt Documentation/devicetree/bindings/arm/atmel-at91.txt Documentation/devicetree/bindings/arm/ccn.txt Documentation/devicetree/bindings/arm/cpus.txt Documentation/devicetree/bindings/arm/gemini.txt Documentation/devicetree/bindings/arm/hisilicon/hisilicon.txt Documentation/devicetree/bindings/arm/keystone/keystone.txt Documentation/devicetree/bindings/arm/mediatek.txt Documentation/devicetree/bindings/arm/rockchip.txt Documentation/devicetree/bindings/arm/shmobile.txt Documentation/devicetree/bindings/arm/tegra.txt Documentation/devicetree/bindings/ata/ahci-fsl-qoriq.txt Documentation/devicetree/bindings/bus/brcm,gisb-arb.txt Documentation/devicetree/bindings/clock/brcm,iproc-clocks.txt Documentation/devicetree/bindings/cpufreq/ti-cpufreq.txt Documentation/devicetree/bindings/gpio/gpio_atmel.txt Documentation/devicetree/bindings/iio/adc/amlogic,meson-saradc.txt Documentation/devicetree/bindings/iio/adc/renesas,gyroadc.txt Documentation/devicetree/bindings/iio/adc/st,stm32-adc.txt Documentation/devicetree/bindings/iio/imu/st_lsm6dsx.txt Documentation/devicetree/bindings/interrupt-controller/allwinner,sunxi-nmi.txt Documentation/devicetree/bindings/interrupt-controller/aspeed,ast2400-vic.txt Documentation/devicetree/bindings/interrupt-controller/mediatek,sysirq.txt Documentation/devicetree/bindings/leds/common.txt Documentation/devicetree/bindings/mfd/hi6421.txt Documentation/devicetree/bindings/mfd/tps65910.txt Documentation/devicetree/bindings/mmc/fsl-esdhc.txt Documentation/devicetree/bindings/mmc/k3-dw-mshc.txt Documentation/devicetree/bindings/mmc/rockchip-dw-mshc.txt Documentation/devicetree/bindings/mmc/ti-omap-hsmmc.txt Documentation/devicetree/bindings/mtd/atmel-nand.txt Documentation/devicetree/bindings/net/dsa/b53.txt Documentation/devicetree/bindings/net/ethernet.txt Documentation/devicetree/bindings/net/macb.txt Documentation/devicetree/bindings/net/marvell-orion-mdio.txt Documentation/devicetree/bindings/net/ti,wilink-st.txt Documentation/devicetree/bindings/net/wireless/ti,wlcore.txt Documentation/devicetree/bindings/nvmem/rockchip-efuse.txt Documentation/devicetree/bindings/opp/opp.txt Documentation/devicetree/bindings/phy/bcm-ns-usb3-phy.txt Documentation/devicetree/bindings/phy/brcm-sata-phy.txt Documentation/devicetree/bindings/phy/meson8b-usb2-phy.txt Documentation/devicetree/bindings/phy/phy-rockchip-inno-usb2.txt Documentation/devicetree/bindings/power/rockchip-io-domain.txt Documentation/devicetree/bindings/power/supply/bq27xxx.txt Documentation/devicetree/bindings/property-units.txt Documentation/devicetree/bindings/regulator/regulator.txt Documentation/devicetree/bindings/serial/8 error: The following untracked working tree files would be overwritten by checkout: Documentation/ABI/testing/sysfs-class-net-phydev Documentation/DocBook/.gitignore Documentation/DocBook/Makefile Documentation/DocBook/filesystems.tmpl Documentation/DocBook/kernel-hacking.tmpl Documentation/DocBook/kernel-locking.tmpl Documentation/DocBook/kgdb.tmpl Documentation/DocBook/libata.tmpl Documentation/DocBook/librs.tmpl Documentation/DocBook/lsm.tmpl Documentation/DocBook/mtdnand.tmpl Documentation/DocBook/networking.tmpl Documentation/DocBook/rapidio.tmpl Documentation/DocBook/s390-drivers.tmpl Documentation/DocBook/scsi.tmpl Documentation/DocBook/sh.tmpl Documentation/DocBook/stylesheet.xsl Documentation/DocBook/w1.tmpl Documentation/DocBook/z8530book.tmpl Documentation/Makefile.sphinx Documentation/RCU/trace.txt Documentation/devicetree/bindings/i2c/i2c-mt6577.txt Documentation/devicetree/bindings/misc/allwinner,syscon.txt Documentation/devicetree/bindings/net/cortina.txt Documentation/devicetree/bindings/net/dsa/ksz.txt Documentation/devicetree/bindings/net/dwmac-sun8i.txt Documentation/devicetree/bindings/net/qca,qca7000.txt Documentation/devicetree/bindings/power/max8903-charger.txt Documentation/devicetree/bindings/power_supply/maxim,max14656.txt Documentation/devicetree/bindings/ptp/brcm,ptp-dte.txt Documentation/devicetree/bindings/timer/moxa,moxart-timer.txt Documentation/doc-guide/docbook.rst Documentation/networking/tls.txt Documentation/prctl/no_new_privs.txt Documentation/prctl/seccomp_filter.txt Documentation/security/00-INDEX Documentation/security/IMA-templates.txt Documentation/security/LSM.txt Documentation/security/LoadPin.txt Documentation/security/SELinux.txt Documentation/security/Smack.txt Documentation/security/Yama.txt Documentation/security/apparmor.txt Documentation/security/conf.py Documentation/security/credentials.txt Documentation/security/keys-ecryptfs.txt Documentation/security/keys-request-key.txt Documentation/security/keys-trusted-encrypted.txt Documentation/security/keys.txt Documentation/security/self-protection.txt Documentation/security/tomoyo.txt Documentation/sphinx/convert_template.sed Documentation/sphinx/post_convert.sed Documentation/sphinx/tmplcvt Documentation/usb/typec.rst Documentation/usb/usb3-debug-port.rst arch/arm/boot/dts/rk1108-evb.dts arch/arm/boot/dts/rk1108.dtsi arch/arm/boot/dts/tegra20-whistler.dts arch/arm/mach-omap2/opp.c arch/arm/mach-omap2/pmu.c arch/ia64/include/asm/siginfo.h arch/m32r/include/uapi/asm/siginfo.h arch/microblaze/include/asm/bitops.h arch/microblaze/include/asm/bug.h arch/microblaze/include/asm/bugs.h arch/microblaze/include/asm/div64.h arch/microblaze/include/asm/emergency-restart.h arch/microblaze/include/asm/fb.h arch/microblaze/include/asm/hardirq.h arch/microblaze/include/asm/irq_regs.h arch/microblaze/include/asm/kdebug.h arch/microblaze/include/asm/kmap_types.h arch/microblaze/include/asm/linkage.h arch/microblaze/include/asm/local.h arch/microblaze/include/asm/local64.h arch/microblaze/include/asm/parport.h arch/microblaze/include/asm/percpu.h arch/microblaze/include/asm/serial.h arch/microblaze/include/asm/shmparam.h arch/microblaze/include/asm/topology.h arch/microblaze/include/asm/ucontext.h arch/microblaze/include/asm/vga.h arch/microblaze/include/asm/xor.h arch/microblaze/include/uapi/asm/bitsperlong.h arch/microblaze/include/uapi/asm/errno.h arch/microblaze/include/uapi/asm/fcntl.h arch/microblaze/include/uapi/asm/ioctl.h arch/microblaze/include/uapi/asm/ioctls.h arch/microblaze/include/uapi/asm/ipcbuf.h arch/microblaze/include/uapi/asm/kvm_para.h arch/microblaze/include/uapi/asm/mman.h arch/microblaze/include/uapi/asm/msgbuf.h arch/microblaze/include/uapi/asm/param.h arch/microblaze/include/uapi/asm/poll.h arch/microblaze/include/uapi/asm/resource.h arch/microblaze/include/uapi/asm/sembuf.h arch/microblaze/include/uapi/asm/shmbuf.h arch/microblaze/include/uapi/asm/siginfo.h arch/microblaze/include/uapi/asm/signal.h arch/microblaze/includ Aborting W dniu 2017-09-20 o 11:45, Paweł Staszewski pisze:quoted
Ok looks like ending bisection Latest bisected kernel when there is no kernel panic 4.12.0+ (from next) - but only this warning: [ 309.030019] NETDEV WATCHDOG: enp4s0f0 (ixgbe): transmit queue 0 timed out [ 309.030034] ------------[ cut here ]------------ [ 309.030040] WARNING: CPU: 35 PID: 0 at dev_watchdog+0xcf/0x139 [ 309.030041] Modules linked in: bonding ipmi_si x86_pkg_temp_thermal [ 309.030045] CPU: 35 PID: 0 Comm: swapper/35 Not tainted 4.12.0+ #5 [ 309.030046] task: ffff88086d98a000 task.stack: ffffc90003378000 [ 309.030048] RIP: 0010:dev_watchdog+0xcf/0x139 [ 309.030049] RSP: 0018:ffff88087fbc3ea8 EFLAGS: 00010246 [ 309.030050] RAX: 000000000000003d RBX: ffff88046b680000 RCX: 0000000000000000 [ 309.030050] RDX: ffff88087fbd2f01 RSI: 0000000000000000 RDI: ffff88087fbcda08 [ 309.030051] RBP: ffff88087fbc3eb8 R08: 0000000000000000 R09: ffff88087ff80a04 [ 309.030051] R10: 0000000000000000 R11: ffff88086d98a001 R12: 0000000000000000 [ 309.030052] R13: ffff88087fbc3ef8 R14: ffff88086d98a000 R15: ffffffff81c06008 [ 309.030053] FS: 0000000000000000(0000) GS:ffff88087fbc0000(0000) knlGS:0000000000000000 [ 309.030054] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 309.030054] CR2: 00007fba600f6098 CR3: 000000086b955000 CR4: 00000000001406e0 [ 309.030055] Call Trace: [ 309.030057] <IRQ> [ 309.030059] ? netif_tx_lock+0x79/0x79 [ 309.030062] call_timer_fn.isra.24+0x17/0x77 [ 309.030063] run_timer_softirq+0x118/0x161 [ 309.030065] ? netif_tx_lock+0x79/0x79 [ 309.030066] ? ktime_get+0x2b/0x42 [ 309.030070] ? lapic_next_deadline+0x21/0x27 [ 309.030073] ? clockevents_program_event+0xa8/0xc5 [ 309.030076] __do_softirq+0xa8/0x19d [ 309.030078] irq_exit+0x5d/0x6b [ 309.030079] smp_apic_timer_interrupt+0x2a/0x36 [ 309.030082] apic_timer_interrupt+0x89/0x90 [ 309.030085] RIP: 0010:mwait_idle+0x4e/0x6a [ 309.030086] RSP: 0018:ffffc9000337be98 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff10 [ 309.030087] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 [ 309.030087] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff88086d98a000 [ 309.030088] RBP: ffffc9000337be98 R08: ffff88046f8279a0 R09: ffff88046f827040 [ 309.030089] R10: ffff88086d98a000 R11: ffff88086d98a000 R12: 0000000000000000 [ 309.030089] R13: ffff88086d98a000 R14: ffff88086d98a000 R15: ffff88086d98a000 [ 309.030090] </IRQ> [ 309.030094] arch_cpu_idle+0xa/0xc [ 309.030095] default_idle_call+0x19/0x1b [ 309.030102] do_idle+0xbc/0x196 [ 309.030104] cpu_startup_entry+0x1d/0x20 [ 309.030105] start_secondary+0xd8/0xdc [ 309.030108] secondary_startup_64+0x9f/0x9f [ 309.030109] Code: cc 75 bd eb 35 48 89 df c6 05 c3 dc 74 00 01 e8 3a 62 fe ff 44 89 e1 48 89 de 48 89 c2 48 c7 c7 0f 65 a4 81 31 c0 e8 3d 4c b5 ff <0f> ff 48 8b 83 e0 01 00 00 48 89 df ff 50 78 48 8b 05 a0 bc 6a [ 309.030128] ---[ end trace 9102cb25703ae2d9 ]--- I just marked it as good - cause this problem above is differend - and im going to: git bisect good Bisecting: 1787 revisions left to test after this (roughly 11 steps) W dniu 2017-09-20 o 10:44, Paweł Staszewski pisze:quoted
Trying to make video from ipmi :) with that results: https://bugzilla.kernel.org/attachment.cgi?id=258521 catched two more lines where it starts - panic from 4.13.2. Now will try tro do some bisection W dniu 2017-09-20 o 09:58, Paweł Staszewski pisze:quoted
Hi Will try bisecting tonight W dniu 2017-09-20 o 05:24, Eric Dumazet pisze:quoted
On Wed, 2017-09-20 at 02:06 +0200, Paweł Staszewski wrote:quoted
Just checked kernel 4.13.2 and same problem Just after start all 6 bgp sessions - and kernel starts to learn routes it panic. https://bugzilla.kernel.org/attachment.cgi?id=258509Unfortunately we have not enough information from these traces. Can you get a full stack trace ? Alternatively, can you bisect ? Thanks.
Re: Latest net-next from GIT panic
From: Paweł Staszewski <hidden>
Date: 2017-09-20 11:04:51
Ok resumed and soo far: Panic: # bad: [9cc9a5cb176ccb4f2cda5ac34da5a659926f125f] datapath: Avoid using stack larger than 1024. git bisect bad 9cc9a5cb176ccb4f2cda5ac34da5a659926f125f No panic: # good: [073cf9e20c333ab29744717a23f9e43ec7512a20] Merge branch 'udp-reduce-cache-pressure' git bisect good 073cf9e20c333ab29744717a23f9e43ec7512a20 W dniu 2017-09-20 o 12:22, Paweł Staszewski pisze:
Soo far bisected and marked: git bisect start # bad: [07dd6cc1fff160143e82cf5df78c1db0b6e03355] Linux 4.13.2 git bisect bad 07dd6cc1fff160143e82cf5df78c1db0b6e03355 # good: [5d7d2e03e0f01a992e3521b180c3d3e67905f269] Linux 4.12.13 git bisect good 5d7d2e03e0f01a992e3521b180c3d3e67905f269 # good: [6f7da290413ba713f0cdd9ff1a2a9bb129ef4f6c] Linux 4.12 git bisect good 6f7da290413ba713f0cdd9ff1a2a9bb129ef4f6c # bad: [ac7b75966c9c86426b55fe1c50ae148aa4571075] Merge tag 'pinctrl-v4.13-1' of git://git.kernel.org/pub/scm/linux/kernel/git/linusw/linux-pinctrl git bisect bad ac7b75966c9c86426b55fe1c50ae148aa4571075 # good: [e24dd9ee5399747b71c1d982a484fc7601795f31] Merge branch 'next' of git://git.kernel.org/pub/scm/linux/kernel/git/jmorris/linux-security git bisect good e24dd9ee5399747b71c1d982a484fc7601795f31 # good: [e24dd9ee5399747b71c1d982a484fc7601795f31] Merge branch 'next' of git://git.kernel.org/pub/scm/linux/kernel/git/jmorris/linux-security git bisect good e24dd9ee5399747b71c1d982a484fc7601795f31 # good: [e24dd9ee5399747b71c1d982a484fc7601795f31] Merge branch 'next' of git://git.kernel.org/pub/scm/linux/kernel/git/jmorris/linux-security git bisect good e24dd9ee5399747b71c1d982a484fc7601795f31 W dniu 2017-09-20 o 12:21, Paweł Staszewski pisze:quoted
Ok kernel crashed with different panic that i didnt catch when i was doing bisect and now my bisection is broken :) git bisect good Bisecting: 1787 revisions left to test after this (roughly 11 steps) error: Your local changes to the following files would be overwritten by checkout: Documentation/00-INDEX Documentation/ABI/stable/sysfs-class-udc Documentation/ABI/testing/configfs-usb-gadget-uac1 Documentation/ABI/testing/ima_policy Documentation/ABI/testing/sysfs-bus-iio Documentation/ABI/testing/sysfs-bus-iio-meas-spec Documentation/ABI/testing/sysfs-bus-iio-timer-stm32 Documentation/ABI/testing/sysfs-class-net Documentation/ABI/testing/sysfs-class-power-twl4030 Documentation/ABI/testing/sysfs-class-typec Documentation/DMA-API.txt Documentation/IRQ-domain.txt Documentation/Makefile Documentation/PCI/MSI-HOWTO.txt Documentation/RCU/00-INDEX Documentation/RCU/Design/Requirements/Requirements.html Documentation/RCU/checklist.txt Documentation/admin-guide/README.rst Documentation/admin-guide/devices.txt Documentation/admin-guide/index.rst Documentation/admin-guide/kernel-parameters.txt Documentation/admin-guide/pm/cpufreq.rst Documentation/admin-guide/pm/intel_pstate.rst Documentation/admin-guide/ras.rst Documentation/arm/Atmel/README Documentation/block/biodoc.txt Documentation/conf.py Documentation/core-api/assoc_array.rst Documentation/core-api/atomic_ops.rst Documentation/core-api/index.rst Documentation/crypto/asymmetric-keys.txt Documentation/dev-tools/index.rst Documentation/dev-tools/sparse.rst Documentation/devicetree/bindings/arm/amlogic.txt Documentation/devicetree/bindings/arm/atmel-at91.txt Documentation/devicetree/bindings/arm/ccn.txt Documentation/devicetree/bindings/arm/cpus.txt Documentation/devicetree/bindings/arm/gemini.txt Documentation/devicetree/bindings/arm/hisilicon/hisilicon.txt Documentation/devicetree/bindings/arm/keystone/keystone.txt Documentation/devicetree/bindings/arm/mediatek.txt Documentation/devicetree/bindings/arm/rockchip.txt Documentation/devicetree/bindings/arm/shmobile.txt Documentation/devicetree/bindings/arm/tegra.txt Documentation/devicetree/bindings/ata/ahci-fsl-qoriq.txt Documentation/devicetree/bindings/bus/brcm,gisb-arb.txt Documentation/devicetree/bindings/clock/brcm,iproc-clocks.txt Documentation/devicetree/bindings/cpufreq/ti-cpufreq.txt Documentation/devicetree/bindings/gpio/gpio_atmel.txt Documentation/devicetree/bindings/iio/adc/amlogic,meson-saradc.txt Documentation/devicetree/bindings/iio/adc/renesas,gyroadc.txt Documentation/devicetree/bindings/iio/adc/st,stm32-adc.txt Documentation/devicetree/bindings/iio/imu/st_lsm6dsx.txt Documentation/devicetree/bindings/interrupt-controller/allwinner,sunxi-nmi.txt Documentation/devicetree/bindings/interrupt-controller/aspeed,ast2400-vic.txt Documentation/devicetree/bindings/interrupt-controller/mediatek,sysirq.txt Documentation/devicetree/bindings/leds/common.txt Documentation/devicetree/bindings/mfd/hi6421.txt Documentation/devicetree/bindings/mfd/tps65910.txt Documentation/devicetree/bindings/mmc/fsl-esdhc.txt Documentation/devicetree/bindings/mmc/k3-dw-mshc.txt Documentation/devicetree/bindings/mmc/rockchip-dw-mshc.txt Documentation/devicetree/bindings/mmc/ti-omap-hsmmc.txt Documentation/devicetree/bindings/mtd/atmel-nand.txt Documentation/devicetree/bindings/net/dsa/b53.txt Documentation/devicetree/bindings/net/ethernet.txt Documentation/devicetree/bindings/net/macb.txt Documentation/devicetree/bindings/net/marvell-orion-mdio.txt Documentation/devicetree/bindings/net/ti,wilink-st.txt Documentation/devicetree/bindings/net/wireless/ti,wlcore.txt Documentation/devicetree/bindings/nvmem/rockchip-efuse.txt Documentation/devicetree/bindings/opp/opp.txt Documentation/devicetree/bindings/phy/bcm-ns-usb3-phy.txt Documentation/devicetree/bindings/phy/brcm-sata-phy.txt Documentation/devicetree/bindings/phy/meson8b-usb2-phy.txt Documentation/devicetree/bindings/phy/phy-rockchip-inno-usb2.txt Documentation/devicetree/bindings/power/rockchip-io-domain.txt Documentation/devicetree/bindings/power/supply/bq27xxx.txt Documentation/devicetree/bindings/property-units.txt Documentation/devicetree/bindings/regulator/regulator.txt Documentation/devicetree/bindings/serial/8 error: The following untracked working tree files would be overwritten by checkout: Documentation/ABI/testing/sysfs-class-net-phydev Documentation/DocBook/.gitignore Documentation/DocBook/Makefile Documentation/DocBook/filesystems.tmpl Documentation/DocBook/kernel-hacking.tmpl Documentation/DocBook/kernel-locking.tmpl Documentation/DocBook/kgdb.tmpl Documentation/DocBook/libata.tmpl Documentation/DocBook/librs.tmpl Documentation/DocBook/lsm.tmpl Documentation/DocBook/mtdnand.tmpl Documentation/DocBook/networking.tmpl Documentation/DocBook/rapidio.tmpl Documentation/DocBook/s390-drivers.tmpl Documentation/DocBook/scsi.tmpl Documentation/DocBook/sh.tmpl Documentation/DocBook/stylesheet.xsl Documentation/DocBook/w1.tmpl Documentation/DocBook/z8530book.tmpl Documentation/Makefile.sphinx Documentation/RCU/trace.txt Documentation/devicetree/bindings/i2c/i2c-mt6577.txt Documentation/devicetree/bindings/misc/allwinner,syscon.txt Documentation/devicetree/bindings/net/cortina.txt Documentation/devicetree/bindings/net/dsa/ksz.txt Documentation/devicetree/bindings/net/dwmac-sun8i.txt Documentation/devicetree/bindings/net/qca,qca7000.txt Documentation/devicetree/bindings/power/max8903-charger.txt Documentation/devicetree/bindings/power_supply/maxim,max14656.txt Documentation/devicetree/bindings/ptp/brcm,ptp-dte.txt Documentation/devicetree/bindings/timer/moxa,moxart-timer.txt Documentation/doc-guide/docbook.rst Documentation/networking/tls.txt Documentation/prctl/no_new_privs.txt Documentation/prctl/seccomp_filter.txt Documentation/security/00-INDEX Documentation/security/IMA-templates.txt Documentation/security/LSM.txt Documentation/security/LoadPin.txt Documentation/security/SELinux.txt Documentation/security/Smack.txt Documentation/security/Yama.txt Documentation/security/apparmor.txt Documentation/security/conf.py Documentation/security/credentials.txt Documentation/security/keys-ecryptfs.txt Documentation/security/keys-request-key.txt Documentation/security/keys-trusted-encrypted.txt Documentation/security/keys.txt Documentation/security/self-protection.txt Documentation/security/tomoyo.txt Documentation/sphinx/convert_template.sed Documentation/sphinx/post_convert.sed Documentation/sphinx/tmplcvt Documentation/usb/typec.rst Documentation/usb/usb3-debug-port.rst arch/arm/boot/dts/rk1108-evb.dts arch/arm/boot/dts/rk1108.dtsi arch/arm/boot/dts/tegra20-whistler.dts arch/arm/mach-omap2/opp.c arch/arm/mach-omap2/pmu.c arch/ia64/include/asm/siginfo.h arch/m32r/include/uapi/asm/siginfo.h arch/microblaze/include/asm/bitops.h arch/microblaze/include/asm/bug.h arch/microblaze/include/asm/bugs.h arch/microblaze/include/asm/div64.h arch/microblaze/include/asm/emergency-restart.h arch/microblaze/include/asm/fb.h arch/microblaze/include/asm/hardirq.h arch/microblaze/include/asm/irq_regs.h arch/microblaze/include/asm/kdebug.h arch/microblaze/include/asm/kmap_types.h arch/microblaze/include/asm/linkage.h arch/microblaze/include/asm/local.h arch/microblaze/include/asm/local64.h arch/microblaze/include/asm/parport.h arch/microblaze/include/asm/percpu.h arch/microblaze/include/asm/serial.h arch/microblaze/include/asm/shmparam.h arch/microblaze/include/asm/topology.h arch/microblaze/include/asm/ucontext.h arch/microblaze/include/asm/vga.h arch/microblaze/include/asm/xor.h arch/microblaze/include/uapi/asm/bitsperlong.h arch/microblaze/include/uapi/asm/errno.h arch/microblaze/include/uapi/asm/fcntl.h arch/microblaze/include/uapi/asm/ioctl.h arch/microblaze/include/uapi/asm/ioctls.h arch/microblaze/include/uapi/asm/ipcbuf.h arch/microblaze/include/uapi/asm/kvm_para.h arch/microblaze/include/uapi/asm/mman.h arch/microblaze/include/uapi/asm/msgbuf.h arch/microblaze/include/uapi/asm/param.h arch/microblaze/include/uapi/asm/poll.h arch/microblaze/include/uapi/asm/resource.h arch/microblaze/include/uapi/asm/sembuf.h arch/microblaze/include/uapi/asm/shmbuf.h arch/microblaze/include/uapi/asm/siginfo.h arch/microblaze/include/uapi/asm/signal.h arch/microblaze/includ Aborting W dniu 2017-09-20 o 11:45, Paweł Staszewski pisze:quoted
Ok looks like ending bisection Latest bisected kernel when there is no kernel panic 4.12.0+ (from next) - but only this warning: [ 309.030019] NETDEV WATCHDOG: enp4s0f0 (ixgbe): transmit queue 0 timed out [ 309.030034] ------------[ cut here ]------------ [ 309.030040] WARNING: CPU: 35 PID: 0 at dev_watchdog+0xcf/0x139 [ 309.030041] Modules linked in: bonding ipmi_si x86_pkg_temp_thermal [ 309.030045] CPU: 35 PID: 0 Comm: swapper/35 Not tainted 4.12.0+ #5 [ 309.030046] task: ffff88086d98a000 task.stack: ffffc90003378000 [ 309.030048] RIP: 0010:dev_watchdog+0xcf/0x139 [ 309.030049] RSP: 0018:ffff88087fbc3ea8 EFLAGS: 00010246 [ 309.030050] RAX: 000000000000003d RBX: ffff88046b680000 RCX: 0000000000000000 [ 309.030050] RDX: ffff88087fbd2f01 RSI: 0000000000000000 RDI: ffff88087fbcda08 [ 309.030051] RBP: ffff88087fbc3eb8 R08: 0000000000000000 R09: ffff88087ff80a04 [ 309.030051] R10: 0000000000000000 R11: ffff88086d98a001 R12: 0000000000000000 [ 309.030052] R13: ffff88087fbc3ef8 R14: ffff88086d98a000 R15: ffffffff81c06008 [ 309.030053] FS: 0000000000000000(0000) GS:ffff88087fbc0000(0000) knlGS:0000000000000000 [ 309.030054] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 309.030054] CR2: 00007fba600f6098 CR3: 000000086b955000 CR4: 00000000001406e0 [ 309.030055] Call Trace: [ 309.030057] <IRQ> [ 309.030059] ? netif_tx_lock+0x79/0x79 [ 309.030062] call_timer_fn.isra.24+0x17/0x77 [ 309.030063] run_timer_softirq+0x118/0x161 [ 309.030065] ? netif_tx_lock+0x79/0x79 [ 309.030066] ? ktime_get+0x2b/0x42 [ 309.030070] ? lapic_next_deadline+0x21/0x27 [ 309.030073] ? clockevents_program_event+0xa8/0xc5 [ 309.030076] __do_softirq+0xa8/0x19d [ 309.030078] irq_exit+0x5d/0x6b [ 309.030079] smp_apic_timer_interrupt+0x2a/0x36 [ 309.030082] apic_timer_interrupt+0x89/0x90 [ 309.030085] RIP: 0010:mwait_idle+0x4e/0x6a [ 309.030086] RSP: 0018:ffffc9000337be98 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff10 [ 309.030087] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 [ 309.030087] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff88086d98a000 [ 309.030088] RBP: ffffc9000337be98 R08: ffff88046f8279a0 R09: ffff88046f827040 [ 309.030089] R10: ffff88086d98a000 R11: ffff88086d98a000 R12: 0000000000000000 [ 309.030089] R13: ffff88086d98a000 R14: ffff88086d98a000 R15: ffff88086d98a000 [ 309.030090] </IRQ> [ 309.030094] arch_cpu_idle+0xa/0xc [ 309.030095] default_idle_call+0x19/0x1b [ 309.030102] do_idle+0xbc/0x196 [ 309.030104] cpu_startup_entry+0x1d/0x20 [ 309.030105] start_secondary+0xd8/0xdc [ 309.030108] secondary_startup_64+0x9f/0x9f [ 309.030109] Code: cc 75 bd eb 35 48 89 df c6 05 c3 dc 74 00 01 e8 3a 62 fe ff 44 89 e1 48 89 de 48 89 c2 48 c7 c7 0f 65 a4 81 31 c0 e8 3d 4c b5 ff <0f> ff 48 8b 83 e0 01 00 00 48 89 df ff 50 78 48 8b 05 a0 bc 6a [ 309.030128] ---[ end trace 9102cb25703ae2d9 ]--- I just marked it as good - cause this problem above is differend - and im going to: git bisect good Bisecting: 1787 revisions left to test after this (roughly 11 steps) W dniu 2017-09-20 o 10:44, Paweł Staszewski pisze:quoted
Trying to make video from ipmi :) with that results: https://bugzilla.kernel.org/attachment.cgi?id=258521 catched two more lines where it starts - panic from 4.13.2. Now will try tro do some bisection W dniu 2017-09-20 o 09:58, Paweł Staszewski pisze:quoted
Hi Will try bisecting tonight W dniu 2017-09-20 o 05:24, Eric Dumazet pisze:quoted
On Wed, 2017-09-20 at 02:06 +0200, Paweł Staszewski wrote:quoted
Just checked kernel 4.13.2 and same problem Just after start all 6 bgp sessions - and kernel starts to learn routes it panic. https://bugzilla.kernel.org/attachment.cgi?id=258509Unfortunately we have not enough information from these traces. Can you get a full stack trace ? Alternatively, can you bisect ? Thanks.
Re: Latest net-next from GIT panic
From: Paweł Staszewski <hidden>
Date: 2017-09-20 12:25:57
Almost there Bisecting: 6 revisions left to test after this (roughly 3 steps) [ad65a2f05695aced349e308193c6e2a6b1d87112] ipv6: call dst_hold_safe() properly W dniu 2017-09-20 o 13:02, Paweł Staszewski pisze:
Ok resumed and soo far: Panic: # bad: [9cc9a5cb176ccb4f2cda5ac34da5a659926f125f] datapath: Avoid using stack larger than 1024. git bisect bad 9cc9a5cb176ccb4f2cda5ac34da5a659926f125f No panic: # good: [073cf9e20c333ab29744717a23f9e43ec7512a20] Merge branch 'udp-reduce-cache-pressure' git bisect good 073cf9e20c333ab29744717a23f9e43ec7512a20 W dniu 2017-09-20 o 12:22, Paweł Staszewski pisze:quoted
Soo far bisected and marked: git bisect start # bad: [07dd6cc1fff160143e82cf5df78c1db0b6e03355] Linux 4.13.2 git bisect bad 07dd6cc1fff160143e82cf5df78c1db0b6e03355 # good: [5d7d2e03e0f01a992e3521b180c3d3e67905f269] Linux 4.12.13 git bisect good 5d7d2e03e0f01a992e3521b180c3d3e67905f269 # good: [6f7da290413ba713f0cdd9ff1a2a9bb129ef4f6c] Linux 4.12 git bisect good 6f7da290413ba713f0cdd9ff1a2a9bb129ef4f6c # bad: [ac7b75966c9c86426b55fe1c50ae148aa4571075] Merge tag 'pinctrl-v4.13-1' of git://git.kernel.org/pub/scm/linux/kernel/git/linusw/linux-pinctrl git bisect bad ac7b75966c9c86426b55fe1c50ae148aa4571075 # good: [e24dd9ee5399747b71c1d982a484fc7601795f31] Merge branch 'next' of git://git.kernel.org/pub/scm/linux/kernel/git/jmorris/linux-security git bisect good e24dd9ee5399747b71c1d982a484fc7601795f31 # good: [e24dd9ee5399747b71c1d982a484fc7601795f31] Merge branch 'next' of git://git.kernel.org/pub/scm/linux/kernel/git/jmorris/linux-security git bisect good e24dd9ee5399747b71c1d982a484fc7601795f31 # good: [e24dd9ee5399747b71c1d982a484fc7601795f31] Merge branch 'next' of git://git.kernel.org/pub/scm/linux/kernel/git/jmorris/linux-security git bisect good e24dd9ee5399747b71c1d982a484fc7601795f31 W dniu 2017-09-20 o 12:21, Paweł Staszewski pisze:quoted
Ok kernel crashed with different panic that i didnt catch when i was doing bisect and now my bisection is broken :) git bisect good Bisecting: 1787 revisions left to test after this (roughly 11 steps) error: Your local changes to the following files would be overwritten by checkout: Documentation/00-INDEX Documentation/ABI/stable/sysfs-class-udc Documentation/ABI/testing/configfs-usb-gadget-uac1 Documentation/ABI/testing/ima_policy Documentation/ABI/testing/sysfs-bus-iio Documentation/ABI/testing/sysfs-bus-iio-meas-spec Documentation/ABI/testing/sysfs-bus-iio-timer-stm32 Documentation/ABI/testing/sysfs-class-net Documentation/ABI/testing/sysfs-class-power-twl4030 Documentation/ABI/testing/sysfs-class-typec Documentation/DMA-API.txt Documentation/IRQ-domain.txt Documentation/Makefile Documentation/PCI/MSI-HOWTO.txt Documentation/RCU/00-INDEX Documentation/RCU/Design/Requirements/Requirements.html Documentation/RCU/checklist.txt Documentation/admin-guide/README.rst Documentation/admin-guide/devices.txt Documentation/admin-guide/index.rst Documentation/admin-guide/kernel-parameters.txt Documentation/admin-guide/pm/cpufreq.rst Documentation/admin-guide/pm/intel_pstate.rst Documentation/admin-guide/ras.rst Documentation/arm/Atmel/README Documentation/block/biodoc.txt Documentation/conf.py Documentation/core-api/assoc_array.rst Documentation/core-api/atomic_ops.rst Documentation/core-api/index.rst Documentation/crypto/asymmetric-keys.txt Documentation/dev-tools/index.rst Documentation/dev-tools/sparse.rst Documentation/devicetree/bindings/arm/amlogic.txt Documentation/devicetree/bindings/arm/atmel-at91.txt Documentation/devicetree/bindings/arm/ccn.txt Documentation/devicetree/bindings/arm/cpus.txt Documentation/devicetree/bindings/arm/gemini.txt Documentation/devicetree/bindings/arm/hisilicon/hisilicon.txt Documentation/devicetree/bindings/arm/keystone/keystone.txt Documentation/devicetree/bindings/arm/mediatek.txt Documentation/devicetree/bindings/arm/rockchip.txt Documentation/devicetree/bindings/arm/shmobile.txt Documentation/devicetree/bindings/arm/tegra.txt Documentation/devicetree/bindings/ata/ahci-fsl-qoriq.txt Documentation/devicetree/bindings/bus/brcm,gisb-arb.txt Documentation/devicetree/bindings/clock/brcm,iproc-clocks.txt Documentation/devicetree/bindings/cpufreq/ti-cpufreq.txt Documentation/devicetree/bindings/gpio/gpio_atmel.txt Documentation/devicetree/bindings/iio/adc/amlogic,meson-saradc.txt Documentation/devicetree/bindings/iio/adc/renesas,gyroadc.txt Documentation/devicetree/bindings/iio/adc/st,stm32-adc.txt Documentation/devicetree/bindings/iio/imu/st_lsm6dsx.txt Documentation/devicetree/bindings/interrupt-controller/allwinner,sunxi-nmi.txt Documentation/devicetree/bindings/interrupt-controller/aspeed,ast2400-vic.txt Documentation/devicetree/bindings/interrupt-controller/mediatek,sysirq.txt Documentation/devicetree/bindings/leds/common.txt Documentation/devicetree/bindings/mfd/hi6421.txt Documentation/devicetree/bindings/mfd/tps65910.txt Documentation/devicetree/bindings/mmc/fsl-esdhc.txt Documentation/devicetree/bindings/mmc/k3-dw-mshc.txt Documentation/devicetree/bindings/mmc/rockchip-dw-mshc.txt Documentation/devicetree/bindings/mmc/ti-omap-hsmmc.txt Documentation/devicetree/bindings/mtd/atmel-nand.txt Documentation/devicetree/bindings/net/dsa/b53.txt Documentation/devicetree/bindings/net/ethernet.txt Documentation/devicetree/bindings/net/macb.txt Documentation/devicetree/bindings/net/marvell-orion-mdio.txt Documentation/devicetree/bindings/net/ti,wilink-st.txt Documentation/devicetree/bindings/net/wireless/ti,wlcore.txt Documentation/devicetree/bindings/nvmem/rockchip-efuse.txt Documentation/devicetree/bindings/opp/opp.txt Documentation/devicetree/bindings/phy/bcm-ns-usb3-phy.txt Documentation/devicetree/bindings/phy/brcm-sata-phy.txt Documentation/devicetree/bindings/phy/meson8b-usb2-phy.txt Documentation/devicetree/bindings/phy/phy-rockchip-inno-usb2.txt Documentation/devicetree/bindings/power/rockchip-io-domain.txt Documentation/devicetree/bindings/power/supply/bq27xxx.txt Documentation/devicetree/bindings/property-units.txt Documentation/devicetree/bindings/regulator/regulator.txt Documentation/devicetree/bindings/serial/8 error: The following untracked working tree files would be overwritten by checkout: Documentation/ABI/testing/sysfs-class-net-phydev Documentation/DocBook/.gitignore Documentation/DocBook/Makefile Documentation/DocBook/filesystems.tmpl Documentation/DocBook/kernel-hacking.tmpl Documentation/DocBook/kernel-locking.tmpl Documentation/DocBook/kgdb.tmpl Documentation/DocBook/libata.tmpl Documentation/DocBook/librs.tmpl Documentation/DocBook/lsm.tmpl Documentation/DocBook/mtdnand.tmpl Documentation/DocBook/networking.tmpl Documentation/DocBook/rapidio.tmpl Documentation/DocBook/s390-drivers.tmpl Documentation/DocBook/scsi.tmpl Documentation/DocBook/sh.tmpl Documentation/DocBook/stylesheet.xsl Documentation/DocBook/w1.tmpl Documentation/DocBook/z8530book.tmpl Documentation/Makefile.sphinx Documentation/RCU/trace.txt Documentation/devicetree/bindings/i2c/i2c-mt6577.txt Documentation/devicetree/bindings/misc/allwinner,syscon.txt Documentation/devicetree/bindings/net/cortina.txt Documentation/devicetree/bindings/net/dsa/ksz.txt Documentation/devicetree/bindings/net/dwmac-sun8i.txt Documentation/devicetree/bindings/net/qca,qca7000.txt Documentation/devicetree/bindings/power/max8903-charger.txt Documentation/devicetree/bindings/power_supply/maxim,max14656.txt Documentation/devicetree/bindings/ptp/brcm,ptp-dte.txt Documentation/devicetree/bindings/timer/moxa,moxart-timer.txt Documentation/doc-guide/docbook.rst Documentation/networking/tls.txt Documentation/prctl/no_new_privs.txt Documentation/prctl/seccomp_filter.txt Documentation/security/00-INDEX Documentation/security/IMA-templates.txt Documentation/security/LSM.txt Documentation/security/LoadPin.txt Documentation/security/SELinux.txt Documentation/security/Smack.txt Documentation/security/Yama.txt Documentation/security/apparmor.txt Documentation/security/conf.py Documentation/security/credentials.txt Documentation/security/keys-ecryptfs.txt Documentation/security/keys-request-key.txt Documentation/security/keys-trusted-encrypted.txt Documentation/security/keys.txt Documentation/security/self-protection.txt Documentation/security/tomoyo.txt Documentation/sphinx/convert_template.sed Documentation/sphinx/post_convert.sed Documentation/sphinx/tmplcvt Documentation/usb/typec.rst Documentation/usb/usb3-debug-port.rst arch/arm/boot/dts/rk1108-evb.dts arch/arm/boot/dts/rk1108.dtsi arch/arm/boot/dts/tegra20-whistler.dts arch/arm/mach-omap2/opp.c arch/arm/mach-omap2/pmu.c arch/ia64/include/asm/siginfo.h arch/m32r/include/uapi/asm/siginfo.h arch/microblaze/include/asm/bitops.h arch/microblaze/include/asm/bug.h arch/microblaze/include/asm/bugs.h arch/microblaze/include/asm/div64.h arch/microblaze/include/asm/emergency-restart.h arch/microblaze/include/asm/fb.h arch/microblaze/include/asm/hardirq.h arch/microblaze/include/asm/irq_regs.h arch/microblaze/include/asm/kdebug.h arch/microblaze/include/asm/kmap_types.h arch/microblaze/include/asm/linkage.h arch/microblaze/include/asm/local.h arch/microblaze/include/asm/local64.h arch/microblaze/include/asm/parport.h arch/microblaze/include/asm/percpu.h arch/microblaze/include/asm/serial.h arch/microblaze/include/asm/shmparam.h arch/microblaze/include/asm/topology.h arch/microblaze/include/asm/ucontext.h arch/microblaze/include/asm/vga.h arch/microblaze/include/asm/xor.h arch/microblaze/include/uapi/asm/bitsperlong.h arch/microblaze/include/uapi/asm/errno.h arch/microblaze/include/uapi/asm/fcntl.h arch/microblaze/include/uapi/asm/ioctl.h arch/microblaze/include/uapi/asm/ioctls.h arch/microblaze/include/uapi/asm/ipcbuf.h arch/microblaze/include/uapi/asm/kvm_para.h arch/microblaze/include/uapi/asm/mman.h arch/microblaze/include/uapi/asm/msgbuf.h arch/microblaze/include/uapi/asm/param.h arch/microblaze/include/uapi/asm/poll.h arch/microblaze/include/uapi/asm/resource.h arch/microblaze/include/uapi/asm/sembuf.h arch/microblaze/include/uapi/asm/shmbuf.h arch/microblaze/include/uapi/asm/siginfo.h arch/microblaze/include/uapi/asm/signal.h arch/microblaze/includ Aborting W dniu 2017-09-20 o 11:45, Paweł Staszewski pisze:quoted
Ok looks like ending bisection Latest bisected kernel when there is no kernel panic 4.12.0+ (from next) - but only this warning: [ 309.030019] NETDEV WATCHDOG: enp4s0f0 (ixgbe): transmit queue 0 timed out [ 309.030034] ------------[ cut here ]------------ [ 309.030040] WARNING: CPU: 35 PID: 0 at dev_watchdog+0xcf/0x139 [ 309.030041] Modules linked in: bonding ipmi_si x86_pkg_temp_thermal [ 309.030045] CPU: 35 PID: 0 Comm: swapper/35 Not tainted 4.12.0+ #5 [ 309.030046] task: ffff88086d98a000 task.stack: ffffc90003378000 [ 309.030048] RIP: 0010:dev_watchdog+0xcf/0x139 [ 309.030049] RSP: 0018:ffff88087fbc3ea8 EFLAGS: 00010246 [ 309.030050] RAX: 000000000000003d RBX: ffff88046b680000 RCX: 0000000000000000 [ 309.030050] RDX: ffff88087fbd2f01 RSI: 0000000000000000 RDI: ffff88087fbcda08 [ 309.030051] RBP: ffff88087fbc3eb8 R08: 0000000000000000 R09: ffff88087ff80a04 [ 309.030051] R10: 0000000000000000 R11: ffff88086d98a001 R12: 0000000000000000 [ 309.030052] R13: ffff88087fbc3ef8 R14: ffff88086d98a000 R15: ffffffff81c06008 [ 309.030053] FS: 0000000000000000(0000) GS:ffff88087fbc0000(0000) knlGS:0000000000000000 [ 309.030054] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 309.030054] CR2: 00007fba600f6098 CR3: 000000086b955000 CR4: 00000000001406e0 [ 309.030055] Call Trace: [ 309.030057] <IRQ> [ 309.030059] ? netif_tx_lock+0x79/0x79 [ 309.030062] call_timer_fn.isra.24+0x17/0x77 [ 309.030063] run_timer_softirq+0x118/0x161 [ 309.030065] ? netif_tx_lock+0x79/0x79 [ 309.030066] ? ktime_get+0x2b/0x42 [ 309.030070] ? lapic_next_deadline+0x21/0x27 [ 309.030073] ? clockevents_program_event+0xa8/0xc5 [ 309.030076] __do_softirq+0xa8/0x19d [ 309.030078] irq_exit+0x5d/0x6b [ 309.030079] smp_apic_timer_interrupt+0x2a/0x36 [ 309.030082] apic_timer_interrupt+0x89/0x90 [ 309.030085] RIP: 0010:mwait_idle+0x4e/0x6a [ 309.030086] RSP: 0018:ffffc9000337be98 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff10 [ 309.030087] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 [ 309.030087] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff88086d98a000 [ 309.030088] RBP: ffffc9000337be98 R08: ffff88046f8279a0 R09: ffff88046f827040 [ 309.030089] R10: ffff88086d98a000 R11: ffff88086d98a000 R12: 0000000000000000 [ 309.030089] R13: ffff88086d98a000 R14: ffff88086d98a000 R15: ffff88086d98a000 [ 309.030090] </IRQ> [ 309.030094] arch_cpu_idle+0xa/0xc [ 309.030095] default_idle_call+0x19/0x1b [ 309.030102] do_idle+0xbc/0x196 [ 309.030104] cpu_startup_entry+0x1d/0x20 [ 309.030105] start_secondary+0xd8/0xdc [ 309.030108] secondary_startup_64+0x9f/0x9f [ 309.030109] Code: cc 75 bd eb 35 48 89 df c6 05 c3 dc 74 00 01 e8 3a 62 fe ff 44 89 e1 48 89 de 48 89 c2 48 c7 c7 0f 65 a4 81 31 c0 e8 3d 4c b5 ff <0f> ff 48 8b 83 e0 01 00 00 48 89 df ff 50 78 48 8b 05 a0 bc 6a [ 309.030128] ---[ end trace 9102cb25703ae2d9 ]--- I just marked it as good - cause this problem above is differend - and im going to: git bisect good Bisecting: 1787 revisions left to test after this (roughly 11 steps) W dniu 2017-09-20 o 10:44, Paweł Staszewski pisze:quoted
Trying to make video from ipmi :) with that results: https://bugzilla.kernel.org/attachment.cgi?id=258521 catched two more lines where it starts - panic from 4.13.2. Now will try tro do some bisection W dniu 2017-09-20 o 09:58, Paweł Staszewski pisze:quoted
Hi Will try bisecting tonight W dniu 2017-09-20 o 05:24, Eric Dumazet pisze:quoted
On Wed, 2017-09-20 at 02:06 +0200, Paweł Staszewski wrote:quoted
Just checked kernel 4.13.2 and same problem Just after start all 6 bgp sessions - and kernel starts to learn routes it panic. https://bugzilla.kernel.org/attachment.cgi?id=258509Unfortunately we have not enough information from these traces. Can you get a full stack trace ? Alternatively, can you bisect ? Thanks.
Re: Latest net-next from GIT panic
From: Paweł Staszewski <hidden>
Date: 2017-09-20 12:51:44
And the last one git bisect good Bisecting: 1 revision left to test after this (roughly 1 step) [1cfb71eeb12047bcdbd3e6730ffed66e810a0855] ipv6: take dst->__refcnt for insertion into fib6 tree With this have kernel panic same as always git bisect bad Bisecting: 0 revisions left to test after this (roughly 0 steps) [b838d5e1c5b6e57b10ec8af2268824041e3ea911] ipv4: mark DST_NOGC and remove the operation of dst_free() W dniu 2017-09-20 o 14:23, Paweł Staszewski pisze:
Almost there Bisecting: 6 revisions left to test after this (roughly 3 steps) [ad65a2f05695aced349e308193c6e2a6b1d87112] ipv6: call dst_hold_safe() properly W dniu 2017-09-20 o 13:02, Paweł Staszewski pisze:quoted
Ok resumed and soo far: Panic: # bad: [9cc9a5cb176ccb4f2cda5ac34da5a659926f125f] datapath: Avoid using stack larger than 1024. git bisect bad 9cc9a5cb176ccb4f2cda5ac34da5a659926f125f No panic: # good: [073cf9e20c333ab29744717a23f9e43ec7512a20] Merge branch 'udp-reduce-cache-pressure' git bisect good 073cf9e20c333ab29744717a23f9e43ec7512a20 W dniu 2017-09-20 o 12:22, Paweł Staszewski pisze:quoted
Soo far bisected and marked: git bisect start # bad: [07dd6cc1fff160143e82cf5df78c1db0b6e03355] Linux 4.13.2 git bisect bad 07dd6cc1fff160143e82cf5df78c1db0b6e03355 # good: [5d7d2e03e0f01a992e3521b180c3d3e67905f269] Linux 4.12.13 git bisect good 5d7d2e03e0f01a992e3521b180c3d3e67905f269 # good: [6f7da290413ba713f0cdd9ff1a2a9bb129ef4f6c] Linux 4.12 git bisect good 6f7da290413ba713f0cdd9ff1a2a9bb129ef4f6c # bad: [ac7b75966c9c86426b55fe1c50ae148aa4571075] Merge tag 'pinctrl-v4.13-1' of git://git.kernel.org/pub/scm/linux/kernel/git/linusw/linux-pinctrl git bisect bad ac7b75966c9c86426b55fe1c50ae148aa4571075 # good: [e24dd9ee5399747b71c1d982a484fc7601795f31] Merge branch 'next' of git://git.kernel.org/pub/scm/linux/kernel/git/jmorris/linux-security git bisect good e24dd9ee5399747b71c1d982a484fc7601795f31 # good: [e24dd9ee5399747b71c1d982a484fc7601795f31] Merge branch 'next' of git://git.kernel.org/pub/scm/linux/kernel/git/jmorris/linux-security git bisect good e24dd9ee5399747b71c1d982a484fc7601795f31 # good: [e24dd9ee5399747b71c1d982a484fc7601795f31] Merge branch 'next' of git://git.kernel.org/pub/scm/linux/kernel/git/jmorris/linux-security git bisect good e24dd9ee5399747b71c1d982a484fc7601795f31 W dniu 2017-09-20 o 12:21, Paweł Staszewski pisze:quoted
Ok kernel crashed with different panic that i didnt catch when i was doing bisect and now my bisection is broken :) git bisect good Bisecting: 1787 revisions left to test after this (roughly 11 steps) error: Your local changes to the following files would be overwritten by checkout: Documentation/00-INDEX Documentation/ABI/stable/sysfs-class-udc Documentation/ABI/testing/configfs-usb-gadget-uac1 Documentation/ABI/testing/ima_policy Documentation/ABI/testing/sysfs-bus-iio Documentation/ABI/testing/sysfs-bus-iio-meas-spec Documentation/ABI/testing/sysfs-bus-iio-timer-stm32 Documentation/ABI/testing/sysfs-class-net Documentation/ABI/testing/sysfs-class-power-twl4030 Documentation/ABI/testing/sysfs-class-typec Documentation/DMA-API.txt Documentation/IRQ-domain.txt Documentation/Makefile Documentation/PCI/MSI-HOWTO.txt Documentation/RCU/00-INDEX Documentation/RCU/Design/Requirements/Requirements.html Documentation/RCU/checklist.txt Documentation/admin-guide/README.rst Documentation/admin-guide/devices.txt Documentation/admin-guide/index.rst Documentation/admin-guide/kernel-parameters.txt Documentation/admin-guide/pm/cpufreq.rst Documentation/admin-guide/pm/intel_pstate.rst Documentation/admin-guide/ras.rst Documentation/arm/Atmel/README Documentation/block/biodoc.txt Documentation/conf.py Documentation/core-api/assoc_array.rst Documentation/core-api/atomic_ops.rst Documentation/core-api/index.rst Documentation/crypto/asymmetric-keys.txt Documentation/dev-tools/index.rst Documentation/dev-tools/sparse.rst Documentation/devicetree/bindings/arm/amlogic.txt Documentation/devicetree/bindings/arm/atmel-at91.txt Documentation/devicetree/bindings/arm/ccn.txt Documentation/devicetree/bindings/arm/cpus.txt Documentation/devicetree/bindings/arm/gemini.txt Documentation/devicetree/bindings/arm/hisilicon/hisilicon.txt Documentation/devicetree/bindings/arm/keystone/keystone.txt Documentation/devicetree/bindings/arm/mediatek.txt Documentation/devicetree/bindings/arm/rockchip.txt Documentation/devicetree/bindings/arm/shmobile.txt Documentation/devicetree/bindings/arm/tegra.txt Documentation/devicetree/bindings/ata/ahci-fsl-qoriq.txt Documentation/devicetree/bindings/bus/brcm,gisb-arb.txt Documentation/devicetree/bindings/clock/brcm,iproc-clocks.txt Documentation/devicetree/bindings/cpufreq/ti-cpufreq.txt Documentation/devicetree/bindings/gpio/gpio_atmel.txt Documentation/devicetree/bindings/iio/adc/amlogic,meson-saradc.txt Documentation/devicetree/bindings/iio/adc/renesas,gyroadc.txt Documentation/devicetree/bindings/iio/adc/st,stm32-adc.txt Documentation/devicetree/bindings/iio/imu/st_lsm6dsx.txt Documentation/devicetree/bindings/interrupt-controller/allwinner,sunxi-nmi.txt Documentation/devicetree/bindings/interrupt-controller/aspeed,ast2400-vic.txt Documentation/devicetree/bindings/interrupt-controller/mediatek,sysirq.txt Documentation/devicetree/bindings/leds/common.txt Documentation/devicetree/bindings/mfd/hi6421.txt Documentation/devicetree/bindings/mfd/tps65910.txt Documentation/devicetree/bindings/mmc/fsl-esdhc.txt Documentation/devicetree/bindings/mmc/k3-dw-mshc.txt Documentation/devicetree/bindings/mmc/rockchip-dw-mshc.txt Documentation/devicetree/bindings/mmc/ti-omap-hsmmc.txt Documentation/devicetree/bindings/mtd/atmel-nand.txt Documentation/devicetree/bindings/net/dsa/b53.txt Documentation/devicetree/bindings/net/ethernet.txt Documentation/devicetree/bindings/net/macb.txt Documentation/devicetree/bindings/net/marvell-orion-mdio.txt Documentation/devicetree/bindings/net/ti,wilink-st.txt Documentation/devicetree/bindings/net/wireless/ti,wlcore.txt Documentation/devicetree/bindings/nvmem/rockchip-efuse.txt Documentation/devicetree/bindings/opp/opp.txt Documentation/devicetree/bindings/phy/bcm-ns-usb3-phy.txt Documentation/devicetree/bindings/phy/brcm-sata-phy.txt Documentation/devicetree/bindings/phy/meson8b-usb2-phy.txt Documentation/devicetree/bindings/phy/phy-rockchip-inno-usb2.txt Documentation/devicetree/bindings/power/rockchip-io-domain.txt Documentation/devicetree/bindings/power/supply/bq27xxx.txt Documentation/devicetree/bindings/property-units.txt Documentation/devicetree/bindings/regulator/regulator.txt Documentation/devicetree/bindings/serial/8 error: The following untracked working tree files would be overwritten by checkout: Documentation/ABI/testing/sysfs-class-net-phydev Documentation/DocBook/.gitignore Documentation/DocBook/Makefile Documentation/DocBook/filesystems.tmpl Documentation/DocBook/kernel-hacking.tmpl Documentation/DocBook/kernel-locking.tmpl Documentation/DocBook/kgdb.tmpl Documentation/DocBook/libata.tmpl Documentation/DocBook/librs.tmpl Documentation/DocBook/lsm.tmpl Documentation/DocBook/mtdnand.tmpl Documentation/DocBook/networking.tmpl Documentation/DocBook/rapidio.tmpl Documentation/DocBook/s390-drivers.tmpl Documentation/DocBook/scsi.tmpl Documentation/DocBook/sh.tmpl Documentation/DocBook/stylesheet.xsl Documentation/DocBook/w1.tmpl Documentation/DocBook/z8530book.tmpl Documentation/Makefile.sphinx Documentation/RCU/trace.txt Documentation/devicetree/bindings/i2c/i2c-mt6577.txt Documentation/devicetree/bindings/misc/allwinner,syscon.txt Documentation/devicetree/bindings/net/cortina.txt Documentation/devicetree/bindings/net/dsa/ksz.txt Documentation/devicetree/bindings/net/dwmac-sun8i.txt Documentation/devicetree/bindings/net/qca,qca7000.txt Documentation/devicetree/bindings/power/max8903-charger.txt Documentation/devicetree/bindings/power_supply/maxim,max14656.txt Documentation/devicetree/bindings/ptp/brcm,ptp-dte.txt Documentation/devicetree/bindings/timer/moxa,moxart-timer.txt Documentation/doc-guide/docbook.rst Documentation/networking/tls.txt Documentation/prctl/no_new_privs.txt Documentation/prctl/seccomp_filter.txt Documentation/security/00-INDEX Documentation/security/IMA-templates.txt Documentation/security/LSM.txt Documentation/security/LoadPin.txt Documentation/security/SELinux.txt Documentation/security/Smack.txt Documentation/security/Yama.txt Documentation/security/apparmor.txt Documentation/security/conf.py Documentation/security/credentials.txt Documentation/security/keys-ecryptfs.txt Documentation/security/keys-request-key.txt Documentation/security/keys-trusted-encrypted.txt Documentation/security/keys.txt Documentation/security/self-protection.txt Documentation/security/tomoyo.txt Documentation/sphinx/convert_template.sed Documentation/sphinx/post_convert.sed Documentation/sphinx/tmplcvt Documentation/usb/typec.rst Documentation/usb/usb3-debug-port.rst arch/arm/boot/dts/rk1108-evb.dts arch/arm/boot/dts/rk1108.dtsi arch/arm/boot/dts/tegra20-whistler.dts arch/arm/mach-omap2/opp.c arch/arm/mach-omap2/pmu.c arch/ia64/include/asm/siginfo.h arch/m32r/include/uapi/asm/siginfo.h arch/microblaze/include/asm/bitops.h arch/microblaze/include/asm/bug.h arch/microblaze/include/asm/bugs.h arch/microblaze/include/asm/div64.h arch/microblaze/include/asm/emergency-restart.h arch/microblaze/include/asm/fb.h arch/microblaze/include/asm/hardirq.h arch/microblaze/include/asm/irq_regs.h arch/microblaze/include/asm/kdebug.h arch/microblaze/include/asm/kmap_types.h arch/microblaze/include/asm/linkage.h arch/microblaze/include/asm/local.h arch/microblaze/include/asm/local64.h arch/microblaze/include/asm/parport.h arch/microblaze/include/asm/percpu.h arch/microblaze/include/asm/serial.h arch/microblaze/include/asm/shmparam.h arch/microblaze/include/asm/topology.h arch/microblaze/include/asm/ucontext.h arch/microblaze/include/asm/vga.h arch/microblaze/include/asm/xor.h arch/microblaze/include/uapi/asm/bitsperlong.h arch/microblaze/include/uapi/asm/errno.h arch/microblaze/include/uapi/asm/fcntl.h arch/microblaze/include/uapi/asm/ioctl.h arch/microblaze/include/uapi/asm/ioctls.h arch/microblaze/include/uapi/asm/ipcbuf.h arch/microblaze/include/uapi/asm/kvm_para.h arch/microblaze/include/uapi/asm/mman.h arch/microblaze/include/uapi/asm/msgbuf.h arch/microblaze/include/uapi/asm/param.h arch/microblaze/include/uapi/asm/poll.h arch/microblaze/include/uapi/asm/resource.h arch/microblaze/include/uapi/asm/sembuf.h arch/microblaze/include/uapi/asm/shmbuf.h arch/microblaze/include/uapi/asm/siginfo.h arch/microblaze/include/uapi/asm/signal.h arch/microblaze/includ Aborting W dniu 2017-09-20 o 11:45, Paweł Staszewski pisze:quoted
Ok looks like ending bisection Latest bisected kernel when there is no kernel panic 4.12.0+ (from next) - but only this warning: [ 309.030019] NETDEV WATCHDOG: enp4s0f0 (ixgbe): transmit queue 0 timed out [ 309.030034] ------------[ cut here ]------------ [ 309.030040] WARNING: CPU: 35 PID: 0 at dev_watchdog+0xcf/0x139 [ 309.030041] Modules linked in: bonding ipmi_si x86_pkg_temp_thermal [ 309.030045] CPU: 35 PID: 0 Comm: swapper/35 Not tainted 4.12.0+ #5 [ 309.030046] task: ffff88086d98a000 task.stack: ffffc90003378000 [ 309.030048] RIP: 0010:dev_watchdog+0xcf/0x139 [ 309.030049] RSP: 0018:ffff88087fbc3ea8 EFLAGS: 00010246 [ 309.030050] RAX: 000000000000003d RBX: ffff88046b680000 RCX: 0000000000000000 [ 309.030050] RDX: ffff88087fbd2f01 RSI: 0000000000000000 RDI: ffff88087fbcda08 [ 309.030051] RBP: ffff88087fbc3eb8 R08: 0000000000000000 R09: ffff88087ff80a04 [ 309.030051] R10: 0000000000000000 R11: ffff88086d98a001 R12: 0000000000000000 [ 309.030052] R13: ffff88087fbc3ef8 R14: ffff88086d98a000 R15: ffffffff81c06008 [ 309.030053] FS: 0000000000000000(0000) GS:ffff88087fbc0000(0000) knlGS:0000000000000000 [ 309.030054] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 309.030054] CR2: 00007fba600f6098 CR3: 000000086b955000 CR4: 00000000001406e0 [ 309.030055] Call Trace: [ 309.030057] <IRQ> [ 309.030059] ? netif_tx_lock+0x79/0x79 [ 309.030062] call_timer_fn.isra.24+0x17/0x77 [ 309.030063] run_timer_softirq+0x118/0x161 [ 309.030065] ? netif_tx_lock+0x79/0x79 [ 309.030066] ? ktime_get+0x2b/0x42 [ 309.030070] ? lapic_next_deadline+0x21/0x27 [ 309.030073] ? clockevents_program_event+0xa8/0xc5 [ 309.030076] __do_softirq+0xa8/0x19d [ 309.030078] irq_exit+0x5d/0x6b [ 309.030079] smp_apic_timer_interrupt+0x2a/0x36 [ 309.030082] apic_timer_interrupt+0x89/0x90 [ 309.030085] RIP: 0010:mwait_idle+0x4e/0x6a [ 309.030086] RSP: 0018:ffffc9000337be98 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff10 [ 309.030087] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 [ 309.030087] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff88086d98a000 [ 309.030088] RBP: ffffc9000337be98 R08: ffff88046f8279a0 R09: ffff88046f827040 [ 309.030089] R10: ffff88086d98a000 R11: ffff88086d98a000 R12: 0000000000000000 [ 309.030089] R13: ffff88086d98a000 R14: ffff88086d98a000 R15: ffff88086d98a000 [ 309.030090] </IRQ> [ 309.030094] arch_cpu_idle+0xa/0xc [ 309.030095] default_idle_call+0x19/0x1b [ 309.030102] do_idle+0xbc/0x196 [ 309.030104] cpu_startup_entry+0x1d/0x20 [ 309.030105] start_secondary+0xd8/0xdc [ 309.030108] secondary_startup_64+0x9f/0x9f [ 309.030109] Code: cc 75 bd eb 35 48 89 df c6 05 c3 dc 74 00 01 e8 3a 62 fe ff 44 89 e1 48 89 de 48 89 c2 48 c7 c7 0f 65 a4 81 31 c0 e8 3d 4c b5 ff <0f> ff 48 8b 83 e0 01 00 00 48 89 df ff 50 78 48 8b 05 a0 bc 6a [ 309.030128] ---[ end trace 9102cb25703ae2d9 ]--- I just marked it as good - cause this problem above is differend - and im going to: git bisect good Bisecting: 1787 revisions left to test after this (roughly 11 steps) W dniu 2017-09-20 o 10:44, Paweł Staszewski pisze:quoted
Trying to make video from ipmi :) with that results: https://bugzilla.kernel.org/attachment.cgi?id=258521 catched two more lines where it starts - panic from 4.13.2. Now will try tro do some bisection W dniu 2017-09-20 o 09:58, Paweł Staszewski pisze:quoted
Hi Will try bisecting tonight W dniu 2017-09-20 o 05:24, Eric Dumazet pisze:quoted
On Wed, 2017-09-20 at 02:06 +0200, Paweł Staszewski wrote:quoted
Just checked kernel 4.13.2 and same problem Just after start all 6 bgp sessions - and kernel starts to learn routes it panic. https://bugzilla.kernel.org/attachment.cgi?id=258509Unfortunately we have not enough information from these traces. Can you get a full stack trace ? Alternatively, can you bisect ? Thanks.
Re: Latest net-next from GIT panic
From: Paweł Staszewski <hidden>
Date: 2017-09-20 13:08:01
hmm But after b838d5e1c5b6e57b10ec8af2268824041e3ea911 is the first bad commit commit b838d5e1c5b6e57b10ec8af2268824041e3ea911 Author: Wei Wang [off-list ref] Date: Sat Jun 17 10:42:32 2017 -0700 ipv4: mark DST_NOGC and remove the operation of dst_free() With the previous preparation patches, we are ready to get rid of the dst gc operation in ipv4 code and release dst based on refcnt only. So this patch adds DST_NOGC flag for all IPv4 dst and remove the calls to dst_free(). At this point, all dst created in ipv4 code do not use the dst gc anymore and will be destroyed at the point when refcnt drops to 0. Signed-off-by: Wei Wang [off-list ref] Acked-by: Martin KaFai Lau [off-list ref] Signed-off-by: David S. Miller [off-list ref] :040000 040000 9b7e7fb641de6531fc7887473ca47ef7cb6a11da 831a73b71d3df1755f3e24c0d3c86d7a93fd55e2 M net Still panic - soo will back to past 3 steps and will try to get again bisect without panic. W dniu 2017-09-20 o 14:49, Paweł Staszewski pisze:
And the last one git bisect good Bisecting: 1 revision left to test after this (roughly 1 step) [1cfb71eeb12047bcdbd3e6730ffed66e810a0855] ipv6: take dst->__refcnt for insertion into fib6 tree With this have kernel panic same as always git bisect bad Bisecting: 0 revisions left to test after this (roughly 0 steps) [b838d5e1c5b6e57b10ec8af2268824041e3ea911] ipv4: mark DST_NOGC and remove the operation of dst_free() W dniu 2017-09-20 o 14:23, Paweł Staszewski pisze:quoted
Almost there Bisecting: 6 revisions left to test after this (roughly 3 steps) [ad65a2f05695aced349e308193c6e2a6b1d87112] ipv6: call dst_hold_safe() properly W dniu 2017-09-20 o 13:02, Paweł Staszewski pisze:quoted
Ok resumed and soo far: Panic: # bad: [9cc9a5cb176ccb4f2cda5ac34da5a659926f125f] datapath: Avoid using stack larger than 1024. git bisect bad 9cc9a5cb176ccb4f2cda5ac34da5a659926f125f No panic: # good: [073cf9e20c333ab29744717a23f9e43ec7512a20] Merge branch 'udp-reduce-cache-pressure' git bisect good 073cf9e20c333ab29744717a23f9e43ec7512a20 W dniu 2017-09-20 o 12:22, Paweł Staszewski pisze:quoted
Soo far bisected and marked: git bisect start # bad: [07dd6cc1fff160143e82cf5df78c1db0b6e03355] Linux 4.13.2 git bisect bad 07dd6cc1fff160143e82cf5df78c1db0b6e03355 # good: [5d7d2e03e0f01a992e3521b180c3d3e67905f269] Linux 4.12.13 git bisect good 5d7d2e03e0f01a992e3521b180c3d3e67905f269 # good: [6f7da290413ba713f0cdd9ff1a2a9bb129ef4f6c] Linux 4.12 git bisect good 6f7da290413ba713f0cdd9ff1a2a9bb129ef4f6c # bad: [ac7b75966c9c86426b55fe1c50ae148aa4571075] Merge tag 'pinctrl-v4.13-1' of git://git.kernel.org/pub/scm/linux/kernel/git/linusw/linux-pinctrl git bisect bad ac7b75966c9c86426b55fe1c50ae148aa4571075 # good: [e24dd9ee5399747b71c1d982a484fc7601795f31] Merge branch 'next' of git://git.kernel.org/pub/scm/linux/kernel/git/jmorris/linux-security git bisect good e24dd9ee5399747b71c1d982a484fc7601795f31 # good: [e24dd9ee5399747b71c1d982a484fc7601795f31] Merge branch 'next' of git://git.kernel.org/pub/scm/linux/kernel/git/jmorris/linux-security git bisect good e24dd9ee5399747b71c1d982a484fc7601795f31 # good: [e24dd9ee5399747b71c1d982a484fc7601795f31] Merge branch 'next' of git://git.kernel.org/pub/scm/linux/kernel/git/jmorris/linux-security git bisect good e24dd9ee5399747b71c1d982a484fc7601795f31 W dniu 2017-09-20 o 12:21, Paweł Staszewski pisze:quoted
Ok kernel crashed with different panic that i didnt catch when i was doing bisect and now my bisection is broken :) git bisect good Bisecting: 1787 revisions left to test after this (roughly 11 steps) error: Your local changes to the following files would be overwritten by checkout: Documentation/00-INDEX Documentation/ABI/stable/sysfs-class-udc Documentation/ABI/testing/configfs-usb-gadget-uac1 Documentation/ABI/testing/ima_policy Documentation/ABI/testing/sysfs-bus-iio Documentation/ABI/testing/sysfs-bus-iio-meas-spec Documentation/ABI/testing/sysfs-bus-iio-timer-stm32 Documentation/ABI/testing/sysfs-class-net Documentation/ABI/testing/sysfs-class-power-twl4030 Documentation/ABI/testing/sysfs-class-typec Documentation/DMA-API.txt Documentation/IRQ-domain.txt Documentation/Makefile Documentation/PCI/MSI-HOWTO.txt Documentation/RCU/00-INDEX Documentation/RCU/Design/Requirements/Requirements.html Documentation/RCU/checklist.txt Documentation/admin-guide/README.rst Documentation/admin-guide/devices.txt Documentation/admin-guide/index.rst Documentation/admin-guide/kernel-parameters.txt Documentation/admin-guide/pm/cpufreq.rst Documentation/admin-guide/pm/intel_pstate.rst Documentation/admin-guide/ras.rst Documentation/arm/Atmel/README Documentation/block/biodoc.txt Documentation/conf.py Documentation/core-api/assoc_array.rst Documentation/core-api/atomic_ops.rst Documentation/core-api/index.rst Documentation/crypto/asymmetric-keys.txt Documentation/dev-tools/index.rst Documentation/dev-tools/sparse.rst Documentation/devicetree/bindings/arm/amlogic.txt Documentation/devicetree/bindings/arm/atmel-at91.txt Documentation/devicetree/bindings/arm/ccn.txt Documentation/devicetree/bindings/arm/cpus.txt Documentation/devicetree/bindings/arm/gemini.txt Documentation/devicetree/bindings/arm/hisilicon/hisilicon.txt Documentation/devicetree/bindings/arm/keystone/keystone.txt Documentation/devicetree/bindings/arm/mediatek.txt Documentation/devicetree/bindings/arm/rockchip.txt Documentation/devicetree/bindings/arm/shmobile.txt Documentation/devicetree/bindings/arm/tegra.txt Documentation/devicetree/bindings/ata/ahci-fsl-qoriq.txt Documentation/devicetree/bindings/bus/brcm,gisb-arb.txt Documentation/devicetree/bindings/clock/brcm,iproc-clocks.txt Documentation/devicetree/bindings/cpufreq/ti-cpufreq.txt Documentation/devicetree/bindings/gpio/gpio_atmel.txt Documentation/devicetree/bindings/iio/adc/amlogic,meson-saradc.txt Documentation/devicetree/bindings/iio/adc/renesas,gyroadc.txt Documentation/devicetree/bindings/iio/adc/st,stm32-adc.txt Documentation/devicetree/bindings/iio/imu/st_lsm6dsx.txt Documentation/devicetree/bindings/interrupt-controller/allwinner,sunxi-nmi.txt Documentation/devicetree/bindings/interrupt-controller/aspeed,ast2400-vic.txt Documentation/devicetree/bindings/interrupt-controller/mediatek,sysirq.txt Documentation/devicetree/bindings/leds/common.txt Documentation/devicetree/bindings/mfd/hi6421.txt Documentation/devicetree/bindings/mfd/tps65910.txt Documentation/devicetree/bindings/mmc/fsl-esdhc.txt Documentation/devicetree/bindings/mmc/k3-dw-mshc.txt Documentation/devicetree/bindings/mmc/rockchip-dw-mshc.txt Documentation/devicetree/bindings/mmc/ti-omap-hsmmc.txt Documentation/devicetree/bindings/mtd/atmel-nand.txt Documentation/devicetree/bindings/net/dsa/b53.txt Documentation/devicetree/bindings/net/ethernet.txt Documentation/devicetree/bindings/net/macb.txt Documentation/devicetree/bindings/net/marvell-orion-mdio.txt Documentation/devicetree/bindings/net/ti,wilink-st.txt Documentation/devicetree/bindings/net/wireless/ti,wlcore.txt Documentation/devicetree/bindings/nvmem/rockchip-efuse.txt Documentation/devicetree/bindings/opp/opp.txt Documentation/devicetree/bindings/phy/bcm-ns-usb3-phy.txt Documentation/devicetree/bindings/phy/brcm-sata-phy.txt Documentation/devicetree/bindings/phy/meson8b-usb2-phy.txt Documentation/devicetree/bindings/phy/phy-rockchip-inno-usb2.txt Documentation/devicetree/bindings/power/rockchip-io-domain.txt Documentation/devicetree/bindings/power/supply/bq27xxx.txt Documentation/devicetree/bindings/property-units.txt Documentation/devicetree/bindings/regulator/regulator.txt Documentation/devicetree/bindings/serial/8 error: The following untracked working tree files would be overwritten by checkout: Documentation/ABI/testing/sysfs-class-net-phydev Documentation/DocBook/.gitignore Documentation/DocBook/Makefile Documentation/DocBook/filesystems.tmpl Documentation/DocBook/kernel-hacking.tmpl Documentation/DocBook/kernel-locking.tmpl Documentation/DocBook/kgdb.tmpl Documentation/DocBook/libata.tmpl Documentation/DocBook/librs.tmpl Documentation/DocBook/lsm.tmpl Documentation/DocBook/mtdnand.tmpl Documentation/DocBook/networking.tmpl Documentation/DocBook/rapidio.tmpl Documentation/DocBook/s390-drivers.tmpl Documentation/DocBook/scsi.tmpl Documentation/DocBook/sh.tmpl Documentation/DocBook/stylesheet.xsl Documentation/DocBook/w1.tmpl Documentation/DocBook/z8530book.tmpl Documentation/Makefile.sphinx Documentation/RCU/trace.txt Documentation/devicetree/bindings/i2c/i2c-mt6577.txt Documentation/devicetree/bindings/misc/allwinner,syscon.txt Documentation/devicetree/bindings/net/cortina.txt Documentation/devicetree/bindings/net/dsa/ksz.txt Documentation/devicetree/bindings/net/dwmac-sun8i.txt Documentation/devicetree/bindings/net/qca,qca7000.txt Documentation/devicetree/bindings/power/max8903-charger.txt Documentation/devicetree/bindings/power_supply/maxim,max14656.txt Documentation/devicetree/bindings/ptp/brcm,ptp-dte.txt Documentation/devicetree/bindings/timer/moxa,moxart-timer.txt Documentation/doc-guide/docbook.rst Documentation/networking/tls.txt Documentation/prctl/no_new_privs.txt Documentation/prctl/seccomp_filter.txt Documentation/security/00-INDEX Documentation/security/IMA-templates.txt Documentation/security/LSM.txt Documentation/security/LoadPin.txt Documentation/security/SELinux.txt Documentation/security/Smack.txt Documentation/security/Yama.txt Documentation/security/apparmor.txt Documentation/security/conf.py Documentation/security/credentials.txt Documentation/security/keys-ecryptfs.txt Documentation/security/keys-request-key.txt Documentation/security/keys-trusted-encrypted.txt Documentation/security/keys.txt Documentation/security/self-protection.txt Documentation/security/tomoyo.txt Documentation/sphinx/convert_template.sed Documentation/sphinx/post_convert.sed Documentation/sphinx/tmplcvt Documentation/usb/typec.rst Documentation/usb/usb3-debug-port.rst arch/arm/boot/dts/rk1108-evb.dts arch/arm/boot/dts/rk1108.dtsi arch/arm/boot/dts/tegra20-whistler.dts arch/arm/mach-omap2/opp.c arch/arm/mach-omap2/pmu.c arch/ia64/include/asm/siginfo.h arch/m32r/include/uapi/asm/siginfo.h arch/microblaze/include/asm/bitops.h arch/microblaze/include/asm/bug.h arch/microblaze/include/asm/bugs.h arch/microblaze/include/asm/div64.h arch/microblaze/include/asm/emergency-restart.h arch/microblaze/include/asm/fb.h arch/microblaze/include/asm/hardirq.h arch/microblaze/include/asm/irq_regs.h arch/microblaze/include/asm/kdebug.h arch/microblaze/include/asm/kmap_types.h arch/microblaze/include/asm/linkage.h arch/microblaze/include/asm/local.h arch/microblaze/include/asm/local64.h arch/microblaze/include/asm/parport.h arch/microblaze/include/asm/percpu.h arch/microblaze/include/asm/serial.h arch/microblaze/include/asm/shmparam.h arch/microblaze/include/asm/topology.h arch/microblaze/include/asm/ucontext.h arch/microblaze/include/asm/vga.h arch/microblaze/include/asm/xor.h arch/microblaze/include/uapi/asm/bitsperlong.h arch/microblaze/include/uapi/asm/errno.h arch/microblaze/include/uapi/asm/fcntl.h arch/microblaze/include/uapi/asm/ioctl.h arch/microblaze/include/uapi/asm/ioctls.h arch/microblaze/include/uapi/asm/ipcbuf.h arch/microblaze/include/uapi/asm/kvm_para.h arch/microblaze/include/uapi/asm/mman.h arch/microblaze/include/uapi/asm/msgbuf.h arch/microblaze/include/uapi/asm/param.h arch/microblaze/include/uapi/asm/poll.h arch/microblaze/include/uapi/asm/resource.h arch/microblaze/include/uapi/asm/sembuf.h arch/microblaze/include/uapi/asm/shmbuf.h arch/microblaze/include/uapi/asm/siginfo.h arch/microblaze/include/uapi/asm/signal.h arch/microblaze/includ Aborting W dniu 2017-09-20 o 11:45, Paweł Staszewski pisze:quoted
Ok looks like ending bisection Latest bisected kernel when there is no kernel panic 4.12.0+ (from next) - but only this warning: [ 309.030019] NETDEV WATCHDOG: enp4s0f0 (ixgbe): transmit queue 0 timed out [ 309.030034] ------------[ cut here ]------------ [ 309.030040] WARNING: CPU: 35 PID: 0 at dev_watchdog+0xcf/0x139 [ 309.030041] Modules linked in: bonding ipmi_si x86_pkg_temp_thermal [ 309.030045] CPU: 35 PID: 0 Comm: swapper/35 Not tainted 4.12.0+ #5 [ 309.030046] task: ffff88086d98a000 task.stack: ffffc90003378000 [ 309.030048] RIP: 0010:dev_watchdog+0xcf/0x139 [ 309.030049] RSP: 0018:ffff88087fbc3ea8 EFLAGS: 00010246 [ 309.030050] RAX: 000000000000003d RBX: ffff88046b680000 RCX: 0000000000000000 [ 309.030050] RDX: ffff88087fbd2f01 RSI: 0000000000000000 RDI: ffff88087fbcda08 [ 309.030051] RBP: ffff88087fbc3eb8 R08: 0000000000000000 R09: ffff88087ff80a04 [ 309.030051] R10: 0000000000000000 R11: ffff88086d98a001 R12: 0000000000000000 [ 309.030052] R13: ffff88087fbc3ef8 R14: ffff88086d98a000 R15: ffffffff81c06008 [ 309.030053] FS: 0000000000000000(0000) GS:ffff88087fbc0000(0000) knlGS:0000000000000000 [ 309.030054] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 309.030054] CR2: 00007fba600f6098 CR3: 000000086b955000 CR4: 00000000001406e0 [ 309.030055] Call Trace: [ 309.030057] <IRQ> [ 309.030059] ? netif_tx_lock+0x79/0x79 [ 309.030062] call_timer_fn.isra.24+0x17/0x77 [ 309.030063] run_timer_softirq+0x118/0x161 [ 309.030065] ? netif_tx_lock+0x79/0x79 [ 309.030066] ? ktime_get+0x2b/0x42 [ 309.030070] ? lapic_next_deadline+0x21/0x27 [ 309.030073] ? clockevents_program_event+0xa8/0xc5 [ 309.030076] __do_softirq+0xa8/0x19d [ 309.030078] irq_exit+0x5d/0x6b [ 309.030079] smp_apic_timer_interrupt+0x2a/0x36 [ 309.030082] apic_timer_interrupt+0x89/0x90 [ 309.030085] RIP: 0010:mwait_idle+0x4e/0x6a [ 309.030086] RSP: 0018:ffffc9000337be98 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff10 [ 309.030087] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 [ 309.030087] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff88086d98a000 [ 309.030088] RBP: ffffc9000337be98 R08: ffff88046f8279a0 R09: ffff88046f827040 [ 309.030089] R10: ffff88086d98a000 R11: ffff88086d98a000 R12: 0000000000000000 [ 309.030089] R13: ffff88086d98a000 R14: ffff88086d98a000 R15: ffff88086d98a000 [ 309.030090] </IRQ> [ 309.030094] arch_cpu_idle+0xa/0xc [ 309.030095] default_idle_call+0x19/0x1b [ 309.030102] do_idle+0xbc/0x196 [ 309.030104] cpu_startup_entry+0x1d/0x20 [ 309.030105] start_secondary+0xd8/0xdc [ 309.030108] secondary_startup_64+0x9f/0x9f [ 309.030109] Code: cc 75 bd eb 35 48 89 df c6 05 c3 dc 74 00 01 e8 3a 62 fe ff 44 89 e1 48 89 de 48 89 c2 48 c7 c7 0f 65 a4 81 31 c0 e8 3d 4c b5 ff <0f> ff 48 8b 83 e0 01 00 00 48 89 df ff 50 78 48 8b 05 a0 bc 6a [ 309.030128] ---[ end trace 9102cb25703ae2d9 ]--- I just marked it as good - cause this problem above is differend - and im going to: git bisect good Bisecting: 1787 revisions left to test after this (roughly 11 steps) W dniu 2017-09-20 o 10:44, Paweł Staszewski pisze:quoted
Trying to make video from ipmi :) with that results: https://bugzilla.kernel.org/attachment.cgi?id=258521 catched two more lines where it starts - panic from 4.13.2. Now will try tro do some bisection W dniu 2017-09-20 o 09:58, Paweł Staszewski pisze:quoted
Hi Will try bisecting tonight W dniu 2017-09-20 o 05:24, Eric Dumazet pisze:quoted
On Wed, 2017-09-20 at 02:06 +0200, Paweł Staszewski wrote:quoted
Just checked kernel 4.13.2 and same problem Just after start all 6 bgp sessions - and kernel starts to learn routes it panic. https://bugzilla.kernel.org/attachment.cgi?id=258509Unfortunately we have not enough information from these traces. Can you get a full stack trace ? Alternatively, can you bisect ? Thanks.
Re: Latest net-next from GIT panic
From: Eric Dumazet <hidden>
Date: 2017-09-20 13:11:19
Sorry for top-posting, but this is to give context to Wei, since Pawel
used a top posting way to report his bisection.
Wei, can you take a look at Pawel report ?
Crash happens in dst_destroy() at following :
if (dst->dev)
dev_put(dst->dev); <<CRASH>>
dst->dev is not NULL, but netdev->pcpu_refcnt is NULL
65 ff 08 decl %gs:(%rax) // CRASH since rax = NULL
Pawel, please share your netdevices and routing setup ?
Thanks !
On Wed, 2017-09-20 at 14:49 +0200, Paweł Staszewski wrote:And the last one git bisect good Bisecting: 1 revision left to test after this (roughly 1 step) [1cfb71eeb12047bcdbd3e6730ffed66e810a0855] ipv6: take dst->__refcnt for insertion into fib6 tree With this have kernel panic same as always git bisect bad Bisecting: 0 revisions left to test after this (roughly 0 steps) [b838d5e1c5b6e57b10ec8af2268824041e3ea911] ipv4: mark DST_NOGC and remove the operation of dst_free() W dniu 2017-09-20 o 14:23, Paweł Staszewski pisze:quoted
Almost there Bisecting: 6 revisions left to test after this (roughly 3 steps) [ad65a2f05695aced349e308193c6e2a6b1d87112] ipv6: call dst_hold_safe() properly W dniu 2017-09-20 o 13:02, Paweł Staszewski pisze:quoted
Ok resumed and soo far: Panic: # bad: [9cc9a5cb176ccb4f2cda5ac34da5a659926f125f] datapath: Avoid using stack larger than 1024. git bisect bad 9cc9a5cb176ccb4f2cda5ac34da5a659926f125f No panic: # good: [073cf9e20c333ab29744717a23f9e43ec7512a20] Merge branch 'udp-reduce-cache-pressure' git bisect good 073cf9e20c333ab29744717a23f9e43ec7512a20 W dniu 2017-09-20 o 12:22, Paweł Staszewski pisze:quoted
Soo far bisected and marked: git bisect start # bad: [07dd6cc1fff160143e82cf5df78c1db0b6e03355] Linux 4.13.2 git bisect bad 07dd6cc1fff160143e82cf5df78c1db0b6e03355 # good: [5d7d2e03e0f01a992e3521b180c3d3e67905f269] Linux 4.12.13 git bisect good 5d7d2e03e0f01a992e3521b180c3d3e67905f269 # good: [6f7da290413ba713f0cdd9ff1a2a9bb129ef4f6c] Linux 4.12 git bisect good 6f7da290413ba713f0cdd9ff1a2a9bb129ef4f6c # bad: [ac7b75966c9c86426b55fe1c50ae148aa4571075] Merge tag 'pinctrl-v4.13-1' of git://git.kernel.org/pub/scm/linux/kernel/git/linusw/linux-pinctrl git bisect bad ac7b75966c9c86426b55fe1c50ae148aa4571075 # good: [e24dd9ee5399747b71c1d982a484fc7601795f31] Merge branch 'next' of git://git.kernel.org/pub/scm/linux/kernel/git/jmorris/linux-security git bisect good e24dd9ee5399747b71c1d982a484fc7601795f31 # good: [e24dd9ee5399747b71c1d982a484fc7601795f31] Merge branch 'next' of git://git.kernel.org/pub/scm/linux/kernel/git/jmorris/linux-security git bisect good e24dd9ee5399747b71c1d982a484fc7601795f31 # good: [e24dd9ee5399747b71c1d982a484fc7601795f31] Merge branch 'next' of git://git.kernel.org/pub/scm/linux/kernel/git/jmorris/linux-security git bisect good e24dd9ee5399747b71c1d982a484fc7601795f31 W dniu 2017-09-20 o 12:21, Paweł Staszewski pisze:quoted
Ok kernel crashed with different panic that i didnt catch when i was doing bisect and now my bisection is broken :) git bisect good Bisecting: 1787 revisions left to test after this (roughly 11 steps) error: Your local changes to the following files would be overwritten by checkout: Documentation/00-INDEX Documentation/ABI/stable/sysfs-class-udc Documentation/ABI/testing/configfs-usb-gadget-uac1 Documentation/ABI/testing/ima_policy Documentation/ABI/testing/sysfs-bus-iio Documentation/ABI/testing/sysfs-bus-iio-meas-spec Documentation/ABI/testing/sysfs-bus-iio-timer-stm32 Documentation/ABI/testing/sysfs-class-net Documentation/ABI/testing/sysfs-class-power-twl4030 Documentation/ABI/testing/sysfs-class-typec Documentation/DMA-API.txt Documentation/IRQ-domain.txt Documentation/Makefile Documentation/PCI/MSI-HOWTO.txt Documentation/RCU/00-INDEX Documentation/RCU/Design/Requirements/Requirements.html Documentation/RCU/checklist.txt Documentation/admin-guide/README.rst Documentation/admin-guide/devices.txt Documentation/admin-guide/index.rst Documentation/admin-guide/kernel-parameters.txt Documentation/admin-guide/pm/cpufreq.rst Documentation/admin-guide/pm/intel_pstate.rst Documentation/admin-guide/ras.rst Documentation/arm/Atmel/README Documentation/block/biodoc.txt Documentation/conf.py Documentation/core-api/assoc_array.rst Documentation/core-api/atomic_ops.rst Documentation/core-api/index.rst Documentation/crypto/asymmetric-keys.txt Documentation/dev-tools/index.rst Documentation/dev-tools/sparse.rst Documentation/devicetree/bindings/arm/amlogic.txt Documentation/devicetree/bindings/arm/atmel-at91.txt Documentation/devicetree/bindings/arm/ccn.txt Documentation/devicetree/bindings/arm/cpus.txt Documentation/devicetree/bindings/arm/gemini.txt Documentation/devicetree/bindings/arm/hisilicon/hisilicon.txt Documentation/devicetree/bindings/arm/keystone/keystone.txt Documentation/devicetree/bindings/arm/mediatek.txt Documentation/devicetree/bindings/arm/rockchip.txt Documentation/devicetree/bindings/arm/shmobile.txt Documentation/devicetree/bindings/arm/tegra.txt Documentation/devicetree/bindings/ata/ahci-fsl-qoriq.txt Documentation/devicetree/bindings/bus/brcm,gisb-arb.txt Documentation/devicetree/bindings/clock/brcm,iproc-clocks.txt Documentation/devicetree/bindings/cpufreq/ti-cpufreq.txt Documentation/devicetree/bindings/gpio/gpio_atmel.txt Documentation/devicetree/bindings/iio/adc/amlogic,meson-saradc.txt Documentation/devicetree/bindings/iio/adc/renesas,gyroadc.txt Documentation/devicetree/bindings/iio/adc/st,stm32-adc.txt Documentation/devicetree/bindings/iio/imu/st_lsm6dsx.txt Documentation/devicetree/bindings/interrupt-controller/allwinner,sunxi-nmi.txt Documentation/devicetree/bindings/interrupt-controller/aspeed,ast2400-vic.txt Documentation/devicetree/bindings/interrupt-controller/mediatek,sysirq.txt Documentation/devicetree/bindings/leds/common.txt Documentation/devicetree/bindings/mfd/hi6421.txt Documentation/devicetree/bindings/mfd/tps65910.txt Documentation/devicetree/bindings/mmc/fsl-esdhc.txt Documentation/devicetree/bindings/mmc/k3-dw-mshc.txt Documentation/devicetree/bindings/mmc/rockchip-dw-mshc.txt Documentation/devicetree/bindings/mmc/ti-omap-hsmmc.txt Documentation/devicetree/bindings/mtd/atmel-nand.txt Documentation/devicetree/bindings/net/dsa/b53.txt Documentation/devicetree/bindings/net/ethernet.txt Documentation/devicetree/bindings/net/macb.txt Documentation/devicetree/bindings/net/marvell-orion-mdio.txt Documentation/devicetree/bindings/net/ti,wilink-st.txt Documentation/devicetree/bindings/net/wireless/ti,wlcore.txt Documentation/devicetree/bindings/nvmem/rockchip-efuse.txt Documentation/devicetree/bindings/opp/opp.txt Documentation/devicetree/bindings/phy/bcm-ns-usb3-phy.txt Documentation/devicetree/bindings/phy/brcm-sata-phy.txt Documentation/devicetree/bindings/phy/meson8b-usb2-phy.txt Documentation/devicetree/bindings/phy/phy-rockchip-inno-usb2.txt Documentation/devicetree/bindings/power/rockchip-io-domain.txt Documentation/devicetree/bindings/power/supply/bq27xxx.txt Documentation/devicetree/bindings/property-units.txt Documentation/devicetree/bindings/regulator/regulator.txt Documentation/devicetree/bindings/serial/8 error: The following untracked working tree files would be overwritten by checkout: Documentation/ABI/testing/sysfs-class-net-phydev Documentation/DocBook/.gitignore Documentation/DocBook/Makefile Documentation/DocBook/filesystems.tmpl Documentation/DocBook/kernel-hacking.tmpl Documentation/DocBook/kernel-locking.tmpl Documentation/DocBook/kgdb.tmpl Documentation/DocBook/libata.tmpl Documentation/DocBook/librs.tmpl Documentation/DocBook/lsm.tmpl Documentation/DocBook/mtdnand.tmpl Documentation/DocBook/networking.tmpl Documentation/DocBook/rapidio.tmpl Documentation/DocBook/s390-drivers.tmpl Documentation/DocBook/scsi.tmpl Documentation/DocBook/sh.tmpl Documentation/DocBook/stylesheet.xsl Documentation/DocBook/w1.tmpl Documentation/DocBook/z8530book.tmpl Documentation/Makefile.sphinx Documentation/RCU/trace.txt Documentation/devicetree/bindings/i2c/i2c-mt6577.txt Documentation/devicetree/bindings/misc/allwinner,syscon.txt Documentation/devicetree/bindings/net/cortina.txt Documentation/devicetree/bindings/net/dsa/ksz.txt Documentation/devicetree/bindings/net/dwmac-sun8i.txt Documentation/devicetree/bindings/net/qca,qca7000.txt Documentation/devicetree/bindings/power/max8903-charger.txt Documentation/devicetree/bindings/power_supply/maxim,max14656.txt Documentation/devicetree/bindings/ptp/brcm,ptp-dte.txt Documentation/devicetree/bindings/timer/moxa,moxart-timer.txt Documentation/doc-guide/docbook.rst Documentation/networking/tls.txt Documentation/prctl/no_new_privs.txt Documentation/prctl/seccomp_filter.txt Documentation/security/00-INDEX Documentation/security/IMA-templates.txt Documentation/security/LSM.txt Documentation/security/LoadPin.txt Documentation/security/SELinux.txt Documentation/security/Smack.txt Documentation/security/Yama.txt Documentation/security/apparmor.txt Documentation/security/conf.py Documentation/security/credentials.txt Documentation/security/keys-ecryptfs.txt Documentation/security/keys-request-key.txt Documentation/security/keys-trusted-encrypted.txt Documentation/security/keys.txt Documentation/security/self-protection.txt Documentation/security/tomoyo.txt Documentation/sphinx/convert_template.sed Documentation/sphinx/post_convert.sed Documentation/sphinx/tmplcvt Documentation/usb/typec.rst Documentation/usb/usb3-debug-port.rst arch/arm/boot/dts/rk1108-evb.dts arch/arm/boot/dts/rk1108.dtsi arch/arm/boot/dts/tegra20-whistler.dts arch/arm/mach-omap2/opp.c arch/arm/mach-omap2/pmu.c arch/ia64/include/asm/siginfo.h arch/m32r/include/uapi/asm/siginfo.h arch/microblaze/include/asm/bitops.h arch/microblaze/include/asm/bug.h arch/microblaze/include/asm/bugs.h arch/microblaze/include/asm/div64.h arch/microblaze/include/asm/emergency-restart.h arch/microblaze/include/asm/fb.h arch/microblaze/include/asm/hardirq.h arch/microblaze/include/asm/irq_regs.h arch/microblaze/include/asm/kdebug.h arch/microblaze/include/asm/kmap_types.h arch/microblaze/include/asm/linkage.h arch/microblaze/include/asm/local.h arch/microblaze/include/asm/local64.h arch/microblaze/include/asm/parport.h arch/microblaze/include/asm/percpu.h arch/microblaze/include/asm/serial.h arch/microblaze/include/asm/shmparam.h arch/microblaze/include/asm/topology.h arch/microblaze/include/asm/ucontext.h arch/microblaze/include/asm/vga.h arch/microblaze/include/asm/xor.h arch/microblaze/include/uapi/asm/bitsperlong.h arch/microblaze/include/uapi/asm/errno.h arch/microblaze/include/uapi/asm/fcntl.h arch/microblaze/include/uapi/asm/ioctl.h arch/microblaze/include/uapi/asm/ioctls.h arch/microblaze/include/uapi/asm/ipcbuf.h arch/microblaze/include/uapi/asm/kvm_para.h arch/microblaze/include/uapi/asm/mman.h arch/microblaze/include/uapi/asm/msgbuf.h arch/microblaze/include/uapi/asm/param.h arch/microblaze/include/uapi/asm/poll.h arch/microblaze/include/uapi/asm/resource.h arch/microblaze/include/uapi/asm/sembuf.h arch/microblaze/include/uapi/asm/shmbuf.h arch/microblaze/include/uapi/asm/siginfo.h arch/microblaze/include/uapi/asm/signal.h arch/microblaze/includ Aborting W dniu 2017-09-20 o 11:45, Paweł Staszewski pisze:quoted
Ok looks like ending bisection Latest bisected kernel when there is no kernel panic 4.12.0+ (from next) - but only this warning: [ 309.030019] NETDEV WATCHDOG: enp4s0f0 (ixgbe): transmit queue 0 timed out [ 309.030034] ------------[ cut here ]------------ [ 309.030040] WARNING: CPU: 35 PID: 0 at dev_watchdog+0xcf/0x139 [ 309.030041] Modules linked in: bonding ipmi_si x86_pkg_temp_thermal [ 309.030045] CPU: 35 PID: 0 Comm: swapper/35 Not tainted 4.12.0+ #5 [ 309.030046] task: ffff88086d98a000 task.stack: ffffc90003378000 [ 309.030048] RIP: 0010:dev_watchdog+0xcf/0x139 [ 309.030049] RSP: 0018:ffff88087fbc3ea8 EFLAGS: 00010246 [ 309.030050] RAX: 000000000000003d RBX: ffff88046b680000 RCX: 0000000000000000 [ 309.030050] RDX: ffff88087fbd2f01 RSI: 0000000000000000 RDI: ffff88087fbcda08 [ 309.030051] RBP: ffff88087fbc3eb8 R08: 0000000000000000 R09: ffff88087ff80a04 [ 309.030051] R10: 0000000000000000 R11: ffff88086d98a001 R12: 0000000000000000 [ 309.030052] R13: ffff88087fbc3ef8 R14: ffff88086d98a000 R15: ffffffff81c06008 [ 309.030053] FS: 0000000000000000(0000) GS:ffff88087fbc0000(0000) knlGS:0000000000000000 [ 309.030054] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 309.030054] CR2: 00007fba600f6098 CR3: 000000086b955000 CR4: 00000000001406e0 [ 309.030055] Call Trace: [ 309.030057] <IRQ> [ 309.030059] ? netif_tx_lock+0x79/0x79 [ 309.030062] call_timer_fn.isra.24+0x17/0x77 [ 309.030063] run_timer_softirq+0x118/0x161 [ 309.030065] ? netif_tx_lock+0x79/0x79 [ 309.030066] ? ktime_get+0x2b/0x42 [ 309.030070] ? lapic_next_deadline+0x21/0x27 [ 309.030073] ? clockevents_program_event+0xa8/0xc5 [ 309.030076] __do_softirq+0xa8/0x19d [ 309.030078] irq_exit+0x5d/0x6b [ 309.030079] smp_apic_timer_interrupt+0x2a/0x36 [ 309.030082] apic_timer_interrupt+0x89/0x90 [ 309.030085] RIP: 0010:mwait_idle+0x4e/0x6a [ 309.030086] RSP: 0018:ffffc9000337be98 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff10 [ 309.030087] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 [ 309.030087] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff88086d98a000 [ 309.030088] RBP: ffffc9000337be98 R08: ffff88046f8279a0 R09: ffff88046f827040 [ 309.030089] R10: ffff88086d98a000 R11: ffff88086d98a000 R12: 0000000000000000 [ 309.030089] R13: ffff88086d98a000 R14: ffff88086d98a000 R15: ffff88086d98a000 [ 309.030090] </IRQ> [ 309.030094] arch_cpu_idle+0xa/0xc [ 309.030095] default_idle_call+0x19/0x1b [ 309.030102] do_idle+0xbc/0x196 [ 309.030104] cpu_startup_entry+0x1d/0x20 [ 309.030105] start_secondary+0xd8/0xdc [ 309.030108] secondary_startup_64+0x9f/0x9f [ 309.030109] Code: cc 75 bd eb 35 48 89 df c6 05 c3 dc 74 00 01 e8 3a 62 fe ff 44 89 e1 48 89 de 48 89 c2 48 c7 c7 0f 65 a4 81 31 c0 e8 3d 4c b5 ff <0f> ff 48 8b 83 e0 01 00 00 48 89 df ff 50 78 48 8b 05 a0 bc 6a [ 309.030128] ---[ end trace 9102cb25703ae2d9 ]--- I just marked it as good - cause this problem above is differend - and im going to: git bisect good Bisecting: 1787 revisions left to test after this (roughly 11 steps) W dniu 2017-09-20 o 10:44, Paweł Staszewski pisze:quoted
Trying to make video from ipmi :) with that results: https://bugzilla.kernel.org/attachment.cgi?id=258521 catched two more lines where it starts - panic from 4.13.2. Now will try tro do some bisection W dniu 2017-09-20 o 09:58, Paweł Staszewski pisze:quoted
Hi Will try bisecting tonight W dniu 2017-09-20 o 05:24, Eric Dumazet pisze:quoted
On Wed, 2017-09-20 at 02:06 +0200, Paweł Staszewski wrote:quoted
Just checked kernel 4.13.2 and same problem Just after start all 6 bgp sessions - and kernel starts to learn routes it panic. https://bugzilla.kernel.org/attachment.cgi?id=258509Unfortunately we have not enough information from these traces. Can you get a full stack trace ? Alternatively, can you bisect ? Thanks.
Re: Latest net-next from GIT panic
From: Paweł Staszewski <hidden>
Date: 2017-09-20 13:11:43
So far path for bisect was: git bisect start # bad: [ac7b75966c9c86426b55fe1c50ae148aa4571075] Merge tag 'pinctrl-v4.13-1' of git://git.kernel.org/pub/scm/linux/kernel/git/linusw/linux-pinctrl git bisect bad ac7b75966c9c86426b55fe1c50ae148aa4571075 # good: [e24dd9ee5399747b71c1d982a484fc7601795f31] Merge branch 'next' of git://git.kernel.org/pub/scm/linux/kernel/git/jmorris/linux-security git bisect good e24dd9ee5399747b71c1d982a484fc7601795f31 # bad: [9cc9a5cb176ccb4f2cda5ac34da5a659926f125f] datapath: Avoid using stack larger than 1024. git bisect bad 9cc9a5cb176ccb4f2cda5ac34da5a659926f125f # good: [073cf9e20c333ab29744717a23f9e43ec7512a20] Merge branch 'udp-reduce-cache-pressure' git bisect good 073cf9e20c333ab29744717a23f9e43ec7512a20 # bad: [8abd5599a520e9f188a750f1bde9dde5fb856230] Merge branch 's390-net-updates-part-2' git bisect bad 8abd5599a520e9f188a750f1bde9dde5fb856230 # good: [2fae5d0e647c6470d206e72b5fc24972bb900f70] Merge branch 'bpf-ctx-narrow' git bisect good 2fae5d0e647c6470d206e72b5fc24972bb900f70 # good: [41500c3e2a19ffcf40a7158fce1774de08e26ba2] rds: tcp: remove cp_outgoing git bisect good 41500c3e2a19ffcf40a7158fce1774de08e26ba2 # bad: [8917a777be3ba566377be05117f71b93a5fd909d] tcp: md5: add TCP_MD5SIG_EXT socket option to set a key address prefix git bisect bad 8917a777be3ba566377be05117f71b93a5fd909d # good: [4a6ce2b6f2ecabbddcfe47e7cf61dd0f00b10e36] net: introduce a new function dst_dev_put() git bisect good 4a6ce2b6f2ecabbddcfe47e7cf61dd0f00b10e36 # bad: [a4c2fd7f78915a0d7c5275e7612e7793157a01f2] net: remove DST_NOCACHE flag git bisect bad a4c2fd7f78915a0d7c5275e7612e7793157a01f2 # bad: [ad65a2f05695aced349e308193c6e2a6b1d87112] ipv6: call dst_hold_safe() properly git bisect bad ad65a2f05695aced349e308193c6e2a6b1d87112 No PANIC # good: [9df16efadd2a8a82731dc76ff656c771e261827f] ipv4: call dst_hold_safe() properly git bisect good 9df16efadd2a8a82731dc76ff656c771e261827f PANIC # bad: [1cfb71eeb12047bcdbd3e6730ffed66e810a0855] ipv6: take dst->__refcnt for insertion into fib6 tree git bisect bad 1cfb71eeb12047bcdbd3e6730ffed66e810a0855 PANIC # bad: [b838d5e1c5b6e57b10ec8af2268824041e3ea911] ipv4: mark DST_NOGC and remove the operation of dst_free() git bisect bad b838d5e1c5b6e57b10ec8af2268824041e3ea911 # first bad commit: [b838d5e1c5b6e57b10ec8af2268824041e3ea911] ipv4: mark DST_NOGC and remove the operation of dst_free() W dniu 2017-09-20 o 15:05, Paweł Staszewski pisze:
hmm But after b838d5e1c5b6e57b10ec8af2268824041e3ea911 is the first bad commit commit b838d5e1c5b6e57b10ec8af2268824041e3ea911 Author: Wei Wang [off-list ref] Date: Sat Jun 17 10:42:32 2017 -0700 ipv4: mark DST_NOGC and remove the operation of dst_free() With the previous preparation patches, we are ready to get rid of the dst gc operation in ipv4 code and release dst based on refcnt only. So this patch adds DST_NOGC flag for all IPv4 dst and remove the calls to dst_free(). At this point, all dst created in ipv4 code do not use the dst gc anymore and will be destroyed at the point when refcnt drops to 0. Signed-off-by: Wei Wang [off-list ref] Acked-by: Martin KaFai Lau [off-list ref] Signed-off-by: David S. Miller [off-list ref] :040000 040000 9b7e7fb641de6531fc7887473ca47ef7cb6a11da 831a73b71d3df1755f3e24c0d3c86d7a93fd55e2 M net Still panic - soo will back to past 3 steps and will try to get again bisect without panic. W dniu 2017-09-20 o 14:49, Paweł Staszewski pisze:quoted
And the last one git bisect good Bisecting: 1 revision left to test after this (roughly 1 step) [1cfb71eeb12047bcdbd3e6730ffed66e810a0855] ipv6: take dst->__refcnt for insertion into fib6 tree With this have kernel panic same as always git bisect bad Bisecting: 0 revisions left to test after this (roughly 0 steps) [b838d5e1c5b6e57b10ec8af2268824041e3ea911] ipv4: mark DST_NOGC and remove the operation of dst_free() W dniu 2017-09-20 o 14:23, Paweł Staszewski pisze:quoted
Almost there Bisecting: 6 revisions left to test after this (roughly 3 steps) [ad65a2f05695aced349e308193c6e2a6b1d87112] ipv6: call dst_hold_safe() properly W dniu 2017-09-20 o 13:02, Paweł Staszewski pisze:quoted
Ok resumed and soo far: Panic: # bad: [9cc9a5cb176ccb4f2cda5ac34da5a659926f125f] datapath: Avoid using stack larger than 1024. git bisect bad 9cc9a5cb176ccb4f2cda5ac34da5a659926f125f No panic: # good: [073cf9e20c333ab29744717a23f9e43ec7512a20] Merge branch 'udp-reduce-cache-pressure' git bisect good 073cf9e20c333ab29744717a23f9e43ec7512a20 W dniu 2017-09-20 o 12:22, Paweł Staszewski pisze:quoted
Soo far bisected and marked: git bisect start # bad: [07dd6cc1fff160143e82cf5df78c1db0b6e03355] Linux 4.13.2 git bisect bad 07dd6cc1fff160143e82cf5df78c1db0b6e03355 # good: [5d7d2e03e0f01a992e3521b180c3d3e67905f269] Linux 4.12.13 git bisect good 5d7d2e03e0f01a992e3521b180c3d3e67905f269 # good: [6f7da290413ba713f0cdd9ff1a2a9bb129ef4f6c] Linux 4.12 git bisect good 6f7da290413ba713f0cdd9ff1a2a9bb129ef4f6c # bad: [ac7b75966c9c86426b55fe1c50ae148aa4571075] Merge tag 'pinctrl-v4.13-1' of git://git.kernel.org/pub/scm/linux/kernel/git/linusw/linux-pinctrl git bisect bad ac7b75966c9c86426b55fe1c50ae148aa4571075 # good: [e24dd9ee5399747b71c1d982a484fc7601795f31] Merge branch 'next' of git://git.kernel.org/pub/scm/linux/kernel/git/jmorris/linux-security git bisect good e24dd9ee5399747b71c1d982a484fc7601795f31 # good: [e24dd9ee5399747b71c1d982a484fc7601795f31] Merge branch 'next' of git://git.kernel.org/pub/scm/linux/kernel/git/jmorris/linux-security git bisect good e24dd9ee5399747b71c1d982a484fc7601795f31 # good: [e24dd9ee5399747b71c1d982a484fc7601795f31] Merge branch 'next' of git://git.kernel.org/pub/scm/linux/kernel/git/jmorris/linux-security git bisect good e24dd9ee5399747b71c1d982a484fc7601795f31 W dniu 2017-09-20 o 12:21, Paweł Staszewski pisze:quoted
Ok kernel crashed with different panic that i didnt catch when i was doing bisect and now my bisection is broken :) git bisect good Bisecting: 1787 revisions left to test after this (roughly 11 steps) error: Your local changes to the following files would be overwritten by checkout: Documentation/00-INDEX Documentation/ABI/stable/sysfs-class-udc Documentation/ABI/testing/configfs-usb-gadget-uac1 Documentation/ABI/testing/ima_policy Documentation/ABI/testing/sysfs-bus-iio Documentation/ABI/testing/sysfs-bus-iio-meas-spec Documentation/ABI/testing/sysfs-bus-iio-timer-stm32 Documentation/ABI/testing/sysfs-class-net Documentation/ABI/testing/sysfs-class-power-twl4030 Documentation/ABI/testing/sysfs-class-typec Documentation/DMA-API.txt Documentation/IRQ-domain.txt Documentation/Makefile Documentation/PCI/MSI-HOWTO.txt Documentation/RCU/00-INDEX Documentation/RCU/Design/Requirements/Requirements.html Documentation/RCU/checklist.txt Documentation/admin-guide/README.rst Documentation/admin-guide/devices.txt Documentation/admin-guide/index.rst Documentation/admin-guide/kernel-parameters.txt Documentation/admin-guide/pm/cpufreq.rst Documentation/admin-guide/pm/intel_pstate.rst Documentation/admin-guide/ras.rst Documentation/arm/Atmel/README Documentation/block/biodoc.txt Documentation/conf.py Documentation/core-api/assoc_array.rst Documentation/core-api/atomic_ops.rst Documentation/core-api/index.rst Documentation/crypto/asymmetric-keys.txt Documentation/dev-tools/index.rst Documentation/dev-tools/sparse.rst Documentation/devicetree/bindings/arm/amlogic.txt Documentation/devicetree/bindings/arm/atmel-at91.txt Documentation/devicetree/bindings/arm/ccn.txt Documentation/devicetree/bindings/arm/cpus.txt Documentation/devicetree/bindings/arm/gemini.txt Documentation/devicetree/bindings/arm/hisilicon/hisilicon.txt Documentation/devicetree/bindings/arm/keystone/keystone.txt Documentation/devicetree/bindings/arm/mediatek.txt Documentation/devicetree/bindings/arm/rockchip.txt Documentation/devicetree/bindings/arm/shmobile.txt Documentation/devicetree/bindings/arm/tegra.txt Documentation/devicetree/bindings/ata/ahci-fsl-qoriq.txt Documentation/devicetree/bindings/bus/brcm,gisb-arb.txt Documentation/devicetree/bindings/clock/brcm,iproc-clocks.txt Documentation/devicetree/bindings/cpufreq/ti-cpufreq.txt Documentation/devicetree/bindings/gpio/gpio_atmel.txt Documentation/devicetree/bindings/iio/adc/amlogic,meson-saradc.txt Documentation/devicetree/bindings/iio/adc/renesas,gyroadc.txt Documentation/devicetree/bindings/iio/adc/st,stm32-adc.txt Documentation/devicetree/bindings/iio/imu/st_lsm6dsx.txt Documentation/devicetree/bindings/interrupt-controller/allwinner,sunxi-nmi.txt Documentation/devicetree/bindings/interrupt-controller/aspeed,ast2400-vic.txt Documentation/devicetree/bindings/interrupt-controller/mediatek,sysirq.txt Documentation/devicetree/bindings/leds/common.txt Documentation/devicetree/bindings/mfd/hi6421.txt Documentation/devicetree/bindings/mfd/tps65910.txt Documentation/devicetree/bindings/mmc/fsl-esdhc.txt Documentation/devicetree/bindings/mmc/k3-dw-mshc.txt Documentation/devicetree/bindings/mmc/rockchip-dw-mshc.txt Documentation/devicetree/bindings/mmc/ti-omap-hsmmc.txt Documentation/devicetree/bindings/mtd/atmel-nand.txt Documentation/devicetree/bindings/net/dsa/b53.txt Documentation/devicetree/bindings/net/ethernet.txt Documentation/devicetree/bindings/net/macb.txt Documentation/devicetree/bindings/net/marvell-orion-mdio.txt Documentation/devicetree/bindings/net/ti,wilink-st.txt Documentation/devicetree/bindings/net/wireless/ti,wlcore.txt Documentation/devicetree/bindings/nvmem/rockchip-efuse.txt Documentation/devicetree/bindings/opp/opp.txt Documentation/devicetree/bindings/phy/bcm-ns-usb3-phy.txt Documentation/devicetree/bindings/phy/brcm-sata-phy.txt Documentation/devicetree/bindings/phy/meson8b-usb2-phy.txt Documentation/devicetree/bindings/phy/phy-rockchip-inno-usb2.txt Documentation/devicetree/bindings/power/rockchip-io-domain.txt Documentation/devicetree/bindings/power/supply/bq27xxx.txt Documentation/devicetree/bindings/property-units.txt Documentation/devicetree/bindings/regulator/regulator.txt Documentation/devicetree/bindings/serial/8 error: The following untracked working tree files would be overwritten by checkout: Documentation/ABI/testing/sysfs-class-net-phydev Documentation/DocBook/.gitignore Documentation/DocBook/Makefile Documentation/DocBook/filesystems.tmpl Documentation/DocBook/kernel-hacking.tmpl Documentation/DocBook/kernel-locking.tmpl Documentation/DocBook/kgdb.tmpl Documentation/DocBook/libata.tmpl Documentation/DocBook/librs.tmpl Documentation/DocBook/lsm.tmpl Documentation/DocBook/mtdnand.tmpl Documentation/DocBook/networking.tmpl Documentation/DocBook/rapidio.tmpl Documentation/DocBook/s390-drivers.tmpl Documentation/DocBook/scsi.tmpl Documentation/DocBook/sh.tmpl Documentation/DocBook/stylesheet.xsl Documentation/DocBook/w1.tmpl Documentation/DocBook/z8530book.tmpl Documentation/Makefile.sphinx Documentation/RCU/trace.txt Documentation/devicetree/bindings/i2c/i2c-mt6577.txt Documentation/devicetree/bindings/misc/allwinner,syscon.txt Documentation/devicetree/bindings/net/cortina.txt Documentation/devicetree/bindings/net/dsa/ksz.txt Documentation/devicetree/bindings/net/dwmac-sun8i.txt Documentation/devicetree/bindings/net/qca,qca7000.txt Documentation/devicetree/bindings/power/max8903-charger.txt Documentation/devicetree/bindings/power_supply/maxim,max14656.txt Documentation/devicetree/bindings/ptp/brcm,ptp-dte.txt Documentation/devicetree/bindings/timer/moxa,moxart-timer.txt Documentation/doc-guide/docbook.rst Documentation/networking/tls.txt Documentation/prctl/no_new_privs.txt Documentation/prctl/seccomp_filter.txt Documentation/security/00-INDEX Documentation/security/IMA-templates.txt Documentation/security/LSM.txt Documentation/security/LoadPin.txt Documentation/security/SELinux.txt Documentation/security/Smack.txt Documentation/security/Yama.txt Documentation/security/apparmor.txt Documentation/security/conf.py Documentation/security/credentials.txt Documentation/security/keys-ecryptfs.txt Documentation/security/keys-request-key.txt Documentation/security/keys-trusted-encrypted.txt Documentation/security/keys.txt Documentation/security/self-protection.txt Documentation/security/tomoyo.txt Documentation/sphinx/convert_template.sed Documentation/sphinx/post_convert.sed Documentation/sphinx/tmplcvt Documentation/usb/typec.rst Documentation/usb/usb3-debug-port.rst arch/arm/boot/dts/rk1108-evb.dts arch/arm/boot/dts/rk1108.dtsi arch/arm/boot/dts/tegra20-whistler.dts arch/arm/mach-omap2/opp.c arch/arm/mach-omap2/pmu.c arch/ia64/include/asm/siginfo.h arch/m32r/include/uapi/asm/siginfo.h arch/microblaze/include/asm/bitops.h arch/microblaze/include/asm/bug.h arch/microblaze/include/asm/bugs.h arch/microblaze/include/asm/div64.h arch/microblaze/include/asm/emergency-restart.h arch/microblaze/include/asm/fb.h arch/microblaze/include/asm/hardirq.h arch/microblaze/include/asm/irq_regs.h arch/microblaze/include/asm/kdebug.h arch/microblaze/include/asm/kmap_types.h arch/microblaze/include/asm/linkage.h arch/microblaze/include/asm/local.h arch/microblaze/include/asm/local64.h arch/microblaze/include/asm/parport.h arch/microblaze/include/asm/percpu.h arch/microblaze/include/asm/serial.h arch/microblaze/include/asm/shmparam.h arch/microblaze/include/asm/topology.h arch/microblaze/include/asm/ucontext.h arch/microblaze/include/asm/vga.h arch/microblaze/include/asm/xor.h arch/microblaze/include/uapi/asm/bitsperlong.h arch/microblaze/include/uapi/asm/errno.h arch/microblaze/include/uapi/asm/fcntl.h arch/microblaze/include/uapi/asm/ioctl.h arch/microblaze/include/uapi/asm/ioctls.h arch/microblaze/include/uapi/asm/ipcbuf.h arch/microblaze/include/uapi/asm/kvm_para.h arch/microblaze/include/uapi/asm/mman.h arch/microblaze/include/uapi/asm/msgbuf.h arch/microblaze/include/uapi/asm/param.h arch/microblaze/include/uapi/asm/poll.h arch/microblaze/include/uapi/asm/resource.h arch/microblaze/include/uapi/asm/sembuf.h arch/microblaze/include/uapi/asm/shmbuf.h arch/microblaze/include/uapi/asm/siginfo.h arch/microblaze/include/uapi/asm/signal.h arch/microblaze/includ Aborting W dniu 2017-09-20 o 11:45, Paweł Staszewski pisze:quoted
Ok looks like ending bisection Latest bisected kernel when there is no kernel panic 4.12.0+ (from next) - but only this warning: [ 309.030019] NETDEV WATCHDOG: enp4s0f0 (ixgbe): transmit queue 0 timed out [ 309.030034] ------------[ cut here ]------------ [ 309.030040] WARNING: CPU: 35 PID: 0 at dev_watchdog+0xcf/0x139 [ 309.030041] Modules linked in: bonding ipmi_si x86_pkg_temp_thermal [ 309.030045] CPU: 35 PID: 0 Comm: swapper/35 Not tainted 4.12.0+ #5 [ 309.030046] task: ffff88086d98a000 task.stack: ffffc90003378000 [ 309.030048] RIP: 0010:dev_watchdog+0xcf/0x139 [ 309.030049] RSP: 0018:ffff88087fbc3ea8 EFLAGS: 00010246 [ 309.030050] RAX: 000000000000003d RBX: ffff88046b680000 RCX: 0000000000000000 [ 309.030050] RDX: ffff88087fbd2f01 RSI: 0000000000000000 RDI: ffff88087fbcda08 [ 309.030051] RBP: ffff88087fbc3eb8 R08: 0000000000000000 R09: ffff88087ff80a04 [ 309.030051] R10: 0000000000000000 R11: ffff88086d98a001 R12: 0000000000000000 [ 309.030052] R13: ffff88087fbc3ef8 R14: ffff88086d98a000 R15: ffffffff81c06008 [ 309.030053] FS: 0000000000000000(0000) GS:ffff88087fbc0000(0000) knlGS:0000000000000000 [ 309.030054] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 309.030054] CR2: 00007fba600f6098 CR3: 000000086b955000 CR4: 00000000001406e0 [ 309.030055] Call Trace: [ 309.030057] <IRQ> [ 309.030059] ? netif_tx_lock+0x79/0x79 [ 309.030062] call_timer_fn.isra.24+0x17/0x77 [ 309.030063] run_timer_softirq+0x118/0x161 [ 309.030065] ? netif_tx_lock+0x79/0x79 [ 309.030066] ? ktime_get+0x2b/0x42 [ 309.030070] ? lapic_next_deadline+0x21/0x27 [ 309.030073] ? clockevents_program_event+0xa8/0xc5 [ 309.030076] __do_softirq+0xa8/0x19d [ 309.030078] irq_exit+0x5d/0x6b [ 309.030079] smp_apic_timer_interrupt+0x2a/0x36 [ 309.030082] apic_timer_interrupt+0x89/0x90 [ 309.030085] RIP: 0010:mwait_idle+0x4e/0x6a [ 309.030086] RSP: 0018:ffffc9000337be98 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff10 [ 309.030087] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 [ 309.030087] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff88086d98a000 [ 309.030088] RBP: ffffc9000337be98 R08: ffff88046f8279a0 R09: ffff88046f827040 [ 309.030089] R10: ffff88086d98a000 R11: ffff88086d98a000 R12: 0000000000000000 [ 309.030089] R13: ffff88086d98a000 R14: ffff88086d98a000 R15: ffff88086d98a000 [ 309.030090] </IRQ> [ 309.030094] arch_cpu_idle+0xa/0xc [ 309.030095] default_idle_call+0x19/0x1b [ 309.030102] do_idle+0xbc/0x196 [ 309.030104] cpu_startup_entry+0x1d/0x20 [ 309.030105] start_secondary+0xd8/0xdc [ 309.030108] secondary_startup_64+0x9f/0x9f [ 309.030109] Code: cc 75 bd eb 35 48 89 df c6 05 c3 dc 74 00 01 e8 3a 62 fe ff 44 89 e1 48 89 de 48 89 c2 48 c7 c7 0f 65 a4 81 31 c0 e8 3d 4c b5 ff <0f> ff 48 8b 83 e0 01 00 00 48 89 df ff 50 78 48 8b 05 a0 bc 6a [ 309.030128] ---[ end trace 9102cb25703ae2d9 ]--- I just marked it as good - cause this problem above is differend - and im going to: git bisect good Bisecting: 1787 revisions left to test after this (roughly 11 steps) W dniu 2017-09-20 o 10:44, Paweł Staszewski pisze:quoted
Trying to make video from ipmi :) with that results: https://bugzilla.kernel.org/attachment.cgi?id=258521 catched two more lines where it starts - panic from 4.13.2. Now will try tro do some bisection W dniu 2017-09-20 o 09:58, Paweł Staszewski pisze:quoted
Hi Will try bisecting tonight W dniu 2017-09-20 o 05:24, Eric Dumazet pisze:quoted
On Wed, 2017-09-20 at 02:06 +0200, Paweł Staszewski wrote:quoted
Just checked kernel 4.13.2 and same problem Just after start all 6 bgp sessions - and kernel starts to learn routes it panic. https://bugzilla.kernel.org/attachment.cgi?id=258509Unfortunately we have not enough information from these traces. Can you get a full stack trace ? Alternatively, can you bisect ? Thanks.
Re: Latest net-next from GIT panic
From: Paweł Staszewski <hidden>
Date: 2017-09-20 13:18:23
Yes sorry for top-posting also. Configuration: Ethernet devices: lspci | grep Etherne 02:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) 02:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) 04:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01) 04:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01) 07:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01) 07:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01) 81:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01) 81:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01) 83:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01) 83:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01) ip l 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: enp2s0f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT qlen 8192 link/ether 00:25:90:e4:97:9a brd ff:ff:ff:ff:ff:ff 3: enp2s0f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT qlen 8192 link/ether 00:25:90:e4:97:9b brd ff:ff:ff:ff:ff:ff 4: enp4s0f0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond1 state UP mode DEFAULT qlen 8192 link/ether 0c:c4:7a:bc:b8:68 brd ff:ff:ff:ff:ff:ff 5: enp4s0f1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP mode DEFAULT qlen 8192 link/ether 0c:c4:7a:bc:b8:69 brd ff:ff:ff:ff:ff:ff 6: enp7s0f0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond1 state UP mode DEFAULT qlen 8192 link/ether 0c:c4:7a:bc:b8:68 brd ff:ff:ff:ff:ff:ff 7: enp7s0f1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP mode DEFAULT qlen 8192 link/ether 0c:c4:7a:bc:b8:69 brd ff:ff:ff:ff:ff:ff 8: enp129s0f0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond1 state UP mode DEFAULT qlen 8192 link/ether 0c:c4:7a:bc:b8:68 brd ff:ff:ff:ff:ff:ff 9: enp129s0f1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP mode DEFAULT qlen 8192 link/ether 0c:c4:7a:bc:b8:69 brd ff:ff:ff:ff:ff:ff 10: enp131s0f0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond1 state UP mode DEFAULT qlen 8192 link/ether 0c:c4:7a:bc:b8:68 brd ff:ff:ff:ff:ff:ff 11: enp131s0f1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP mode DEFAULT qlen 8192 link/ether 0c:c4:7a:bc:b8:69 brd ff:ff:ff:ff:ff:ff 12: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN mode DEFAULT qlen 1000 link/sit 0.0.0.0 brd 0.0.0.0 13: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT qlen 1000 link/ether 0c:c4:7a:bc:b8:69 brd ff:ff:ff:ff:ff:ff 14: bond1: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT qlen 1000 link/ether 0c:c4:7a:bc:b8:68 brd ff:ff:ff:ff:ff:ff 15: vlan4091@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT qlen 1000 link/ether 0c:c4:7a:bc:b8:69 brd ff:ff:ff:ff:ff:ff 16: vlan4032@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT qlen 1000 link/ether 0c:c4:7a:bc:b8:69 brd ff:ff:ff:ff:ff:ff 17: vlan514@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT qlen 1000 link/ether 0c:c4:7a:bc:b8:69 brd ff:ff:ff:ff:ff:ff 18: vlan87@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT qlen 1000 link/ether 0c:c4:7a:bc:b8:69 brd ff:ff:ff:ff:ff:ff 19: vlan518@bond1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT qlen 1000 link/ether 0c:c4:7a:bc:b8:68 brd ff:ff:ff:ff:ff:ff 20: vlan646@bond1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT qlen 1000 link/ether 0c:c4:7a:bc:b8:68 brd ff:ff:ff:ff:ff:ff 21: vlan370@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT qlen 1000 link/ether 0c:c4:7a:bc:b8:69 brd ff:ff:ff:ff:ff:ff 22: vlan3212@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT qlen 1000 link/ether 0c:c4:7a:bc:b8:69 brd ff:ff:ff:ff:ff:ff 23: vlan746@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT qlen 1000 link/ether 0c:c4:7a:bc:b8:69 brd ff:ff:ff:ff:ff:ff There are bonds: cat /proc/net/bonding/bond0 Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011) Bonding Mode: load balancing (round-robin) MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 0 Down Delay (ms): 0 Slave Interface: enp4s0f1 MII Status: up Speed: 10000 Mbps Duplex: full Link Failure Count: 1 Permanent HW addr: 0c:c4:7a:bc:b8:69 Slave queue ID: 0 Slave Interface: enp7s0f1 MII Status: up Speed: 10000 Mbps Duplex: full Link Failure Count: 0 Permanent HW addr: 00:25:90:e3:dd:9d Slave queue ID: 0 Slave Interface: enp129s0f1 MII Status: up Speed: 10000 Mbps Duplex: full Link Failure Count: 0 Permanent HW addr: 00:25:90:e3:da:e1 Slave queue ID: 0 Slave Interface: enp131s0f1 MII Status: up Speed: 10000 Mbps Duplex: full Link Failure Count: 0 Permanent HW addr: 0c:c4:7a:bc:b1:fd Slave queue ID: 0 cat /proc/net/bonding/bond1 Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011) Bonding Mode: load balancing (round-robin) MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 0 Down Delay (ms): 0 Slave Interface: enp4s0f0 MII Status: up Speed: 10000 Mbps Duplex: full Link Failure Count: 2 Permanent HW addr: 0c:c4:7a:bc:b8:68 Slave queue ID: 0 Slave Interface: enp7s0f0 MII Status: up Speed: 10000 Mbps Duplex: full Link Failure Count: 1 Permanent HW addr: 00:25:90:e3:dd:9c Slave queue ID: 0 Slave Interface: enp129s0f0 MII Status: up Speed: 10000 Mbps Duplex: full Link Failure Count: 1 Permanent HW addr: 00:25:90:e3:da:e0 Slave queue ID: 0 Slave Interface: enp131s0f0 MII Status: up Speed: 10000 Mbps Duplex: full Link Failure Count: 1 Permanent HW addr: 0c:c4:7a:bc:b1:fc Slave queue ID: 0 About routing - installed frr with bgp/zebra support 6x BGP sessions with full BGP table ~600k prefixes Ando some clients bgp sessions where prefixes from upstreams are advertised. About 20 L3 ipv4 nexthops W dniu 2017-09-20 o 15:11, Eric Dumazet pisze:
Sorry for top-posting, but this is to give context to Wei, since Pawel used a top posting way to report his bisection. Wei, can you take a look at Pawel report ? Crash happens in dst_destroy() at following : if (dst->dev) dev_put(dst->dev); <<CRASH>> dst->dev is not NULL, but netdev->pcpu_refcnt is NULL 65 ff 08 decl %gs:(%rax) // CRASH since rax = NULL Pawel, please share your netdevices and routing setup ? Thanks ! On Wed, 2017-09-20 at 14:49 +0200, Paweł Staszewski wrote:quoted
And the last one git bisect good Bisecting: 1 revision left to test after this (roughly 1 step) [1cfb71eeb12047bcdbd3e6730ffed66e810a0855] ipv6: take dst->__refcnt for insertion into fib6 tree With this have kernel panic same as always git bisect bad Bisecting: 0 revisions left to test after this (roughly 0 steps) [b838d5e1c5b6e57b10ec8af2268824041e3ea911] ipv4: mark DST_NOGC and remove the operation of dst_free() W dniu 2017-09-20 o 14:23, Paweł Staszewski pisze:quoted
Almost there Bisecting: 6 revisions left to test after this (roughly 3 steps) [ad65a2f05695aced349e308193c6e2a6b1d87112] ipv6: call dst_hold_safe() properly W dniu 2017-09-20 o 13:02, Paweł Staszewski pisze:quoted
Ok resumed and soo far: Panic: # bad: [9cc9a5cb176ccb4f2cda5ac34da5a659926f125f] datapath: Avoid using stack larger than 1024. git bisect bad 9cc9a5cb176ccb4f2cda5ac34da5a659926f125f No panic: # good: [073cf9e20c333ab29744717a23f9e43ec7512a20] Merge branch 'udp-reduce-cache-pressure' git bisect good 073cf9e20c333ab29744717a23f9e43ec7512a20 W dniu 2017-09-20 o 12:22, Paweł Staszewski pisze:quoted
Soo far bisected and marked: git bisect start # bad: [07dd6cc1fff160143e82cf5df78c1db0b6e03355] Linux 4.13.2 git bisect bad 07dd6cc1fff160143e82cf5df78c1db0b6e03355 # good: [5d7d2e03e0f01a992e3521b180c3d3e67905f269] Linux 4.12.13 git bisect good 5d7d2e03e0f01a992e3521b180c3d3e67905f269 # good: [6f7da290413ba713f0cdd9ff1a2a9bb129ef4f6c] Linux 4.12 git bisect good 6f7da290413ba713f0cdd9ff1a2a9bb129ef4f6c # bad: [ac7b75966c9c86426b55fe1c50ae148aa4571075] Merge tag 'pinctrl-v4.13-1' of git://git.kernel.org/pub/scm/linux/kernel/git/linusw/linux-pinctrl git bisect bad ac7b75966c9c86426b55fe1c50ae148aa4571075 # good: [e24dd9ee5399747b71c1d982a484fc7601795f31] Merge branch 'next' of git://git.kernel.org/pub/scm/linux/kernel/git/jmorris/linux-security git bisect good e24dd9ee5399747b71c1d982a484fc7601795f31 # good: [e24dd9ee5399747b71c1d982a484fc7601795f31] Merge branch 'next' of git://git.kernel.org/pub/scm/linux/kernel/git/jmorris/linux-security git bisect good e24dd9ee5399747b71c1d982a484fc7601795f31 # good: [e24dd9ee5399747b71c1d982a484fc7601795f31] Merge branch 'next' of git://git.kernel.org/pub/scm/linux/kernel/git/jmorris/linux-security git bisect good e24dd9ee5399747b71c1d982a484fc7601795f31 W dniu 2017-09-20 o 12:21, Paweł Staszewski pisze:quoted
Ok kernel crashed with different panic that i didnt catch when i was doing bisect and now my bisection is broken :) git bisect good Bisecting: 1787 revisions left to test after this (roughly 11 steps) error: Your local changes to the following files would be overwritten by checkout: Documentation/00-INDEX Documentation/ABI/stable/sysfs-class-udc Documentation/ABI/testing/configfs-usb-gadget-uac1 Documentation/ABI/testing/ima_policy Documentation/ABI/testing/sysfs-bus-iio Documentation/ABI/testing/sysfs-bus-iio-meas-spec Documentation/ABI/testing/sysfs-bus-iio-timer-stm32 Documentation/ABI/testing/sysfs-class-net Documentation/ABI/testing/sysfs-class-power-twl4030 Documentation/ABI/testing/sysfs-class-typec Documentation/DMA-API.txt Documentation/IRQ-domain.txt Documentation/Makefile Documentation/PCI/MSI-HOWTO.txt Documentation/RCU/00-INDEX Documentation/RCU/Design/Requirements/Requirements.html Documentation/RCU/checklist.txt Documentation/admin-guide/README.rst Documentation/admin-guide/devices.txt Documentation/admin-guide/index.rst Documentation/admin-guide/kernel-parameters.txt Documentation/admin-guide/pm/cpufreq.rst Documentation/admin-guide/pm/intel_pstate.rst Documentation/admin-guide/ras.rst Documentation/arm/Atmel/README Documentation/block/biodoc.txt Documentation/conf.py Documentation/core-api/assoc_array.rst Documentation/core-api/atomic_ops.rst Documentation/core-api/index.rst Documentation/crypto/asymmetric-keys.txt Documentation/dev-tools/index.rst Documentation/dev-tools/sparse.rst Documentation/devicetree/bindings/arm/amlogic.txt Documentation/devicetree/bindings/arm/atmel-at91.txt Documentation/devicetree/bindings/arm/ccn.txt Documentation/devicetree/bindings/arm/cpus.txt Documentation/devicetree/bindings/arm/gemini.txt Documentation/devicetree/bindings/arm/hisilicon/hisilicon.txt Documentation/devicetree/bindings/arm/keystone/keystone.txt Documentation/devicetree/bindings/arm/mediatek.txt Documentation/devicetree/bindings/arm/rockchip.txt Documentation/devicetree/bindings/arm/shmobile.txt Documentation/devicetree/bindings/arm/tegra.txt Documentation/devicetree/bindings/ata/ahci-fsl-qoriq.txt Documentation/devicetree/bindings/bus/brcm,gisb-arb.txt Documentation/devicetree/bindings/clock/brcm,iproc-clocks.txt Documentation/devicetree/bindings/cpufreq/ti-cpufreq.txt Documentation/devicetree/bindings/gpio/gpio_atmel.txt Documentation/devicetree/bindings/iio/adc/amlogic,meson-saradc.txt Documentation/devicetree/bindings/iio/adc/renesas,gyroadc.txt Documentation/devicetree/bindings/iio/adc/st,stm32-adc.txt Documentation/devicetree/bindings/iio/imu/st_lsm6dsx.txt Documentation/devicetree/bindings/interrupt-controller/allwinner,sunxi-nmi.txt Documentation/devicetree/bindings/interrupt-controller/aspeed,ast2400-vic.txt Documentation/devicetree/bindings/interrupt-controller/mediatek,sysirq.txt Documentation/devicetree/bindings/leds/common.txt Documentation/devicetree/bindings/mfd/hi6421.txt Documentation/devicetree/bindings/mfd/tps65910.txt Documentation/devicetree/bindings/mmc/fsl-esdhc.txt Documentation/devicetree/bindings/mmc/k3-dw-mshc.txt Documentation/devicetree/bindings/mmc/rockchip-dw-mshc.txt Documentation/devicetree/bindings/mmc/ti-omap-hsmmc.txt Documentation/devicetree/bindings/mtd/atmel-nand.txt Documentation/devicetree/bindings/net/dsa/b53.txt Documentation/devicetree/bindings/net/ethernet.txt Documentation/devicetree/bindings/net/macb.txt Documentation/devicetree/bindings/net/marvell-orion-mdio.txt Documentation/devicetree/bindings/net/ti,wilink-st.txt Documentation/devicetree/bindings/net/wireless/ti,wlcore.txt Documentation/devicetree/bindings/nvmem/rockchip-efuse.txt Documentation/devicetree/bindings/opp/opp.txt Documentation/devicetree/bindings/phy/bcm-ns-usb3-phy.txt Documentation/devicetree/bindings/phy/brcm-sata-phy.txt Documentation/devicetree/bindings/phy/meson8b-usb2-phy.txt Documentation/devicetree/bindings/phy/phy-rockchip-inno-usb2.txt Documentation/devicetree/bindings/power/rockchip-io-domain.txt Documentation/devicetree/bindings/power/supply/bq27xxx.txt Documentation/devicetree/bindings/property-units.txt Documentation/devicetree/bindings/regulator/regulator.txt Documentation/devicetree/bindings/serial/8 error: The following untracked working tree files would be overwritten by checkout: Documentation/ABI/testing/sysfs-class-net-phydev Documentation/DocBook/.gitignore Documentation/DocBook/Makefile Documentation/DocBook/filesystems.tmpl Documentation/DocBook/kernel-hacking.tmpl Documentation/DocBook/kernel-locking.tmpl Documentation/DocBook/kgdb.tmpl Documentation/DocBook/libata.tmpl Documentation/DocBook/librs.tmpl Documentation/DocBook/lsm.tmpl Documentation/DocBook/mtdnand.tmpl Documentation/DocBook/networking.tmpl Documentation/DocBook/rapidio.tmpl Documentation/DocBook/s390-drivers.tmpl Documentation/DocBook/scsi.tmpl Documentation/DocBook/sh.tmpl Documentation/DocBook/stylesheet.xsl Documentation/DocBook/w1.tmpl Documentation/DocBook/z8530book.tmpl Documentation/Makefile.sphinx Documentation/RCU/trace.txt Documentation/devicetree/bindings/i2c/i2c-mt6577.txt Documentation/devicetree/bindings/misc/allwinner,syscon.txt Documentation/devicetree/bindings/net/cortina.txt Documentation/devicetree/bindings/net/dsa/ksz.txt Documentation/devicetree/bindings/net/dwmac-sun8i.txt Documentation/devicetree/bindings/net/qca,qca7000.txt Documentation/devicetree/bindings/power/max8903-charger.txt Documentation/devicetree/bindings/power_supply/maxim,max14656.txt Documentation/devicetree/bindings/ptp/brcm,ptp-dte.txt Documentation/devicetree/bindings/timer/moxa,moxart-timer.txt Documentation/doc-guide/docbook.rst Documentation/networking/tls.txt Documentation/prctl/no_new_privs.txt Documentation/prctl/seccomp_filter.txt Documentation/security/00-INDEX Documentation/security/IMA-templates.txt Documentation/security/LSM.txt Documentation/security/LoadPin.txt Documentation/security/SELinux.txt Documentation/security/Smack.txt Documentation/security/Yama.txt Documentation/security/apparmor.txt Documentation/security/conf.py Documentation/security/credentials.txt Documentation/security/keys-ecryptfs.txt Documentation/security/keys-request-key.txt Documentation/security/keys-trusted-encrypted.txt Documentation/security/keys.txt Documentation/security/self-protection.txt Documentation/security/tomoyo.txt Documentation/sphinx/convert_template.sed Documentation/sphinx/post_convert.sed Documentation/sphinx/tmplcvt Documentation/usb/typec.rst Documentation/usb/usb3-debug-port.rst arch/arm/boot/dts/rk1108-evb.dts arch/arm/boot/dts/rk1108.dtsi arch/arm/boot/dts/tegra20-whistler.dts arch/arm/mach-omap2/opp.c arch/arm/mach-omap2/pmu.c arch/ia64/include/asm/siginfo.h arch/m32r/include/uapi/asm/siginfo.h arch/microblaze/include/asm/bitops.h arch/microblaze/include/asm/bug.h arch/microblaze/include/asm/bugs.h arch/microblaze/include/asm/div64.h arch/microblaze/include/asm/emergency-restart.h arch/microblaze/include/asm/fb.h arch/microblaze/include/asm/hardirq.h arch/microblaze/include/asm/irq_regs.h arch/microblaze/include/asm/kdebug.h arch/microblaze/include/asm/kmap_types.h arch/microblaze/include/asm/linkage.h arch/microblaze/include/asm/local.h arch/microblaze/include/asm/local64.h arch/microblaze/include/asm/parport.h arch/microblaze/include/asm/percpu.h arch/microblaze/include/asm/serial.h arch/microblaze/include/asm/shmparam.h arch/microblaze/include/asm/topology.h arch/microblaze/include/asm/ucontext.h arch/microblaze/include/asm/vga.h arch/microblaze/include/asm/xor.h arch/microblaze/include/uapi/asm/bitsperlong.h arch/microblaze/include/uapi/asm/errno.h arch/microblaze/include/uapi/asm/fcntl.h arch/microblaze/include/uapi/asm/ioctl.h arch/microblaze/include/uapi/asm/ioctls.h arch/microblaze/include/uapi/asm/ipcbuf.h arch/microblaze/include/uapi/asm/kvm_para.h arch/microblaze/include/uapi/asm/mman.h arch/microblaze/include/uapi/asm/msgbuf.h arch/microblaze/include/uapi/asm/param.h arch/microblaze/include/uapi/asm/poll.h arch/microblaze/include/uapi/asm/resource.h arch/microblaze/include/uapi/asm/sembuf.h arch/microblaze/include/uapi/asm/shmbuf.h arch/microblaze/include/uapi/asm/siginfo.h arch/microblaze/include/uapi/asm/signal.h arch/microblaze/includ Aborting W dniu 2017-09-20 o 11:45, Paweł Staszewski pisze:quoted
Ok looks like ending bisection Latest bisected kernel when there is no kernel panic 4.12.0+ (from next) - but only this warning: [ 309.030019] NETDEV WATCHDOG: enp4s0f0 (ixgbe): transmit queue 0 timed out [ 309.030034] ------------[ cut here ]------------ [ 309.030040] WARNING: CPU: 35 PID: 0 at dev_watchdog+0xcf/0x139 [ 309.030041] Modules linked in: bonding ipmi_si x86_pkg_temp_thermal [ 309.030045] CPU: 35 PID: 0 Comm: swapper/35 Not tainted 4.12.0+ #5 [ 309.030046] task: ffff88086d98a000 task.stack: ffffc90003378000 [ 309.030048] RIP: 0010:dev_watchdog+0xcf/0x139 [ 309.030049] RSP: 0018:ffff88087fbc3ea8 EFLAGS: 00010246 [ 309.030050] RAX: 000000000000003d RBX: ffff88046b680000 RCX: 0000000000000000 [ 309.030050] RDX: ffff88087fbd2f01 RSI: 0000000000000000 RDI: ffff88087fbcda08 [ 309.030051] RBP: ffff88087fbc3eb8 R08: 0000000000000000 R09: ffff88087ff80a04 [ 309.030051] R10: 0000000000000000 R11: ffff88086d98a001 R12: 0000000000000000 [ 309.030052] R13: ffff88087fbc3ef8 R14: ffff88086d98a000 R15: ffffffff81c06008 [ 309.030053] FS: 0000000000000000(0000) GS:ffff88087fbc0000(0000) knlGS:0000000000000000 [ 309.030054] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 309.030054] CR2: 00007fba600f6098 CR3: 000000086b955000 CR4: 00000000001406e0 [ 309.030055] Call Trace: [ 309.030057] <IRQ> [ 309.030059] ? netif_tx_lock+0x79/0x79 [ 309.030062] call_timer_fn.isra.24+0x17/0x77 [ 309.030063] run_timer_softirq+0x118/0x161 [ 309.030065] ? netif_tx_lock+0x79/0x79 [ 309.030066] ? ktime_get+0x2b/0x42 [ 309.030070] ? lapic_next_deadline+0x21/0x27 [ 309.030073] ? clockevents_program_event+0xa8/0xc5 [ 309.030076] __do_softirq+0xa8/0x19d [ 309.030078] irq_exit+0x5d/0x6b [ 309.030079] smp_apic_timer_interrupt+0x2a/0x36 [ 309.030082] apic_timer_interrupt+0x89/0x90 [ 309.030085] RIP: 0010:mwait_idle+0x4e/0x6a [ 309.030086] RSP: 0018:ffffc9000337be98 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff10 [ 309.030087] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 [ 309.030087] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff88086d98a000 [ 309.030088] RBP: ffffc9000337be98 R08: ffff88046f8279a0 R09: ffff88046f827040 [ 309.030089] R10: ffff88086d98a000 R11: ffff88086d98a000 R12: 0000000000000000 [ 309.030089] R13: ffff88086d98a000 R14: ffff88086d98a000 R15: ffff88086d98a000 [ 309.030090] </IRQ> [ 309.030094] arch_cpu_idle+0xa/0xc [ 309.030095] default_idle_call+0x19/0x1b [ 309.030102] do_idle+0xbc/0x196 [ 309.030104] cpu_startup_entry+0x1d/0x20 [ 309.030105] start_secondary+0xd8/0xdc [ 309.030108] secondary_startup_64+0x9f/0x9f [ 309.030109] Code: cc 75 bd eb 35 48 89 df c6 05 c3 dc 74 00 01 e8 3a 62 fe ff 44 89 e1 48 89 de 48 89 c2 48 c7 c7 0f 65 a4 81 31 c0 e8 3d 4c b5 ff <0f> ff 48 8b 83 e0 01 00 00 48 89 df ff 50 78 48 8b 05 a0 bc 6a [ 309.030128] ---[ end trace 9102cb25703ae2d9 ]--- I just marked it as good - cause this problem above is differend - and im going to: git bisect good Bisecting: 1787 revisions left to test after this (roughly 11 steps) W dniu 2017-09-20 o 10:44, Paweł Staszewski pisze:quoted
Trying to make video from ipmi :) with that results: https://bugzilla.kernel.org/attachment.cgi?id=258521 catched two more lines where it starts - panic from 4.13.2. Now will try tro do some bisection W dniu 2017-09-20 o 09:58, Paweł Staszewski pisze:quoted
Hi Will try bisecting tonight W dniu 2017-09-20 o 05:24, Eric Dumazet pisze:quoted
On Wed, 2017-09-20 at 02:06 +0200, Paweł Staszewski wrote:quoted
Just checked kernel 4.13.2 and same problem Just after start all 6 bgp sessions - and kernel starts to learn routes it panic. https://bugzilla.kernel.org/attachment.cgi?id=258509Unfortunately we have not enough information from these traces. Can you get a full stack trace ? Alternatively, can you bisect ? Thanks.
Re: Latest net-next from GIT panic
From: Eric Dumazet <hidden>
Date: 2017-09-20 13:34:41
Could you try this debug patch ?
diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
index f535779d9dc1dfe36934c2abba4e43d053ac5d6f..1eaa3553a724dc8c048f67b556337072d5addc82 100644
--- a/include/linux/netdevice.h
+++ b/include/linux/netdevice.h@@ -3331,7 +3331,14 @@ void netdev_run_todo(void); */ static inline void dev_put(struct net_device *dev) { - this_cpu_dec(*dev->pcpu_refcnt); + int __percpu *pref = READ_ONCE(dev->pcpu_refcnt); + + if (!pref) { + pr_err("no pcpu_refcnt on dev %p(%s) state %d dismantle %d\n", + dev, dev->name, dev->reg_state, dev->dismantle); + BUG(); + } + this_cpu_dec(*pref); } /**
Re: Latest net-next from GIT panic
From: Eric Dumazet <hidden>
Date: 2017-09-20 13:37:40
On Wed, 2017-09-20 at 06:34 -0700, Eric Dumazet wrote:
quoted hunk
Could you try this debug patch ?diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index f535779d9dc1dfe36934c2abba4e43d053ac5d6f..1eaa3553a724dc8c048f67b556337072d5addc82 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h@@ -3331,7 +3331,14 @@ void netdev_run_todo(void); */ static inline void dev_put(struct net_device *dev) { - this_cpu_dec(*dev->pcpu_refcnt); + int __percpu *pref = READ_ONCE(dev->pcpu_refcnt); + + if (!pref) { + pr_err("no pcpu_refcnt on dev %p(%s) state %d dismantle %d\n", + dev, dev->name, dev->reg_state, dev->dismantle); + BUG(); + } + this_cpu_dec(*pref); } /**
And since the console will be filled by stack trace, maybe instead of BUG() use some infinite loop ? for (;;) cpu_relax();
Re: Latest net-next from GIT panic
From: Paweł Staszewski <hidden>
Date: 2017-09-20 13:42:05
W dniu 2017-09-20 o 15:34, Eric Dumazet pisze:
quoted hunk
Could you try this debug patch ?diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index f535779d9dc1dfe36934c2abba4e43d053ac5d6f..1eaa3553a724dc8c048f67b556337072d5addc82 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h@@ -3331,7 +3331,14 @@ void netdev_run_todo(void); */ static inline void dev_put(struct net_device *dev) { - this_cpu_dec(*dev->pcpu_refcnt); + int __percpu *pref = READ_ONCE(dev->pcpu_refcnt); + + if (!pref) { + pr_err("no pcpu_refcnt on dev %p(%s) state %d dismantle %d\n", + dev, dev->name, dev->reg_state, dev->dismantle); + BUG(); + } + this_cpu_dec(*pref); } /**
You want me to add this patch to what kernel version ? currently im after git bisect reset - so mainline stable
Re: Latest net-next from GIT panic
From: Eric Dumazet <hidden>
Date: 2017-09-20 13:45:01
On Wed, 2017-09-20 at 15:39 +0200, Paweł Staszewski wrote:
W dniu 2017-09-20 o 15:34, Eric Dumazet pisze:quoted
Could you try this debug patch ?diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index f535779d9dc1dfe36934c2abba4e43d053ac5d6f..1eaa3553a724dc8c048f67b556337072d5addc82 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h@@ -3331,7 +3331,14 @@ void netdev_run_todo(void); */ static inline void dev_put(struct net_device *dev) { - this_cpu_dec(*dev->pcpu_refcnt); + int __percpu *pref = READ_ONCE(dev->pcpu_refcnt); + + if (!pref) { + pr_err("no pcpu_refcnt on dev %p(%s) state %d dismantle %d\n", + dev, dev->name, dev->reg_state, dev->dismantle); + BUG(); + } + this_cpu_dec(*pref); } /**You want me to add this patch to what kernel version ? currently im after git bisect reset - so mainline stable
Simply us the latest net-next as mentioned in the thread title, thanks.
Re: Latest net-next from GIT panic
From: Paweł Staszewski <hidden>
Date: 2017-09-20 14:05:49
Nit much more after adding this patch https://bugzilla.kernel.org/attachment.cgi?id=258529 W dniu 2017-09-20 o 15:44, Eric Dumazet pisze:
On Wed, 2017-09-20 at 15:39 +0200, Paweł Staszewski wrote:quoted
W dniu 2017-09-20 o 15:34, Eric Dumazet pisze:quoted
Could you try this debug patch ?diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index f535779d9dc1dfe36934c2abba4e43d053ac5d6f..1eaa3553a724dc8c048f67b556337072d5addc82 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h@@ -3331,7 +3331,14 @@ void netdev_run_todo(void); */ static inline void dev_put(struct net_device *dev) { - this_cpu_dec(*dev->pcpu_refcnt); + int __percpu *pref = READ_ONCE(dev->pcpu_refcnt); + + if (!pref) { + pr_err("no pcpu_refcnt on dev %p(%s) state %d dismantle %d\n", + dev, dev->name, dev->reg_state, dev->dismantle); + BUG(); + } + this_cpu_dec(*pref); } /**You want me to add this patch to what kernel version ? currently im after git bisect reset - so mainline stableSimply us the latest net-next as mentioned in the thread title, thanks.
Re: Latest net-next from GIT panic
From: Eric Dumazet <hidden>
Date: 2017-09-20 14:40:54
On Wed, 2017-09-20 at 16:03 +0200, Paweł Staszewski wrote:
Nit much more after adding this patch https://bugzilla.kernel.org/attachment.cgi?id=258529
This is why I suggested to replace the BUG() in another mail So :
diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
index f535779d9dc1dfe36934c2abba4e43d053ac5d6f..220cd12456754876edf2d3ef13195e82d70d5c74 100644
--- a/include/linux/netdevice.h
+++ b/include/linux/netdevice.h@@ -3331,7 +3331,15 @@ void netdev_run_todo(void); */ static inline void dev_put(struct net_device *dev) { - this_cpu_dec(*dev->pcpu_refcnt); + int __percpu *pref = READ_ONCE(dev->pcpu_refcnt); + + if (!pref) { + pr_err("no pcpu_refcnt on dev %p(%s) state %d dismantle %d\n", + dev, dev->name, dev->reg_state, dev->dismantle); + for (;;) + cpu_relax(); + } + this_cpu_dec(*pref); } /**
Re: Latest net-next from GIT panic
From: Paweł Staszewski <hidden>
Date: 2017-09-20 15:07:53
W dniu 2017-09-20 o 16:40, Eric Dumazet pisze:
quoted hunk
On Wed, 2017-09-20 at 16:03 +0200, Paweł Staszewski wrote:quoted
Nit much more after adding this patch https://bugzilla.kernel.org/attachment.cgi?id=258529This is why I suggested to replace the BUG() in another mail So :diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index f535779d9dc1dfe36934c2abba4e43d053ac5d6f..220cd12456754876edf2d3ef13195e82d70d5c74 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h@@ -3331,7 +3331,15 @@ void netdev_run_todo(void); */ static inline void dev_put(struct net_device *dev) { - this_cpu_dec(*dev->pcpu_refcnt); + int __percpu *pref = READ_ONCE(dev->pcpu_refcnt); + + if (!pref) { + pr_err("no pcpu_refcnt on dev %p(%s) state %d dismantle %d\n", + dev, dev->name, dev->reg_state, dev->dismantle); + for (;;) + cpu_relax(); + } + this_cpu_dec(*pref); } /**
Full panic https://bugzilla.kernel.org/attachment.cgi?id=258531 I will change patch and apply but later today cause now cant use backup router as testlab - Internet rush hours if something happens this will be bed when second router will have bugged kernel :)
Re: Latest net-next from GIT panic
From: Wei Wang <hidden>
Date: 2017-09-20 17:47:00
quoted
This is why I suggested to replace the BUG() in another mail So :diff --git a/include/linux/netdevice.h b/include/linux/netdevice.hindex f535779d9dc1dfe36934c2abba4e43d053ac5d6f..220cd12456754876edf2d3ef13195e82d70d5c74 100644--- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h@@ -3331,7 +3331,15 @@ void netdev_run_todo(void); */ static inline void dev_put(struct net_device *dev) { - this_cpu_dec(*dev->pcpu_refcnt); + int __percpu *pref = READ_ONCE(dev->pcpu_refcnt); + + if (!pref) { + pr_err("no pcpu_refcnt on dev %p(%s) state %d dismantle%d\n", + dev, dev->name, dev->reg_state, dev->dismantle); + for (;;) + cpu_relax(); + } + this_cpu_dec(*pref); } /**
Thanks a lot Eric for the debug patch. Pawel, I want to confirm with you about the last good commit when you did bisection. You mentioned:
And the last one git bisect good Bisecting: 1 revision left to test after this (roughly 1 step) [1cfb71eeb12047bcdbd3e6730ffed66e810a0855] ipv6: take dst->__refcnt for insertion into fib6 tree With this have kernel panic same as always git bisect bad Bisecting: 0 revisions left to test after this (roughly 0 steps) [b838d5e1c5b6e57b10ec8af2268824041e3ea911] ipv4: mark DST_NOGC and remove the operation of dst_free()
So it breaks right at: [b838d5e1c5b6e57b10ec8af2268824041e3ea911] ipv4: mark DST_NOGC and remove the operation of dst_free() Right? If you sync the image to one commit before the above one: [9df16efadd2a8a82731dc76ff656c771e261827f] ipv4: call dst_hold_safe() properly Does it crash? And could you confirm that your config does not have any IPv6 addresses or routes configured? Thanks. Wei 6:03 +0200, Paweł Staszewski wrote:
quoted
quoted
Nit much more after adding this patch https://bugzilla.kernel.org/attachment.cgi?id=258529This is why I suggested to replace the BUG() in another mail So :diff --git a/include/linux/netdevice.h b/include/linux/netdevice.hindex f535779d9dc1dfe36934c2abba4e43d053ac5d6f..220cd12456754876edf2d3ef13195e82d70d5c74 100644--- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h@@ -3331,7 +3331,15 @@ void netdev_run_todo(void); */ static inline void dev_put(struct net_device *dev) { - this_cpu_dec(*dev->pcpu_refcnt); + int __percpu *pref = READ_ONCE(dev->pcpu_refcnt); + + if (!pref) { + pr_err("no pcpu_refcnt on dev %p(%s) state %d dismantle%d\n", + dev, dev->name, dev->reg_state, dev->dismantle); + for (;;) + cpu_relax(); + } + this_cpu_dec(*pref); } /**Full panic https://bugzilla.kernel.org/attachment.cgi?id=258531 I will change patch and apply but later today cause now cant use backup router as testlab - Internet rush hours if something happens this will be bed when second router will have bugged kernel :)
Re: Latest net-next from GIT panic
From: Cong Wang <hidden>
Date: 2017-09-20 17:50:42
On Wed, Sep 20, 2017 at 6:11 AM, Eric Dumazet [off-list ref] wrote:
Sorry for top-posting, but this is to give context to Wei, since Pawel
used a top posting way to report his bisection.
Wei, can you take a look at Pawel report ?
Crash happens in dst_destroy() at following :
if (dst->dev)
dev_put(dst->dev); <<CRASH>>
dst->dev is not NULL, but netdev->pcpu_refcnt is NULL
65 ff 08 decl %gs:(%rax) // CRASH since rax = NULL
Pawel, please share your netdevices and routing setup ?Looks like a double dev_put() on some dev... Pawel, do you have any idea how this is triggered? Does your test try to remove some network device? If so which one? I noticed you have at least multiple vlan, bond and ixgbe devices.
Re: Latest net-next from GIT panic
From: Eric Dumazet <hidden>
Date: 2017-09-20 17:59:10
On Wed, 2017-09-20 at 10:50 -0700, Cong Wang wrote:
On Wed, Sep 20, 2017 at 6:11 AM, Eric Dumazet [off-list ref] wrote:quoted
Sorry for top-posting, but this is to give context to Wei, since Pawel used a top posting way to report his bisection. Wei, can you take a look at Pawel report ? Crash happens in dst_destroy() at following : if (dst->dev) dev_put(dst->dev); <<CRASH>> dst->dev is not NULL, but netdev->pcpu_refcnt is NULL 65 ff 08 decl %gs:(%rax) // CRASH since rax = NULL Pawel, please share your netdevices and routing setup ?Looks like a double dev_put() on some dev... Pawel, do you have any idea how this is triggered? Does your test try to remove some network device? If so which one? I noticed you have at least multiple vlan, bond and ixgbe devices.
Or a missing dev_hold() somewhere.
Re: Latest net-next from GIT panic
From: Paweł Staszewski <hidden>
Date: 2017-09-20 18:00:59
W dniu 2017-09-20 o 19:46, Wei Wang pisze:
quoted
quoted
This is why I suggested to replace the BUG() in another mail So :diff --git a/include/linux/netdevice.h b/include/linux/netdevice.hindex f535779d9dc1dfe36934c2abba4e43d053ac5d6f..220cd12456754876edf2d3ef13195e82d70d5c74 100644--- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h@@ -3331,7 +3331,15 @@ void netdev_run_todo(void); */ static inline void dev_put(struct net_device *dev) { - this_cpu_dec(*dev->pcpu_refcnt); + int __percpu *pref = READ_ONCE(dev->pcpu_refcnt); + + if (!pref) { + pr_err("no pcpu_refcnt on dev %p(%s) state %d dismantle%d\n", + dev, dev->name, dev->reg_state, dev->dismantle); + for (;;) + cpu_relax(); + } + this_cpu_dec(*pref); } /**Thanks a lot Eric for the debug patch. Pawel, I want to confirm with you about the last good commit when you did bisection. You mentioned:quoted
And the last one git bisect good Bisecting: 1 revision left to test after this (roughly 1 step) [1cfb71eeb12047bcdbd3e6730ffed66e810a0855] ipv6: take dst->__refcnt for insertion into fib6 tree With this have kernel panic same as always git bisect bad Bisecting: 0 revisions left to test after this (roughly 0 steps) [b838d5e1c5b6e57b10ec8af2268824041e3ea911] ipv4: mark DST_NOGC and remove the operation of dst_free()So it breaks right at: [b838d5e1c5b6e57b10ec8af2268824041e3ea911] ipv4: mark DST_NOGC and remove the operation of dst_free() Right? If you sync the image to one commit before the above one: [9df16efadd2a8a82731dc76ff656c771e261827f] ipv4: call dst_hold_safe() properly Does it crash?
Later today i will repeat last three steps - in about next 3 hours after rush hours of internet traffic - now i cant touch backup router :)
And could you confirm that your config does not have any IPv6 addresses or routes configured?
There is ipv6 enabled And yes there are some ipv6 ip's One interface have ipv6 enabled with one static route but no ipv6 bgp sessions - so nt many ipv6 prefixes and ipv6 fib is almost empty ip -6 r ls | wc -l 57
Thanks. Wei 6:03 +0200, Paweł Staszewski wrote:quoted
quoted
quoted
Nit much more after adding this patch https://bugzilla.kernel.org/attachment.cgi?id=258529This is why I suggested to replace the BUG() in another mail So :diff --git a/include/linux/netdevice.h b/include/linux/netdevice.hindex f535779d9dc1dfe36934c2abba4e43d053ac5d6f..220cd12456754876edf2d3ef13195e82d70d5c74 100644--- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h@@ -3331,7 +3331,15 @@ void netdev_run_todo(void); */ static inline void dev_put(struct net_device *dev) { - this_cpu_dec(*dev->pcpu_refcnt); + int __percpu *pref = READ_ONCE(dev->pcpu_refcnt); + + if (!pref) { + pr_err("no pcpu_refcnt on dev %p(%s) state %d dismantle%d\n", + dev, dev->name, dev->reg_state, dev->dismantle); + for (;;) + cpu_relax(); + } + this_cpu_dec(*pref); } /**Full panic https://bugzilla.kernel.org/attachment.cgi?id=258531 I will change patch and apply but later today cause now cant use backup router as testlab - Internet rush hours if something happens this will be bed when second router will have bugged kernel :)