On Wed, 26 Aug 2015 08:46:59 +0000
Shaun Crampton [off-list ref] wrote:
Testing our app at scale on Google¹s GCE, running ~1000 CoreOS hosts: over
approximately 1 hour, I see about 1 in 50 hosts hit one of the Oopses
below and then reboot (I¹m not sure if the different oopses are related to
each other).
The app is Project Calico, which is a datacenter networking fabric.
calico-felix, the process named below, is our per-host agent. The
per-host agent is responsible for reading the network information from a
central server and applying "ip route² and "iptables" updates to the
kernel. We¹re running on CoreOS, with about 100 docker containers/veths
pairs running on each host. calico-felix is running inside one of those
containers. We also run the BIRD BGP stack to redistribute routes around
the datacenter. The errors happen more frequently while Calico is under
load.
I¹m not sure where to go from here. I can reproduce these issues easily
at that scale but I haven¹t managed to boil it down to a small-scale repro
scenario for further investigation (yet).
What in the world is going on with those call traces? E.g.:
There are two functions in the call trace that the kernel knows
nothing about. How did they get in there?
And there is really executable code in there, as can be seen from a
later trace:
Presumably the same two functions as before (loaded at a different
base address but same offsets, 0xd81 and 0x72b). And then nf_iterate
call into another unknown function, and there really is code there
and it's consistent with the oops. And the kernel thinks it's
outside of any normal text section, so it does not try to dump any
code from before the instruction pointer.
0: 48 8b 88 40 03 00 00 mov 0x340(%rax),%rcx
7: e8 1d dd dd ff callq 0xffffffffffdddd29
c: 5d pop %rbp
d: c3 retq
Did you write your own module loader or something?
And the kernel thinks it's
outside of any normal text section, so it does not try to dump any
code from before the instruction pointer.
0: 48 8b 88 40 03 00 00 mov 0x340(%rax),%rcx
7: e8 1d dd dd ff callq 0xffffffffffdddd29
c: 5d pop %rbp
d: c3 retq
Did you write your own module loader or something?
We certainly didn't but CoreOS may have. I've asked CoreOS if they know
what's going on.
Are there any extra diagnostics I can gather from a CoreOS system to help
figure out what's going on there? Is there anything I can do to get more
useful diagnostics when one of these failures occur? As noted, I can
reproduce the issue but it's expensive, requiring hundreds of VMs to
hammer away for an hour or so.
From: Michael Marineau <hidden> Date: 2015-08-26 20:54:37
On Wed, Aug 26, 2015 at 4:49 AM, Chuck Ebbert [off-list ref] wrote:
On Wed, 26 Aug 2015 08:46:59 +0000
Shaun Crampton [off-list ref] wrote:
quoted
Testing our app at scale on Google¹s GCE, running ~1000 CoreOS hosts: over
approximately 1 hour, I see about 1 in 50 hosts hit one of the Oopses
below and then reboot (I¹m not sure if the different oopses are related to
each other).
The app is Project Calico, which is a datacenter networking fabric.
calico-felix, the process named below, is our per-host agent. The
per-host agent is responsible for reading the network information from a
central server and applying "ip route² and "iptables" updates to the
kernel. We¹re running on CoreOS, with about 100 docker containers/veths
pairs running on each host. calico-felix is running inside one of those
containers. We also run the BIRD BGP stack to redistribute routes around
the datacenter. The errors happen more frequently while Calico is under
load.
I¹m not sure where to go from here. I can reproduce these issues easily
at that scale but I haven¹t managed to boil it down to a small-scale repro
scenario for further investigation (yet).
What in the world is going on with those call traces? E.g.:
There are two functions in the call trace that the kernel knows
nothing about. How did they get in there?
And there is really executable code in there, as can be seen from a
later trace:
Presumably the same two functions as before (loaded at a different
base address but same offsets, 0xd81 and 0x72b). And then nf_iterate
call into another unknown function, and there really is code there
and it's consistent with the oops. And the kernel thinks it's
outside of any normal text section, so it does not try to dump any
code from before the instruction pointer.
0: 48 8b 88 40 03 00 00 mov 0x340(%rax),%rcx
7: e8 1d dd dd ff callq 0xffffffffffdddd29
c: 5d pop %rbp
d: c3 retq
Did you write your own module loader or something?
These are stock kernels, with the exception that we include the secure
boot patch set:
https://github.com/coreos/coreos-overlay/tree/master/sys-kernel/coreos-sources/files/4.1
Been a while since kmod got updated so CoreOS is currently shipping
with kmod-15 but beyond being a bit old there isn't anything special
about the module loader.
So nothing particularly magical going on here that I know of.
For reference the original bug report includes a few more varieties of
stack traces: https://github.com/coreos/bugs/issues/435
From: Eric Dumazet <hidden> Date: 2015-08-27 13:00:12
On Wed, 2015-08-26 at 13:54 -0700, Michael Marineau wrote:
On Wed, Aug 26, 2015 at 4:49 AM, Chuck Ebbert [off-list ref] wrote:
quoted
On Wed, 26 Aug 2015 08:46:59 +0000
Shaun Crampton [off-list ref] wrote:
quoted
Testing our app at scale on Google¹s GCE, running ~1000 CoreOS hosts: over
approximately 1 hour, I see about 1 in 50 hosts hit one of the Oopses
below and then reboot (I¹m not sure if the different oopses are related to
each other).
The app is Project Calico, which is a datacenter networking fabric.
calico-felix, the process named below, is our per-host agent. The
per-host agent is responsible for reading the network information from a
central server and applying "ip route² and "iptables" updates to the
kernel. We¹re running on CoreOS, with about 100 docker containers/veths
pairs running on each host. calico-felix is running inside one of those
containers. We also run the BIRD BGP stack to redistribute routes around
the datacenter. The errors happen more frequently while Calico is under
load.
I¹m not sure where to go from here. I can reproduce these issues easily
at that scale but I haven¹t managed to boil it down to a small-scale repro
scenario for further investigation (yet).
What in the world is going on with those call traces? E.g.:
There are two functions in the call trace that the kernel knows
nothing about. How did they get in there?
And there is really executable code in there, as can be seen from a
later trace:
Presumably the same two functions as before (loaded at a different
base address but same offsets, 0xd81 and 0x72b). And then nf_iterate
call into another unknown function, and there really is code there
and it's consistent with the oops. And the kernel thinks it's
outside of any normal text section, so it does not try to dump any
code from before the instruction pointer.
0: 48 8b 88 40 03 00 00 mov 0x340(%rax),%rcx
7: e8 1d dd dd ff callq 0xffffffffffdddd29
c: 5d pop %rbp
d: c3 retq
Did you write your own module loader or something?
These are stock kernels, with the exception that we include the secure
boot patch set:
https://github.com/coreos/coreos-overlay/tree/master/sys-kernel/coreos-sources/files/4.1
Been a while since kmod got updated so CoreOS is currently shipping
with kmod-15 but beyond being a bit old there isn't anything special
about the module loader.
So nothing particularly magical going on here that I know of.
For reference the original bug report includes a few more varieties of
stack traces: https://github.com/coreos/bugs/issues/435
One of these traces mentions ipv4_dst_destroy()
Make sure you backported commit
10e2eb878f3ca07ac2f05fa5ca5e6c4c9174a27a
("udp: fix dst races with multicast early demux")
From: Michael Marineau <hidden> Date: 2015-08-27 16:16:10
On Thu, Aug 27, 2015 at 6:00 AM, Eric Dumazet [off-list ref] wrote:
On Wed, 2015-08-26 at 13:54 -0700, Michael Marineau wrote:
quoted
On Wed, Aug 26, 2015 at 4:49 AM, Chuck Ebbert [off-list ref] wrote:
quoted
On Wed, 26 Aug 2015 08:46:59 +0000
Shaun Crampton [off-list ref] wrote:
quoted
Testing our app at scale on Google¹s GCE, running ~1000 CoreOS hosts: over
approximately 1 hour, I see about 1 in 50 hosts hit one of the Oopses
below and then reboot (I¹m not sure if the different oopses are related to
each other).
The app is Project Calico, which is a datacenter networking fabric.
calico-felix, the process named below, is our per-host agent. The
per-host agent is responsible for reading the network information from a
central server and applying "ip route² and "iptables" updates to the
kernel. We¹re running on CoreOS, with about 100 docker containers/veths
pairs running on each host. calico-felix is running inside one of those
containers. We also run the BIRD BGP stack to redistribute routes around
the datacenter. The errors happen more frequently while Calico is under
load.
I¹m not sure where to go from here. I can reproduce these issues easily
at that scale but I haven¹t managed to boil it down to a small-scale repro
scenario for further investigation (yet).
What in the world is going on with those call traces? E.g.:
There are two functions in the call trace that the kernel knows
nothing about. How did they get in there?
And there is really executable code in there, as can be seen from a
later trace:
Presumably the same two functions as before (loaded at a different
base address but same offsets, 0xd81 and 0x72b). And then nf_iterate
call into another unknown function, and there really is code there
and it's consistent with the oops. And the kernel thinks it's
outside of any normal text section, so it does not try to dump any
code from before the instruction pointer.
0: 48 8b 88 40 03 00 00 mov 0x340(%rax),%rcx
7: e8 1d dd dd ff callq 0xffffffffffdddd29
c: 5d pop %rbp
d: c3 retq
Did you write your own module loader or something?
These are stock kernels, with the exception that we include the secure
boot patch set:
https://github.com/coreos/coreos-overlay/tree/master/sys-kernel/coreos-sources/files/4.1
Been a while since kmod got updated so CoreOS is currently shipping
with kmod-15 but beyond being a bit old there isn't anything special
about the module loader.
So nothing particularly magical going on here that I know of.
For reference the original bug report includes a few more varieties of
stack traces: https://github.com/coreos/bugs/issues/435
One of these traces mentions ipv4_dst_destroy()
Make sure you backported commit
10e2eb878f3ca07ac2f05fa5ca5e6c4c9174a27a
("udp: fix dst races with multicast early demux")
Oh, interesting. Looks like that patch didn't get CC'd to stable
though, is there a reason for that or just oversight?
From: Eric Dumazet <hidden> Date: 2015-08-27 16:30:22
On Thu, 2015-08-27 at 09:16 -0700, Michael Marineau wrote:
Oh, interesting. Looks like that patch didn't get CC'd to stable
though, is there a reason for that or just oversight?
We never CC stable for networking patches.
David Miller prefers to take care of this himself.
( this is in Documentation/networking/netdev-FAQ.txt )
Q: How can I tell what patches are queued up for backporting to the
various stable releases?
A: Normally Greg Kroah-Hartman collects stable commits himself, but
for networking, Dave collects up patches he deems critical for the
networking subsystem, and then hands them off to Greg.
There is a patchworks queue that you can see here:
http://patchwork.ozlabs.org/bundle/davem/stable/?state=*
It contains the patches which Dave has selected, but not yet handed
off to Greg. If Greg already has the patch, then it will be here:
http://git.kernel.org/cgit/linux/kernel/git/stable/stable-queue.git
A quick way to find whether the patch is in this stable-queue is
to simply clone the repo, and then git grep the mainline commit ID, e.g.
stable-queue$ git grep -l 284041ef21fdf2e
releases/3.0.84/ipv6-fix-possible-crashes-in-ip6_cork_release.patch
releases/3.4.51/ipv6-fix-possible-crashes-in-ip6_cork_release.patch
releases/3.9.8/ipv6-fix-possible-crashes-in-ip6_cork_release.patch
stable/stable-queue$
From: Michael Marineau <hidden> Date: 2015-08-27 16:32:40
On Thu, Aug 27, 2015 at 9:30 AM, Eric Dumazet [off-list ref] wrote:
On Thu, 2015-08-27 at 09:16 -0700, Michael Marineau wrote:
quoted
Oh, interesting. Looks like that patch didn't get CC'd to stable
though, is there a reason for that or just oversight?
We never CC stable for networking patches.
David Miller prefers to take care of this himself.
Ah, right, sorry. forgot about that. :)
( this is in Documentation/networking/netdev-FAQ.txt )
Q: How can I tell what patches are queued up for backporting to the
various stable releases?
A: Normally Greg Kroah-Hartman collects stable commits himself, but
for networking, Dave collects up patches he deems critical for the
networking subsystem, and then hands them off to Greg.
There is a patchworks queue that you can see here:
http://patchwork.ozlabs.org/bundle/davem/stable/?state=*
It contains the patches which Dave has selected, but not yet handed
off to Greg. If Greg already has the patch, then it will be here:
http://git.kernel.org/cgit/linux/kernel/git/stable/stable-queue.git
A quick way to find whether the patch is in this stable-queue is
to simply clone the repo, and then git grep the mainline commit ID, e.g.
stable-queue$ git grep -l 284041ef21fdf2e
releases/3.0.84/ipv6-fix-possible-crashes-in-ip6_cork_release.patch
releases/3.4.51/ipv6-fix-possible-crashes-in-ip6_cork_release.patch
releases/3.9.8/ipv6-fix-possible-crashes-in-ip6_cork_release.patch
stable/stable-queue$
From: David Miller <davem@davemloft.net> Date: 2015-08-27 16:40:15
From: Michael Marineau <redacted>
Date: Thu, 27 Aug 2015 09:16:06 -0700
On Thu, Aug 27, 2015 at 6:00 AM, Eric Dumazet [off-list ref] wrote:
quoted
Make sure you backported commit
10e2eb878f3ca07ac2f05fa5ca5e6c4c9174a27a
("udp: fix dst races with multicast early demux")
Oh, interesting. Looks like that patch didn't get CC'd to stable
though, is there a reason for that or just oversight?
All networking bug fixes are submitted to -stable by hand by me at a
time of my choosing. We do not use the "CC: stable" facility, as I
feel it pushes patches into -stable way too quickly and before the
change gets sufficient exposure for regressions in Linus's tree.
The patch in question got submitted last night.
From: Michael Marineau <hidden> Date: 2015-08-27 16:47:50
On Thu, Aug 27, 2015 at 9:40 AM, David Miller [off-list ref] wrote:
From: Michael Marineau <redacted>
Date: Thu, 27 Aug 2015 09:16:06 -0700
quoted
On Thu, Aug 27, 2015 at 6:00 AM, Eric Dumazet [off-list ref] wrote:
quoted
Make sure you backported commit
10e2eb878f3ca07ac2f05fa5ca5e6c4c9174a27a
("udp: fix dst races with multicast early demux")
Oh, interesting. Looks like that patch didn't get CC'd to stable
though, is there a reason for that or just oversight?
All networking bug fixes are submitted to -stable by hand by me at a
time of my choosing. We do not use the "CC: stable" facility, as I
feel it pushes patches into -stable way too quickly and before the
change gets sufficient exposure for regressions in Linus's tree.
The patch in question got submitted last night.
Looking at this one, I am still puzzeled where 0xffffffffa008772b and
0xffffffffa008772b comes from ... some driver, bridge ...? Also the call
to inet_del_offload() seems a bit odd. Even in 4.1, there's only one (buggy)
instance that calls inet_del_offload(), which is ipv6_exthdrs_offload_init(),
but IPPROTO_ROUTING shouldn't have much of an effect on the v4 table as
far as I can see. Maybe rather a false positive that address, hmm? Perhaps
some callback/infrastructure vanished underneath us as ip/rip is both null
... maybe due to that also 0xffffffffa008772b / 0xffffffffa008772b don't
resolve?
Looking at this one, I am still puzzeled where 0xffffffffa008772b and
0xffffffffa008772b comes from ... some driver, bridge ...?
Is there anything I can do on a running system to help figure this out?
Some sort of kernel equivalent to pmap to find out what module or device
owns that chunk of memory?
From: Daniel Borkmann <daniel@iogearbox.net> Date: 2015-09-03 09:03:59
On 09/03/2015 10:13 AM, Shaun Crampton wrote:
...
Is there anything I can do on a running system to help figure this out?
Some sort of kernel equivalent to pmap to find out what module or device
owns that chunk of memory?
Hmm, perhaps /proc/kallsyms could point to something. 0xffffffffa0087d81
and 0xffffffffa008772b could be from the same module, if any.
Is there anything I can do on a running system to help figure this out?
Some sort of kernel equivalent to pmap to find out what module or device
owns that chunk of memory?
Hmm, perhaps /proc/kallsyms could point to something. 0xffffffffa0087d81
and 0xffffffffa008772b could be from the same module, if any.
From: Eric Dumazet <hidden> Date: 2015-09-03 12:10:46
On Thu, 2015-09-03 at 10:09 +0000, Shaun Crampton wrote:
quoted
...
quoted
Is there anything I can do on a running system to help figure this out?
Some sort of kernel equivalent to pmap to find out what module or device
owns that chunk of memory?
Hmm, perhaps /proc/kallsyms could point to something. 0xffffffffa0087d81
and 0xffffffffa008772b could be from the same module, if any.