From: Daniel Borkmann <daniel@iogearbox.net> Date: 2015-08-05 09:06:20
[ please cc netdev ]
On 08/05/2015 10:56 AM, Zang MingJie wrote:
Hi:
I found a bug when remove an ip address which is referenced by a routing entry.
step to reproduce:
ip li add type dummy
ip li set dummy0 up
ip ad add 10.0.0.1/24 dev dummy0
ip ad add 10.0.0.2/24 dev dummy0
ip ro add default via 10.0.0.2/24
ip ad del 10.0.0.2/24 dev dummy0
after deleting the secondary ip address, the routing entry still
pointing to 10.0.0.2
# ip ro
default via 10.0.0.2 dev dummy0
10.0.0.0/24 dev dummy0 proto kernel scope link src 10.0.0.1
but actually, kernel considers the default route is directly connected.
# ip ro get 1.1.1.1
1.1.1.1 dev dummy0 src 10.0.0.1
cache
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Alexander Duyck <hidden> Date: 2015-08-05 17:45:21
On 08/05/2015 02:06 AM, Daniel Borkmann wrote:
[ please cc netdev ]
On 08/05/2015 10:56 AM, Zang MingJie wrote:
quoted
Hi:
I found a bug when remove an ip address which is referenced by a
routing entry.
step to reproduce:
ip li add type dummy
ip li set dummy0 up
ip ad add 10.0.0.1/24 dev dummy0
ip ad add 10.0.0.2/24 dev dummy0
Okay, so up to this point you have 2 addresses on the same subnet that
are now on dummy0.
quoted
ip ro add default via 10.0.0.2/24
This makes the default route go through 10.0.0.2.
quoted
ip ad del 10.0.0.2/24 dev dummy0
Then you remove 10.0.0.2 from the local system, however since 10.0.0.1
is on the same subnet dummy0 would still be the correct interface to
access 10.0.0.2 it is just no longer local to the system.
quoted
after deleting the secondary ip address, the routing entry still
pointing to 10.0.0.2
You didn't delete the default routing entry so why would you expect it
to change? All you did is remove 10.0.0.2 from the local system. I
believe the assumption is that 10.0.0.2 is still out there somewhere, it
just isn't on the local system anymore.
quoted
# ip ro
default via 10.0.0.2 dev dummy0
10.0.0.0/24 dev dummy0 proto kernel scope link src 10.0.0.1
This matches up with what I would expect. 10.0.0.2 is the default
gateway and it is accessible from dummy0 since 10.0.0.0/24 is accessible
from dummy0.
quoted
but actually, kernel considers the default route is directly connected.
# ip ro get 1.1.1.1
1.1.1.1 dev dummy0 src 10.0.0.1
cache
I'm not sure how you came to the "directly connected" conclusion. It is
still routing things out through 10.0.0.2 from 10.0.0.1.
Maybe your example would work better if you used 10.0.0.1 and 10.0.1.1
instead. Then I think you might be able to better see that when you
delete the second address the route would be broken.
- Alex
On Thu, Aug 6, 2015 at 1:45 AM, Alexander Duyck
[off-list ref] wrote:
On 08/05/2015 02:06 AM, Daniel Borkmann wrote:
quoted
[ please cc netdev ]
On 08/05/2015 10:56 AM, Zang MingJie wrote:
quoted
Hi:
I found a bug when remove an ip address which is referenced by a routing
entry.
step to reproduce:
ip li add type dummy
ip li set dummy0 up
ip ad add 10.0.0.1/24 dev dummy0
ip ad add 10.0.0.2/24 dev dummy0
Okay, so up to this point you have 2 addresses on the same subnet that are
now on dummy0.
quoted
quoted
ip ro add default via 10.0.0.2/24
This makes the default route go through 10.0.0.2.
quoted
quoted
ip ad del 10.0.0.2/24 dev dummy0
Then you remove 10.0.0.2 from the local system, however since 10.0.0.1 is on
the same subnet dummy0 would still be the correct interface to access
10.0.0.2 it is just no longer local to the system.
quoted
quoted
after deleting the secondary ip address, the routing entry still
pointing to 10.0.0.2
You didn't delete the default routing entry so why would you expect it to
change? All you did is remove 10.0.0.2 from the local system. I believe
the assumption is that 10.0.0.2 is still out there somewhere, it just isn't
on the local system anymore.
Yes, 10.0.0.2 is migrated to somewhere else
quoted
quoted
# ip ro
default via 10.0.0.2 dev dummy0
10.0.0.0/24 dev dummy0 proto kernel scope link src 10.0.0.1
This matches up with what I would expect. 10.0.0.2 is the default gateway
and it is accessible from dummy0 since 10.0.0.0/24 is accessible from
dummy0.
This means 0.0.0.0/0 is accessible via 10.0.0.2 on the network of dummy0
quoted
quoted
but actually, kernel considers the default route is directly connected.
# ip ro get 1.1.1.1
1.1.1.1 dev dummy0 src 10.0.0.1
cache
I'm not sure how you came to the "directly connected" conclusion. It is
still routing things out through 10.0.0.2 from 10.0.0.1.
Maybe your example would work better if you used 10.0.0.1 and 10.0.1.1
instead. Then I think you might be able to better see that when you delete
the second address the route would be broken.
No, it isn't. when ping 1.1.1.1, kernel will directly send arp request
braodcast to 1.1.1.1, this is not what I expect. it should send arp
request to 10.0.0.2, following should be the correct routing entry:
# ip ro get 1.1.1.1
1.1.1.1 via 10.0.0.2 dev dummy0 src 10.0.0.1
cache
From: Alexander Duyck <hidden> Date: 2015-08-06 22:36:45
On 08/06/2015 03:13 AM, Zang MingJie wrote:
On Thu, Aug 6, 2015 at 1:45 AM, Alexander Duyck
[off-list ref] wrote:
quoted
On 08/05/2015 02:06 AM, Daniel Borkmann wrote:
quoted
[ please cc netdev ]
On 08/05/2015 10:56 AM, Zang MingJie wrote:
quoted
Hi:
I found a bug when remove an ip address which is referenced by a routing
entry.
step to reproduce:
ip li add type dummy
ip li set dummy0 up
ip ad add 10.0.0.1/24 dev dummy0
ip ad add 10.0.0.2/24 dev dummy0
Okay, so up to this point you have 2 addresses on the same subnet that are
now on dummy0.
quoted
quoted
ip ro add default via 10.0.0.2/24
This makes the default route go through 10.0.0.2.
quoted
quoted
ip ad del 10.0.0.2/24 dev dummy0
Then you remove 10.0.0.2 from the local system, however since 10.0.0.1 is on
the same subnet dummy0 would still be the correct interface to access
10.0.0.2 it is just no longer local to the system.
quoted
quoted
after deleting the secondary ip address, the routing entry still
pointing to 10.0.0.2
You didn't delete the default routing entry so why would you expect it to
change? All you did is remove 10.0.0.2 from the local system. I believe
the assumption is that 10.0.0.2 is still out there somewhere, it just isn't
on the local system anymore.
Yes, 10.0.0.2 is migrated to somewhere else
The address might have migrated, but the interface is still up and
10.0.0.1 is still present on the same subnet. Because you made a local
address the default gateway the assumption is any routes not
specifically called out on other interfaces are directly accessible to
this interface.
The bug indicates that the kernel is doing something to make the table
inconsistent, but a default route that is a local interface address does
essentially the same thing.
quoted
quoted
quoted
# ip ro
default via 10.0.0.2 dev dummy0
10.0.0.0/24 dev dummy0 proto kernel scope link src 10.0.0.1
This matches up with what I would expect. 10.0.0.2 is the default gateway
and it is accessible from dummy0 since 10.0.0.0/24 is accessible from
dummy0.
This means 0.0.0.0/0 is accessible via 10.0.0.2 on the network of dummy0
Yes, but at the time you specified it 10.0.0.2 was a local address which
belonged to dummy0. This means that dummy0 can access anything not
specified elsewhere via pretty much any address it wants. So it is
perfectly valid if it wants to use a source address of 10.0.0.1 to send
packets to 1.1.1.1 over dummy0.
quoted
quoted
quoted
but actually, kernel considers the default route is directly connected.
# ip ro get 1.1.1.1
1.1.1.1 dev dummy0 src 10.0.0.1
cache
I'm not sure how you came to the "directly connected" conclusion. It is
still routing things out through 10.0.0.2 from 10.0.0.1.
Maybe your example would work better if you used 10.0.0.1 and 10.0.1.1
instead. Then I think you might be able to better see that when you delete
the second address the route would be broken.
No, it isn't. when ping 1.1.1.1, kernel will directly send arp request
braodcast to 1.1.1.1, this is not what I expect. it should send arp
request to 10.0.0.2, following should be the correct routing entry:
# ip ro get 1.1.1.1
1.1.1.1 via 10.0.0.2 dev dummy0 src 10.0.0.1
cache
I see what you are trying to say, but the example provided is a bit
lacking. Assuming you could ping 1.1.1.1 via dummy0 before with
10.0.0.2 as your default gateway, that shouldn't change if 10.0.0.2 is
migrated to another address. That is, unless there is an issue on the
system 10.0.0.2 was migrated to.
Now if I move away from using dummy interface and instead using a real
network interface things can get a bit more interesting. So if we
follow your example and use 2 different subnets on the two systems then
pings continue to work after we remove the addresses. However if we
flip things a bit and add the default route, and then the local address
for the gateway they don't. So something like below:
ip li set eth0 up
ip ad add 10.0.0.1/24 dev eth0
ip ro add default via 10.0.0.2
ip ad add 10.0.0.2/24 dev eth0
What you end up with is eth0 sending arp requests looking for 10.0.0.2
even though it is a local address on the system.
My question would be what is the correct behavior for this? If a local
address is removed or added that is being used as a gateway address
should we delete the route, or update the scope of the next hop?
- Alex
IMO, the routing decision is determined, given a specific routing
table and local network the result MUST be determined, independence of
how/what order the routing entry is added.
Now there are two ways to configure the system resulting EXACTLY the
same routing table and local addresses, but the routing decision is
totally different.
SAME routing table, DIFFERENT routing decision, there MUST be bugs in kernel.
On Thu, Aug 6, 2015 at 3:43 PM, Alexander Duyck
[off-list ref] wrote:
On 08/06/2015 03:13 AM, Zang MingJie wrote:
quoted
On Thu, Aug 6, 2015 at 1:45 AM, Alexander Duyck
[off-list ref] wrote:
quoted
On 08/05/2015 02:06 AM, Daniel Borkmann wrote:
quoted
[ please cc netdev ]
On 08/05/2015 10:56 AM, Zang MingJie wrote:
quoted
Hi:
I found a bug when remove an ip address which is referenced by a
routing
entry.
step to reproduce:
ip li add type dummy
ip li set dummy0 up
ip ad add 10.0.0.1/24 dev dummy0
ip ad add 10.0.0.2/24 dev dummy0
Okay, so up to this point you have 2 addresses on the same subnet that
are
now on dummy0.
quoted
quoted
ip ro add default via 10.0.0.2/24
This makes the default route go through 10.0.0.2.
quoted
quoted
ip ad del 10.0.0.2/24 dev dummy0
Then you remove 10.0.0.2 from the local system, however since 10.0.0.1 is
on
the same subnet dummy0 would still be the correct interface to access
10.0.0.2 it is just no longer local to the system.
quoted
quoted
after deleting the secondary ip address, the routing entry still
pointing to 10.0.0.2
You didn't delete the default routing entry so why would you expect it to
change? All you did is remove 10.0.0.2 from the local system. I believe
the assumption is that 10.0.0.2 is still out there somewhere, it just
isn't
on the local system anymore.
Yes, 10.0.0.2 is migrated to somewhere else
The address might have migrated, but the interface is still up and 10.0.0.1
is still present on the same subnet. Because you made a local address the
default gateway the assumption is any routes not specifically called out on
other interfaces are directly accessible to this interface.
The bug indicates that the kernel is doing something to make the table
inconsistent, but a default route that is a local interface address does
essentially the same thing.
quoted
quoted
quoted
quoted
# ip ro
default via 10.0.0.2 dev dummy0
10.0.0.0/24 dev dummy0 proto kernel scope link src 10.0.0.1
This matches up with what I would expect. 10.0.0.2 is the default
gateway
and it is accessible from dummy0 since 10.0.0.0/24 is accessible from
dummy0.
This means 0.0.0.0/0 is accessible via 10.0.0.2 on the network of dummy0
Yes, but at the time you specified it 10.0.0.2 was a local address which
belonged to dummy0. This means that dummy0 can access anything not
specified elsewhere via pretty much any address it wants. So it is
perfectly valid if it wants to use a source address of 10.0.0.1 to send
packets to 1.1.1.1 over dummy0.
quoted
quoted
quoted
quoted
but actually, kernel considers the default route is directly connected.
# ip ro get 1.1.1.1
1.1.1.1 dev dummy0 src 10.0.0.1
cache
I'm not sure how you came to the "directly connected" conclusion. It is
still routing things out through 10.0.0.2 from 10.0.0.1.
Maybe your example would work better if you used 10.0.0.1 and 10.0.1.1
instead. Then I think you might be able to better see that when you
delete
the second address the route would be broken.
No, it isn't. when ping 1.1.1.1, kernel will directly send arp request
braodcast to 1.1.1.1, this is not what I expect. it should send arp
request to 10.0.0.2, following should be the correct routing entry:
# ip ro get 1.1.1.1
1.1.1.1 via 10.0.0.2 dev dummy0 src 10.0.0.1
cache
I see what you are trying to say, but the example provided is a bit lacking.
Assuming you could ping 1.1.1.1 via dummy0 before with 10.0.0.2 as your
default gateway, that shouldn't change if 10.0.0.2 is migrated to another
address. That is, unless there is an issue on the system 10.0.0.2 was
migrated to.
Now if I move away from using dummy interface and instead using a real
network interface things can get a bit more interesting. So if we follow
your example and use 2 different subnets on the two systems then pings
continue to work after we remove the addresses. However if we flip things a
bit and add the default route, and then the local address for the gateway
they don't. So something like below:
ip li set eth0 up
ip ad add 10.0.0.1/24 dev eth0
ip ro add default via 10.0.0.2
ip ad add 10.0.0.2/24 dev eth0
What you end up with is eth0 sending arp requests looking for 10.0.0.2 even
though it is a local address on the system.
My question would be what is the correct behavior for this? If a local
address is removed or added that is being used as a gateway address should
we delete the route, or update the scope of the next hop?
- Alex
From: Alexander Duyck <hidden> Date: 2015-08-07 16:08:38
On 08/07/2015 01:23 AM, Zang MingJie wrote:
IMO, the routing decision is determined, given a specific routing
table and local network the result MUST be determined, independence of
how/what order the routing entry is added.
Now there are two ways to configure the system resulting EXACTLY the
same routing table and local addresses, but the routing decision is
totally different.
SAME routing table, DIFFERENT routing decision, there MUST be bugs in kernel
I wasn't arguing that the behavior is undesirable, but the likelihood of
having a default route assigned to a local address should be pretty
low. If the system is the default route of others then it should have a
different default gateway than itself. For example an office router
would end up pointing to the ISP as the gateway, and the ISP would
either point to some other provider or run a BGP configuration. So in
the case of the default route transitioning to us we should end up
having to delete and update the default route anyway. This is likely
one of the reasons why there hasn't been any issues reported with this
behavior until now.
I'm just wondering if the work involved to fix it is going to be worth
it. We have to keep in mind that this will result in a change of
behavior for existing users and we don't know if anyone might be
expecting this type of behavior.
We basically are looking at one of three options. The first one is to
just delete the route if you add the gateway as a local address or
remove it. That would be consistent with what you might see if the
address was the sole address on an interface of its own. The second
option is to update the nh_scope which I believe should be transitioned
between RT_SCOPE_HOST to RT_SCOPE_LINK if I am understanding things
correctly. The third option is we don't change the behavior and just
document it. This would then require manually deleting and restoring
any routes that use a recently modified address as their gateway.
Based on your feedback I'm assuming you would probably prefer the second
option. I'm just waiting to see if there are any other opinions on the
matter before I act.
Thanks.
- Alex
From: Hannes Frederic Sowa <hidden> Date: 2015-08-07 17:00:28
Hello,
Alexander Duyck [off-list ref] writes:
On 08/07/2015 01:23 AM, Zang MingJie wrote:
quoted
IMO, the routing decision is determined, given a specific routing
table and local network the result MUST be determined, independence of
how/what order the routing entry is added.
Now there are two ways to configure the system resulting EXACTLY the
same routing table and local addresses, but the routing decision is
totally different.
SAME routing table, DIFFERENT routing decision, there MUST be bugs in kernel
I wasn't arguing that the behavior is undesirable, but the likelihood of
having a default route assigned to a local address should be pretty
low. If the system is the default route of others then it should have a
different default gateway than itself. For example an office router
would end up pointing to the ISP as the gateway, and the ISP would
either point to some other provider or run a BGP configuration. So in
the case of the default route transitioning to us we should end up
having to delete and update the default route anyway. This is likely
one of the reasons why there hasn't been any issues reported with this
behavior until now.
I'm just wondering if the work involved to fix it is going to be worth
it. We have to keep in mind that this will result in a change of
behavior for existing users and we don't know if anyone might be
expecting this type of behavior.
We basically are looking at one of three options. The first one is to
just delete the route if you add the gateway as a local address or
remove it. That would be consistent with what you might see if the
address was the sole address on an interface of its own. The second
option is to update the nh_scope which I believe should be transitioned
between RT_SCOPE_HOST to RT_SCOPE_LINK if I am understanding things
correctly. The third option is we don't change the behavior and just
document it. This would then require manually deleting and restoring
any routes that use a recently modified address as their gateway.
Based on your feedback I'm assuming you would probably prefer the second
option. I'm just waiting to see if there are any other opinions on the
matter before I act.
The semantics behind this are not easy and the result might well break
other people's system. I would leave the current resolution logic as-is
and merely change the way iproute presents those information.
Currently we resolve the nexthop during route setup time and install the
resulting information into the FIB. This is very common on other OS, too.
In case we would reevaluate the nexthop part of a route during local
address changes on one of the interfaces, we could get the system very
well in a situation where it would have to remove its default route
because the network would not be reachable via ip subnetting any more,
but neighboring information would still keep the machine connected. And
this could happen with setups where someone did not configure their
routes to their own addresses, which are much more widespread.
The change wouldn't be in contradiction with weak end system behavior,
but I very much don't want to make other people's machines unreachable
because of such a change.
If we could rewind time, we could make local nexthops -EINVAL.
Bye,
Hannes