I observe that UDP sockets listening on ANY
send response packets with ip addr derived from
ip address of interface which is used to send 'em
instead of using dst ip address of client's packet.
I was bitten by this with DNS and NTP.
This is a test setup:
{net 172.16/16}----- 172.16.22.2 [multihomed] 195.66.192.167 ----....
This is a UDP server on multihomed box:
# echo test | nc -vvvnu -l -p 222
listening on [any] 222 ...
connect to [172.16.22.2] from (UNKNOWN) [172.16.42.177] 333
ping
sent 5, rcvd 5 : Connection refused
#
This is client on 172.16.42.177:
# echo ping | nc -vvvnu -p 333 195.66.192.167 222
(UNKNOWN) [195.66.192.167] 222 (?) open
it waits forever, never getting 'test' string from server.
This is a tcpdump of this event:
13:58:00.555207 172.16.42.177.333 > 195.66.192.167.222: udp 5 (DF)
13:58:00.559849 172.16.22.2.222 > 172.16.42.177.333: udp 5 (DF)
^^^^^^^^^^^
13:58:00.560457 172.16.42.177 > 172.16.22.2: icmp: 172.16.42.177 udp port 333 unreachable [tos 0xc0]
This is why 172.16.42.177 thinks that port 333 is unreachable:
# lsof -nP | grep 333
ntpd 418 root mem REG 3,4 133316 68184 /.share/usr/lib/all-lib/libreadline.so.3.0
most 7559 root txt REG 3,4 46828 24333 /.share/usr/app/most-4.9.2/bin/most
nc 15085 root 3u IPv4 29381 UDP 172.16.42.177:333->195.66.192.167:222
# ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
kernel does not think that 172.16.22.2.222 > 172.16.42.177.333 packet
belong to this socket, and it is right imo.
Test works ok if I remove -u from both netcats (i.e. TCP works).
client:
# uname -a
Linux firebird 2.6.6-rc1-bk1 #1 Wed May 26 11:03:11 EEST 2004 i686 unknown
server:
# uname -a
Linux oldie 2.4.22-rc2_QoS #1 Tue Nov 18 15:55:00 GMT-2 2003 i586 unknown
root@oldie:/.share/home/vda#
I saw this behavior on 2.6 based machines as well.
--
vda
I observe that UDP sockets listening on ANY
send response packets with ip addr derived from
ip address of interface which is used to send 'em
instead of using dst ip address of client's packet.
I was bitten by this with DNS and NTP.
The problem is in the apps. You have some options:
- use sockets listening on ANY and using sendmsg with IP_PKTINFO
and providing the desired src IP in ipi_spec_dst
- you can also create many listener sockets listening on
particular src IP and to reply using the socket where the
request is received, because the kernel does not know any
association between request and reply packets if the listener
is bound to ANY.
Regards
--
Julian Anastasov [off-list ref]
In article [off-list ref] (at Wed, 9 Jun 2004 14:25:39 +0300), Denis Vlasenko [off-list ref] says:
I observe that UDP sockets listening on ANY
send response packets with ip addr derived from
ip address of interface which is used to send 'em
instead of using dst ip address of client's packet.
use IP_PKTINFO when responding the client.
--yoshfuji
From: Martijn van Oosterhout <hidden> Date: 2004-06-09 12:27:46
On Wed, Jun 09, 2004 at 02:25:39PM +0300, Denis Vlasenko wrote:
I observe that UDP sockets listening on ANY
send response packets with ip addr derived from
ip address of interface which is used to send 'em
instead of using dst ip address of client's packet.
I was bitten by this with DNS and NTP.
This is the responsibility of the program. Unless the UDP packet is
bound to a particular address, or the program specifies an IP, the
kernel will pick one. For this reason both the BIND and NTP daemons
open a socket for each interface so they can control this. netstat on
my machine shows:
udp 0 0 192.168.1.225:123 0.0.0.0:*
udp 0 0 127.0.0.1:123 0.0.0.0:*
udp 0 0 0.0.0.0:123 0.0.0.0:*
for the NTP server. The DNS has similar machanism. Remember, UDP
doesn't involve connections, so there is no concept of "replying" to a
packet, the program has to manage that itself.
In your example, if you tell netcat which address to bind to, it will
work.
Hope this helps,
--
Martijn van Oosterhout [off-list ref] http://svana.org/kleptog/
Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a
tool for doing 5% of the work and then sitting around waiting for someone
else to do the other 95% so you can sue them.
On Wednesday 09 June 2004 15:18, Julian Anastasov wrote:
Hello,
On Wed, 9 Jun 2004, Denis Vlasenko wrote:
quoted
I observe that UDP sockets listening on ANY
send response packets with ip addr derived from
ip address of interface which is used to send 'em
instead of using dst ip address of client's packet.
I was bitten by this with DNS and NTP.
The problem is in the apps. You have some options:
- use sockets listening on ANY and using sendmsg with IP_PKTINFO
and providing the desired src IP in ipi_spec_dst
- you can also create many listener sockets listening on
particular src IP and to reply using the socket where the
request is received, because the kernel does not know any
association between request and reply packets if the listener
is bound to ANY.
Aha! That's why ntpd is so eager to bind to everything imaginable!
(Which does not help BTW with non-permanent interfaces like
ppp, tun etc).
/me googling for that IP_PKTINFO thing...
--
vda
On Wednesday 09 June 2004 15:24, YOSHIFUJI Hideaki wrote:
Denis Vlasenko [off-list ref] says:
quoted
I observe that UDP sockets listening on ANY
send response packets with ip addr derived from
ip address of interface which is used to send 'em
instead of using dst ip address of client's packet.
use IP_PKTINFO when responding the client.
Thanks!
With your help and some googling I've found and adapted
code to get dst ip of UDP packet.
Small test program successfully ran and reported correct
dst addresses of incoming UDP packets.
Now, I am trying to fix (or shall I say 'improve'?) dnscache.
You may find some code below my sig. It's a start.
The problem is, how to _send replies_ with correct src ip?
I can bind a temporary socket to needed src address,
do a sendto(), then close socket. This will work,
but this can introduce a race - any incoming
packet to this (ip,port) will inadvertently
be classified as belonging to temp socket!
This is going to be a nasty bug, manifesting
itself only under load.
I looked into sendmsg(). Looks like ther is no way to
indicate source ip.
Shall I use some other technique?
--
vda
#if defined IP_RECVDSTADDR
# define DSTADDR_SOCKOPT IP_RECVDSTADDR
# define DSTADDR_DATASIZE (CMSG_SPACE(sizeof(struct in_addr)))
# define dstaddr(x) (CMSG_DATA(x))
#elif defined IP_PKTINFO
# define DSTADDR_SOCKOPT IP_PKTINFO
# define DSTADDR_DATASIZE (CMSG_SPACE(sizeof(struct in_pktinfo)))
# define dstaddr(x) (&(((struct in_pktinfo *)(CMSG_DATA(x)))->ipi_addr))
#else
# error "can't determine socket option"
#endif
int socket_recv4_dst(int s,char *buf,int len,char ip[4],uint16 *port, char ipdst[4])
{
int r;
struct iovec iov[1];
struct sockaddr_in sa;
union control_data cmsg;
struct cmsghdr *cmsgptr;
struct msghdr msg;
iov[0].iov_base = buf;
iov[0].iov_len = len;
memset(&msg, 0, sizeof msg);
msg.msg_name = &sa;
msg.msg_namelen = sizeof sa;
msg.msg_iov = iov;
msg.msg_iovlen = 1;
msg.msg_control = &cmsg;
msg.msg_controllen = sizeof cmsg;
{ // FIXME: we need to do it ONCE! move it into socket_bind4_dstaddropt()
int sockopt;
sockopt = 1;
if (setsockopt(s, IPPROTO_IP, DSTADDR_SOCKOPT, &sockopt, sizeof sockopt) == -1)
return -1;
}
//r = recvfrom(s,buf,len,0,(struct sockaddr *) &sa,&dummy);
r = recvmsg(s, &msg, 0);
if (r == -1) return -1;
// Here we retrieve destination IP and memorize it
for (cmsgptr = CMSG_FIRSTHDR(&msg);
cmsgptr != NULL;
cmsgptr = CMSG_NXTHDR(&msg, cmsgptr)) {
if (cmsgptr->cmsg_level == IPPROTO_IP
&& cmsgptr->cmsg_type == DSTADDR_SOCKOPT) {
byte_copy(ipdst,4,(char *) dstaddr(cmsgptr));
}
}
return r;
}
I looked into sendmsg(). Looks like ther is no way to
indicate source ip.
Shall I use some other technique?
IP_PKTINFO works just as well there. Look at the ip_cmsg_send call
in udp_sendmsg.
int udp_sendmsg(...)
{
...
ipc.addr = inet->saddr;
ipc.oif = sk->sk_bound_dev_if;
if (msg->msg_controllen) {
err = ip_cmsg_send(msg, &ipc);
if (err)
return err;
if (ipc.opt)
free = 1;
connected = 0;
}
if (!ipc.opt)
ipc.opt = inet->opt;
saddr = ipc.addr;
...
int ip_cmsg_send(struct msghdr *msg, struct ipcm_cookie *ipc)
{
...
case IP_PKTINFO:
{
struct in_pktinfo *info;
if (cmsg->cmsg_len != CMSG_LEN(sizeof(struct in_pktinfo)))
return -EINVAL;
info = (struct in_pktinfo *)CMSG_DATA(cmsg);
ipc->oif = info->ipi_ifindex;
ipc->addr = info->ipi_spec_dst.s_addr;
manpage:
IP_PKTINFO
Pass an IP_PKTINFO ancillary message that contains a pktinfo structure that supplies some information about the
incoming packet. This only works for datagram oriented sockets.
struct in_pktinfo
{
unsigned int ipi_ifindex; /* Interface index */
struct in_addr ipi_spec_dst; /* Routing destination address */
struct in_addr ipi_addr; /* Header Destination address */
};
Hmmm... do I have to set a *routing dest address* field
to set src ip address of my UDP packet?
--
vda
On Friday 11 June 2004 16:53, Julian Anastasov wrote:
Hello,
On Fri, 11 Jun 2004, Denis Vlasenko wrote:
quoted
Hmmm... do I have to set a *routing dest address* field
to set src ip address of my UDP packet?
Try such function:
[snip]
Thanks for your excellent help Julian.
It helped me to finally make my code work.
Pity I did not keep it that way, because
it wouldn't be portable. I converted it
to 'man cmsg' compliant way.
On Friday 11 June 2004 17:14, Henrik Nordstrom wrote:
[snip]
quoted
I am trying to figure it out, currently I'm getting
-EINVAL in sendmsg in this code:
# define SRCADDR_SOCKOPT IP_PKTINFO
# define SRCADDR_DATASIZE (CMSG_SPACE(sizeof(struct in_pktinfo)))
# define srcaddr(x) (&(((struct in_pktinfo
*)(CMSG_DATA(x)))->ipi_spec_dst))
Drop the defines. These are only obfuscating your code and the source to
your error. You need to pass a in_pktinfo structure in the message, and
They were there in the example recv code from the Net. That code tried
to use IP_DSTADDR (IIRC) option (a BSDism or something) if it is #defined.
Dropped #defines.
I ended up with following patch for djbdns. Run tested.
Thanks again for your help guys.
--
vda