Le mardi 08 novembre 2011 à 21:28 -0800, Manavalan Krishnan a écrit :
Hi All
I have two systems with two network interfaces each(eth0 and eth1). I
am running linux-HA (heartbeat deamon) on both the systems and they
use eth0 for exchanging heartbeats. I have NFS mounted directory in
one system and the NFS client uses the interface eth1.
I try to copy a large file to NFS mounted directory. But the heartbeat
daemons misses the heartbeat packets from peers while copy is under
progress. I did tcpdump and found that the heartbeat packets are
delayed for few seconds before sent out on eth0. When I stop the file
copy, the heartbeats are delivered properly. It seems linux kernel
somehow giving priority for NFS packets(generated from the file copy)
over other application packets.
Any thoughts on this behavior? Is there any way we can avoid this so
that application packets get equal chance while large file copy to NFS
mounted directory under progress?
CC netdev
1) Is your NFS using UDP or TCP ?
2) Is your eth0 dedicated to heartbeats and eth1 to NFS traffic ?
3) How do you know heartbeats are delayed ?
4) Is your server CPU bounded ?
Thanks
(1) NFS is using TCP
(2) yes eth0 is dedicated to heartbeat and eth1 is dedicated to NFS
(3) I notice the following at the system where file copy is occuring
The kernel Recv-Q of the heartbeat application socket grows but not delivered to the socket recv call.
Here is the netstat output.
Proto Recv-Q Send-Q Local Address Foreign Address
udp 11522 0 *:23435 *:*
As soon as I stop the file transfer, the socket recv call receives the packets and Recv-Q goes 0.
(4) The server has 4 cpu cores and 25G RAM
________________________________
From: Eric Dumazet <redacted>
To: Manavalan Krishnan <redacted>
Cc: "linux-kernel@vger.kernel.org" <redacted>; netdev <redacted>
Sent: Tuesday, November 8, 2011 10:24 PM
Subject: Re: Large file copy to NFS mounted directory causes delay in other application packets
Le mardi 08 novembre 2011 à 21:28 -0800, Manavalan Krishnan a écrit :
Hi All
I have two systems with two network interfaces each(eth0 and eth1). I
am running linux-HA (heartbeat deamon) on both the systems and they
use eth0 for exchanging heartbeats. I have NFS mounted directory in
one system and the NFS client uses the interface eth1.
I try to copy a large file to NFS mounted directory. But the heartbeat
daemons misses the heartbeat packets from peers while copy is under
progress. I did tcpdump and found that the heartbeat packets are
delayed for few seconds before sent out on eth0. When I stop the file
copy, the heartbeats are delivered properly. It seems linux kernel
somehow giving priority for NFS packets(generated from the file copy)
over other application packets.
Any thoughts on this behavior? Is there any way we can avoid this so
that application packets get equal chance while large file copy to NFS
mounted directory under progress?
CC netdev
1) Is your NFS using UDP or TCP ?
2) Is your eth0 dedicated to heartbeats and eth1 to NFS traffic ?
3) How do you know heartbeats are delayed ?
4) Is your server CPU bounded ?
Thanks
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Please dont top post on these lists, thanks.
Le mercredi 09 novembre 2011 à 00:13 -0800, Manavalan Krishnan a écrit :
(1) NFS is using TCP
(2) yes eth0 is dedicated to heartbeat and eth1 is dedicated to NFS
(3) I notice the following at the system where file copy is occuring
The kernel Recv-Q of the heartbeat application socket grows but not delivered to the socket recv call.
Here is the netstat output.
Proto Recv-Q Send-Q Local Address Foreign Address
udp 11522 0 *:23435 *:*
OK so the sending side is OK : The delay is at receiver side.
Note that since netstat shows receive queue has some skbs, it should be
available to heartbeat daemon immediately.
As soon as I stop the file transfer, the socket recv call receives the packets and Recv-Q goes 0.
(4) The server has 4 cpu cores and 25G RAM
1) How many nfsd threads are running ?
grep th /proc/net/rpc/nfsd
2) WHat kind of NIC do you use ?
lsmod , lspci
3) Hmm, are IRQ to eth0/eth1 handled by same cpu ?
grep eth /proc/interrupts
4) You could try to cpu affine all nfsd to cpu0,cpu1,cpu2 and heartbeat
daemon to cpu3.
man taskset
5) You could 'strace -ttt' heartbeat daemon to check if it is not
blocked on some local disk access (it competes with all nfsd threads)