Francois Romieu wrote:
Boris B. Zhmurov [off-list ref] :
[...]
quoted
Any ideas are welcome.
Usually my Pavlov answer is "disable iptables, run latest kernel, renice
ksoftirqd like hell, see if it makes a difference and keep netdev Cced".
Of course there is no iptables. And linux-2.6.18 shows even worse
results with iperf + smallsized packets, than 2.6.16.29. Here some
results from linux-2.6.18. Reniceing ksoftirqd to "-19" makes network
performance to ~ 30% worse... From 26Mbit/s to 16-18Mbit/s. My .config
attached.
What we have (legend):
SRV -> Server with FreeBSD-6.1 release and iperf as daemon. This server
receives packets (Intel Pro 1000)
CLIENT1 -> FreeBSD-6.1 release and `iperf -uc 10.1.0.5 -b1073741824 -l
64 -t 300 -i 1` as client (Realtek8169)
CLIENT2 -> Linux-2.6.18 and `iperf -uc 10.1.0.5 -b1073741824 -l 64 -t
300 -i 1` as client (Realtek8169)
Client1 and Client2 has no different with hardware. Both - P4@3Ghz + HT,
1Gb Ram and RTL8169 Gbit NIC.
########################################
Start sending udp packets from CLIENT1 to SRV.
########################################
1) CLIENT1 output:
l9# iperf -uc 10.1.0.5 -b1073741824 -l 64 -t 300 -i 1
------------------------------------------------------------
Client connecting to 10.1.0.5, UDP port 5001
Sending 64 byte datagrams
UDP buffer size: 9.00 KByte (default)
------------------------------------------------------------
[ 3] local 10.1.0.2 port 65121 connected with 10.1.0.5 port 5001
[ 3] 0.0- 1.0 sec 8.97 MBytes 71.7 Mbits/sec
[ 3] 1.0- 2.0 sec 8.97 MBytes 71.7 Mbits/sec
[ 3] 2.0- 3.0 sec 8.98 MBytes 71.9 Mbits/sec
[ 3] 3.0- 4.0 sec 8.98 MBytes 71.9 Mbits/sec
2) SRV's netstat output:
l7# netstat -I em0 -w 1
input (em0) output
packets errs bytes packets errs bytes colls
138928 0 15282034 1 0 218 0
139086 0 15299370 1 0 170 0
138999 0 15289798 1 0 170 0
139004 0 15290394 1 0 170 0
########################################
Start sending udp packets from CLIENT2 to SRV.
########################################
1) CLIENT2 output:
[root@bb-work ~]# iperf -uc 10.1.0.5 -b1073741824 -l 64 -t 300 -i 1
------------------------------------------------------------
Client connecting to 10.1.0.5, UDP port 5001
Sending 64 byte datagrams
UDP buffer size: 64.0 KByte (default)
------------------------------------------------------------
[ 3] local 10.1.0.6 port 32776 connected with 10.1.0.5 port 5001
[ 3] 0.0- 1.0 sec 2.32 MBytes 19.5 Mbits/sec
[ 3] 1.0- 2.0 sec 2.12 MBytes 17.8 Mbits/sec
[ 3] 2.0- 3.0 sec 2.35 MBytes 19.7 Mbits/sec
[ 3] 3.0- 4.0 sec 2.15 MBytes 18.1 Mbits/sec
2) SRV's netstat output:
l7# netstat -I em0 -w 1
input (em0) output
packets errs bytes packets errs bytes colls
38630 0 4248702 1 0 206 0
35849 0 3943298 1 0 170 0
38264 0 4208994 1 0 170 0
36265 0 3989058 1 0 170 0
Is there any chance to get linux worked at least as fast, as freebsd-6.1
with small udp packets?
Any help will be more than welcome!
Please, CC: me, due to I'm not subscribed to netdev@vger.kernel.org.
--
Boris B. Zhmurov
System/Network Administrator
mailto: bb@kernelpanic.ru
"wget http://kernelpanic.ru/bb_public_key.pgp -O - | gpg --import"
Boris B. Zhmurov [off-list ref] :
[...]
Is there any chance to get linux worked at least as fast, as freebsd-6.1
with small udp packets?
Any help will be more than welcome!
Please, CC: me, due to I'm not subscribed to netdev@vger.kernel.org.
Can you publish dmesg, vmstat 1 and /proc/interrupts somewhere ?
# Thu Sep 21 10:31:58 2006
#
CONFIG_X86_32=y
CONFIG_GENERIC_TIME=y
CONFIG_LOCKDEP_SUPPORT=y
Disable
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_SEMAPHORE_SLEEPERS=y
Disable
CONFIG_X86=y
CONFIG_MMU=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_IOMAP=y
CONFIG_GENERIC_HWEIGHT=y
CONFIG_ARCH_MAY_HAVE_PC_FDC=y
CONFIG_DMI=y
CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"
#
# Code maturity level options
#
CONFIG_EXPERIMENTAL=y
Disable
CONFIG_LOCK_KERNEL=y
CONFIG_INIT_ENV_ARG_LIMIT=32
[...]
CONFIG_AUDIT=y
CONFIG_AUDITSYSCALL=y
Disable
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
[...]
CONFIG_M686=y
# CONFIG_MPENTIUMII is not set
# CONFIG_MPENTIUMIII is not set
# CONFIG_MPENTIUMM is not set
# CONFIG_MPENTIUM4 is not set
?
[...]
# CONFIG_PREEMPT_NONE is not set
Enable
CONFIG_PREEMPT_VOLUNTARY=y
Disable
# CONFIG_PREEMPT is not set
CONFIG_PREEMPT_BKL=y
Disable
[...]
# CONFIG_NOHIGHMEM is not set
CONFIG_HIGHMEM4G=y
# CONFIG_HIGHMEM64G is not set
CONFIG_PAGE_OFFSET=0xC0000000
CONFIG_HIGHMEM=y
CONFIG_SELECT_MEMORY_MODEL=y
CONFIG_FLATMEM_MANUAL=y
# CONFIG_DISCONTIGMEM_MANUAL is not set
# CONFIG_SPARSEMEM_MANUAL is not set
CONFIG_FLATMEM=y
CONFIG_FLAT_NODE_MEM_MAP=y
# CONFIG_SPARSEMEM_STATIC is not set
CONFIG_SPLIT_PTLOCK_CPUS=4
# CONFIG_RESOURCES_64BIT is not set
CONFIG_HIGHPTE=y
# CONFIG_MATH_EMULATION is not set
CONFIG_MTRR=y
# CONFIG_EFI is not set
# CONFIG_IRQBALANCE is not set
CONFIG_REGPARM=y
# CONFIG_SECCOMP is not set
CONFIG_HZ_100=y
# CONFIG_HZ_250 is not set
# CONFIG_HZ_1000 is not set
CONFIG_HZ=100
CONFIG_HZ_1000
[...]
#
# CPU Frequency scaling
#
CONFIG_CPU_FREQ=y
CONFIG_CPU_FREQ_TABLE=y
# CONFIG_CPU_FREQ_DEBUG is not set
CONFIG_CPU_FREQ_STAT=m
CONFIG_CPU_FREQ_STAT_DETAILS=y
# CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE is not set
CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE=y
CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
CONFIG_CPU_FREQ_GOV_POWERSAVE=m
CONFIG_CPU_FREQ_GOV_USERSPACE=y
CONFIG_CPU_FREQ_GOV_ONDEMAND=m
CONFIG_CPU_FREQ_GOV_CONSERVATIVE=m
...
#
# CPUFreq processor drivers
#
CONFIG_X86_ACPI_CPUFREQ=m
# CONFIG_X86_POWERNOW_K6 is not set
CONFIG_X86_POWERNOW_K7=y
CONFIG_X86_POWERNOW_K8=y
# CONFIG_X86_GX_SUSPMOD is not set
CONFIG_X86_SPEEDSTEP_CENTRINO=y
CONFIG_X86_SPEEDSTEP_CENTRINO_TABLE=y
CONFIG_X86_SPEEDSTEP_ICH=m
CONFIG_X86_SPEEDSTEP_SMI=m
CONFIG_X86_P4_CLOCKMOD=m
...
[...]
# Instrumentation Support
#
# CONFIG_PROFILING is not set
# CONFIG_KPROBES is not set
#
# Kernel hacking
#
CONFIG_TRACE_IRQFLAGS_SUPPORT=y
# CONFIG_PRINTK_TIME is not set
# CONFIG_MAGIC_SYSRQ is not set
CONFIG_UNUSED_SYMBOLS=y
CONFIG_DEBUG_KERNEL=y
Disable
--
Ueimor
Francois Romieu wrote:
quoted
Is there any chance to get linux worked at least as fast, as freebsd-6.1
with small udp packets?
Any help will be more than welcome!
Please, CC: me, due to I'm not subscribed to netdev@vger.kernel.org.
Can you publish dmesg, vmstat 1 and /proc/interrupts somewhere ?
ftp://bb-work.kernelpanic.ru/pub/dmesg
ftp://bb-work.kernelpanic.ru/pub/vmstat
ftp://bb-work.kernelpanic.ru/pub/interrupts
quoted
# Thu Sep 21 10:31:58 2006
#
CONFIG_X86_32=y
CONFIG_GENERIC_TIME=y
CONFIG_LOCKDEP_SUPPORT=y
Disable
[......]
Thanks Francois, after your suggestions, network performance was twice
increased, but I'm still loosing to freebsd:
previous .config (server's input):
input (em0) output
packets errs bytes packets errs bytes colls
38630 0 4248702 1 0 206 0
35849 0 3943298 1 0 170 0
38264 0 4208994 1 0 170 0
new .config (server's input):
input (em0) output
packets errs bytes packets errs bytes colls
90431 0 9947370 1 0 182 0
90504 0 9955400 1 0 294 0
90494 0 9954300 1 0 182 0
P.S. I couldn't disable the following things due to bzImage build errors:
CONFIG_GENERIC_TIME
CONFIG_X86=y
CONFIG_MMU=y
CONFIG_GENERIC_IOMAP=y
CONFIG_DMI=y
--
Boris B. Zhmurov
System/Network Administrator
mailto: bb@kernelpanic.ru
"wget http://kernelpanic.ru/bb_public_key.pgp -O - | gpg --import"
Boris B. Zhmurov [off-list ref] :
[...]
Thanks Francois, after your suggestions, network performance was twice
increased, but I'm still loosing to freebsd:
Ok, now that the Linux kernel is not in heavy debugging mode, it takes
some ~10 us/(sent packet). It is not _too_ far from the usual syscall
tax.
Can you send a typical excerpt of an 'strace -T' for your client iperf
process and compare it against the FreeBSD one ?
[...]
P.S. I couldn't disable the following things due to bzImage build errors:
CONFIG_GENERIC_TIME
CONFIG_X86=y
CONFIG_MMU=y
CONFIG_GENERIC_IOMAP=y
CONFIG_DMI=y
Oops, I was a bit terse: you were only supposed to disable the outlined
options (i.e. mostly debugging and/or unneeded stuff). As long as the
kernel performs better, it should not be a big issue though.
--
Ueimor