From: Roy Sigurd Karlsbakk <hidden> Date: 2002-10-23 10:04:38
hi
I've got this video server serving video for VoD. problem is the P4 1.8 seems
to be maxed out by a few system calls. The below output is for ~50 clients
streaming at ~4.5Mbps. if trying to increase this to ~70, the CPU maxes out.
Does anyone have an idea?
bash-2.05# readprofile | sort -rn +2 | head -30
154203 default_idle 2409.4219
212723 csum_partial_copy_generic 916.9095
100164 handle_IRQ_event 695.5833
24979 system_call 390.2969
37300 e1000_intr 388.5417
119699 ide_intr 340.0540
30598 skb_release_data 273.1964
40740 do_softirq 195.8654
131818 do_wp_page 164.7725
9935 fget 155.2344
24747 kfree 154.6687
10911 del_timer 113.6562
11683 ip_conntrack_find_get 91.2734
4120 sock_poll 85.8333
9357 ip_ct_find_proto 83.5446
5194 sock_wfree 81.1562
4929 add_wait_queue 77.0156
8361 flush_tlb_page 74.6518
4571 remove_wait_queue 71.4219
2191 __brelse 68.4688
29477 skb_clone 68.2338
8562 do_gettimeofday 59.4583
5673 process_timeout 59.0938
11097 tcp_v4_send_check 57.7969
6124 kfree_skbmem 54.6786
17115 tcp_poll 53.4844
21130 nf_hook_slow 52.8250
8299 ip_ct_refresh 51.8687
15429 __kfree_skb 50.7533
1059 lru_cache_del 46.0435
roy
--
Roy Sigurd Karlsbakk, Datavaktmester
ProntoTV AS - http://www.pronto.tv/
Tel: +47 9801 3356
Computers are like air conditioners.
They stop working when you open Windows.
From: Roy Sigurd Karlsbakk <hidden> Date: 2002-10-23 10:52:44
I've got this video server serving video for VoD. problem is the P4 1.8
seems to be maxed out by a few system calls. The below output is for ~50
clients streaming at ~4.5Mbps. if trying to increase this to ~70, the CPU
maxes out.
Does anyone have an idea?
On Wed, Oct 23, 2002 at 01:06:18PM +0200, Roy Sigurd Karlsbakk wrote:
quoted
I've got this video server serving video for VoD. problem is the P4 1.8
seems to be maxed out by a few system calls. The below output is for ~50
clients streaming at ~4.5Mbps. if trying to increase this to ~70, the CPU
maxes out.
'50 clients *each* streaming at ~4.4MBps', better make that clear, otherwise
something is *very* broken. Also mention that you have an e1000 card which
does not do outgoing checksumming.
You'd think that a kernel would be able to do 250megabits of TCP checksums
though.
...adding the whole profile output - sorted by the first column this time...
905182 total 0.4741
121426 csum_partial_copy_generic 474.3203
93633 default_idle 1800.6346
74665 do_wp_page 111.1086
Perhaps the 'copy' also entails grabbing the page from disk, leading to
inflated csum_partial_copy_generic stats?
Where are you serving from?
Regards,
bert
--
http://www.PowerDNS.com Versatile DNS Software & Services
http://lartc.org Linux Advanced Routing & Traffic Control HOWTO
From: David S. Miller <hidden> Date: 2002-10-23 13:03:50
On Wed, 2002-10-23 at 06:01, bert hubert wrote:
Also mention that you have an e1000 card which
does not do outgoing checksumming.
The e1000 can very well do hardware checksumming on transmit.
The missing piece of the puzzle is that his application is not
using sendfile(), without which no transmit checksum offload
can take place.
From: Roy Sigurd Karlsbakk <hidden> Date: 2002-10-23 13:29:16
The e1000 can very well do hardware checksumming on transmit.
The missing piece of the puzzle is that his application is not
using sendfile(), without which no transmit checksum offload
can take place.
As far as I've understood, sendfile() won't do much good with large files. Is
this right?
We're talking of 3-6GB files here ...
roy
--
Roy Sigurd Karlsbakk, Datavaktmester
ProntoTV AS - http://www.pronto.tv/
Tel: +47 9801 3356
Computers are like air conditioners.
They stop working when you open Windows.
On Wed, Oct 23, 2002 at 03:42:48PM +0200, Roy Sigurd Karlsbakk wrote:
quoted
The e1000 can very well do hardware checksumming on transmit.
The missing piece of the puzzle is that his application is not
using sendfile(), without which no transmit checksum offload
can take place.
As far as I've understood, sendfile() won't do much good with large files. Is
this right?
I still refuse to believe that a 1.8GHz Pentium4 can only checksum
250megabits/second. MD Raid5 does better and they probably don't use a
checksum as braindead as that used by TCP.
If the checksumming is not the problem, the copying is, which would be a
weakness of your hardware. The function profiled does both the copying and
the checksumming.
But 250megabits/second also seems low.
Dave?
Regards,
bert
--
http://www.PowerDNS.com Versatile DNS Software & Services
http://lartc.org Linux Advanced Routing & Traffic Control HOWTO
From: Richard B. Johnson <hidden> Date: 2002-10-23 17:03:00
On Wed, 23 Oct 2002, bert hubert wrote:
On Wed, Oct 23, 2002 at 03:42:48PM +0200, Roy Sigurd Karlsbakk wrote:
quoted
quoted
The e1000 can very well do hardware checksumming on transmit.
The missing piece of the puzzle is that his application is not
using sendfile(), without which no transmit checksum offload
can take place.
As far as I've understood, sendfile() won't do much good with large files. Is
this right?
I still refuse to believe that a 1.8GHz Pentium4 can only checksum
250megabits/second. MD Raid5 does better and they probably don't use a
checksum as braindead as that used by TCP.
If the checksumming is not the problem, the copying is, which would be a
weakness of your hardware. The function profiled does both the copying and
the checksumming.
But 250megabits/second also seems low.
Dave?
Ordinary DUAL Pentium 400 MHz machine does this...
Calculating CPU speed...done
Testing checksum speed...done
Testing RAM copy...done
Testing I/O port speed...done
CPU Clock = 400 MHz
checksum speed = 685 Mb/s
RAM copy = 1549 Mb/s
I/O port speed = 654 kb/s
This is 685 megaBYTES per second.
checksum speed = 685 Mb/s
Cheers,
Dick Johnson
Penguin : Linux version 2.4.18 on an i686 machine (797.90 BogoMips).
Bush : The Fourth Reich of America
From: Ben Greear <hidden> Date: 2002-10-23 17:06:51
bert hubert wrote:
I still refuse to believe that a 1.8GHz Pentium4 can only checksum
250megabits/second. MD Raid5 does better and they probably don't use a
checksum as braindead as that used by TCP.
For what it's worth, I have been able to send and receive 400+ Mbps
of traffic, by directional, on the same machine (ie, about 1600 Mbps
of payload across the PCI bus)
So, it's probably not the e1000 or networking code that is slowing you down.
(This was on a 64/66 PCI, Dual-AMD 2Ghz machine though,
are you running only 32/33 PCI? If not, where did you find this motherboard!)
Have you tried just reading the information from disk and doing everying except
the final 'send/write/sendto' ? That would help determine if it is your
file reads that are killing you.
Ben
--
Ben Greear [off-list ref] <Ben_Greear AT excite.com>
President of Candela Technologies Inc http://www.candelatech.com
ScryMUD: http://scry.wanfear.comhttp://scry.wanfear.com/~greear
I still refuse to believe that a 1.8GHz Pentium4 can only checksum
250megabits/second. MD Raid5 does better and they probably don't use a
checksum as braindead as that used by TCP.
If the checksumming is not the problem, the copying is, which would be a
weakness of your hardware. The function profiled does both the copying and
the checksumming.
Yep, its not so much the checksumming as the fact that this is
done over each byte of data and copied.
thanks,
Nivedita
From: Richard B. Johnson <hidden> Date: 2002-10-23 17:48:47
On Wed, 23 Oct 2002, Nivedita Singhvi wrote:
bert hubert wrote:
quoted
I still refuse to believe that a 1.8GHz Pentium4 can only checksum
250megabits/second. MD Raid5 does better and they probably don't use a
checksum as braindead as that used by TCP.
If the checksumming is not the problem, the copying is, which would be a
weakness of your hardware. The function profiled does both the copying and
the checksumming.
Yep, its not so much the checksumming as the fact that this is
done over each byte of data and copied.
thanks,
Nivedita
No. It's done over each word (short int) and the actual summation
takes place during the address calculation of the next word. This
gets you a checksum that is practically free.
A 400 MHz ix86 CPU will checksum/copy at 685 megabytes per second.
It will copy at 1,549 megabytes per second. Those are megaBYTES!
If you have slow network performance it has nothing to do with
either copy or checksum. Data transmission acts like a low-pass
filter. The dominant pole of that transfer function determines
the speed, that's why it's called dominant. If you measure
a data-rate of 10 megabytes/second. Nothing you do with copy
or checksum will affect it to any significant extent.
If you have a data-rate of 100 megabytes per second, then any
tinkering with copy will have an effective improvement ratio
of 100/1,559 ~= 0.064. If you have a data rate of 100 megabytes
per second and you tinker with checksum, you get an improvement
ratio of 100/685 ~=0.14. These are just not the things that are
affecting your performance.
If you were to double the checksumming speed, you increase the
throughput by 2 * 0.14 = 0.28 with the parameters shown.
The TCP/IP checksum is quite nice. It may have been discovered
by accident, but it's still nice. It works regardless of whether
you have a little endian or big endian machine. It also doesn't
wrap so you don't (usually) show a good checksum when the data
is bad. It does have the characteristic that if all the bits are
inverted, it will checksum good. However, there are not too many
real-world scenarios that would result in this inversion. So it's
not "brain-dead" as you state. A hardware checksum is really
quick because it's really easy.
Cheers,
Dick Johnson
Penguin : Linux version 2.4.18 on an i686 machine (797.90 BogoMips).
Bush : The Fourth Reich of America
No. It's done over each word (short int) and the actual summation
takes place during the address calculation of the next word. This
gets you a checksum that is practically free.
Yep, sorry, word, not byte. My bad. The cost is in the fact
that this whole process involves loading each word of the data
stream into a register. Which is why I also used to consider
the checksum cost as negligible.
A 400 MHz ix86 CPU will checksum/copy at 685 megabytes per second.
It will copy at 1,549 megabytes per second. Those are megaBYTES!
But then why the difference in the checksum/copy and copy?
Are you saying the checksum is not costing you 864 megabytes
a second??
thanks,
Nivedita
From: Richard B. Johnson <hidden> Date: 2002-10-23 18:21:45
On Wed, 23 Oct 2002, Nivedita Singhvi wrote:
"Richard B. Johnson" wrote:
quoted
No. It's done over each word (short int) and the actual summation
takes place during the address calculation of the next word. This
gets you a checksum that is practically free.
Yep, sorry, word, not byte. My bad. The cost is in the fact
that this whole process involves loading each word of the data
stream into a register. Which is why I also used to consider
the checksum cost as negligible.
quoted
A 400 MHz ix86 CPU will checksum/copy at 685 megabytes per second.
It will copy at 1,549 megabytes per second. Those are megaBYTES!
But then why the difference in the checksum/copy and copy?
Are you saying the checksum is not costing you 864 megabytes
a second??
Costing you 864 megabytes per second?
Lets say the checksum was free. You are then able to INF bytes/per/sec.
So it's costing you INF bytes/per/sec? No, it's costing you nothing.
If we were not dealing with INF, then 'Cost' is approximately 1/N, not
N. Cost is work_done_without_checksum - work_done_with_checksum. Because
of the low-pass filter pole, these numbers are practically the same.
But, you can get a measurable difference between any two large numbers.
This makes the 'cost' seem high. You need to make it relative to make
any sense, so a 'goodness' can be expressed as a ratio of the cost and
the work having been done.
Cheers,
Dick Johnson
Penguin : Linux version 2.4.18 on an i686 machine (797.90 BogoMips).
Bush : The Fourth Reich of America
From: David S. Miller <hidden> Date: 2002-10-24 03:52:58
On Wed, 2002-10-23 at 06:42, Roy Sigurd Karlsbakk wrote:
As far as I've understood, sendfile() won't do much good with large files. Is
this right?
There is always a benefit to using sendfile(), when you use
sendfile() the cpu doesn't touch one byte of the data if
the network card support TX checksumming. The disk DMAs
to ram, then the net card DMAs from ram. Simple as that.
From: Roy Sigurd Karlsbakk <hidden> Date: 2002-10-24 10:16:40
On Thursday 24 October 2002 06:11, David S. Miller wrote:
On Wed, 2002-10-23 at 06:42, Roy Sigurd Karlsbakk wrote:
quoted
As far as I've understood, sendfile() won't do much good with large
files. Is this right?
There is always a benefit to using sendfile(), when you use
sendfile() the cpu doesn't touch one byte of the data if
the network card support TX checksumming. The disk DMAs
to ram, then the net card DMAs from ram. Simple as that.
Are there any plans of implementing sendfile64() or sendfile() support for
-D_FILE_OFFSET_BITS=64?
(from man 2 sendfile)
ssize_t sendfile(int out_fd, int in_fd, off_t *offset, size_t count);
int main() {
ssize_t s1;
size_t count;
off_t offset;
printf("sizeof ssize_t: %d\n", sizeof s1);
printf("sizeof size_t: %d\n", sizeof count);
printf("sizeof off_t: %d\n", sizeof offset);
return 0;
}
$ make
...
$ ./sendfile_test
sizeof ssize_t: 4
sizeof size_t: 4
sizeof off_t: 4
$
and - when attempting to build this with -D_FILE_OFFSET_BITS=64
[roy@roy-sin micro_httpd-O_DIRECT]$ make sendfile_test
gcc -D_DEBUG -Wall -W -D_GNU_SOURCE -D_NO_DIR_ACCESS -D_FILE_OFFSET_BITS=64
-D_LARGEFILE_SOURCE -DUSE_O_DIRECT -DINETD -Wno-unused -O0 -ggdb -c
sendfile_test.c
In file included from sendfile_test.c:1:
/usr/include/sys/sendfile.h:26: #error "<sys/sendfile.h> cannot be used with
_FILE_OFFSET_BITS=64"
make: *** [sendfile_test.o] Error 1
--
Roy Sigurd Karlsbakk, Datavaktmester
ProntoTV AS - http://www.pronto.tv/
Tel: +47 9801 3356
Computers are like air conditioners.
They stop working when you open Windows.
From: Roy Sigurd Karlsbakk <hidden> Date: 2002-10-24 10:53:31
On Thursday 24 October 2002 12:47, David S. Miller wrote:
On Thu, 2002-10-24 at 03:30, Roy Sigurd Karlsbakk wrote:
quoted
Are there any plans of implementing sendfile64() or sendfile() support
for -D_FILE_OFFSET_BITS=64?
This is old hat, and appears in every current vendor kernel I am
aware of and is in 2.5.x as well.
then where can I find these patches? I cannot use 2.5, and I usually try to
stick with an official kernel.
and - if this patch has been around all this time...
why isn't it in the official kernel yet?
--
Roy Sigurd Karlsbakk, Datavaktmester
ProntoTV AS - http://www.pronto.tv/
Tel: +47 9801 3356
Computers are like air conditioners.
They stop working when you open Windows.
From: Roy Sigurd Karlsbakk <hidden> Date: 2002-10-23 13:27:19
'50 clients *each* streaming at ~4.4MBps', better make that clear,
otherwise something is *very* broken. Also mention that you have an e1000
card which does not do outgoing checksumming.
just to clearify
s/MBps/Mbps/
s/bps/bits per second/
You'd think that a kernel would be able to do 250megabits of TCP checksums
though.
quoted
...adding the whole profile output - sorted by the first column this
time...
905182 total 0.4741
121426 csum_partial_copy_generic 474.3203
93633 default_idle 1800.6346
74665 do_wp_page 111.1086
Perhaps the 'copy' also entails grabbing the page from disk, leading to
inflated csum_partial_copy_generic stats?
I really don't know. Just to clearify a little more - the server app uses
O_DIRECT to read the data before tossing it to the socket.
Where are you serving from?
What do you mean?
roy
--
Roy Sigurd Karlsbakk, Datavaktmester
ProntoTV AS - http://www.pronto.tv/
Tel: +47 9801 3356
Computers are like air conditioners.
They stop working when you open Windows.
...adding the whole profile output - sorted by the first column this time...
905182 total 0.4741
121426 csum_partial_copy_generic 474.3203
93633 default_idle 1800.6346
74665 do_wp_page 111.1086
Perhaps the 'copy' also entails grabbing the page from disk, leading to
inflated csum_partial_copy_generic stats?
I think this is strictly a copy from user space->kernel and vice versa.
This shouldnt include the disk access etc.
thanks,
Nivedita
From: Roy Sigurd Karlsbakk <hidden> Date: 2002-10-23 15:12:37
On Wednesday 23 October 2002 16:59, Nivedita Singhvi wrote:
bert hubert wrote:
quoted
quoted
...adding the whole profile output - sorted by the first column this
time...
905182 total 0.4741
121426 csum_partial_copy_generic 474.3203
93633 default_idle 1800.6346
74665 do_wp_page 111.1086
Perhaps the 'copy' also entails grabbing the page from disk, leading to
inflated csum_partial_copy_generic stats?
I think this is strictly a copy from user space->kernel and vice versa.
This shouldnt include the disk access etc.
hm
I'm doing O_DIRECT read (from disk), so it needs to be user -> kernel, then.
any chance of using O_DIRECT to the socket?
--
Roy Sigurd Karlsbakk, Datavaktmester
ProntoTV AS - http://www.pronto.tv/
Tel: +47 9801 3356
Computers are like air conditioners.
They stop working when you open Windows.
I'm doing O_DIRECT read (from disk), so it needs to be user -> kernel, then.
any chance of using O_DIRECT to the socket?
Hmm, I'm still not clear on why you cannot use sendfile()?
I was not aware of any upper limit to the file size in order
for sendfile() to be used? From what little I know, this
is exactly the kind of situation that sendfile was intended
to benefit.
thanks,
Nivedita
From: Roy Sigurd Karlsbakk <hidden> Date: 2002-10-24 10:00:54
Hmm, I'm still not clear on why you cannot use sendfile()?
I was not aware of any upper limit to the file size in order
for sendfile() to be used? From what little I know, this
is exactly the kind of situation that sendfile was intended
to benefit.
I can't use sendfile(). I'm working with files > 4GB, and from man 2 sendfile:
ssize_t sendfile(int out_fd, int in_fd, off_t *offset, size_t count);
int main() {
ssize_t s1;
off_t offset;
size_t count;
printf("sizeof ssize_t: %d\n", sizeof s1);
printf("sizeof size_t: %d\n", sizeof count);
printf("sizeof off_t: %d\n", sizeof offset);
return 0;
}
running it
$ ./sendfile_test
sizeof ssize_t: 4
sizeof size_t: 4
sizeof off_t: 4
$
as far as I'm concerned, this will not allow me to address files past the 4GB
limit (or was it 2?)
roy
--
Roy Sigurd Karlsbakk, Datavaktmester
ProntoTV AS - http://www.pronto.tv/
Tel: +47 9801 3356
Computers are like air conditioners.
They stop working when you open Windows.
From: David S. Miller <hidden> Date: 2002-10-24 10:28:42
On Thu, 2002-10-24 at 03:14, Roy Sigurd Karlsbakk wrote:
I can't use sendfile(). I'm working with files > 4GB, and from man 2 sendfile:
That's what sendfile64() is for. In fact every vendor I am aware
of is shipping the sys_sendfile64() patch in their kernels and
an appropriately fixed up glibc.
On 23 October 2002 09:06, Roy Sigurd Karlsbakk wrote:
quoted
I've got this video server serving video for VoD. problem is the P4
1.8 seems to be maxed out by a few system calls. The below output
is for ~50 clients streaming at ~4.5Mbps. if trying to increase
this to ~70, the CPU maxes out.
Does anyone have an idea?
...adding the whole profile output - sorted by the first column this
time...
905182 total 0.4741
121426 csum_partial_copy_generic 474.3203
Well, maybe take a look at this func and try to optimize it?
You have 1 ide_intr per 2 csum_partial_copy_generic... hmmm...
how large is your readahead? I assume you'd like to fetch
more sectors from ide per interrupt. (I hope you do DMA ;)
do_wp_page is Defined as a function in: mm/memory.c
comments from the file:
/*
* This routine handles present pages, when users try to write
* to a shared page. It is done by copying the page to a new address
* and decrementing the shared-page counter for the old page.
*
* Goto-purists beware: the only reason for goto's here is that it results
* in better assembly code.. The "default" path will see no jumps at all.
*
* Note that this routine assumes that the protection checks have been
* done by the caller (the low-level page fault routine in most cases).
* Thus we can safely just mark it writable once we've done any necessary
* COW.
*
* We also mark the page dirty at this point even though the page will
* change only once the write actually happens. This avoids a few races,
* and potentially makes it more efficient.
*
* We hold the mm semaphore and the page_table_lock on entry and exit
* with the page_table_lock released.
*/
quoted
65857 ide_intr 184.9916
You have 1 ide_intr per 2 csum_partial_copy_generic... hmmm...
how large is your readahead? I assume you'd like to fetch
more sectors from ide per interrupt. (I hope you do DMA ;)
doing DMA - RAID-0 with 1MB chunk size on 4 disks.
I know zero about networking, but why 120 000 csum_partial_copy_generic
and inly 20 000 nic interrupts? That may be abnormal.
sorry
I don't know
--
Roy Sigurd Karlsbakk, Datavaktmester
ProntoTV AS - http://www.pronto.tv/
Tel: +47 9801 3356
Computers are like air conditioners.
They stop working when you open Windows.
do_wp_page is Defined as a function in: mm/memory.c
comments from the file:
[snip]
Please delete memory.o, rerun make bzImage, capture gcc
command used for compiling memory.c, modify it:
gcc ... -o memory.o -> gcc ... -S -o memory.s ...
and examine assembler code. Maybe something will stick out
(or use objdump to disassemble memory.o, I recall nice
option to produce assembler output with C code intermixed
as comments!) (send disasmed listing to me offlist).
quoted
quoted
65857 ide_intr 184.9916
You have 1 ide_intr per 2 csum_partial_copy_generic... hmmm...
how large is your readahead? I assume you'd like to fetch
more sectors from ide per interrupt. (I hope you do DMA ;)
doing DMA - RAID-0 with 1MB chunk size on 4 disks.
You should aim at maxing out IDE performance.
Please find out how many sectors you read in one go.
Maybe:
# cat /proc/interrupts
# dd bs=1m count=1 if=/dev/hda of=/dev/null
# cat /proc/interrupts
and calculate how many IDE interrupts happened. (1mb = 2048 sectors)
--
vda