From: Russell King <hidden> Date: 2003-03-07 09:32:06
On Fri, Mar 07, 2003 at 01:29:05AM +0000, Chris Dukes wrote:
If IBM can fit a kernel and a ramdisk containing all the utilities you
describe and more in smaller than 5M of file for tftp, one would think
that it could be done on Linux.
Wow. 5MB eh? We currently do NFS-root in 690K.
quoted
ipconfig.c does more than just configure networking. It's a far smaller
solution to NFS-root than any userspace implementation could ever hope
to be.
That's nice. Would you mind explaining to us where that would be a
benefit? Aside from dead header space in elf executables, I'm at
a loss as to how a usermode implementation must be significantly
larger than kernel code.
If you're suggesting above that "5MB isn't significantly larger than
the size Linux can do this" then I think I've just proven you wrong.
Lets see - building an ramdisk to mount a root filesystem out of existing
binaries would require from my exisitng systems probably something like:
text data bss dec hex filename
1093047 21224 15560 1129831 113d67 /lib/libc.so.6
515890 22320 16640 554850 87762 /bin/sh
58540 2436 9776 70752 11460 /lib/libresolv.so.2
53685 1476 5488 60649 ece9 /bin/mount
45511 672 432 46615 b617 /bin/sed
42830 624 40 43494 a9e6 /sbin/pump
10783 500 104 11387 2c7b /lib/libtermcap.so.2
8765 444 28 9237 2415 /lib/libdl.so.2
pump isn't really suitable for the task, but I don't have dhcpcd around.
dhcpcd is even larger than pump however.
That's getting on for 2MB vs:
2620 2012 0 4632 1218 fs/nfs/nfsroot.o
8016 380 80 8476 211c net/ipv4/ipconfig.o
about 13K.
Which version is overly bloated?
Which version is huge?
Which version is compact?
Even the klibc ipconfig version is significantly larger than the in-kernel
version - and klibc and its binaries are written to be small.
Note: I *do* agree that ipconfig.c needs to die before 2.6 but I do not
agree that today is the right day.
--
Russell King (rmk@arm.linux.org.uk) The developer of ARM Linux
http://www.arm.linux.org.uk/personal/aboutme.html
Which version is overly bloated?
Which version is huge?
Which version is compact?
... and the size is not important only because we want to make everything
smaller, but because of how it's commonly used (at least in the clustering
world from which I come):
the mainboard BIOS or NIC PROC contains PXE/DHCP client; data is
transferred through UDP, with very poor (if any) congestion control.
Congestion control means here both extreme situations: if packets don't
arrive to the client, it might not ask again, ask only a limited number of
times or give up after some timeout; if the server has some faster NIC to
be able to handle more such requests, it might also send too fast for a
single client which might drop packets. In some cases, if such situation
occurs, the client just blocks there printing an error message on the
console, without trying to restart the whole process and the only way to
make it do something is to press the Reset button or plug in a keyboard...
When you have tens or hundreds of such nodes, it's not a pleasure !
Booting a bunch of such nodes would become problematic if they need
to transfer more data (=initrd) to start the kernel and so network booting
would become less reliable. Please note that I'm not saying "ipconfig has
to stay" - just that any solution should not dramatically increase the
size of data transferred before the jump to kernel code.
--
Bogdan Costescu
IWR - Interdisziplinaeres Zentrum fuer Wissenschaftliches Rechnen
Universitaet Heidelberg, INF 368, D-69120 Heidelberg, GERMANY
Telephone: +49 6221 54 8869, Telefax: +49 6221 54 8868
E-mail: Bogdan.Costescu@IWR.Uni-Heidelberg.De
From: William Lee Irwin III <hidden> Date: 2003-03-07 21:38:06
On Fri, Mar 07, 2003 at 09:42:35AM +0000, Russell King wrote:
That's getting on for 2MB vs:
2620 2012 0 4632 1218 fs/nfs/nfsroot.o
8016 380 80 8476 211c net/ipv4/ipconfig.o
about 13K.
There's a cap on the maximum size of things various bootloaders can
load via tftp; 2MB is relatively certain to blow it. ISTR the limit
being something near 1MB for 2 of my boxen.
-- wli
From: Chris Friesen <hidden> Date: 2003-03-07 21:51:23
William Lee Irwin III wrote:
On Fri, Mar 07, 2003 at 09:42:35AM +0000, Russell King wrote:
quoted
That's getting on for 2MB vs:
2620 2012 0 4632 1218 fs/nfs/nfsroot.o
8016 380 80 8476 211c net/ipv4/ipconfig.o
about 13K.
There's a cap on the maximum size of things various bootloaders can
load via tftp; 2MB is relatively certain to blow it. ISTR the limit
being something near 1MB for 2 of my boxen.
Since this is totally machine/architecture specific (we're tftp'ing 10MB
kernel/ramdisk images to embedded PPC machines here) it might be a good
idea to ask around and find what the most restrictive requirements are.
Is 1MB the worst-case or does it get even tighter?
Chris
--
Chris Friesen | MailStop: 043/33/F10
Nortel Networks | work: (613) 765-0557
3500 Carling Avenue | fax: (613) 765-2986
Nepean, ON K2H 8E9 Canada | email: cfriesen@nortelnetworks.com
From: Eric W. Biederman <hidden> Date: 2003-03-08 01:53:33
Bogdan Costescu [off-list ref] writes:
On Fri, 7 Mar 2003, Russell King wrote:
quoted
Which version is overly bloated?
Which version is huge?
Which version is compact?
... and the size is not important only because we want to make everything
smaller, but because of how it's commonly used (at least in the clustering
world from which I come):
the mainboard BIOS or NIC PROC contains PXE/DHCP client; data is
transferred through UDP, with very poor (if any) congestion control.
Only because the implementations suck. See etherboot.
Congestion control means here both extreme situations: if packets don't
arrive to the client, it might not ask again, ask only a limited number of
times or give up after some timeout; if the server has some faster NIC to
be able to handle more such requests, it might also send too fast for a
single client which might drop packets. In some cases, if such situation
occurs, the client just blocks there printing an error message on the
console, without trying to restart the whole process and the only way to
make it do something is to press the Reset button or plug in a keyboard...
When you have tens or hundreds of such nodes, it's not a pleasure !
But this is all before the kernel is loaded. Having booted a 1000 node
cluster with TFTP and DHCP. From a single host with even being in the same
town, I think I have some room to talk.
Booting a bunch of such nodes would become problematic if they need
to transfer more data (=initrd) to start the kernel and so network booting
would become less reliable. Please note that I'm not saying "ipconfig has
to stay" - just that any solution should not dramatically increase the
size of data transferred before the jump to kernel code.
Right. But I would suggest fixing your NBP (what PXE load) which must
be < 64K anyway if you have noticeable reliability problems. Not that
I even suggest using PXE for production use anyway. But sometimes
you are stuck with what you can do.
Eric
Only because the implementations suck. See etherboot.
Agreed, but as you rightly say at the end of your message...
But sometimes you are stuck with what you can do.
... and you can't go use etherboot or whatever, you have to deal with it.
You can deal with it today because ipconfig is small, you might not be
able to deal with it tomorrow if you'll have to transfer twice as much
because of a big initrd.
But this is all before the kernel is loaded.
But that's exactly my point. The ipconfig functionality is needed and what
I ask for is that whatever means (if any) are chosen to replace it, they
should keep the low size.
Having booted a 1000 node cluster with TFTP and DHCP.
I do not doubt this, but I'm afraid that you (or we) might not be able to
do it again tomorrow. And probably this is an ideal case where you have
used the better solution as client (etherboot)...
--
Bogdan Costescu
IWR - Interdisziplinaeres Zentrum fuer Wissenschaftliches Rechnen
Universitaet Heidelberg, INF 368, D-69120 Heidelberg, GERMANY
Telephone: +49 6221 54 8869, Telefax: +49 6221 54 8868
E-mail: Bogdan.Costescu@IWR.Uni-Heidelberg.De
From: Eric W. Biederman <hidden> Date: 2003-03-08 15:57:51
Bogdan Costescu [off-list ref] writes:
On 7 Mar 2003, Eric W. Biederman wrote:
quoted
Only because the implementations suck. See etherboot.
Agreed, but as you rightly say at the end of your message...
quoted
But sometimes you are stuck with what you can do.
... and you can't go use etherboot or whatever, you have to deal with it.
At the very least I can use etherboot as a NBP in PXE terms. So I have
a reasonable client after the first tftp transaction.
You can deal with it today because ipconfig is small, you might not be
able to deal with it tomorrow if you'll have to transfer twice as much
because of a big initrd.
I routinely support an initrd with:
glibc.
/bin/bash
dhclient
mke2fs
mkreiserfs
parted
sfdisk
mount
pivot_root
etc. (All binaries were striped though).
And I usually have to pass an ramdisk_size=XXX option to the kernel or
my decompressed initial ramdisk is to large. I use it for setting up
a local filesystem on a cluster node. And I was able to setup an
entire cluster 1000 node cluster in about 15-20 minutes. (Multicast cuts
down on the bandwidth requirements which is very nice).
With a good bootloader it does not much how big your initrd is. I
totally agree that small is good and important. At the same time
ipconfig.c is wrong. It is great during development and on systems
with a single NIC. But the hard coded policies can be bad for
production systems. Not that hard coded policies are bad in general
just the kernel is the wrong place to put them.
quoted
But this is all before the kernel is loaded.
But that's exactly my point. The ipconfig functionality is needed and what
I ask for is that whatever means (if any) are chosen to replace it, they
should keep the low size.
Similar functionality is definitely needed.
quoted
Having booted a 1000 node cluster with TFTP and DHCP.
I do not doubt this, but I'm afraid that you (or we) might not be able to
do it again tomorrow. And probably this is an ideal case where you have
used the better solution as client (etherboot)...
True. But when things are important and the there is GPL'd firmware
available that actually works properly. It is worth putting it on the
requirements list of things to do.
Eric