Re: [Bugme-new] [Bug 6613] New: iptables broken on 32-bit PReP (ARCH=ppc)

bugme-daemon@bugzilla.kernel.org wrote:

http://bugzilla.kernel.org/show_bug.cgi?id=6613

           Summary: iptables broken on 32-bit PReP (ARCH=ppc)
    Kernel Version: 2.6.17-rc4
            Status: NEW
          Severity: normal
             Owner: laforge@gnumonks.org
         Submitter: mroos@linux.ee


Most recent kernel where this bug did not occur: none known, this is a fresh 
install
Distribution: Debian unstable
Hardware Environment: 32-bit PowerPC 604 with PReP subarch (using old 
ARCH=ppc)
Software Environment: usual 32-bit ppc userspace, gcc 4.0.3
Problem Description: iptables operations usually just give "Incalida 
operation". modprobe iptable_filter and adding rules to the nat table have 
failed in testing while iptable_nat can be modprobed and listed.

Steps to reproduce:
modprobe iptable_filter (errors out with Invalid Argument)
iptables -t nat -A POSTROUTING -s 10.0.0.0/8 -j SNAT --to 192.168.1.1 (usually 
errors out with Invalid Argument, sometimes succeeds, when succeeds then the 
rule works fine)

Andrew Morton wrote:

bugme-daemon@bugzilla.kernel.org wrote:

quoted

http://bugzilla.kernel.org/show_bug.cgi?id=6613

          Summary: iptables broken on 32-bit PReP (ARCH=ppc)
   Kernel Version: 2.6.17-rc4
           Status: NEW
         Severity: normal
            Owner: laforge@gnumonks.org
        Submitter: mroos@linux.ee


Most recent kernel where this bug did not occur: none known, this is a fresh 
install
Distribution: Debian unstable
Hardware Environment: 32-bit PowerPC 604 with PReP subarch (using old 
ARCH=ppc)
Software Environment: usual 32-bit ppc userspace, gcc 4.0.3
Problem Description: iptables operations usually just give "Incalida 
operation". modprobe iptable_filter and adding rules to the nat table have 
failed in testing while iptable_nat can be modprobed and listed.

Steps to reproduce:
modprobe iptable_filter (errors out with Invalid Argument)
iptables -t nat -A POSTROUTING -s 10.0.0.0/8 -j SNAT --to 192.168.1.1 (usually 
errors out with Invalid Argument, sometimes succeeds, when succeeds then the 
rule works fine)

Meelis, it would really help if you could try 2.6.16 and in case
that doesn't work 2.6.15 to give an idea about whether this is a
recent regression or an old problem. We had a number of changes
in this area in the last two kernel versions that could be related.

Meelis, it would really help if you could try 2.6.16 and in case
that doesn't work 2.6.15 to give an idea about whether this is a
recent regression or an old problem. We had a number of changes
in this area in the last two kernel versions that could be related.

Yes, I'm still compiling 2.6.16, since just before sending the report. Will let you know ASAP.

-- 
Meelis Roos (mroos@linux.ee)

quoted

quoted

http://bugzilla.kernel.org/show_bug.cgi?id=6613

Meelis, it would really help if you could try 2.6.16 and in case
that doesn't work 2.6.15 to give an idea about whether this is a
recent regression or an old problem. We had a number of changes
in this area in the last two kernel versions that could be related.

2.6.16 doesn't work either.

Tried 2.6.8-3 from sarge package, it is working.

Compiling 2.6.15 now...

-- 
Meelis Roos (mroos@linux.ee)

Meelis, it would really help if you could try 2.6.16 and in case
that doesn't work 2.6.15 to give an idea about whether this is a
recent regression or an old problem. We had a number of changes
in this area in the last two kernel versions that could be related.

Unfortunatlety, 2.6.15 does not boot on this machine so I'm locked out remotely at the moment. Will see if I can find the boot cure - there used to be a Motorola Powerstack-specific patch to make it boot that Debian 2.6.18 and IIRC 2.6.12 packages included and that was integrated somewhere later - maybe it's missing fom 2.6.15.

-- 
Meelis Roos (mroos@linux.ee)

Hi Meelis,

Unfortunatlety, 2.6.15 does not boot on this machine so I'm locked out 
remotely at the moment.

Here it my paranoid boot setup:

1. Use "lilo -R new-kernel", to boot a kernel only
    once and reboot the default kernel next time.

2. Force reboot on any panic after 10 seconds:
	append="panic=10" in /etc/lilo.conf

3. Schedule automatic reboot in case of impossible login
	echo "/bin/sync; /sbin/reboot -f "|at now + 15min

4. Put "sysctl -w kernel.panic_on_oops=1" as early as possible
     in your boot scripts[1].

And now reboot into the new kernel, try to login and delete the reboot
cronjob. If this doesn't work, just wait 15min and have the last stable kernel
booted automatically.

This method saved me and our customers a lot of time already :-)


Regards

Ingo Oeser

[1] This should be the default and should be disabled by the init scripts 
      as soon as we reach the desired runlevel (S99oops_not_fatal).

quoted

Unfortunatlety, 2.6.15 does not boot on this machine so I'm locked out
remotely at the moment.

Here it my paranoid boot setup:

Thanks, but it's not much use here, since the machine is a PReP powerpc machine that can boot one kernel from disk (directly loaded from boot partition, no fancy bootloader) or netboot via serial console for test kernels. However, if the test kernel hangs, it hangs and I would need remote power cycling device that I do not have.

-- 
Meelis Roos (mroos@linux.ee)

Ingo Oeser wrote:

Hi Meelis,

quoted

Unfortunatlety, 2.6.15 does not boot on this machine so I'm locked out 
remotely at the moment.

Here it my paranoid boot setup:

1. Use "lilo -R new-kernel", to boot a kernel only
    once and reboot the default kernel next time.

2. Force reboot on any panic after 10 seconds:
	append="panic=10" in /etc/lilo.conf

3. Schedule automatic reboot in case of impossible login
	echo "/bin/sync; /sbin/reboot -f "|at now + 15min

Instead of this, I usually use a system startup script like this:

case "$(cat /proc/cmdline)" in
 *linux-test*)
   (sleep 300; [ -f /var/run/noreboot ] || reboot) &
   ;;
esac

which means that if the kernel image is named 'linux-test', it will
be rebooted in 15 minutes after booting if no /var/run/noreboot file
exist.  So if I'm able to log in, i just touch /var/run/noreboot and
be done with it.

And oh, yes, for this to work, in lilo.conf the new entry should be
labeled linux-test -- ie, install new kernel, add new entry into lilo.conf
with label=linux-test, run `lilo && lilo -R linux-test && init 6' and..
wait ;)  After successeful reboot (and touching /var/run/noreboot), edit
lilo.conf, restore the proper label, set proper order of entries if needed
and re-run lilo.

/mjt

4. Put "sysctl -w kernel.panic_on_oops=1" as early as possible
     in your boot scripts[1].

You can as well boot with oops=panic

-Andi

Hi Andi,

Andi Kleen wrote:

quoted

4. Put "sysctl -w kernel.panic_on_oops=1" as early as possible
     in your boot scripts[1].

You can as well boot with oops=panic

Only on x86_64 as of Linux 2.6.16.
But maybe this could be put into kernel/panic.c instead :-)

Regards

Ingo Oeser

Meelis Roos wrote:

quoted

quoted

Unfortunatlety, 2.6.15 does not boot on this machine so I'm locked out
remotely at the moment.

Here it my paranoid boot setup:

Thanks, but it's not much use here, since the machine is a PReP powerpc machine that can boot one kernel from disk (directly loaded from boot partition, no fancy bootloader) or netboot via serial console for test kernels. However, if the test kernel hangs, it hangs and I would need remote power cycling device that I do not have.

I did a lot of this at one time, and used lilo in just the way described. I did have a remote reboot device, however, an operator (1st shift), janitor (2nd shift), or security guard (3rd/wkend shift) who had been instructed to push the clearly marked reset button on demand "when the weird guy in New York tells you."

IBM rack units, like x345 and such, can have an "RSA" card which allows remote hardware monitor and reboot with a separate IP address for control. Worth its weight in gold! The latest will let you do remote console as well.

-- 
Bill Davidsen [off-list ref]
  Obscure bug of 2004: BASH BUFFER OVERFLOW - if bash is being run by a
normal user and is setuid root, with the "vi" line edit mode selected,
and the character set is "big5," an off-by-one errors occurs during
wildcard (glob) expansion.

quoted

quoted

modprobe iptable_filter (errors out with Invalid Argument)
iptables -t nat -A POSTROUTING -s 10.0.0.0/8 -j SNAT --to 192.168.1.1 (usually
errors out with Invalid Argument, sometimes succeeds, when succeeds then the
rule works fine)

Meelis, it would really help if you could try 2.6.16 and in case
that doesn't work 2.6.15 to give an idea about whether this is a
recent regression or an old problem. We had a number of changes
in this area in the last two kernel versions that could be related.

Have not gotten 2.6.15 to work with one evening of tinkering - the irq patch was not sufficent, there is something more broken in booting that I dodn't figure out yet. So no test results for 2.6.15 yet.

-- 
Meelis Roos (mroos@linux.ee)

Meelis Roos wrote:

quoted

Meelis, it would really help if you could try 2.6.16 and in case
that doesn't work 2.6.15 to give an idea about whether this is a
recent regression or an old problem. We had a number of changes
in this area in the last two kernel versions that could be related.

Have not gotten 2.6.15 to work with one evening of tinkering - the irq
patch was not sufficent, there is something more broken in booting that
I dodn't figure out yet. So no test results for 2.6.15 yet.

Then lets try something different. Please enable the
DEBUG_IP_FIREWALL_USER define in net/ipv4/netfilter/ip_tables.c and
post the results, if any.

Then lets try something different. Please enable the
DEBUG_IP_FIREWALL_USER define in net/ipv4/netfilter/ip_tables.c and
post the results, if any.

On bootup I get this in dmesg (one Bad offset has been added):

ip_tables: (C) 2000-2006 Netfilter Core Team
Netfilter messages via NETLINK v0.30.
ip_conntrack version 2.4 (1536 buckets, 12288 max) - 224 bytes per conntrack
translate_table: size 632
Bad offset cb437924
ip_nat_init: can't setup rules.

And on iptables -t nat -L

translate_table: size 632
Bad offset cb4368f4
ip_nat_init: can't setup rules.
translate_table: size 632
Bad offset cb4368f4
ip_nat_init: can't setup rules.

Seems iptable_nat does not load at all this time.

Modprobe iptable_filter still fails, dmesg contains
translate_table: size 632
Finished chain 1
Finished chain 2
Finished chain 3

Next modprobe iptable_nat gives

translate_table: size 632
Bad offset c8e01944
ip_nat_init: can't setup rules.

-- 
Meelis Roos (mroos@linux.ee)

Meelis Roos wrote:

quoted

Then lets try something different. Please enable the
DEBUG_IP_FIREWALL_USER define in net/ipv4/netfilter/ip_tables.c and
post the results, if any.

On bootup I get this in dmesg (one Bad offset has been added):

ip_tables: (C) 2000-2006 Netfilter Core Team
Netfilter messages via NETLINK v0.30.
ip_conntrack version 2.4 (1536 buckets, 12288 max) - 224 bytes per
conntrack
translate_table: size 632
Bad offset cb437924
ip_nat_init: can't setup rules.

And on iptables -t nat -L

translate_table: size 632
Bad offset cb4368f4
ip_nat_init: can't setup rules.
translate_table: size 632
Bad offset cb4368f4
ip_nat_init: can't setup rules.

Seems iptable_nat does not load at all this time.

Modprobe iptable_filter still fails, dmesg contains
translate_table: size 632
Finished chain 1
Finished chain 2
Finished chain 3

Next modprobe iptable_nat gives

translate_table: size 632
Bad offset c8e01944
ip_nat_init: can't setup rules.

Very strange, this means that the initial table data must somehow
be wrong, but for some reason it still seems to get past the
size and offset checks for the filter table. I can't see how
loading the filter table could fail after the "Finished chain .."
messages without another message. Which kernel version did you
perform these test on?

Very strange, this means that the initial table data must somehow
be wrong, but for some reason it still seems to get past the
size and offset checks for the filter table. I can't see how
loading the filter table could fail after the "Finished chain .."
messages without another message. Which kernel version did you
perform these test on?

Yesterdays 2.6.17-rc5+git.

-- 
Meelis Roos (mroos@linux.ee)

Meelis Roos wrote:

quoted

Very strange, this means that the initial table data must somehow
be wrong, but for some reason it still seems to get past the
size and offset checks for the filter table. I can't see how
loading the filter table could fail after the "Finished chain .."
messages without another message. Which kernel version did you
perform these test on?

Yesterdays 2.6.17-rc5+git.

Please enable DEBUG_IP_FIREWALL_USER in net/netfilter/x_tables.c as well
and retry. Results of the raw or mangle table would also be interesting
because they contain a different number of built-in chains.

Please enable DEBUG_IP_FIREWALL_USER in net/netfilter/x_tables.c as well
and retry. Results of the raw or mangle table would also be interesting
because they contain a different number of built-in chains.

Sorry it took so long, I was away. Adding this define does not seem to do much (table->private->number prints only):

On boot (1 nat rule):
ip_tables: (C) 2000-2006 Netfilter Core Team
Netfilter messages via NETLINK v0.30.
ip_conntrack version 2.4 (1536 buckets, 12288 max) - 224 bytes per conntrack
translate_table: size 632
Finished chain 0
Finished chain 3
Finished chain 4
table->private->number = 4
t->private->number = 4
translate_table: size 800
Bad offset cba528d4

modprobe iptable_nat succeeded in manual modprobe.

modprobe iptable_filter:
translate_table: size 632
Bad offset cbbd910c

modprobe iptable_mangle:
translate_table: size 936
Bad offset cbbd80dc

modprobe iptable_raw:
translate_table: size 480
Bad offset cb8abd44

Retrying ifup and ifdown that tried to do iptables -D and iptables -I:
t->private->number = 4
t->private->number = 4
t->private->number = 4
translate_table: size 800
Bad offset cbbd80dc
t->private->number = 4

And retrying it more (succeeded this time):

t->private->number = 4
t->private->number = 4
translate_table: size 800
Finished chain 0
Finished chain 3
Finished chain 4
ip_tables: Translated table
do_replace: oldnum=4, initnum=4, newnum=5
t->private->number = 5

-- 
Meelis Roos (mroos@linux.ee)