[BUG] SNAT sometimes allows packets to pass through unchanged

I'm seeing some strange behavior on my firewall, which is running 
Fedora 8's version of 2.6.26.  Every so often a packet with a private 
source address is sent out the public interface unchanged, when it 
should be dropped.

This happens when internal hosts are slow to close their end of a TCP 
connection.  For example:

	Internal host A (using a private address) initiates a TCP
	connection to an external server B.

	Data is sent back and forth.

	External host B sends a FIN and host A responds with ACK.

	Several minutes later (after the tracking for this connection
	has expired), host A sends a FIN to host B.  This packet
	goes through the firewall unchanged and is sent out the
	public interface with the private source address intact.

Now I would expect that such packets would be dropped, because they 
don't belong to an existing connection and they can't be the start of a 
new connection.  The fact that this doesn't happen indicates there is a 
bug in the netfilter code somewhere.

For reference, here is a lightly-edited extract from my iptables 
script on the firewall:

------------------------------------------------------------------------
# Use SNAT for new outgoing connections from private addresses.
OUTIP=...  # The IP address of the firewall's public interface, eth1
iptables -t nat -N startnat
iptables -t nat -A startnat -p tcp -j SNAT --to-source $OUTIP:7000-65500
iptables -t nat -A startnat -p udp -j SNAT --to-source $OUTIP:7000-65500
iptables -t nat -A startnat -p icmp -j SNAT --to-source $OUTIP
iptables -t nat -A startnat -j DROP

iptables -t nat -A POSTROUTING -o eth1 -s 10.0.0.0/8 -j startnat
------------------------------------------------------------------------

Can anyone help figure out what's wrong?

Alan Stern

Alan Stern wrote:

I'm seeing some strange behavior on my firewall, which is running Fedora 8's version of 2.6.26.  Every so often a packet with a private source address is sent out the public interface unchanged, when it should be dropped.

This happens when internal hosts are slow to close their end of a TCP connection.  For example:

	Internal host A (using a private address) initiates a TCP
	connection to an external server B.

	Data is sent back and forth.

	External host B sends a FIN and host A responds with ACK.

	Several minutes later (after the tracking for this connection
	has expired), host A sends a FIN to host B.  This packet
	goes through the firewall unchanged and is sent out the
	public interface with the private source address intact.

Now I would expect that such packets would be dropped, because they don't belong to an existing connection and they can't be the start of a new connection.  The fact that this doesn't happen indicates there is a bug in the netfilter code somewhere.

If the connection has already timed out (from conntracks perspective),
it has lost its state. Unless connection pickup is enabled, the packet will be marked as INVALID because it doesn't belong to a connection.
You can control dropping of these packets yourself by adding the
appropriate "-m state --state INVALID" rules. That said, there were
some bugs in the past few releases that caused some bad interaction
between TCP and TCP conntrack (not sure anymore which one of both was
to blame). Its possible that this is the root cause for this, so
you might want to consider a kernel update.

On Thu, 12 Feb 2009, Patrick McHardy wrote:

If the connection has already timed out (from conntracks perspective),
it has lost its state. Unless connection pickup is enabled, the packet 
will be marked as INVALID because it doesn't belong to a connection.
You can control dropping of these packets yourself by adding the
appropriate "-m state --state INVALID" rules.

I tried adding a rule to log these unaccounted-for packets.  Nothing 
showed up, even when I could see the packets being sent.

That said, there were
some bugs in the past few releases that caused some bad interaction
between TCP and TCP conntrack (not sure anymore which one of both was
to blame). Its possible that this is the root cause for this, so
you might want to consider a kernel update.

It does sound like the result of a bug.  Do you have any pointers to 
patches or locations to check in the source?

Alan Stern

Alan Stern wrote:

On Thu, 12 Feb 2009, Patrick McHardy wrote:

quoted

If the connection has already timed out (from conntracks perspective),
it has lost its state. Unless connection pickup is enabled, the packet will be marked as INVALID because it doesn't belong to a connection.
You can control dropping of these packets yourself by adding the
appropriate "-m state --state INVALID" rules.

I tried adding a rule to log these unaccounted-for packets.  Nothing showed up, even when I could see the packets being sent.

Where (table/chain/position) did you add this rule?

quoted

That said, there were
some bugs in the past few releases that caused some bad interaction
between TCP and TCP conntrack (not sure anymore which one of both was
to blame). Its possible that this is the root cause for this, so
you might want to consider a kernel update.

It does sound like the result of a bug.  Do you have any pointers to patches or locations to check in the source?

Sorry, there were quite a few patches and I don't remember which
ones exactly are related.

On Mon, 16 Feb 2009, Patrick McHardy wrote:

Alan Stern wrote:

quoted

On Thu, 12 Feb 2009, Patrick McHardy wrote:

quoted

If the connection has already timed out (from conntracks perspective),
it has lost its state. Unless connection pickup is enabled, the packet 
will be marked as INVALID because it doesn't belong to a connection.
You can control dropping of these packets yourself by adding the
appropriate "-m state --state INVALID" rules.

I tried adding a rule to log these unaccounted-for packets.  Nothing 
showed up, even when I could see the packets being sent.

Where (table/chain/position) did you add this rule?

In the first position of the POSTROUTING chain in the nat table.  I
don't remember exactly what rules I used, but at one point I tried
something very much like this:

iptables -t nat -I POSTROUTING 1 -s 10.0.0.0/8 -p tcp ! --syn

The counter for this rule remained at 0 even after packets with private 
source addresses were sent through the public interface.

quoted

quoted

That said, there were
some bugs in the past few releases that caused some bad interaction
between TCP and TCP conntrack (not sure anymore which one of both was
to blame). Its possible that this is the root cause for this, so
you might want to consider a kernel update.

It does sound like the result of a bug.  Do you have any pointers to 
patches or locations to check in the source?

Sorry, there were quite a few patches and I don't remember which
ones exactly are related.

I tried using 2 6.27 kernel but the problem remained.  Building a later 
version won't be easy because of the need to create the proper config.  
Can you remember in which version these bugs got fixed?

Alan Stern

Alan Stern wrote:

On Mon, 16 Feb 2009, Patrick McHardy wrote:

quoted

quoted

I tried adding a rule to log these unaccounted-for packets.  Nothing showed up, even when I could see the packets being sent.

Where (table/chain/position) did you add this rule?

In the first position of the POSTROUTING chain in the nat table.  I
don't remember exactly what rules I used, but at one point I tried
something very much like this:

iptables -t nat -I POSTROUTING 1 -s 10.0.0.0/8 -p tcp ! --syn

The counter for this rule remained at 0 even after packets with private source addresses were sent through the public interface.

The NAT table only sees the first packet of every connection
and never INVALID packets. The mangle table should work fine.

You can also enable conntrack-internal logging of invalid packets:

echo 255 >/proc/sys/net/netfilter/nf_conntrack_log_invalid

quoted

Sorry, there were quite a few patches and I don't remember which
ones exactly are related.

I tried using 2 6.27 kernel but the problem remained.  Building a later version won't be easy because of the need to create the proper config.  Can you remember in which version these bugs got fixed?

Sorry, no.

On Mon, 16 Feb 2009, Patrick McHardy wrote:

Alan Stern wrote:

quoted

On Mon, 16 Feb 2009, Patrick McHardy wrote:

quoted

quoted

I tried adding a rule to log these unaccounted-for packets.  Nothing 
showed up, even when I could see the packets being sent.

Where (table/chain/position) did you add this rule?

In the first position of the POSTROUTING chain in the nat table.  I
don't remember exactly what rules I used, but at one point I tried
something very much like this:

iptables -t nat -I POSTROUTING 1 -s 10.0.0.0/8 -p tcp ! --syn

The counter for this rule remained at 0 even after packets with private 
source addresses were sent through the public interface.

The NAT table only sees the first packet of every connection
and never INVALID packets. The mangle table should work fine.

Okay, I'll try it instead.  Are these facts documented anywhere?  FWIW, 
I don't recall ever seeing anywhere a description of what packets go 
through the mangle table, or at what stage of processing.

You can also enable conntrack-internal logging of invalid packets:

echo 255 >/proc/sys/net/netfilter/nf_conntrack_log_invalid

I did this (except I used the value of IPPROTO_TCP instead of 255).  
Several things did get logged, but no messages were produced for the
packets I'm concerned about.  _That_ definitely seemed like a bug...

quoted

I tried using 2 6.27 kernel but the problem remained.  Building a later 
version won't be easy because of the need to create the proper config.  
Can you remember in which version these bugs got fixed?

Sorry, no.

Hm.  Maybe I'll try to set up a test system with a 2.6.29-rc kernel, to 
see if the problem really has been fixed yet.

Alan Stern

On Mon, 16 Feb 2009, Patrick McHardy wrote:

The NAT table only sees the first packet of every connection
and never INVALID packets. The mangle table should work fine.

I ended up adding a rule to the FORWARD chain of the filter table.  
The trick was to select based on the state.  That worked; it saw all
those un-NATed packets and was able to eliminate them.  In case you're
curious, the rule was essentially this:

iptables -A FORWARD -o eth1 -s 10.0.0.0/8 -m state --state INVALID -j DROP

Ideally, the rule should select all the packets which haven't been
altered by SNAT, not just the ones marked INVALID.  Is there any way to
do this?

Alan Stern

Alan Stern wrote:

On Mon, 16 Feb 2009, Patrick McHardy wrote:

quoted

The NAT table only sees the first packet of every connection
and never INVALID packets. The mangle table should work fine.

I ended up adding a rule to the FORWARD chain of the filter table.  The trick was to select based on the state.  That worked; it saw all
those un-NATed packets and was able to eliminate them.  In case you're
curious, the rule was essentially this:

iptables -A FORWARD -o eth1 -s 10.0.0.0/8 -m state --state INVALID -j DROP

Ideally, the rule should select all the packets which haven't been
altered by SNAT, not just the ones marked INVALID.  Is there any way to
do this?

Not in the sense that you could somehow catch valid packets "missed"
by SNAT, that would be a bug. The conntrack match supports matching
on whether the state says that a packet should be NATed.

On Wed, 18 Feb 2009, Patrick McHardy wrote:

Alan Stern wrote:

quoted

On Mon, 16 Feb 2009, Patrick McHardy wrote:

quoted

The NAT table only sees the first packet of every connection
and never INVALID packets. The mangle table should work fine.

I ended up adding a rule to the FORWARD chain of the filter table.  
The trick was to select based on the state.  That worked; it saw all
those un-NATed packets and was able to eliminate them.  In case you're
curious, the rule was essentially this:

iptables -A FORWARD -o eth1 -s 10.0.0.0/8 -m state --state INVALID -j DROP

 >

quoted

Ideally, the rule should select all the packets which haven't been
altered by SNAT, not just the ones marked INVALID.  Is there any way to
do this?

Not in the sense that you could somehow catch valid packets "missed"
by SNAT, that would be a bug. The conntrack match supports matching
on whether the state says that a packet should be NATed.

You mean I should do something like this?

iptables -A FORWARD -o eth1 -s 10.0.0.0/8 -m conntrack --ctstate ! SNAT -j DROP

The man page says that --ctstate doesn't support using ! for negation,
but that detail is easily worked around.

Alan Stern