Re: OOM killer invoked on 2.6.24.4

On Mon, 2008-06-16 at 16:55 -0300, Marcelo Tosatti wrote:

On Sat, Jun 14, 2008 at 05:31:18PM +0200, Bart Van Assche wrote:

quoted

On Sat, Jun 14, 2008 at 12:13 PM, Marco Barbero [off-list ref] wrote:

quoted

samba/slapd/heartbeat/drbd/mon

...

quoted

Jun 14 01:00:17 pippo kernel: Out of memory: kill process 6873 (slapd)
score 5003 or a child
Jun 14 01:00:17 pippo kernel: Killed process 6873 (slapd)

Anyone can help me in understanding what went wrong? And if I need to
upgrade to last kernel version?

One or more processes on your system used too much memory. Dumping the
output of the following command periodically (e.g. every 10 minutes)
to a file will tell you which process is using too much memory:

{ ps aux | head -n 1; ps aux | sort -n +4 | tail -n 10; }

There were lots of swap free:

Jun 14 01:00:17 pippo kernel: Free swap  = 3903752kB
Jun 14 01:00:17 pippo kernel: Total swap = 3903752kB
Jun 14 01:00:17 pippo kernel: Free swap:       3903752kB

Something is wrong.

It was a normal GFP_KERNEL allocation:

Jun 14 01:00:17 pippo kernel: sendmail-mta invoked oom-killer:
gfp_mask=0x800d0, order=0, oomkilladj=0

And the system /is/ low on memory:

Jun 14 01:00:17 pippo kernel: DMA free:3548kB min:68kB low:84kB
high:100kB active:0kB inactive:4kB present:16256kB pages_scanned:25
all_unreclaimable? yes
Jun 14 01:00:17 pippo kernel: lowmem_reserve[]: 0 873 4810 4810
Jun 14 01:00:17 pippo kernel: Normal free:3736kB min:3744kB low:4680kB
high:5616kB active:4616kB inactive:4672kB present:894080kB
pages_scanned:14277 all_unreclaimable? yes
Jun 14 01:00:17 pippo kernel: lowmem_reserve[]: 0 0 31496 31496
Jun 14 01:00:17 pippo kernel: HighMem free:2975684kB min:512kB
low:4736kB high:8960kB active:248580kB inactive:21644kB
present:4031488kB pages_scanned:0 all_unreclaimable? no

Still high memory available, and swap, but ZONE_DMA and ZONE_NORMAL were
all used and "all_unreclaimable? yes" . So __alloc_pages(GFP_KERNEL)
fails.

It might be tcp buffers- see
http://marc.info/?l=linux-netdev&m=121362441431941&w=2

In which case, use this workaround:

echo "98304 131072 196608" > /proc/sys/net/ipv4/tcp_mem

Mike.