On Fri, Jul 22, 2011 at 1:58 AM, Andrew Lutomirski [off-list ref] wrote:

On Thu, Jul 21, 2011 at 12:42 PM, Minchan Kim [off-list ref] wrote:

quoted

On Thu, Jul 21, 2011 at 12:36:11PM -0400, Andrew Lutomirski wrote:

quoted

On Thu, Jul 21, 2011 at 12:24 PM, Minchan Kim [off-list ref] wrote:

quoted

On Thu, Jul 21, 2011 at 05:09:59PM +0100, Mel Gorman wrote:

quoted

On Fri, Jul 22, 2011 at 12:37:22AM +0900, Minchan Kim wrote:

quoted

On Fri, Jun 24, 2011 at 03:44:53PM +0100, Mel Gorman wrote:

quoted

(Built this time and passed a basic sniff-test.)

During allocator-intensive workloads, kswapd will be woken frequently
causing free memory to oscillate between the high and min watermark.
This is expected behaviour.  Unfortunately, if the highest zone is
small, a problem occurs.

This seems to happen most with recent sandybridge laptops but it's
probably a co-incidence as some of these laptops just happen to have
a small Normal zone. The reproduction case is almost always during
copying large files that kswapd pegs at 100% CPU until the file is
deleted or cache is dropped.

The problem is mostly down to sleeping_prematurely() keeping kswapd
awake when the highest zone is small and unreclaimable and compounded
by the fact we shrink slabs even when not shrinking zones causing a lot
of time to be spent in shrinkers and a lot of memory to be reclaimed.

Patch 1 corrects sleeping_prematurely to check the zones matching
  the classzone_idx instead of all zones.

Patch 2 avoids shrinking slab when we are not shrinking a zone.

Patch 3 notes that sleeping_prematurely is checking lower zones against
  a high classzone which is not what allocators or balance_pgdat()
  is doing leading to an artifical believe that kswapd should be
  still awake.

Patch 4 notes that when balance_pgdat() gives up on a high zone that the
  decision is not communicated to sleeping_prematurely()

This problem affects 2.6.38.8 for certain and is expected to affect
2.6.39 and 3.0-rc4 as well. If accepted, they need to go to -stable
to be picked up by distros and this series is against 3.0-rc4. I've
cc'd people that reported similar problems recently to see if they
still suffer from the problem and if this fixes it.

Good!
This patch solved the problem.
But there is still a mystery.

In log, we could see excessive shrink_slab calls.

Yes, because shrink_slab() was called on each loop through
balance_pgdat() even if the zone was balanced.

quoted

And as you know, we had merged patch which adds cond_resched where last of the function
in shrink_slab. So other task should get the CPU and we should not see
100% CPU of kswapd, I think.

cond_resched() is not a substitute for going to sleep.

Of course, it's not equal with sleep but other task should get CPU and conusme their time slice
So we should never see 100% CPU consumption of kswapd.
No?

If the rest of the system is idle, then kswapd will happily use 100%
CPU.  (Or on a multi-core system, kswapd will use close to 100% of one

Of course. But at least, we have a test program and I think it's not idle.

The test program I used was 'top', which is pretty close to idle.

quoted

quoted

CPU even if another task is using the other one.  This is bad enough
on a desktop, but on a laptop you start to notice when your battery
dies.)

Of course it's bad. :)
What I want to know is just what's exact cause of 100% CPU usage.
It might be not 100% but we might use the word sloppily.

Well, if you want to pedantic, my laptop can, in theory, demonstrate
true 100% CPU usage.  Trigger the bug, suspend every other thread, and
listen to the laptop fan spin and feel the laptop get hot.  (The fan
is controlled by the EC and takes no CPU.)

In practice, the usage was close enough to 100% that it got rounded.

The cond_resched was enough to at least make the system responsive
instead of the hard freeze I used to get.

I don't want to be pedantic. :)
What I have a thought about 100% CPU usage was that it doesn't yield
CPU and spins on the CPU but as I heard your example(ie, cond_resched
makes the system responsive), it's not the case. It was just to use
most of time in kswapd, not 100%. It seems I was paranoid about the
word, sorry for that.

-- 
Kind regards,
Minchan Kim

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>