Re: compaction of zspages

On Wed, Aug 27, 2014 at 5:17 PM, Minchan Kim [off-list ref] wrote:

Hey Luigi,

On Wed, Aug 27, 2014 at 04:25:52PM -0700, Luigi Semenzato wrote:

quoted

Thank you Seth!

On Wed, Aug 27, 2014 at 3:09 PM, Seth Jennings [off-list ref] wrote:

quoted

On Wed, Aug 27, 2014 at 02:42:52PM -0700, Luigi Semenzato wrote:

quoted

Hello Minchan and others,

I just noticed that the data structures used by zsmalloc have the
potential to tie up memory unnecessarily.  I don't call it "leaking"
because that memory can be reused, but it's not necessarily returned
to the system upon freeing.

Yes, this is a known condition in zsmalloc.

Yeb, I discussed it with Seth and Dan two years ago but I didn't have
a number how it's significat problem for real practice and no time to
look at it.

quoted

quoted

Compaction is not a simple as it seems because zsmalloc returns a handle
to the user that encodes the pfn.  In order the implement a compaction
system, there would need to be some notification method to the alert the
user that their allocation has moved and provide a new handle so the
user can update its structures.  This is very non-trivial and I'm not
sure that it can be done safely (i.e.  without races).

Since the handles are opaque, we can add a level of indirection
without affecting users.  Assuming that the overhead is tolerable, or
anyway less than what we're wasting now.  (For some definition of
"less".)

Yeb, my idea was same.
We could add indirection layer and it wouldn't be hard to implement.
It would add a bit overhead for memory footprint and performance
but I think it's is worth to try and see the result.

Well I don't even know if this is really a problem, so I'll try to
determine that first.

I hope I'd really like to implement it.

I am not sure you mean what you wrote, but I hope so too! :-)

quoted

I agree that notification + update would be a huge pain, not really acceptable.

quoted

I looked at it a while back and it would be a significant effort.

And yes, if you could do such a thing, you would not want the compaction
triggered by the shrinkers as the users of zsmalloc are only active
under memory pressure.  Something like a periodic compaction kthread
would be the best way (after two minutes of thinking about it).

Seth

quoted

I have no idea if this has any impact in practice, but I plan to run a
test in the near future.  Also, I am not sure that doing compaction in
the shrinkers (as planned according to a comment) is the best
approach, because the shrinkers won't be called unless there is
considerable pressure, but the compaction would be more effective when
there is less pressure.

If we add the feature, basically, I'd like to open the interface(ex, zs_compact)
to user because when we need to compact depends on user's usecase and then
we could add up more smart things (ex, zs_set_auto_compaction(frag_ratio))
based on it.

OK.

quoted

quoted

quoted

Some more detail here:

https://code.google.com/p/chromium/issues/detail?id=408221

Should I open a bug on some other tracker?

I don't think it's a bug, every allocator have a same problem(fragmentation).

Right, I don't think so either---we tend to use the term "bug" too
loosely,  Our bug tracker is really an issue tracker, including
feature requests, investigations, etc.

So, the ball is on my side.  I'll instrument the allocator and try to
get some numbers out of it, then I will let you know.  Thanks!

Thanks for the report!

quoted

quoted

quoted

Thank you very much!
Luigi

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

--
Kind regards,
Minchan Kim

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>