zcache/ramster rewrite and promotion

Here finally is the long promised rewrite of zcache (and ramster).

I know that we are concentrating on moving zcache from staging,
and not ramster. However the amount of duplicate code that ramster
used from zcache is astonishing so when I did the rewrite I thought
why not kill two birds with one stone - since both are in the
staging directory.

Of notable interest to the broader mm community, I am proposing, when review
is complete, to place zcache in a new subdirectory of mm, called "tmem"
(short for transcendent memory).  Zcache is truly memory management,
not a hardware driver, and it interfaces with mm/swap/vfs through mm/cleancache.c
and mm/frontswap.c (which possibly should move to the new tmem directory
in the future as well).

This is a major rewrite for zcache, not a sequence of small patches.
So those who are interested in understanding, reviewing, and commenting
in detail on the design and the functioning of the code can find it at:
 
 git://oss.oracle.com/git/djm/tmem.git #zcache-120731

For those who prefer to review and comment line-by-line, it's not clear
yet how best to post the ~10K lines of code to ensure reviewer productivity.
Konrad suggested an IRC talk on Monday to talk about this so we can figure
out what is the proper option.

(If you are not familiar with the tmem terminology, you can review
it here: http://lwn.net/Articles/454795/ )
 
Some of the highlights of this git branch:
 1. Merge of zcache and ramster.  Zcache and ramster had a great deal of
    duplicate code which is now merged.  In essence, zcache *is* ramster
    but with no remote machine available, but !CONFIG_RAMSTER will avoid
    compiling lots of ramster-specific code.
 2. Allocator.  Previously, persistent pools used zsmalloc and ephemeral pools
    used zbud.  Now a completely rewritten zbud is used for both.  Notably
    this zbud maintains all persistent (frontswap) and ephemeral (cleancache)
    pageframes in separate queues in LRU order.
 3. Interaction with page allocator.  Zbud does no page allocation/freeing,
    it is done entirely in zcache where it can be tracked more effectively.

83 more lines

On Tue, Jul 31, 2012 at 11:18 PM, Dan Magenheimer
[off-list ref] wrote:

diffstat vs 3.5:
 drivers/staging/ramster/Kconfig       |    2
 drivers/staging/ramster/Makefile      |    2
 drivers/staging/zcache/Kconfig        |    2
 drivers/staging/zcache/Makefile       |    2
 mm/Kconfig                            |    2
 mm/Makefile                           |    4
 mm/tmem/Kconfig                       |   33
 mm/tmem/Makefile                      |    5
 mm/tmem/tmem.c                        |  894 +++++++++++++
 mm/tmem/tmem.h                        |  259 +++
 mm/tmem/zbud.c                        | 1060 +++++++++++++++
 mm/tmem/zbud.h                        |   33
 mm/tmem/zcache-main.c                 | 1686 +++++++++++++++++++++++++
 mm/tmem/zcache.h                      |   53
 mm/tmem/ramster.h                     |   59
 mm/tmem/ramster/heartbeat.c           |  462 ++++++
 mm/tmem/ramster/heartbeat.h           |   87 +
 mm/tmem/ramster/masklog.c             |  155 ++
 mm/tmem/ramster/masklog.h             |  220 +++
 mm/tmem/ramster/nodemanager.c         |  995 +++++++++++++++
 mm/tmem/ramster/nodemanager.h         |   88 +
 mm/tmem/ramster/r2net.c               |  414 ++++++
 mm/tmem/ramster/ramster.c             |  985 ++++++++++++++
 mm/tmem/ramster/ramster.h             |  161 ++
 mm/tmem/ramster/ramster_nodemanager.h |   39
 mm/tmem/ramster/tcp.c                 | 2253 ++++++++++++++++++++++++++++++++++
 mm/tmem/ramster/tcp.h                 |  159 ++
 mm/tmem/ramster/tcp_internal.h        |  248 +++
28 files changed, 10358 insertions(+), 4 deletions(-)

So it's basically this commit, right?

https://oss.oracle.com/git/djm/tmem.git/?p=djm/tmem.git;a=commitdiff;h=22844fe3f52d912247212408294be330a867937c

Why on earth would you want to move that under the mm directory?

                        Pekka

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

On Tue, Jul 31, 2012 at 11:53:57PM +0300, Pekka Enberg wrote:

On Tue, Jul 31, 2012 at 11:18 PM, Dan Magenheimer
[off-list ref] wrote:

quoted

diffstat vs 3.5:
 drivers/staging/ramster/Kconfig       |    2
 drivers/staging/ramster/Makefile      |    2
 drivers/staging/zcache/Kconfig        |    2
 drivers/staging/zcache/Makefile       |    2
 mm/Kconfig                            |    2
 mm/Makefile                           |    4
 mm/tmem/Kconfig                       |   33
 mm/tmem/Makefile                      |    5
 mm/tmem/tmem.c                        |  894 +++++++++++++
 mm/tmem/tmem.h                        |  259 +++
 mm/tmem/zbud.c                        | 1060 +++++++++++++++
 mm/tmem/zbud.h                        |   33
 mm/tmem/zcache-main.c                 | 1686 +++++++++++++++++++++++++
 mm/tmem/zcache.h                      |   53
 mm/tmem/ramster.h                     |   59
 mm/tmem/ramster/heartbeat.c           |  462 ++++++
 mm/tmem/ramster/heartbeat.h           |   87 +
 mm/tmem/ramster/masklog.c             |  155 ++
 mm/tmem/ramster/masklog.h             |  220 +++
 mm/tmem/ramster/nodemanager.c         |  995 +++++++++++++++
 mm/tmem/ramster/nodemanager.h         |   88 +
 mm/tmem/ramster/r2net.c               |  414 ++++++
 mm/tmem/ramster/ramster.c             |  985 ++++++++++++++
 mm/tmem/ramster/ramster.h             |  161 ++
 mm/tmem/ramster/ramster_nodemanager.h |   39
 mm/tmem/ramster/tcp.c                 | 2253 ++++++++++++++++++++++++++++++++++
 mm/tmem/ramster/tcp.h                 |  159 ++
 mm/tmem/ramster/tcp_internal.h        |  248 +++
28 files changed, 10358 insertions(+), 4 deletions(-)

So it's basically this commit, right?

Yeah, one big RFC patch.

https://oss.oracle.com/git/djm/tmem.git/?p=djm/tmem.git;a=commitdiff;h=22844fe3f52d912247212408294be330a867937c

Why on earth would you want to move that under the mm directory?

If you take aside that problem that it is one big patch instead
of being split up in more reasonable pieces - would you recommend
that it reside in a different directory?

Or is that it does not make sense b/c it has other components in it - such
as tcp/nodemaneger/hearbeat/etc so it should go under the refactor knife?

And if you rip out the ramster from this and just concentrate on zcache -
should that go in drivers/mm or mm/tmem/zcache?

                        Pekka

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

From: Pekka Enberg [mailto:penberg@kernel.org]
Sent: Tuesday, July 31, 2012 2:54 PM

On Tue, Jul 31, 2012 at 11:18 PM, Dan Magenheimer
[off-list ref] wrote:

quoted

diffstat vs 3.5:
 drivers/staging/ramster/Kconfig       |    2
 drivers/staging/ramster/Makefile      |    2
 drivers/staging/zcache/Kconfig        |    2
 drivers/staging/zcache/Makefile       |    2
 mm/Kconfig                            |    2
 mm/Makefile                           |    4
 mm/tmem/Kconfig                       |   33
 mm/tmem/Makefile                      |    5
 mm/tmem/tmem.c                        |  894 +++++++++++++
 mm/tmem/tmem.h                        |  259 +++
 mm/tmem/zbud.c                        | 1060 +++++++++++++++
 mm/tmem/zbud.h                        |   33
 mm/tmem/zcache-main.c                 | 1686 +++++++++++++++++++++++++
 mm/tmem/zcache.h                      |   53
 mm/tmem/ramster.h                     |   59
 mm/tmem/ramster/heartbeat.c           |  462 ++++++
 mm/tmem/ramster/heartbeat.h           |   87 +
 mm/tmem/ramster/masklog.c             |  155 ++
 mm/tmem/ramster/masklog.h             |  220 +++
 mm/tmem/ramster/nodemanager.c         |  995 +++++++++++++++
 mm/tmem/ramster/nodemanager.h         |   88 +
 mm/tmem/ramster/r2net.c               |  414 ++++++
 mm/tmem/ramster/ramster.c             |  985 ++++++++++++++
 mm/tmem/ramster/ramster.h             |  161 ++
 mm/tmem/ramster/ramster_nodemanager.h |   39
 mm/tmem/ramster/tcp.c                 | 2253 ++++++++++++++++++++++++++++++++++
 mm/tmem/ramster/tcp.h                 |  159 ++
 mm/tmem/ramster/tcp_internal.h        |  248 +++
28 files changed, 10358 insertions(+), 4 deletions(-)

So it's basically this commit, right?

https://oss.oracle.com/git/djm/tmem.git/?p=djm/tmem.git;a=commitdiff;h=22844fe3f52d912247212408294be33
0a867937c

2 more quoted lines

Hi Pekka --

Thanks for your reply and question.

MM means "memory management" and zcache manages physical memory
to allow more pages of data to be stored in RAM.  So it seems a
logical place.  It's not a block driver, or a network driver,
or a device driver, or a filesystem... do you have a different
location in the kernel in mind?

Zcache does it a bit differently than all the other parts of mm
because it needs to; because all the other parts of mm try to
maximize the amount of physical memory that is directly addressable
by threads but one can't directly address pages that have been compressed.
So zcache uses the transcendent memory approach (via cleancache
and frontswap) to compress/decompress clean pagecache pages and
swap pages "on demand".  The tmem design also nicely handles
both the fact that the degree of compression is unpredictable and
the fact that the fraction of fixed total RAM used for compressed
pages vs "normal uncompressed mm" pages needs to be very dynamic.

Ramster does the same thing but manages it peer-to-peer across
multiple systems using kernel sockets.  One could argue that
the dependency on sockets makes it more of a driver than "mm"
but ramster is "memory management" too, just a bit more exotic.

Thanks,
Dan

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Hi Konrad,

On Tue, Jul 31, 2012 at 11:53:57PM +0300, Pekka Enberg wrote:

quoted

Why on earth would you want to move that under the mm directory?

On Wed, Aug 1, 2012 at 12:04 AM, Konrad Rzeszutek Wilk
[off-list ref] wrote:

If you take aside that problem that it is one big patch instead
of being split up in more reasonable pieces - would you recommend
that it reside in a different directory?

Or is that it does not make sense b/c it has other components in it - such
as tcp/nodemaneger/hearbeat/etc so it should go under the refactor knife?

And if you rip out the ramster from this and just concentrate on zcache -
should that go in drivers/mm or mm/tmem/zcache?

I definitely think mm/zcache.c makes sense. I hate the fact that it's
now riddled with references to "tmem" and "ramster" but that's probably
fixable. I also hate the fact that you've now gone and rewritten
everything so we lose all the change history zcache has had under
staging.

As for ramster, it might make sense to have its core in mm/ramster.c and
move the TCP weirdness somewhere else. The exact location depends on
what kind of userspace ABIs you expose, I suppose. I mean, surely you
need to configure the thing somehow?

                        Pekka

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Hi Dan,

On Wed, Aug 1, 2012 at 12:13 AM, Dan Magenheimer
[off-list ref] wrote:

Ramster does the same thing but manages it peer-to-peer across
multiple systems using kernel sockets.  One could argue that
the dependency on sockets makes it more of a driver than "mm"
but ramster is "memory management" too, just a bit more exotic.

How do you configure it? Can we move parts of the network protocol under
net/ramster or something?

                        Pekka

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

From: Pekka Enberg [mailto:penberg@kernel.org]
Subject: Re: [RFC/PATCH] zcache/ramster rewrite and promotion

Hi Dan,

On Wed, Aug 1, 2012 at 12:13 AM, Dan Magenheimer
[off-list ref] wrote:

quoted

Ramster does the same thing but manages it peer-to-peer across
multiple systems using kernel sockets.  One could argue that
the dependency on sockets makes it more of a driver than "mm"
but ramster is "memory management" too, just a bit more exotic.

How do you configure it?

Hi Pekka --

It looks like the build/configuration how-to at
https://oss.oracle.com/projects/tmem/dist/files/RAMster/HOWTO-v5-120214 
is out-of-date and I need to fix some things in it.  I'll post
a link to it after I update it.

Can we move parts of the network protocol under
net/ramster or something?

Ramster is built on top of kernel sockets.  Both that networking
part and the configuration part of the ramster code are heavily
leveraged from ocfs2 and I suspect there is a lot of similarity
to gfs code as well.  In the code for both of those filesystems
I think the network and configuration code lives in the same
directory with the file system, so that was the model I was following.

I'm OK with placing it wherever kernel developers want to put
it, as long as the reason is not NIMBY-ness. [1]  My preference
is to keep all the parts together, at least for the review phase,
but if there is a consensus that it belongs someplace else,
I will be happy to move it.

Dan

[1] http://en.wikipedia.org/wiki/NIMBY

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

On Mon, Aug 6, 2012 at 5:07 PM, Dan Magenheimer
[off-list ref] wrote:

I'm OK with placing it wherever kernel developers want to put
it, as long as the reason is not NIMBY-ness. [1]  My preference
is to keep all the parts together, at least for the review phase,
but if there is a consensus that it belongs someplace else,
I will be happy to move it.

I'd go for core code in mm/zcache.c and mm/ramster.c, and move the
clustering code under net/ramster or drivers/ramster.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

From: Pekka Enberg [mailto:penberg@kernel.org]
Subject: Re: [RFC/PATCH] zcache/ramster rewrite and promotion

On Mon, Aug 6, 2012 at 5:07 PM, Dan Magenheimer
[off-list ref] wrote:

quoted

I'm OK with placing it wherever kernel developers want to put
it, as long as the reason is not NIMBY-ness. [1]  My preference
is to keep all the parts together, at least for the review phase,
but if there is a consensus that it belongs someplace else,
I will be happy to move it.

I'd go for core code in mm/zcache.c and mm/ramster.c, and move the
clustering code under net/ramster or drivers/ramster.

Hi Pekka --

Thanks for the quick feedback!

Hmmm.. there's also zbud.c and tmem.c which are critical components
of both zcache and ramster.  And there are header files as well which
will need to either be in mm/ or somewhere in include/linux/

Is there a reason or rule that mm/ can't have subdirectories?

Since zcache has at least three .c files plus ramster.c, and
since mm/frontswap.c and mm/cleancache.c are the foundation on
which all of these are built, I was thinking grouping all six
(plus headers) in the same mm/tmem/ subdirectory was a good
way to keep mm/ from continuing to get more cluttered... not counting
new zcache and ramster files, there are now 74 .c files in mm/!
(Personally, I think a directory has too many files in it if
"ls" doesn't fit in a 25x80 window.)

Thoughts?
Dan

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

On Mon, Aug 6, 2012 at 7:10 PM, Dan Magenheimer
[off-list ref] wrote:

Hmmm.. there's also zbud.c and tmem.c which are critical components
of both zcache and ramster.  And there are header files as well which
will need to either be in mm/ or somewhere in include/linux/

Is there a reason or rule that mm/ can't have subdirectories?

Since zcache has at least three .c files plus ramster.c, and
since mm/frontswap.c and mm/cleancache.c are the foundation on
which all of these are built, I was thinking grouping all six
(plus headers) in the same mm/tmem/ subdirectory was a good
way to keep mm/ from continuing to get more cluttered... not counting
new zcache and ramster files, there are now 74 .c files in mm/!
(Personally, I think a directory has too many files in it if
"ls" doesn't fit in a 25x80 window.)

Thoughts?

There's no reason we can't have subdirectories. That said, I really
don't see the point of having a separate directory called 'tmem'. It
might make sense to have mm/zcache and/or mm/ramster but I suspect
you can just fold the core code in mm/zcache.c and mm/ramster.c by
slimming down the weird Solaris-like 'tmem' abstractions.

                        Pekka

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

From: Pekka Enberg [mailto:penberg@kernel.org]
Subject: Re: [RFC/PATCH] zcache/ramster rewrite and promotion

On Mon, Aug 6, 2012 at 7:10 PM, Dan Magenheimer
[off-list ref] wrote:

quoted

Hmmm.. there's also zbud.c and tmem.c which are critical components
of both zcache and ramster.  And there are header files as well which
will need to either be in mm/ or somewhere in include/linux/

Is there a reason or rule that mm/ can't have subdirectories?

Since zcache has at least three .c files plus ramster.c, and
since mm/frontswap.c and mm/cleancache.c are the foundation on
which all of these are built, I was thinking grouping all six
(plus headers) in the same mm/tmem/ subdirectory was a good
way to keep mm/ from continuing to get more cluttered... not counting
new zcache and ramster files, there are now 74 .c files in mm/!
(Personally, I think a directory has too many files in it if
"ls" doesn't fit in a 25x80 window.)

Thoughts?

There's no reason we can't have subdirectories. That said, I really
don't see the point of having a separate directory called 'tmem'. It
might make sense to have mm/zcache and/or mm/ramster but I suspect
you can just fold the core code in mm/zcache.c and mm/ramster.c by
slimming down the weird Solaris-like 'tmem' abstractions.

I'm not sure I understand... what is Solaris-like about tmem?
And what would you slim down?

While I agree one can often glom three separate 1000-line .c files
into a single 3000-line .c file, I recently spent some time moving
the other direction to, I thought, improve readability.  Do kernel
developers have a preference for huge .c files rather than smaller
logically-separated moderate-sized files in a subdirectory?

Thanks,
Dan

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>