From: Dan Magenheimer <hidden> Date: 2011-07-05 15:54:33
In working on a kernel project called RAMster* (where RAM on a
remote system may be used for clean page cache pages and for swap
pages), I found I have need for a kernel socket to be used when
in non-preemptible state. I admit to being a networking idiot,
but I have been successfully using the following small patch.
I'm not sure whether I am lucky so far... perhaps more
sockets or larger/different loads will require a lot more
changes (or maybe even make my objective impossible).
So I thought I'd post it for comment. I'd appreciate
any thoughts or suggestions.
Thanks,
Dan
* http://events.linuxfoundation.org/events/linuxcon/magenheimer
diff -Napur linux-2.6.37/net/core/sock.c linux-2.6.37-ramster/net/core/sock.c
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Eric Dumazet <hidden> Date: 2011-07-05 16:30:47
Le mardi 05 juillet 2011 à 08:54 -0700, Dan Magenheimer a écrit :
quoted hunk
In working on a kernel project called RAMster* (where RAM on a
remote system may be used for clean page cache pages and for swap
pages), I found I have need for a kernel socket to be used when
in non-preemptible state. I admit to being a networking idiot,
but I have been successfully using the following small patch.
I'm not sure whether I am lucky so far... perhaps more
sockets or larger/different loads will require a lot more
changes (or maybe even make my objective impossible).
So I thought I'd post it for comment. I'd appreciate
any thoughts or suggestions.
Thanks,
Dan
* http://events.linuxfoundation.org/events/linuxcon/magenheimer
diff -Napur linux-2.6.37/net/core/sock.c linux-2.6.37-ramster/net/core/sock.c
for (;;) {
prepare_to_wait_exclusive(&sk->sk_lock.wq, &wait,
@@ -1623,7 +1631,8 @@ static void __release_sock(struct sock * * This is safe to do because we've taken the backlog * queue private: */- cond_resched_softirq();+ if (preemptible())+ cond_resched_softirq(); skb = next; } while (skb != NULL);
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
-----Original Message-----
From: netdev-owner@vger.kernel.org [mailto:netdev-
owner@vger.kernel.org] On Behalf Of Dan Magenheimer
Sent: July 05, 2011 11:54 AM
To: netdev@vger.kernel.org
Cc: Konrad Wilk; linux-mm
Subject: [RFC] non-preemptible kernel socket for RAMster
In working on a kernel project called RAMster* (where RAM on a
remote system may be used for clean page cache pages and for swap
pages), I found I have need for a kernel socket to be used when
How is RAMster+swap different than NBD's (pending etc?)support for SWAP
over NBD?
Chetan Loke
From: Dan Magenheimer <hidden> Date: 2011-07-05 17:25:35
From: Eric Dumazet [mailto:eric.dumazet@gmail.com]
Sent: Tuesday, July 05, 2011 10:31 AM
To: Dan Magenheimer
Cc: netdev@vger.kernel.org; Konrad Wilk; linux-mm
Subject: Re: [RFC] non-preemptible kernel socket for RAMster
Le mardi 05 juillet 2011 à 08:54 -0700, Dan Magenheimer a écrit :
quoted
In working on a kernel project called RAMster* (where RAM on a
remote system may be used for clean page cache pages and for swap
pages), I found I have need for a kernel socket to be used when
in non-preemptible state. I admit to being a networking idiot,
but I have been successfully using the following small patch.
I'm not sure whether I am lucky so far... perhaps more
sockets or larger/different loads will require a lot more
changes (or maybe even make my objective impossible).
So I thought I'd post it for comment. I'd appreciate
any thoughts or suggestions.
Thanks,
Dan
* http://events.linuxfoundation.org/events/linuxcon/magenheimer
diff -Napur linux-2.6.37/net/core/sock.c linux-2.6.37-ramster/net/core/sock.c
Hi Eric --
Thanks for the reply!
I hadn't tested UP in awhile so am testing now, and it seems to
work OK so far. However, I am just testing my socket, *not* testing
sockets in general. Are you implying that this patch will
break (kernel) sockets in general on a UP machine? If so,
could you be more specific as to why? (Again, I said
I am a networking idiot. ;-) I played a bit with adding
a new SOCK_ flag and triggering off of that, but this
version of the patch seemed much simpler.
Thanks,
Dan
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Dan Magenheimer <hidden> Date: 2011-07-05 17:25:45
From: Loke, Chetan [mailto:Chetan.Loke@netscout.com]
Sent: Tuesday, July 05, 2011 10:37 AM
To: Dan Magenheimer; netdev@vger.kernel.org
Cc: Konrad Wilk; linux-mm
Subject: RE: [RFC] non-preemptible kernel socket for RAMster
quoted
In working on a kernel project called RAMster* (where RAM on a
remote system may be used for clean page cache pages and for swap
pages), I found I have need for a kernel socket to be used when
How is RAMster+swap different than NBD's (pending etc?)support for SWAP
over NBD?
Hi Chetan --
Thanks for your question.
I may be ignorant of details about NBD, but did some quick
research using google. If I understand correctly, swap over
NBD is still writing to a configured swap disk on the remote
machine. RAMster is swapping to *RAM* on the remote machine.
The idea is that most machines are very overprovisioned in
RAM, and are rarely using all of their RAM, especially when
a machine is (mostly) idle. In other words, the "max of
the sums" of RAM usage on a group of machines is much lower
than the "sum of the max" of RAM usage.
So if the network is sufficiently faster than disk for
moving a page of data, RAMster provides a significant
performance improvement. OR RAMster may allow a significant
reduction in the total amount of RAM across a data center.
The version of RAMster I am working on now is really
a proof-of-concept that works over sockets, using the
ocfs2 cluster layer. One can easily envision a future
"exo-fabric" which allows one machine to write to the
RAM of another machine... for this future hardware,
RAMster becomes much more interesting.
Thanks,
Dan
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
-----Original Message-----
From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
Sent: July 05, 2011 1:25 PM
To: Loke, Chetan; netdev@vger.kernel.org
Cc: Konrad Wilk; linux-mm
Subject: RE: [RFC] non-preemptible kernel socket for RAMster
quoted
From: Loke, Chetan [mailto:Chetan.Loke@netscout.com]
Sent: Tuesday, July 05, 2011 10:37 AM
To: Dan Magenheimer; netdev@vger.kernel.org
Cc: Konrad Wilk; linux-mm
Subject: RE: [RFC] non-preemptible kernel socket for RAMster
quoted
In working on a kernel project called RAMster* (where RAM on a
remote system may be used for clean page cache pages and for swap
pages), I found I have need for a kernel socket to be used when
How is RAMster+swap different than NBD's (pending etc?)support for
SWAP
quoted
over NBD?
Hi Chetan --
Thanks for your question.
I may be ignorant of details about NBD, but did some quick
research using google. If I understand correctly, swap over
NBD is still writing to a configured swap disk on the remote
Hi - I thought NBD-server needs a backing store(a file).
Now the file itself could reside on a RAM-drive or disk-drive etc.
And so a remote NBD(disk or RAM) can be mounted locally as a swap
device.
The local client should still see it as a block device.
I haven't used the RAM-drive feature myself but you may want to check if
it
works or even borrow that logic in your code.
machine. RAMster is swapping to *RAM* on the remote machine.
The idea is that most machines are very overprovisioned in
RAM, and are rarely using all of their RAM, especially when
a machine is (mostly) idle. In other words, the "max of
the sums" of RAM usage on a group of machines is much lower
than the "sum of the max" of RAM usage.
So if the network is sufficiently faster than disk for
moving a page of data, RAMster provides a significant
performance improvement. OR RAMster may allow a significant
reduction in the total amount of RAM across a data center.
The version of RAMster I am working on now is really
a proof-of-concept that works over sockets, using the
ocfs2 cluster layer. One can easily envision a future
"exo-fabric" which allows one machine to write to the
RAM of another machine... for this future hardware,
RAMster becomes much more interesting.
Or you can also try scst-in-RAM mode(if you want to experiment with
different fabrics).
From: Eric Dumazet <hidden> Date: 2011-07-05 18:24:08
Le mardi 05 juillet 2011 à 10:25 -0700, Dan Magenheimer a écrit :
quoted
From: Eric Dumazet [mailto:eric.dumazet@gmail.com]
Sent: Tuesday, July 05, 2011 10:31 AM
To: Dan Magenheimer
Cc: netdev@vger.kernel.org; Konrad Wilk; linux-mm
Subject: Re: [RFC] non-preemptible kernel socket for RAMster
Le mardi 05 juillet 2011 à 08:54 -0700, Dan Magenheimer a écrit :
quoted
In working on a kernel project called RAMster* (where RAM on a
remote system may be used for clean page cache pages and for swap
pages), I found I have need for a kernel socket to be used when
in non-preemptible state. I admit to being a networking idiot,
but I have been successfully using the following small patch.
I'm not sure whether I am lucky so far... perhaps more
sockets or larger/different loads will require a lot more
changes (or maybe even make my objective impossible).
So I thought I'd post it for comment. I'd appreciate
any thoughts or suggestions.
Thanks,
Dan
* http://events.linuxfoundation.org/events/linuxcon/magenheimer
diff -Napur linux-2.6.37/net/core/sock.c linux-2.6.37-ramster/net/core/sock.c
Hi Eric --
Thanks for the reply!
I hadn't tested UP in awhile so am testing now, and it seems to
work OK so far. However, I am just testing my socket, *not* testing
sockets in general. Are you implying that this patch will
break (kernel) sockets in general on a UP machine? If so,
could you be more specific as to why? (Again, I said
I am a networking idiot. ;-) I played a bit with adding
a new SOCK_ flag and triggering off of that, but this
version of the patch seemed much simpler.
Say you have two processes and socket S
One process locks socket S, and is preempted by another process.
This second process is non preemptible and try to lock same socket.
-> deadlock, since P1 never releases socket S
Hi Eric --
Thanks for the reply!
I hadn't tested UP in awhile so am testing now, and it seems to
work OK so far. However, I am just testing my socket, *not* testing
sockets in general. Are you implying that this patch will
break (kernel) sockets in general on a UP machine? If so,
could you be more specific as to why? (Again, I said
I am a networking idiot. ;-) I played a bit with adding
a new SOCK_ flag and triggering off of that, but this
version of the patch seemed much simpler.
Say you have two processes and socket S
One process locks socket S, and is preempted by another process.
This second process is non preemptible and try to lock same socket.
-> deadlock, since P1 never releases socket S
Oh, OK. My use model is that a socket that is used non-preemptible
must always be used non-preemptible. In other words, this kind
of socket is an extreme form of non-blocking. Doesn't that seem
like a reasonable constraint?
Thanks,
Dan
From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
Subject: RE: [RFC] non-preemptible kernel socket for RAMster
quoted
From: Loke, Chetan [mailto:Chetan.Loke@netscout.com]
Sent: Tuesday, July 05, 2011 10:37 AM
To: Dan Magenheimer; netdev@vger.kernel.org
Cc: Konrad Wilk; linux-mm
Subject: RE: [RFC] non-preemptible kernel socket for RAMster
quoted
In working on a kernel project called RAMster* (where RAM on a
remote system may be used for clean page cache pages and for swap
pages), I found I have need for a kernel socket to be used when
How is RAMster+swap different than NBD's (pending etc?)support for
SWAP over NBD?
I may be ignorant of details about NBD, but did some quick
research using google. If I understand correctly, swap over
NBD is still writing to a configured swap disk on the remote
Hi - I thought NBD-server needs a backing store(a file).
Now the file itself could reside on a RAM-drive or disk-drive etc.
And so a remote NBD(disk or RAM) can be mounted locally as a swap
device.
The local client should still see it as a block device.
I haven't used the RAM-drive feature myself but you may want to check if
it
works or even borrow that logic in your code.
Actually, RAMster is using a much more flexible type of
RAM-drive; it is built on top of Transcendent Memory
and on top of zcache (and thus on top of cleancache and
frontswap). A RAM-drive is fixed size so is not very suitable
for the flexibility required for RAMster. For example,
suppose you have two machines A and B. At one point in
time A is overcommitted and needs to swap and B is relatively
idle. Then later, B is overcommitted and needs to swap and
A is relatively idle. RAMster can handle this entirely
dynamically, a RAM-drive cannot.
quoted
machine. RAMster is swapping to *RAM* on the remote machine.
The idea is that most machines are very overprovisioned in
RAM, and are rarely using all of their RAM, especially when
a machine is (mostly) idle. In other words, the "max of
the sums" of RAM usage on a group of machines is much lower
than the "sum of the max" of RAM usage.
So if the network is sufficiently faster than disk for
moving a page of data, RAMster provides a significant
performance improvement. OR RAMster may allow a significant
reduction in the total amount of RAM across a data center.
The version of RAMster I am working on now is really
a proof-of-concept that works over sockets, using the
ocfs2 cluster layer. One can easily envision a future
"exo-fabric" which allows one machine to write to the
RAM of another machine... for this future hardware,
RAMster becomes much more interesting.
Or you can also try scst-in-RAM mode(if you want to experiment with
different fabrics).
Thanks. Could you provide a pointer for this? I found
the SCST sourceforge page but no obvious references to
scst-in-ram-mode. (But also, since it appears to be
SCSI-related, I wonder if it also assumes a fixed size
target device, RAM or disk or ??)
Dan
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
-----Original Message-----
From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
Sent: July 05, 2011 3:19 PM
To: Loke, Chetan; netdev@vger.kernel.org
Cc: Konrad Wilk; linux-mm
Subject: RE: [RFC] non-preemptible kernel socket for RAMster
Actually, RAMster is using a much more flexible type of
RAM-drive; it is built on top of Transcendent Memory
and on top of zcache (and thus on top of cleancache and
frontswap). A RAM-drive is fixed size so is not very suitable
for the flexibility required for RAMster. For example,
suppose you have two machines A and B. At one point in
time A is overcommitted and needs to swap and B is relatively
idle. Then later, B is overcommitted and needs to swap and
A is relatively idle. RAMster can handle this entirely
dynamically, a RAM-drive cannot.
Again, iff NBD works with a ram-drive then you really wouldn't need to
do anything. How often are you going to re-size your remote-SWAP? Plus,
you can make nbd-server listen on multiple ports - Google(Linux NBD)
returned: http://www.fi.muni.cz/~kripac/orac-nbd/ . Look at the
nbd-server code to see if it launches multiple kernel-threads for
servicing different ports. If not, one can enhance it and scale that way
too. But nbd-server today can service multiple-ports(that is effectively
servicing multiple clients). So why not add NBD-filesystem-filters to
make it point to local/remote swap?
Thanks. Could you provide a pointer for this? I found
the SCST sourceforge page but no obvious references to
scst-in-ram-mode. (But also, since it appears to be
SCSI-related, I wonder if it also assumes a fixed size
target device, RAM or disk or ??)
Yes, it is SCSI. You should be looking for SCST I/O modes. Read some
docs and then send an email to the scst-mailing-list. If you speak about
block-IO-performance then FC(in its class of price/performance factor)
is more than capable of handling any workload. FC is a protocol designed
for storage. No exotic fabric other than FC is needed.
Folks who start with ethernet for block-IO, always start with bare
minimal code and then for squeezing block-IO performance(aka version 2
of the product), keep hacking repeatedly or go for a link-speed upgrade.
Start with FC, period.
From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
quoted
Actually, RAMster is using a much more flexible type of
RAM-drive; it is built on top of Transcendent Memory
and on top of zcache (and thus on top of cleancache and
frontswap). A RAM-drive is fixed size so is not very suitable
for the flexibility required for RAMster. For example,
suppose you have two machines A and B. At one point in
time A is overcommitted and needs to swap and B is relatively
idle. Then later, B is overcommitted and needs to swap and
A is relatively idle. RAMster can handle this entirely
dynamically, a RAM-drive cannot.
Again, iff NBD works with a ram-drive then you really wouldn't need to
do anything. How often are you going to re-size your remote-SWAP? Plus,
you can make nbd-server listen on multiple ports - Google(Linux NBD)
returned: http://www.fi.muni.cz/~kripac/orac-nbd/ . Look at the
nbd-server code to see if it launches multiple kernel-threads for
servicing different ports. If not, one can enhance it and scale that way
too. But nbd-server today can service multiple-ports(that is effectively
servicing multiple clients). So why not add NBD-filesystem-filters to
make it point to local/remote swap?
Well, we may be talking past each other, but the RAMster answer to:
How often are you going to re-size your remote-SWAP?
is "as often as the working set changes on any machine in the
cluster", meaning *constantly*, entirely dynamically! How
about a more specific example: Suppose you have 2 machines,
each with 8GB of memory. 99% of the time each machine is
chugging along just fine and doesn't really need more than 4GB,
and may even use less than 1GB a large part of the time.
But very now and then, one of the machines randomly needs
9GB, 10GB, maybe even 12GB of memory. This would normally
result in swapping. (Most system administrators won't even
have this much information... they'll just know they are
seeing swapping and decide they need to buy more RAM.)
With NBD to a ram-drive, each machine would need to pre-allocate
4GB of RAM for the RAM-drive, leaving only 4GB of RAM for
the "local" RAM. The result will actually be MORE swapping
because a fixed amount of RAM has been pre-reserved for the
other machine's swap. With RAMster, everything is done dynamically,
so all that matters is the maximum of the sum of the RAM
used. You may even be able to *remove* ~2GB of RAM from each
of the systems and still never see any swapping to disk.
quoted
Thanks. Could you provide a pointer for this? I found
the SCST sourceforge page but no obvious references to
scst-in-ram-mode. (But also, since it appears to be
SCSI-related, I wonder if it also assumes a fixed size
target device, RAM or disk or ??)
Yes, it is SCSI. You should be looking for SCST I/O modes. Read some
docs and then send an email to the scst-mailing-list. If you speak about
block-IO-performance then FC(in its class of price/performance factor)
is more than capable of handling any workload. FC is a protocol designed
for storage. No exotic fabric other than FC is needed.
Folks who start with ethernet for block-IO, always start with bare
minimal code and then for squeezing block-IO performance(aka version 2
of the product), keep hacking repeatedly or go for a link-speed upgrade.
Start with FC, period.
My point was that block I/O devices (AFAIK) always present a fixed
"size" to the kernel, and if this is also true of scst-in-ram-mode,
the same problem as swap-over-NBD occurs... it's not dynamic.
RAMster does not present a block-I/O storage-like interface;
it's using the Transcendent Memory interface, which is designed
for "slow RAM" of an unknown-and-dynamic size.
I'm not a storage expert either, but I do wonder if "no exotic
fabric other than FC" isn't an oxymoron ;-) FC is certainly
too exotic for me.
Dan
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
quoted
How often are you going to re-size your remote-SWAP?
is "as often as the working set changes on any machine in the
cluster", meaning *constantly*, entirely dynamically! How
about a more specific example: Suppose you have 2 machines,
each with 8GB of memory. 99% of the time each machine is
chugging along just fine and doesn't really need more than 4GB,
and may even use less than 1GB a large part of the time.
But very now and then, one of the machines randomly needs
9GB, 10GB, maybe even 12GB of memory. This would normally
result in swapping. (Most system administrators won't even
have this much information... they'll just know they are
seeing swapping and decide they need to buy more RAM.)
Ok, I understand there is interest in implementing
'remote-volatile-ballooning-variant' but how do you pick a remote
candidate(hypervisor)? Let's say, memory could be available on remote
system but what if the remote-p{NIC,CPU} is overloaded? Sure, sysadmins
won't have this info because this so dynamic(and it's quite possible as
you mentioned above). But does the trans-remote-API know about this
resource-availability before opening a remote-channel?
Stressing the remote-p{NIC/CPU} might trick hypervisor-vmotion-plugin to
vmotion VM[s] to another hypervisor. How is trans-remote-API integrating
with remote/global vmotion policies to avoid this false vmotion?
From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
quoted
How often are you going to re-size your remote-SWAP?
is "as often as the working set changes on any machine in the
cluster", meaning *constantly*, entirely dynamically! How
about a more specific example: Suppose you have 2 machines,
each with 8GB of memory. 99% of the time each machine is
chugging along just fine and doesn't really need more than 4GB,
and may even use less than 1GB a large part of the time.
But very now and then, one of the machines randomly needs
9GB, 10GB, maybe even 12GB of memory. This would normally
result in swapping. (Most system administrators won't even
have this much information... they'll just know they are
seeing swapping and decide they need to buy more RAM.)
Ok, I understand there is interest in implementing
'remote-volatile-ballooning-variant' but how do you pick a remote
candidate(hypervisor)? Let's say, memory could be available on remote
system but what if the remote-p{NIC,CPU} is overloaded? Sure, sysadmins
won't have this info because this so dynamic(and it's quite possible as
you mentioned above). But does the trans-remote-API know about this
resource-availability before opening a remote-channel?
Stressing the remote-p{NIC/CPU} might trick hypervisor-vmotion-plugin to
vmotion VM[s] to another hypervisor. How is trans-remote-API integrating
with remote/global vmotion policies to avoid this false vmotion?
Hi Chetan --
Thanks for the continued discussion.
First, let me clarify that RAMster does not depend on virtualization.
At some time in the future, it may be a nice addition for KVM*,
but the version I am developing currently only works on a
cluster of physical machines. So vmotion/migration is not
an issue right now
As for choosing the remote machine, another key feature of
the Transcendent Memory mechanism is that any and every page
can be rejected. If rejected, the page remains local. In
essence, on *every* page-to-be-swapped, machine A *asks*
machine B, "can you take this page"? If the answer is no,
machine A can choose another machine (C), or may choose to
swap the page to its own slow swap disk. (Currently,
only the latter is implemented, but more complicated
policy could certainly be implemented.)
Dan
* Xen doesn't have drivers so RAMster-over-network is not an option
for Xen. A future RAMster-over-exofabric might work with Xen though.)
And, by the way, the Transcendent Memory implementation on Xen
does handle vmotion/migration so it is a solvable problem.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>