From: Alexander Schreiber <hidden> Date: 2011-01-29 21:08:15
On Sat, Jan 29, 2011 at 12:23:14PM -0200, Denis wrote:
2011/1/29 Alexander Schreiber [off-list ref]
quoted
plain disk performance for writes, while reads should be reasonably
close to the plain disk performance - drbd optimizes reads by just reading
from the local disk if it can.
However, I have not used it with active-active fashion. Have you? if yes,
what is your overall experience?
We are using drbd to provide mirrored disks for virtual machines running
under Xen. 99% of the time, the drbd devices run in primary/secondary
mode (aka active/passive), but they are switched to primary/primary
(aka active/active) for live migrations of domains, as that needs the
disks to be available on both nodes. From our experience, if the drbd
device is healthy, this is very reliable. No experience with running
drbd in primary/primary config for any extended period of time, though
(the live migrations are usually over after a few seconds to a minute at
most, then the drbd devices go back to primary/secondary).
Kind regards,
Alex.
--
"Opportunity is missed by most people because it is dressed in overalls and
looks like work." -- Thomas A. Edison
From: John Robinson <hidden> Date: 2011-01-29 21:54:55
On 29/01/2011 21:08, Alexander Schreiber wrote:
On Sat, Jan 29, 2011 at 12:23:14PM -0200, Denis wrote:
quoted
2011/1/29 Alexander Schreiber[off-list ref]
quoted
plain disk performance for writes, while reads should be reasonably
close to the plain disk performance - drbd optimizes reads by just reading
from the local disk if it can.
However, I have not used it with active-active fashion. Have you? if yes,
what is your overall experience?
We are using drbd to provide mirrored disks for virtual machines running
under Xen. 99% of the time, the drbd devices run in primary/secondary
mode (aka active/passive), but they are switched to primary/primary
(aka active/active) for live migrations of domains, as that needs the
disks to be available on both nodes. From our experience, if the drbd
device is healthy, this is very reliable. No experience with running
drbd in primary/primary config for any extended period of time, though
(the live migrations are usually over after a few seconds to a minute at
most, then the drbd devices go back to primary/secondary).
Now that is interesting, to me at least. More as a thought experiment
for now, I was wondering how one would go about setting up a small
cluster of commodity servers (maybe 8 machines) running Xen (or perhaps
now KVM) VMs, such that if one (or potentially two) of the machines
died, the VMs could be picked up by the other machines in the cluster,
and only using locally-attached SATA/SAS discs in each machine.
I guess I'm talking about RAIN or RAIS rather than RAID so maybe I'd
better start reading the Wikipedia pages on those and not talk about it
on this list...
Cheers,
John.
From: Stan Hoeppner <hidden> Date: 2011-01-29 23:04:31
John Robinson put forth on 1/29/2011 3:54 PM:
Now that is interesting, to me at least. More as a thought experiment for now, I
was wondering how one would go about setting up a small cluster of commodity
servers (maybe 8 machines) running Xen (or perhaps now KVM) VMs, such that if
one (or potentially two) of the machines died, the VMs could be picked up by the
other machines in the cluster, and only using locally-attached SATA/SAS discs in
each machine.
Doing N-way active replication with DRBD increases network utilization
substantially. With two DRBD active nodes you will have a maximum of _2_
simultaneous data streams, one in each direction. With 8 active nodes you will
have a maximum of _56_ simultaneous data streams. Your scenario requires all
nodes be active.
This may work for a hobby cluster or something with very low volume of data
being written to disk. This solution most likely won't scale for a cluster with
any amount of real traffic. GbE peaks at 100 MB/s. Therefore each node will
have only about 12 MB/s of bidirectional bandwidth for each other cluster member
if my math is correct. A single SATA disk run about 80-120 MB/s, so your
network DRBD disk bandwidth is about 1/7th to 1/10th that of a single local
disk. In a 2 node cluster it's closer to 1:1. For you scenario to actually be
feasible, you'd need at least bonded quad GbE interfaces if not single 10 GbE
interfaces to get all the bandwidth you'd need.
You'd be _MUCH_ better off using 2 active DRBD mirrored NFS servers with GFS2
filesystems and having the aforementioned 8 nodes do their data sharing via NFS.
In this setup each node only writes once (to NFS) dramatically reducing network
bandwidth required per node, with only 16 maximum data streams instead of 56.
If you need more bandwidth or IOPS than a single disk NFS server can produce,
simply RAID 4-10 disks on each NFS server via RAID 10, then mirror the two RAIDs
with DRBD.
You may need 2-4 GbE interfaces between the two NFS servers just for DRBD
traffic, but the cost of that is much less than having the same number of
interfaces in each of 8 cluster nodes. This will also give you much better
performance after a node or two fails and you have to boot their VM guests on
other hosts. Having fast central RAID storage will allow those guests to boot
much more quickly and without causing degraded performance on the other nodes
due to lack of disk bandwidth in your suggested model.
--
Stan
From: Miles Fidelman <hidden> Date: 2011-01-29 23:06:06
John Robinson wrote:
Now that is interesting, to me at least. More as a thought experiment
for now, I was wondering how one would go about setting up a small
cluster of commodity servers (maybe 8 machines) running Xen (or
perhaps now KVM) VMs, such that if one (or potentially two) of the
machines died, the VMs could be picked up by the other machines in the
cluster, and only using locally-attached SATA/SAS discs in each machine.
I do that now - albeit only on a 2-node cluster. DRBD works just fine
using locally attached drives.
--
In theory, there is no difference between theory and practice.
In<fnord> practice, there is. .... Yogi Berra
From: Alexander Schreiber <hidden> Date: 2011-01-30 01:43:58
On Sat, Jan 29, 2011 at 09:54:55PM +0000, John Robinson wrote:
On 29/01/2011 21:08, Alexander Schreiber wrote:
quoted
On Sat, Jan 29, 2011 at 12:23:14PM -0200, Denis wrote:
quoted
2011/1/29 Alexander Schreiber[off-list ref]
quoted
plain disk performance for writes, while reads should be reasonably
close to the plain disk performance - drbd optimizes reads by just reading
from the local disk if it can.
However, I have not used it with active-active fashion. Have you? if yes,
what is your overall experience?
We are using drbd to provide mirrored disks for virtual machines running
under Xen. 99% of the time, the drbd devices run in primary/secondary
mode (aka active/passive), but they are switched to primary/primary
(aka active/active) for live migrations of domains, as that needs the
disks to be available on both nodes. From our experience, if the drbd
device is healthy, this is very reliable. No experience with running
drbd in primary/primary config for any extended period of time, though
(the live migrations are usually over after a few seconds to a minute at
most, then the drbd devices go back to primary/secondary).
Now that is interesting, to me at least. More as a thought
experiment for now, I was wondering how one would go about setting
up a small cluster of commodity servers (maybe 8 machines) running
Xen (or perhaps now KVM) VMs, such that if one (or potentially two)
of the machines died, the VMs could be picked up by the other
machines in the cluster, and only using locally-attached SATA/SAS
discs in each machine.
I guess I'm talking about RAIN or RAIS rather than RAID so maybe I'd
better start reading the Wikipedia pages on those and not talk about
it on this list...
For the "survive single node total machine failure" case your problem has
already been solved: http://code.google.com/p/ganeti/
We run a large number of clusters with that and the VMs routinely survive
disk failures and recover (come back from what looks like a power failure
to the VM) from node failure.
Kind regards,
Alex.
--
"Opportunity is missed by most people because it is dressed in overalls and
looks like work." -- Thomas A. Edison
On Sat, Jan 29, 2011 at 12:23:14PM -0200, Denis wrote:
quoted
2011/1/29 Alexander Schreiber [off-list ref]
quoted
plain disk performance for writes, while reads should be reasonably
close to the plain disk performance - drbd optimizes reads by just reading
from the local disk if it can.
However, I have not used it with active-active fashion. Have you? if yes,
what is your overall experience?
We are using drbd to provide mirrored disks for virtual machines running
under Xen. 99% of the time, the drbd devices run in primary/secondary
mode (aka active/passive), but they are switched to primary/primary
(aka active/active) for live migrations of domains, as that needs the
disks to be available on both nodes. From our experience, if the drbd
device is healthy, this is very reliable. No experience with running
drbd in primary/primary config for any extended period of time, though
(the live migrations are usually over after a few seconds to a minute at
most, then the drbd devices go back to primary/secondary).
What filesystem are you using to enable the primary-primary mode? Have
you evaluated it against any other available option?
Kind regards,
Alex.
--
"Opportunity is missed by most people because it is dressed in overalls and
looks like work." -- Thomas A. Edison
cheers!
--
Denis Anjos,
www.versatushpc.com.br
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
From: Alexander Schreiber <hidden> Date: 2011-01-31 13:03:50
On Mon, Jan 31, 2011 at 06:42:44AM -0200, Denis wrote:
2011/1/29 Alexander Schreiber [off-list ref]:
quoted
On Sat, Jan 29, 2011 at 12:23:14PM -0200, Denis wrote:
quoted
2011/1/29 Alexander Schreiber [off-list ref]
quoted
plain disk performance for writes, while reads should be reasonably
close to the plain disk performance - drbd optimizes reads by just reading
from the local disk if it can.
However, I have not used it with active-active fashion. Have you? if yes,
what is your overall experience?
We are using drbd to provide mirrored disks for virtual machines running
under Xen. 99% of the time, the drbd devices run in primary/secondary
mode (aka active/passive), but they are switched to primary/primary
(aka active/active) for live migrations of domains, as that needs the
disks to be available on both nodes. From our experience, if the drbd
device is healthy, this is very reliable. No experience with running
drbd in primary/primary config for any extended period of time, though
(the live migrations are usually over after a few seconds to a minute at
most, then the drbd devices go back to primary/secondary).
What filesystem are you using to enable the primary-primary mode? Have
you evaluated it against any other available option?
The filesystem is whatever the VM is using, usually ext3. But the
filesystem doesn't matter in our use case at all, because:
- the backing store for drbd are logical volumes
- the drbd block devices are directly exported as block devices
to the VMs
The filesystem is only active inside the VM - and the VM is not aware of
the drbd primary/secondary -> primary/primary -> primary/secondary dance
that happens "outside" to enable live migration.
Kind regards,
Alex.
--
"Opportunity is missed by most people because it is dressed in overalls and
looks like work." -- Thomas A. Edison
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
i think filesystem is a problem...
you can't have two writers over a filesystem that allow only one, or
you will have filesystem crash (a lot of fsck repair... local cache
and other's features), maybe a gfs ocfs or another is a better
solution...
2011/1/31 Alexander Schreiber [off-list ref]:
On Mon, Jan 31, 2011 at 06:42:44AM -0200, Denis wrote:
quoted
2011/1/29 Alexander Schreiber [off-list ref]:
quoted
On Sat, Jan 29, 2011 at 12:23:14PM -0200, Denis wrote:
quoted
2011/1/29 Alexander Schreiber [off-list ref]
quoted
plain disk performance for writes, while reads should be reasonably
close to the plain disk performance - drbd optimizes reads by just reading
from the local disk if it can.
However, I have not used it with active-active fashion. Have you? if yes,
what is your overall experience?
We are using drbd to provide mirrored disks for virtual machines running
under Xen. 99% of the time, the drbd devices run in primary/secondary
mode (aka active/passive), but they are switched to primary/primary
(aka active/active) for live migrations of domains, as that needs the
disks to be available on both nodes. From our experience, if the drbd
device is healthy, this is very reliable. No experience with running
drbd in primary/primary config for any extended period of time, though
(the live migrations are usually over after a few seconds to a minute at
most, then the drbd devices go back to primary/secondary).
What filesystem are you using to enable the primary-primary mode? Have
you evaluated it against any other available option?
The filesystem is whatever the VM is using, usually ext3. But the
filesystem doesn't matter in our use case at all, because:
- the backing store for drbd are logical volumes
- the drbd block devices are directly exported as block devices
to the VMs
The filesystem is only active inside the VM - and the VM is not aware of
the drbd primary/secondary -> primary/primary -> primary/secondary dance
that happens "outside" to enable live migration.
Kind regards,
Alex.
--
"Opportunity is missed by most people because it is dressed in overalls and
looks like work." -- Thomas A. Edison
--
Roberto Spadim
Spadim Technology / SPAEmpresarial
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
From: Alexander Schreiber <hidden> Date: 2011-01-31 16:15:26
On Mon, Jan 31, 2011 at 12:45:31PM -0200, Roberto Spadim wrote:
i think filesystem is a problem...
you can't have two writers over a filesystem that allow only one, or
you will have filesystem crash (a lot of fsck repair... local cache
and other's features), maybe a gfs ocfs or another is a better
solution...
No, for _our_ use case (replicated disks for VMs running under Xen
with live migration) the fileystem just _does_ _not_ _matter_ _at_
_all_. Due to the way Xen live migration works, there is only one
writer at any one time: the VM "owning" the virtual disk provided
by drbd.
To illustrate the point, a very short summary of what happens during
Xen live migration in our setup:
- VM is to be migrated from host A to host B, with the virtual block
device for the instance being provided by a drbd pair running on
those hosts
- host A/B are configured primary/secondary
- we reconfigure drbd to primary/primary
- start Xen live migration
- Xen creates a target VM on host B, this VM is not yet running
- Xen syncs live VM memory from host A to host B
- when most of the memory is synced over, Xen suspends execution of
the VM on host A
- Xen copies the remaining dirty VM memory from host A to host B
- Xen resumes VM execution on host B, destroys the source VM
on host A, Xen live migration is completed
- we reconfigure drbd on hosts A/B to secondary/primary
There is no concurrent access to the virtual block device here anywhere.
And the only reason we go primary/primary during live migration is that
for Xen to attach the disks to the target VM, they have to be available
and accessible on the target node - as well as on the source node where
they are currently attached to the source VM.
Now, if you were doing things like, say, use an primary/primary drbd
setup for NFS servers serving in parallel from two hosts, then yes,
you'd have to take special steps with a proper parallel filesystem
to avoid corruption. But this is a completely different problem.
Kidn regards,
Alex.
2011/1/31 Alexander Schreiber [off-list ref]:
quoted
On Mon, Jan 31, 2011 at 06:42:44AM -0200, Denis wrote:
quoted
2011/1/29 Alexander Schreiber [off-list ref]:
quoted
On Sat, Jan 29, 2011 at 12:23:14PM -0200, Denis wrote:
quoted
2011/1/29 Alexander Schreiber [off-list ref]
quoted
plain disk performance for writes, while reads should be reasonably
close to the plain disk performance - drbd optimizes reads by just reading
from the local disk if it can.
However, I have not used it with active-active fashion. Have you? if yes,
what is your overall experience?
We are using drbd to provide mirrored disks for virtual machines running
under Xen. 99% of the time, the drbd devices run in primary/secondary
mode (aka active/passive), but they are switched to primary/primary
(aka active/active) for live migrations of domains, as that needs the
disks to be available on both nodes. From our experience, if the drbd
device is healthy, this is very reliable. No experience with running
drbd in primary/primary config for any extended period of time, though
(the live migrations are usually over after a few seconds to a minute at
most, then the drbd devices go back to primary/secondary).
What filesystem are you using to enable the primary-primary mode? Have
you evaluated it against any other available option?
The filesystem is whatever the VM is using, usually ext3. But the
filesystem doesn't matter in our use case at all, because:
- the backing store for drbd are logical volumes
- the drbd block devices are directly exported as block devices
to the VMs
The filesystem is only active inside the VM - and the VM is not aware of
the drbd primary/secondary -> primary/primary -> primary/secondary dance
that happens "outside" to enable live migration.
--
"Opportunity is missed by most people because it is dressed in overalls and
looks like work." -- Thomas A. Edison
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
nice, you don´t have two writers.
2011/1/31 Alexander Schreiber [off-list ref]:
On Mon, Jan 31, 2011 at 12:45:31PM -0200, Roberto Spadim wrote:
quoted
i think filesystem is a problem...
you can't have two writers over a filesystem that allow only one, or
you will have filesystem crash (a lot of fsck repair... local cache
and other's features), maybe a gfs ocfs or another is a better
solution...
No, for _our_ use case (replicated disks for VMs running under Xen
with live migration) the fileystem just _does_ _not_ _matter_ _at_
_all_. Due to the way Xen live migration works, there is only one
writer at any one time: the VM "owning" the virtual disk provided
by drbd.
To illustrate the point, a very short summary of what happens during
Xen live migration in our setup:
- VM is to be migrated from host A to host B, with the virtual block
device for the instance being provided by a drbd pair running on
those hosts
- host A/B are configured primary/secondary
- we reconfigure drbd to primary/primary
- start Xen live migration
- Xen creates a target VM on host B, this VM is not yet running
- Xen syncs live VM memory from host A to host B
- when most of the memory is synced over, Xen suspends execution of
the VM on host A
- Xen copies the remaining dirty VM memory from host A to host B
- Xen resumes VM execution on host B, destroys the source VM
on host A, Xen live migration is completed
- we reconfigure drbd on hosts A/B to secondary/primary
There is no concurrent access to the virtual block device here anywhere.
And the only reason we go primary/primary during live migration is that
for Xen to attach the disks to the target VM, they have to be available
and accessible on the target node - as well as on the source node where
they are currently attached to the source VM.
Now, if you were doing things like, say, use an primary/primary drbd
setup for NFS servers serving in parallel from two hosts, then yes,
you'd have to take special steps with a proper parallel filesystem
to avoid corruption. But this is a completely different problem.
Kidn regards,
Alex.
quoted
2011/1/31 Alexander Schreiber [off-list ref]:
quoted
On Mon, Jan 31, 2011 at 06:42:44AM -0200, Denis wrote:
quoted
2011/1/29 Alexander Schreiber [off-list ref]:
quoted
On Sat, Jan 29, 2011 at 12:23:14PM -0200, Denis wrote:
quoted
2011/1/29 Alexander Schreiber [off-list ref]
quoted
plain disk performance for writes, while reads should be reasonably
close to the plain disk performance - drbd optimizes reads by just reading
from the local disk if it can.
However, I have not used it with active-active fashion. Have you? if yes,
what is your overall experience?
We are using drbd to provide mirrored disks for virtual machines running
under Xen. 99% of the time, the drbd devices run in primary/secondary
mode (aka active/passive), but they are switched to primary/primary
(aka active/active) for live migrations of domains, as that needs the
disks to be available on both nodes. From our experience, if the drbd
device is healthy, this is very reliable. No experience with running
drbd in primary/primary config for any extended period of time, though
(the live migrations are usually over after a few seconds to a minute at
most, then the drbd devices go back to primary/secondary).
What filesystem are you using to enable the primary-primary mode? Have
you evaluated it against any other available option?
The filesystem is whatever the VM is using, usually ext3. But the
filesystem doesn't matter in our use case at all, because:
- the backing store for drbd are logical volumes
- the drbd block devices are directly exported as block devices
to the VMs
The filesystem is only active inside the VM - and the VM is not aware of
the drbd primary/secondary -> primary/primary -> primary/secondary dance
that happens "outside" to enable live migration.
--
"Opportunity is missed by most people because it is dressed in overalls and
looks like work." -- Thomas A. Edison
--
Roberto Spadim
Spadim Technology / SPAEmpresarial
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html