Re: raid over ethernet

On Mon, Jan 31, 2011 at 12:45:31PM -0200, Roberto Spadim wrote:

i think filesystem is a problem...
you can't have two writers over a filesystem that allow only one, or
you will have filesystem crash (a lot of fsck repair... local cache
and other's features), maybe a gfs ocfs or another is a better
solution...

No, for _our_ use case (replicated disks for VMs running under Xen
with live migration) the fileystem just _does_ _not_ _matter_ _at_
_all_. Due to the way Xen live migration works, there is only one
writer at any one time: the VM "owning" the virtual disk provided
by drbd. 

To illustrate the point, a very short summary of what happens during
Xen live migration in our setup:
 - VM is to be migrated from host A to host B, with the virtual block
   device for the instance being provided by a drbd pair running on
   those hosts
 - host A/B are configured primary/secondary
 - we reconfigure drbd to primary/primary
 - start Xen live migration
 - Xen creates a target VM on host B, this VM is not yet running
 - Xen syncs live VM memory from host A to host B
 - when most of the memory is synced over, Xen suspends execution of
   the VM on host A
 - Xen copies the remaining dirty VM memory from host A to host B
 - Xen resumes VM execution on host B, destroys the source VM
   on host A, Xen live migration is completed
 - we reconfigure drbd on hosts A/B to secondary/primary

There is no concurrent access to the virtual block device here anywhere.
And the only reason we go primary/primary during live migration is that
for Xen to attach the disks to the target VM, they have to be available
and accessible on the target node - as well as on the source node where
they are currently attached to the source VM.

Now, if you were doing things like, say, use an primary/primary drbd
setup for NFS servers serving in parallel from two hosts, then yes, 
you'd have to take special steps with a proper parallel filesystem
to avoid corruption. But this is a completely different problem.

Kidn regards,
          Alex.

2011/1/31 Alexander Schreiber [off-list ref]:

quoted

On Mon, Jan 31, 2011 at 06:42:44AM -0200, Denis wrote:

quoted

2011/1/29 Alexander Schreiber [off-list ref]:

quoted

On Sat, Jan 29, 2011 at 12:23:14PM -0200, Denis wrote:

quoted

2011/1/29 Alexander Schreiber [off-list ref]

quoted

plain disk performance for writes, while reads should be reasonably
close to the plain disk performance - drbd optimizes reads by just reading
from the local disk if it can.

 However, I have not used it with active-active fashion. Have you? if yes,
what is your overall experience?

We are using drbd to provide mirrored disks for virtual machines running
under Xen. 99% of the time, the drbd devices run in primary/secondary
mode (aka active/passive), but they are switched to primary/primary
(aka active/active) for live migrations of domains, as that needs the
disks to be available on both nodes. From our experience, if the drbd
device is healthy, this is very reliable. No experience with running
drbd in primary/primary config for any extended period of time, though
(the live migrations are usually over after a few seconds to a minute at
most, then the drbd devices go back to primary/secondary).

What filesystem are you using to enable the primary-primary mode? Have
you evaluated it against any other available option?

The filesystem is whatever the VM is using, usually ext3. But the
filesystem doesn't matter in our use case at all, because:
 - the backing store for drbd  are logical volumes
 - the drbd block devices are directly exported as block devices
  to the VMs
The filesystem is only active inside the VM - and the VM is not aware of
the drbd primary/secondary -> primary/primary -> primary/secondary dance
that happens "outside" to enable live migration.

-- 
"Opportunity is missed by most people because it is dressed in overalls and
 looks like work."                                      -- Thomas A. Edison
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html