Re: RAID performance

Stan Hoeppner [off-list ref] wrote:

On 2/11/2013 11:33 PM, Adam Goryachev wrote:

quoted

I'm assuming MPIO requires the following:
SAN must have multiple physical links over 'disconnected' networks
(ie, different networks) on different subnets.
iSCSI client must meet the same requirements.

I fubar'd this.  See below for a thorough explanation.  The IPs should
all be in the same subnet.

quoted

OK, what about this option:

Install dual port ethernet card into each of the 8 xen boxes
Install 2 x quad port ethernet card into each of the san boxes

Connect one port from each of the xen boxes plus 4 ports from each
san box to a single switch (16ports)

Connect the second port from each of the xen boxes plus 4 ports from
each san box to a second switch (16 ports)

Connect the motherboard port (existing) from each of the xen boxes
plus one port from each of the SAN boxes (management port) to a single
switch (10 ports).

Total of 42 ports.

Leave the existing motherboard port configured with existing

IP's/etc,

quoted

and dedicate this as the management/user network (RDP/SMB/etc).

Keeping the LAN and SAN traffic on different segments is a big plus.
But I still wonder if a single link for SMB traffic is enough for that
greedy bloke moving 50GB files over the network.

quoted

We then configure the SAN boxes with two bond devices, each
consisting of a set of 4 x 1Gbps as balance-alb, with one IP
address each (from 2 new subnets).

Use MPIO (multipath) only.  Do not use channel bonding.  MPIO runs
circles around channel bonding.  Read this carefully.  I'm pretty sure
you'll like this.

Ok, so this fibre channel guy has been brushing up a bit on iSCSI
multipath, and it looks like the IP subnetting is a non issue, and
after
thinking it through I've put palm to forehead.  As long as you have an
IP path between ethernet ports, the network driver uses the MAC address
from that point forward, DUH!.  Remember that IP addresses exist solely
for routing packets from one network to another.  But within a network
the hardware address is used, i.e. the MAC address.  This has been true
for the 30+ years of networking.  Palm to forehead again. ;)

So, pick a unique subnet for SAN traffic, and assign an IP and
appropriate mask to each physical port in all the machines' iSCSI
ports.

There are a couple of problems I'm having with this solution.

I've created 8 IP's (one on each eth interface on san1, all in the same subnet /24) and another 2 IP's on xen1 for it's 2 eth interfaces, again on the same subnet.

The initial reason I knew this wouldn;t work is that linux will see the arp request (broadcast) and respond from any interface. For example, from xen1 I ping each IP on san1, then do a arp -an and see the same MAC address for every IP (or sometimes I will see two or even three different MAC addresses, but mostly the same. I do arp -d in between to restart the test clean each time).

So, after some initial reading (since I remembered from years ago this was solvable) I did:
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce

Now I get a unique MAC for each IP (perfect)

What about balancing the reverse traffic. 
Assuming I set the same options on xen1, then san1 will reply to the different IP's on different ethernet ports for xen1.
Except, when xen1 sends any TCP request to any of san1 IP's, it will always come from the same IP, so san1 will always respond to the same IP. This was why I suggested using two distinct subnets/LAN's/switches. 

This means I'm limited to only 1Gbps outbound, and therefore only 1Gbps inbound since the san1 will always reply to the same IP.....

If I use two subnets, multipath will use the first ethernet interface/IP to talk to the san1 on it's first IP and the second ethernet/ip to talk to san1 second subnet/ip

I thought to use bonding with balance-alb for the 4 ports on san1 so that it will use a max of 1 out of 4 ports to talk to one client (avoid overloading the client) and also dynamically balance all clients for inbound/outbound with fancy arp announcing.

The rest of the iSCSI setup you already know how to do.  The only
advice I can give you here is to expose every server target LUN out
every physical port so the Xen box ports see the LUNs on every server
port.  I assume you've already done this with the current IP subnet, as
it's required to live migrate your VMs amongst the Xen servers.  So you
just need to change it over for the new SAN specific subnet/ports. 
Now,
when you run 'multipath -ll' on each Xen box it'll see all the LUNs on
all 8 ports of each iSCSI server (as well as local disks), and
automatically do round robin fanning of SCSI block IO packets across
all
8 server ports.  You may need to blacklist local devices.  You'll
obviously want to keep the LUNs on the standby iSCSI server masked, or
simply not used until needed.

You only install the multipath driver on the initiators (Xen clients),
--NOT ON THE TARGETS (servers)-- .  All block IO transactions are
initiated by the client (think desktop PC with single SATA drive--who
talks first, mobo or drive?).  The iSCSI server will always reply on
the
port a packet arrived on.  So, you get automatic perfect block IO
scaling on all server ports, all the time, no matter how many clients
are talking.  Told ya you'd like this. ;)  Here's a relatively
informative read on multipath:

http://linfrastructure.blogspot.com/2008/02/multipath-and-equallogic-iscsi.html

OK, so I need to use iscsiadm and bind it to the individual mac/interface... now I see how this will work

quoted

Add a "floating" IP to the current primary SAN on each of the bond
interfaces from the new subnets.

No, see above.

quoted

We configure each of the xen boxes with two new ethernets with one IP
address each (from the 2 new subnets).

Configure multipath to talk to the two floating IP's

So, to do the failover, I just need to stop ietd on san1, and start ietd on san2.... as long as I did a discovery at some point while san2 was running so that it knows that is a possible path to the devices...

quoted

I'm thinking 200MB/s should be enough performance for any one machine
disk access, and 1Gbps for any single user side network access should

be

quoted

ample given this is the same as what they had previously.

Coincidentally, the last 'decent' size network I managed had 525'ish
users, but our 4 Citrix servers were bare metal blades.  All our CIFS
traffic hit a single blade's GbE port.  That blade, ESX3, hosted our DC
file server VM and 6 other Linux and Windows VMs, some of which had
significant traffic.  User traffic HBA was single GbE, and the SAN HBA
was 2Gb/s fibre channel.  Same bandwidth as your soon-to-be setup.
Though the backend was different, one FasTt600 and one SataBlade, each
with a single 2Gb FC link.

I'll bet it was a lot neater too :)

quoted

The only question left is what will happen when there is only one xen
box asking to read data from the SAN? Will the SAN attempt to send

the

quoted

data at 8Gbps, flooding the 2Gbps that the client can handle, and

You're not using balance-rr inappropriately here.  So, no, this isn't
an
issue.  In a request/reply chain, responses will only go out at the
same
rate requests come in.  With two ports making requests to 8 ports, each
of the 8 will receive and reply to 1/4th of the total requests,
generating ~1/4th of the total bandwidth.  200/8=25, so each of the 8
ports will transmit ~25MB/s in replies.  Reply packets will be larger
due to the data payload, but the packet queuing, window scaling, etc in
the receiver TCP stack FOR EACH PORT will slow down the sender when
necessary.

And you're now wondering why TCP packet queuing didn't kick in with
balance-rr, causing all of those ethernet pause frames and other
issues.
Answer:  I think the problem was that the TCP back off features were
short circuited.  When you were using balance-rr, packets were likely
arriving wildly out of sequence, from the same session, but from
different MAC addresses.  You were sending from one IP stack out 4 MACs
to one 1 IP stack on one MAC.

With multipath iSCSI, each MAC has its own IP address and own TCP
stack,
so all packets always arrive in order or are easily reordered, allowing
packet queuing, window scaling, etc, to work properly.  Balance-rr
works
in the cluster scenario up top because the packets still arrive in
sequence, even though on different ports from different MACs.  Say
1/2/3/4, 5/6/7/8, etc.  Previously you probably had packet ordering
something like 4/1/3/2/7/8/6/5 on occasion.  This short circuited the
receiving TCP stack preventing it from sending back offs.  The TCP
stack
on the server thought all was fine and kept slinging packets until the
switch started sending back ethernet pause frames.  I'm not enough of
an
ethernet or TCP expert to explain what happens next, but I'd bet those
Windows write errors are related to this.

Again, ya don't have to worry about any of this mess using multipath.
And, you get full port balanced bandwidth on all Xen hosts, and all
server ports, all the time.  Pretty slick.

quoted

generate all the pause messages, or is this not relevant and it will
"just work". Actually, I think from reading the docs, it will only

use

quoted

one link out of each group of 4 to send the data, hence it won't

attempt

quoted

to send at more than 2Gbps to each client....

See above.

I'm not confident, but will give it a go and see.... since we will send one small request to each of the 8 san1 IP's, and each of those can reply at 1Gbps, and the reply will be bigger than the request (reads). Though I suppose we won't submit the next read request until after we get the first one, so perhaps this will keep things under control .... I'll let you know how it goes....

quoted

I don't think this system will scale any further than this, I can

only

quoted

add additional single Gbps ports to the xen hosts, and I can only add
one extra 4 x 1Gbps ports to each SAN server.... Best case is add 4 x
10Gbps to the SAN, 2 single 1Gbps ports to each xen, providing a full
32Gbps to the clients, each client gets max 4Gbps. In any case, I

think

quoted

that would be one kick-ass network, besides being a pain to try and
debug, keep cabling neat and tidy, etc... Oh, and the current SSD's
wouldn't be that fast... At 400MB/s read, times 7 data disks is
2800GB/s, actually, damn, that's fast.

You could easily get by with a single quad port NIC on iSCSI target
duty
in each server.  That's 800MB/s duplex, same as 4Gb fibre channel.
That's more than sufficient for your 8 Xen nodes, especially given the
bulk of your traffic is SMB, which is limited to ~100MB/s, which means
~100MB/s on the SAN, with ~300MB/s breathing room.  Use the other quad
port NIC direct connected with x-over cables to the other server for
DRBD using balance-rr.

Yes, at some point I'm going to need to increase the connection for DRBD from the current 1Gbps, but one thing at a time :)

quoted

The only additional future upgrade I would plan is to upgrade the
secondary san to use SSD's matching the primary. Or add additional

SSD's

quoted

to expand storage capacity and I guess speed. I may also need to add
additional ethernet ports to both SAN1 and SAN2 to increase the DRBD
cross connects, but these would I assume be configured using linux
bonding in balance-rr since there is no switch in between.

See above.
And more than sufficient hardware.  I was under the impression that
this much capital was not available, or I'd had different
recommendations.
One being very similar to what you came up with here.

So did I, until they said "Just fix it, whatever you need...." so that's when I had to make sure to purchase everything I might need in one go, and make sure it would work the first time .....

Two words:  Murphy's law

Thanks, I thought I was in trouble after installing the equipment into san1, there was no keyboard, eventually I pulled both quad port ethernets, still keyboard was really unreliable, pulled the new sata controller, same thing.... Eventually tried a different keyboard, and it was perfect.... I can't believe a USB keyboard would fail right in the middle of a major upgrade. Thankfully there was a spare USB keyboard in it's box available or I might have taken me hours longer to sort it out!

quoted

No problem, this has been a definite learning experience for me and I
appreciate all the time and effort you've put into assisting.

There are millions of folks spewing vitriol at one another at any
moment
on the net.  I prefer to be constructive, help people out when I can,
pass a little knowledge and creativity when possible, learn things
myself.  That's not to say I don't pop at folks now and then when
frustration boils. ;)  I'm human too.

Absolutely agree with all that :)

quoted

BTW, I went last night (monday night) and removed one dual port card
from the san2, installed into the xen host running the DC VM.

Configured

quoted

the two new ports on the xen box as active-backup (couldn't get LACP

to

quoted

work since the switch only supports max of 4 LAG's anyway). Removed

one

quoted

port from the LAG on san1, and setup the three ports (1 x san + 2 x
xen1) as a VLAN with private IP address on a new subnet.  Today,
complaints have been non-existant, mostly relating to issues they had
yesterday but didn't bother to call until today. It's now 4:30pm, so

I'm

quoted

thinking that the problem is solved just with that done.

So the biggest part of the problem was simply SMB and iSCSI on the same
link on the DC.  Let's see how the new system does with that user in
need of a clue stick, doing his 50GB SMB xfer when all users are
humming away.

Well, I'm onsite now, in progress, got 3 hours to finish (by 7am) so better go and get it sorted!

I appreciate the fact you posted the topic.  I had to go (re)learn a
little bit myself.  I just hope read you this email before trying to
stick MPIO on top of a channel bond. ;)

Only just in time, I had already installed some of the equipment when I got this.... 

Send me a picture of the racked gear when it's all done, front, and
back, so I can see how ugly that Medusa is, and remind myself of one of
many reasons I prefer fibre channel. ;)

It is a real mess, I forgot to order extra cables, so pulled random lengths of second hand cables from the spares cupboard here... there are no more spares now, but I think they are all working.... Will send some pics, but I will need to come back another time with some new cables, cable ties, and some sort of cable labelling equipment to fix this up!

Thanks again, off to finish implementing now


Regards,
Adam

--
Adam Goryachev
Website Managers
www.websitemanagers.com.au