From: Jonathan Tripathy <hidden> Date: 2012-08-12 21:10:09
Hi Everyone,
Our server currently has 6 SATA ports on the motherboard filled with 6 drives in RAID10. I have a need to install an additional PCIe SATA controller in a server in order to obtain 2 more SATA ports to install an additional 2 new drives (As I want a new RAID1 array). However I would like to move one of the current drives onto the new controller, and one of the new drives onto the motherboard. This should give me a little redundancy in case the new controller card were to fail (I don't want the new controller to bring down 2 drives).
When setting up MD RAID in this configuration, how would I go about tell MD RAID which disk is which in the array (referring to the near/far configurations)? I'm assuming by default that MD RAID just uses alphabetical order in order to determine the drive order?
Let's say the RAID10 drives will be sda sdb sdc sdd sde sdg*, where sdg is the second of the RAID0 strip of sde.
And the two new RAID1 drives are sdf sdh*.
The drives with the asterisks beside them will be on the new PCIe controller.
Do I have to do anything special for this configuration? Will there be a performance hit since one drive in the RAID10 array will be on a different controller?
Thanks
From: Peter Grandi <hidden> Date: 2012-08-12 23:03:59
Our server currently has 6 SATA ports on the motherboard
filled with 6 drives in RAID10. [ ... ] additional PCIe SATA
controller in a server [ ... ] give me a little redundancy in
case the new controller card were to fail (I don't want the
new controller to bring down 2 drives).
It would be nice to split each pair of your new RAID1 and the
existing RAID10 over the two host adapters (the controllers are
the PCBs on the bottom of the drive...) you will have.
As in one member of each pair on the motherboard and the other
on the PCIe card.
Ideally therefore get a 4-drive PCIe card, so you can put on it
3 members of your existing RAID10 set, plus one member of your
new RAID1 set. SPAN.com tend to have reasonable non-RAID SATA
cards.
When setting up MD RAID in this configuration, how would I go
about tell MD RAID which disk is which in the array (referring
to the near/far configurations)?
If you are using a recent MD setup each member block device (not
necessarily disk) has an MD descriptor block with an UUID (or a
name) for the array and one for the member, and ideally you
would use that.
You can use 'mdadm -E /dev/sd?' to get the relevant information,
and them put the relevant UUID (or name) in '/etc/mdadm.conf'
I'm assuming by default that MD RAID just uses alphabetical
order in order to determine the drive order? Let's say the
RAID10 drives will be sda sdb sdc sdd sde sdg* [ ... ]
Drive names can change on any reboot. You cannot rely on them.
Will there be a performance hit since one drive in the RAID10
array will be on a different controller?
PCIe host adapters tend to have lower host bus bandwith than
motherboard ones, but this should not matter in your case (if
your host adapter has a 4-lane PCIe interface).
Jonathan,
On 12-08-12 23:10, Jonathan Tripathy wrote:
Hi Everyone,
Our server currently has 6 SATA ports on the motherboard filled with 6 drives in RAID10. I have a need to install an additional PCIe SATA controller in a server in order to obtain 2 more SATA ports to install an additional 2 new drives (As I want a new RAID1 array). However I would like to move one of the current drives onto the new controller, and one of the new drives onto the motherboard. This should give me a little redundancy in case the new controller card were to fail (I don't want the new controller to bring down 2 drives).
When setting up MD RAID in this configuration, how would I go about tell MD RAID which disk is which in the array (referring to the near/far configurations)? I'm assuming by default that MD RAID just uses alphabetical order in order to determine the drive order?
mdadm does not care about the disk ordering once the raid has been defined. It uses the raid metadata on the disks to figure out which disk is doing what. This includes which disk belongs to which raid.
There are 2 options:
- use metadata .90 (no longer suggested) and have the kernel autodetect during boot
- use a later metadata (1.2 is default now), and have mdadm in the initrd. This will use a config file that has mdadm search on UID. Again, independent of which sdx the disks are.
You could test by swapping disk order in your current system. Although i would suggest to do testing with a test machine ;)
Cheers,
Rudy
From: Jonathan Tripathy <hidden> Date: 2012-08-12 23:47:53
On 13/08/2012 00:16, Rudy Zijlstra wrote:
Jonathan,
On 12-08-12 23:10, Jonathan Tripathy wrote:
quoted
Hi Everyone,
Our server currently has 6 SATA ports on the motherboard filled with 6 drives in RAID10. I have a need to install an additional PCIe SATA controller in a server in order to obtain 2 more SATA ports to install an additional 2 new drives (As I want a new RAID1 array). However I would like to move one of the current drives onto the new controller, and one of the new drives onto the motherboard. This should give me a little redundancy in case the new controller card were to fail (I don't want the new controller to bring down 2 drives).
When setting up MD RAID in this configuration, how would I go about tell MD RAID which disk is which in the array (referring to the near/far configurations)? I'm assuming by default that MD RAID just uses alphabetical order in order to determine the drive order?
mdadm does not care about the disk ordering once the raid has been defined. It uses the raid metadata on the disks to figure out which disk is doing what. This includes which disk belongs to which raid.
There are 2 options:
- use metadata .90 (no longer suggested) and have the kernel autodetect during boot
- use a later metadata (1.2 is default now), and have mdadm in the initrd. This will use a config file that has mdadm search on UID. Again, independent of which sdx the disks are.
You could test by swapping disk order in your current system. Although i would suggest to do testing with a test machine ;)
Thanks for your reply.
Currently, we use CentOS and use the installer to set up the MD RAID. It is using metadata version 0.90. The only thing in mdadm.conf is the UUID of the array and the number of disks per array. It's good to know that mdadm doesn't look as the sdx value once the RAID is setup. What I would like to know is how to configure the disk order in the array. Or maybe it's not very important to do this? For example, going by these diagrams: http://en.wikipedia.org/wiki/Non-standard_RAID_levels#Linux_MD_RAID_10
how do I know/configure which disk is A1 and which is A2? I just want to know the disk order after i split my array across different adapters.
Thanks
From: Phil Turmel <hidden> Date: 2012-08-12 23:57:11
Hi Jonathan,
On 08/12/2012 05:10 PM, Jonathan Tripathy wrote:
Hi Everyone,
Our server currently has 6 SATA ports on the motherboard filled with 6
drives in RAID10. I have a need to install an additional PCIe SATA
controller in a server in order to obtain 2 more SATA ports to install
an additional 2 new drives (As I want a new RAID1 array). However I
would like to move one of the current drives onto the new controller,
and one of the new drives onto the motherboard. This should give me a
little redundancy in case the new controller card were to fail (I don't
want the new controller to bring down 2 drives).
Reasonable plan.
When setting up MD RAID in this configuration, how would I go about tell
MD RAID which disk is which in the array (referring to the near/far
configurations)? I'm assuming by default that MD RAID just uses
alphabetical order in order to determine the drive order?
No, MD uses the order they were specified in the "--create" operation,
and records that in metadata on each device. Re-assembly can specify
them in any order--MD will put them into the correct role.
Let's say the RAID10 drives will be sda sdb sdc sdd sde sdg*, where sdg
is the second of the RAID0 strip of sde.
And the two new RAID1 drives are sdf sdh*.
The drives with the asterisks beside them will be on the new PCIe
controller.
Probe order during boot is not guaranteed--you might discover that the
new controller gets /dev/sda and /dev/sdb. Or with parallel probing you
might get /dev/sda[acefgh] on the mobo and /dev/sd[bd] on the new
controller. (Not terribly likely, in my limited experience.)
You should consider recording what drive serial number is hooked to what
port, and compare that to your device names. You might find my "lsdrv"
script useful for this, as that was its original purpose.[1]
Or just print a copy of "ls -l /dev/disk/by-id/".
Do I have to do anything special for this configuration? Will there be a
performance hit since one drive in the RAID10 array will be on a
different controller?
From: Jonathan Tripathy <hidden> Date: 2012-08-13 00:04:37
No, MD uses the order they were specified in the "--create" operation,
and records that in metadata on each device. Re-assembly can specify
them in any order--MD will put them into the correct role.
Thanks for this. So basically, the naming of the device is only important at the time of creating the array (as that defines the layout). As soon as I create the array, mdadm puts it's own metadata on the drive so it can identify them afterwards? That sounds very good indeed.
Now, onto the topic of PCIe adapter. I was looking at this: http://www.cclonline.com/product/42541/PEXSAT32/IDE-SATA-SCSI-Cards/StarTech-2-Port-SATA-6-Gbps-PCI-Express-SATA-Controller-Card/CNT1075/
Is that a bad model? Seems a little on the cheap side? Can anyone suggest a good model?
Many Thanks for all the help
Amazon users aren't impressed. But the limitation is really the PCIe x1
v2 bus interface: 500MB/sec. Two SSDs would certainly overwhelm it, and
any other similar card.
I have a similar card, the Highpoint 620, in an older system at home
(PCIe v1): it maxes out at 200MB/sec.
So, it depends on what you want it to do.
{I'll let others make card recommendations--I prefer to *listen* to those.}
Phil
Amazon users aren't impressed. But the limitation is really the PCIe x1
v2 bus interface: 500MB/sec. Two SSDs would certainly overwhelm it, and
any other similar card.
I have a similar card, the Highpoint 620, in an older system at home
(PCIe v1): it maxes out at 200MB/sec.
So, it depends on what you want it to do.
{I'll let others make card recommendations--I prefer to *listen* to those.}