On 05/21/2017 12:04 PM, Chris Schanzle wrote:
On 05/18/2017 11:39 PM, Ram Ramesh wrote:
quoted
On 05/18/2017 08:34 PM, Adam Goryachev wrote:
quoted
On 19/05/17 07:37, Ram Ramesh wrote:
quoted
Any one have a method to backup large volumes like md raid6 (16TB)? Since the backup will not fit in one disk (in many cases and mine too) I am wondering, if there is a known/easy technique to backup using multiple usb hard drives. I googed and found a few fancy backup utils/systems like Amanda etc. They are overkill for me. I am choosing not to back up simply because of the complexity of setup as the data in my RAID volume is NOT precious and can be replaced with a week of effort.
If any one can think of some thing simple, please point me to it and I will do the reading to figure out.
I suppose it depends on what you want to achieve. There are various options such as mirroring to another raid6 array (ie, RAID60) but really that isn't a backup, it's another replica.
I use backuppc for my backups, it works well under linux with rsync, I'm sure there are many various options (including amanda which I've used in the past). Ultimately, it depends on your requirements, backups vary significantly depending on needs/etc.
Regards,
Adam
Here is a summary of what I like to do. I want to backup files on to (multiple) disks that will be loaded on to a USB dock. Simple one to one copy is all I am looking for. I am not interested in full vs. incremental or keeping versions of files for restore. My data is just movies and songs. All I want is a SW that understands links (to avoid duplicates) and copy files in batch on to multiple disks. I want content of each (backup) disk to be independent. This way if one backup disk dies, I have all other files unaffected by this failure. The only reason I did not try multi-volume tar is the lack of independence across disks.
I suppose this is a backup question rather than RAID question. I asked here because the size of RAID volumes make it impossible to back up to a single drive and this is a more common problem RAID world rather than general user forum in a backup mailing list.
Ramesh
Thanks for the additional info on your requirements.
Rsync will be your best/fastest tool, as writing 16 TB of data over USB will likely take days for a full backup...and that seems silly when you probably already have 90% of the data already.
If one drive failures are a concern, why not use RAID5 for your group of USB backup disks? Might take you a bit more effort starting/stopping the array, but I'm sure you'll enjoy the large single volume/filesystem in the long run. Use a fairly large --chunk= size for your large files. I have had very positive experience with XFS for large multimedia filesystems (mythtv user and 20TB backup servers).
If the drives you choose to use are desktop grade, be sure to increase the timeouts. Search this list archives for Chris Murphy's subject: "Re: URE, link resets, user hostile defaults".
The only problem with raid is that all disks need to be present to create md. I only have the ability to load two disks and the disks are fairly small compared to RAID l (3TB vs. 16TB) So RAID5 won't work. Besides, I do not want to use RAID to backup RAID as my worry is md going bad and preventing access to my files. It is not disk that I am worried about mdadm layout and requirement that many disk to be alive to get even one file.