busy day, reading up on unraid->flexRaid->disParity :)
sorry if already asked:
1. can I add new drives, fill them, then add them to the parity set (related to below)?
2. does it support offline drives? I have a sata-hotswap cage:
- sys drive
- parity drive (500GB)
- swappable drive bay (500GB drives)
thanks!
edit: I guess I need to ask do all the drives that make up the array need to be online at once in order to do a recovery, or is it done in passes?
(parity XOR drive2, then XOR drive3, then XOR drive4 = drive1)
Currently if you add a new drive, you must rebuild the parity snapshot. It would be fairly trivial to fix it so that you can incrementally add a new drive without a full rebuild, but I haven't gotten around to it yet.
Yes, during a recover, all drives need to be accessible (except the drive being recovered, of course.)
Thanks for the info!
It sounds like for my particular situation I need to keep looking.
So is the problem that you have more drives than you can connect to the system at the same time?
I think I see...you want a solution where you can keep swapping in drive after drive and add each one to parity separately. Yeah disParity really can't do that. Not sure what options there are out there for something like that. Good luck in your hunt!
Yep, that's it exactly.
I was thinking that I would dedicate one 500GB parity drive per 5-10 data drives, depending on if the time between swapping drives in and out for a build/restore was reasonable. (better than no backup at all)
any chance of seeing the source? (C#/.net/Java guy :) )
thanks.
Yeah, sorry, I'm not ready to share the source. I haven't made a decision about that. Kinda depends on where I wind up going with this.
But hey if you are a programmer, this stuff is REALLY not that hard at all. I whipped up the initial parity snapshot algorithm in just a few hours. Supporting subsequent updates (adds/edit/deletes) was a lot more work, but maybe you don't need all that. An "incremental add" algorithm where you add one drive at a time to a parity set would be pretty darn simple.
Implementing restores without access to all drives simultaneously would be a pain, though.
Thanks for all the info, good stuff!
With regards to complications of not having all drives once at once, can you explain?
In a high level way I imagined that for a restore of drive1, I could maybe take the parity file, XOR drive2 into it, then XOR drive3, then XOR drive4, each a separate pass, building up the parity file back into the missing drive. (the initally parity fil being built up the same way, scan drive1, make init parity file, then OR drive2 onto it, OR drive3,etc)
It sounds like from what you said, that wouldn't work, can you explain what I missed or overlooked?
thanks!
Yeah, you would have to do it pretty much like you said. I guess now that you put it into words, it doesn't seem so bad. The only hard part I suppose is at the end, now you've got a bunch of parity files that have your data, but they aren't broken up into the original files. So you'd need to also implement a final pass at the end to re-create all the original files again from the parity files.
The problem with that approach is that each recovered file will take n+1 writes to the target drive in order to be created.
Hi again,
Bit the bullet and bought a cheap 4xSata pci card, so I now have all my drives online.
Tried disParity last night, and it went very well, 2.64TB in 6 hrs, 133MB/s @ 10% cpu, very reasonable. What kind of speeds are others getting?
I'm thinking of juggling my drives and sata channels around to see if I can get faster speeds... I'm pretty sure my bottleneck right now is the sata pci card, but might be able to work around it.
Roland, thanks for writing this cool and needed app, please keep up dev in restore options, and any optimization if possible.
So glad to hear you are finding disParity useful! I do have another update cooking right now, but it's not quite ready for primetime yet. Didn't have as much time to work on it over the holidays as I'd hoped. :(
As for performance, the hard drives are always going to be the bottleneck, and I find it's very hard to predict how the whole system is going to react when an app like disParity tries to read from them all simultaneously as fast as possible. 133MB/sec is good, I can't get much more than that on my system either. I've experimented with a bunch of different I/O algorithms and haven't been able to find any way to speed things up faster than they are currently...in fact just about everything I've tried so far always makes it worse!
I ran verify on 4x 1.5TB drives (1 parity, 3 data, 1.3TB of parity) yesterday to verify everything after building a new server and moving the drives. I saw speeds in the same overall average of about ~130MB/s, with peaks up to 160MB/s for a while. I have the drives spread across the builtin nForce4 controller and two Silicon Image 3132 PCI Express controllers.
-BK
Thanks for the info BlkKnight, good to know I'm getting about the same as others. I've got a Syba Silcon Image based 3124 4xSata2.
The 133MB/s sounds like a pci limit that I hit, but I ran another test after rearranging my drives, and got it up to 140MB/s (even with parity drive on a remote gigabit net drive) , which is about 6 hrs @ 2.6 TB and that I can live with, start the backup at night, go to sleep, wake up all safe and protected :)
Next, sell off 500GB drives, upgrade to 1 / 1.5 TB drives... yikes.
You must log in to post.