disParity

disParity » General Discussion

1. Xyvotha
   Member

   Hi,

   I have a somewhat big (12TB) library stored in a archive server. This data grows periodically but is mostly not modified.
   I wanted a cheap offline/offsite copy of the data and decided to use 3TB HDDs. The only thing that concerns me is data rot, that's how I landed in disParity's page ;)

   Question is: With disParity could I attach each data HDD to a machine, make parity "snapshots" of to a dedicated (online) drive then disconnect the data HDD to be stored it in a closet?

   I hope I made myself clear, english is not my first language.

   TIA!

   Posted 2 years ago #

2. Roland
   Roland

   I don't think disParity will work for what you want to do, because it expects all the drives to be accessible when it updates the backup. So you would need a way to connect all of the 3TB drives to the machine at once in order to make the snapshot. After that you'd be free of course to disconnect everything and put it away, but you'd have to reconnect it all again every time you wanted to update.

   Posted 2 years ago #

3. Xyvotha
   Member

   Thanks for the answer.
   I can't believe in 2013 the only option for secure backup is still tape, no cheap option for the rest of us :(

   Posted 2 years ago #

4. Roland
   Roland

   I should have mentioned that even if there was a way to add drives to parity one at a time while the others are disconnected (which is theoretically possible to implement) there would still be no way to recover a lost drive, since the recovery process requires access to all drives simultaneously.

   It sounds like you are talking about only 4 or 5 3TB drives that need backing up. If you are contemplating tape as a backup, you should also consider just building a cheap generic wintel box and putting all the drives in that box at once so they can be protected using a solution like disParity. That should only cost a few hundred dollars, and you can power down the machine (and even put it away) when you don't need access to the backup. That's what I would do anyway. Good luck!

   Posted 2 years ago #

5. Xyvotha
   Member

   Yeah, it seems that's the way i should go. Thanks again for the advice!

   Posted 2 years ago #

6. Xyvotha
   Member

   Here's another scenario: In the scheme you suggest I'm limited by available SATA ports in the MB (no multiport controllers in my country :/).

   I was thinking on making "disParity sets", like this: if I have only 5 SATA ports available I could set 4 HDDs for data and 1 for parity, calling it "set 1". Once all data drives are filled I just disconnect them and put in a drawer. Then I get another 5 HDDs and build "set 2".

   The question is: where does disParity store the array configuration? Is it possible to save that config file somewhere else before building a fresh set (thus creating a new config)?

   I hope my explanation is clear :S

   Thanks!

   Posted 2 years ago #

7. Roland
   Roland

   Yes, that should work. As you say you would just have to swap out the config files each time. Under Vista and later, the disParity config file is here:

   C:\Users\(username)\AppData\Local\disParity\Config.xml

   Everything else is stored on the parity drive itself.

   Posted 2 years ago #

8. Xyvotha
   Member

   That's great, thanks!

   Posted 2 years ago #

9. bgeery
   Member

   FlexRAID has a Multi-Step Parity function to create/restore data just for this use-case. It has you swap the drives in one by one (or in batches) as it builds or restores from the parity data. See http://wiki.flexraid.com/2011/03/24/multi-step-parity/ for an explanation.

   I wish disParity and SnapRAID would add these feature.

   You should also run SpinRite on the drives every year or so at level 2 to insure all the sectors remain readable over time.

   Posted 2 years ago #

10. Roland
    Roland

    Interesting feature! I see how it could be used to make a single static snapshot of read-only media like DVDs easily enough, but I don't understand how it could be used to do snapshot updates in the case of multiple large external USB drives that are not all plugged in at once. Every time you did an update, if there were any deletes you would have to be constantly plugging/un-plugging the various drives as disParity needs access to them in order to reconstruct the parity data. It seems like it would be an enormous hassle, but maybe I'm missing something.

    Posted 2 years ago #

11. bgeery
    Member

    Yes, you need to swap all the media in, either hard drives or disks, any time your create/restore from the backup.

    However, in the case only one disk (or a few) have changed, you are able to remove the changed drive from the backup (multi-step-remove feature noted in the wiki) and then should be able to just add it back into the backup. This I believe only then involves the drive you are adding/removing and the parity drive. I could ask over at the flexRAID forums for more information if this doesn't sound right to you.

    Posted 2 years ago #

12. Roland
    Roland

    Yes, if a disk is still accessible, it's possible to remove it from the backup without reading all the other drives (disParity does this now if you right-click on a drive and choose "Remove.") It's only when the data is NOT accessible that you need access to the other drives.

    So the problem occurs when the changes you want to process include edited or deleted files. In that case, the old file data is no longer available to do a quick XOR with parity, so the block must be reconstructed by reading all the other drives, even if you are now removing the entire drive at once. I'm not sure how flexRAID would handle that case.

    Posted 2 years ago #

13. bgeery
    Member

    Yes, I see your point, and the parity data corruption caused in your example. If I edit the data on a drive that fails, I can still restore the data to the point where I last took the parity snapshot. If I edit data but instead another of the drives fails, I will not be able to recreate all the data from the failed drive.

    Isn't this exactly the same situation that I would have if all the drives had been online together? You should not change the files that are part of a parity snapshot in either case (without first removing.) Or an I confused?

    The use case I had in mind (and I think Brahim had in mind when he implemented the feature) was the user should not edit or delete the files once added, and if this must be done, use the multi-step-remove feature to first remove those drives *before* editing or deleting any files from the drives, then re-add them after making any changes. Adding files can be done at any time, periodically updating the parity data to include the newly added files. This should only involve the currently "in use" data drive and the parity drive.

    In the case of needing to truly restore a drive from parity using multi-step-parity, you need at least two connections available on the PC. One for the replacement data drive, and one to swap in the old parity/data drives to restore from.

    A lot of businesses archive data files to shelved hard drives, and they have no cost-efficient way of protecting that data (other than flexRAID, if they knew about it.) I currently have my media archived to 20 drives sitting on a shelf. I have a 3-in-2 hot-swap bay on the PC that I use to swap and read drives as needed. My current protection procedure is running the drives through SpinRite level 2 scans every six months to combat bit-rot, but this does nothing for true drive failure. I do purchase at least two identical drives, so I can at least swap electronic boards if needed, but I'm still left open to damage to the mechanical parts or internal electronics.

    Posted 2 years ago #

14. Roland
    Roland

    Yes, all parity snapshot solutions are subject to the possibility of data loss if data has been modified but the snapshot hasn't been updated. In the case of added files, you only risk losing the files just added, but in the case of edited or deleted files, you risk losing data on another drive if it fails (although I believe SnapRAID has the added ability to protect against this risk by using an optional second parity drive.)

    In my example I wasn't trying to point out this risk, which as you say applies whether or not the drives are all currently online. My point was only that, in order to update the snapshot, if data has been removed, then access to all of the over drives will be required, which could be quite a hassle if you have to be constantly swapping drives in/out during the update. It's also true that, as you say, if the only changes are adding of new data, then you will not need access to all of the drives.

    Anyway, I certainly appreciate the need for a proper solution for the multiple offline hard drive case, it's just that this scenario is fundamentally different in several ways from the one disParity was designed to solve. The core parity algorithms in disParity could be used to implement such a mode, but so many other key aspects of the program's logic flow would have to be modified, to the point where it almost becomes a new app.

    Posted 2 years ago #

RSS feed for this topic

Reply

You must log in to post.