disParity

disParity » General Discussion

1. cartman
   Member

   Hello, a few days ago i became aware of this program so i would like to thank the author for making a tool that could be useful to many people with home media servers.

   After testing the program on a few folders, I figured that there is a fundamental problem with the current implementation: the parity data will only guarantee recovery if the data on the other disks is not touched after an update. If for example 10GB is changed on a disk and then another disk fails, then at least 10GB of that disk's data will not be recoverable. Depending on the files that are changed and the algorithm of the program, that number could increase a lot.

   So I think the program needs to have a constant monitoring and update function, maybe evolving into a driver or something, that gets block-level access to disks if possible. A deamon that intercepts everything that is written to a disk and quietly writes parity data would be perfect, ensuring that data is recoverable 100% of the time. There would probably be something like a 50% performance hit, as the program would have to read the parity disk and calculate and write the new parity data, but i think it would be worth it(if 1TB disks write at 80MB/sec and we can get 40MB/sec parity-protected writes i would be extremely happy).

   As someone that has just probably lost a full 1 TB of data from my 8 TB linux file server, I would be very eager to help in testing and nagging for new features :P

   I still have not tested it with samba shares, does anybody have experience with that? Also with what amount of data have you tested? When i get the failed disk from RMA it will become the new parity disk, so how fast would it be with 7-8TB of data over a gigabit network?

   Of course the ideal thing for linux users would be a kernel module that does the job, but because i write to my file server from only one windows pc, i wont have a problem running the parity protection from the windows pc.

   Posted 6 years ago #

2. Roland
   Roland

   A real time low-level driver-based approach like you described would be great, but I think it is beyond the scope of a utility like disParity. If you really need real-time protection like that right now, you should probably consider a commercial product like unRaid. The whole point of a lightweight software utility like disParity is that it is intended for media servers where the file data changes relatively rarely, and the most frequent type of change is the addition of new files, which don't threaten the integrity of the snapshot.

   I've talked about this briefly before in another thread, but I think the farthest disParity will probably ever go is a "psuedo real time" mode where it watches disks for changes, and automatically updates the parity shortly after files are changed. This "lazy update" approach does mean there is a small window after a change where there is a chance of losing some data if a disk were to crash, but I think that small level of exposure is acceptable for most people using a utility like this one.

   I think I mentioned this in the readme, but really crucial data (e.g. your photos and music collection) should always be backed up some other way than just with a parity snapshot utility. Even with perfect updates, it can't protect you against things like failure of more than 1 disk at a time. I use mirroring (complete copy of every file, updated daily) for my crucial data, things like the contents of my main PC's drive (which has my documents, email, etc.) Even that's really not good enough. For the really critical stuff I think you need to go one step beyond mirroring and look into offsite internet backup services, because no matter how great your local backups are, they're still no good in the face of a true disaster, like a fire.

   Samba: haven't tried it myself, but as long as the drive is mounted as a normal network drive, I assume it should work fine. Note that use of network drives will usually slow down disParity, because the LAN becomes a bottleneck. The theoretical maximum throughput of a gigabit network I think is about 100MB/sec, although in practice I haven't been able to get mine above about 60MB/sec (could be any number of reasons for that, though.) If you are both reading the data to be protected off the LAN, and writing the parity back out over the LAN to another network drive, that would be the slowest possible configuration. I wouldn't expect to get more than maybe 20-30 MB/sec continuous over the whole snapshot process in that case. Of course these are all wild guesses. YMMV.

   If your media server runs linux and you would rather run your parity protection on the linux machine, you could also check out flexRAID, which I believe supports linux natively. I was orignally using flexRAID myself, but development and support for flexRAID has been spotty over the last few months, which is why I wrote disParity in the first place.

   Personally I am using disParity to protect about 2.6TB of data spread across 7 drives. I'm not sure what other people are doing. It takes about 6 hours for a full snapshot to run on my server. Updates only take a few minutes at most.

   Posted 6 years ago #

3. cartman
   Member

   Something like lazy update would be fine for me, my server usually has writes and deletes of a few multi-gigabyte files. In fact lazy update could be an advantage if it worked only when there is no disk activity, eliminating any slowdowns that the parity generation would cause.

   About samba, i have got 110MB/sec reads or writes when testing from multiple disks on the windows pc to multiple disks on the server. As the net is full duplex i do not expect significant slow downs, although i have not tested anything more than 50mb/sec bidirectionally. In my case, data would have to be read from 7 disks at the same time and written to one, so reads would be done at 110/7=16MB/sec per disk, so i would presume that the parity would be written at 16MB/sec, so 1TB would take something like 17hours, which i find acceptable. Maybe a cheap second gigabit ethernet pci card in the server would help halve that time if i do bonding.

   Posted 6 years ago #

4. cartman
   Member

   I have sent the disk to RMA, and waiting for 2 more WD10EACS 1TB. So in a few days i will hope to start testing with samba and parity over 1 or 2 gb ethernet. PS. Does parity have to be recalculated if i add an empty drive?

   Posted 6 years ago #

5. Roland
   Roland

   Any time you change the setup in config.txt (add a new drive, remove a drive, or change one of the paths) you need to rebuild the snapshot.

   Posted 6 years ago #

6. cartman
   Member

   Yes, i understand that. But does the rebuilding start from scratch, discarding the previous snapshot?

   Posted 6 years ago #

7. Roland
   Roland

   Yes, that's what I meant by "rebuild". You have to do a "disparity create" command after changing the config. This deletes the old snapshot and replaces it with a new one. You can't do an "update" after adding a new drive, it won't let you.

   It's not ideal, of course, and supporting the addition of a new drive without having to rebuild the entire snapshot is on my list.

   Posted 6 years ago #

8. cartman
   Member

   I finally got to test disparity on my samba shares. It works, but has a very big problem: Veeeery slow parity writes, under 1MByte/sec. I even tried having as source drives 2 local disks from my windows computer and only having the parity drive on samba, and still writes were incredibly slow. When the parity drive is local then everything is much faster, tens of MBytes/sec, even when the source drives are from samba (so reads are fast, although i am not 100% sure that i did that test- i will check when i go home later).

   Of course i have tested reading and writing to the parity drive over samba and it always gets over 50Mbyte/sec, so the problem seems to be with the way your program accesses samba shares.

   Posted 6 years ago #

9. cartman
   Member

   PS Tomorrow i will try to test your program with WINE, so that it runs on the linux machine

   Posted 6 years ago #

10. Roland
    Roland

    Very strange! DisParity writes the parity files using normal, standard, completely uninteresting vanilla Windows .NET calls. It doesn't do anything special for network drives, or samba shares. It just creates the file wherever config.txt says it should go and writes to it. Can't imagine what it is about disparity that would cause writing to a samba drive to be so slow. Perhaps the .NET framework I/O classes have a problem with samba? Who knows.

    I have no linux system set up here so no way to test samba. :(

    Would appreciate hearing about any other clues you can discover about this mystery.

    Posted 6 years ago #

11. cartman
    Member

    Lets see... writing was done in bursts, getting tens of megabytes written at once and then much slower (but without stopping), then 5-10 seconds later a burst again etc. I will do more testing in a few hours and report tomorrow

    Do you have any windows machines on your local network? you could map a shared drive from it and see what happens.

    Posted 6 years ago #

12. cartman
    Member

    ah, i also remember that when the parity file is created it starts with a burst, of about 30mbytes

    Posted 6 years ago #

13. Roland
    Roland

    Yes I have several Windows machines on my LAN.

    I just tried a test config of backing up two local drives with a network drive for the parity. I'm getting 20-25 MB/sec writing continuous to the parity drive over the network. I'm not seeing the "bursty" behavior you are describing. Very strange indeed.

    Posted 6 years ago #

14. Fernando
    Member

    How do you measure writing speed?

    Posted 6 years ago #

15. Roland
    Roland

    Well in this particular case, since I was writing to a network drive, I simply opened Windows Task Manager and switched to the Networking tab and watched the bits fly.

    If it were a local drive it wouldn't be that easy of course.

    Posted 6 years ago #

16. Fernando
    Member

    I see. I thought there was a software that let you know speeds for each drive. It would be nice!

    Posted 6 years ago #

17. cartman
    Member

    some results after testing:
    1) Confirmed that only writes to the samba share are slow, reads are ok
    2) writes happen at 1-2Mbyte/sec for 3 seconds, then burst to 9-10mbytes/sec for 1 second, then get back to 1-2mbytes etc...
    3) Linux has iostat that shows all io that is done to each drive. For windows i use task manager too. You can add I/O read and IO write bytes columns to the process tab.
    4) no amount of tweaking with samba settings, ethernet MTU values changed this.
    5) Is there any tool that i can run while running disparity that can generate a log useful to you? something like http://technet.microsoft.com/en-us/sysinternals/bb896645.aspx process monitor ?

    Posted 6 years ago #

18. Roland
    Roland

    I think that realistically the only way I could get to the bottom of this is by setting up a linux machine here, installing samba, and debugging this myself. I'll be honest and say this isn't really a high priority for me right now, I think that having the parity disk on a remote linux machine is going to be a pretty unusual configuration for most users of disParity. I only have a couple hours a day at most to work on disParity and I still have a bunch of other things I'd like to do first.

    Posted 6 years ago #

19. Roland
    Roland

    Cartman, can you test write performance for the "update" command? Update uses a slightly different algorithm for writing to parity. So create a snapshot, then add a big file (e.g. 1GB) to one of the drives, and then do an "update". I'm curious to see if you get any different performance. Thanks!

    Posted 6 years ago #

20. cartman
    Member

    Ok, i will test that when i get back home. I've been trying to setup wine+dotnet on linux, but i have some problems with free space as the OS is setup on a 1 gb usb stick. kinda funny when you have free space problems on a 10TB machine!

    Posted 6 years ago #

RSS feed for this topic

Reply »

You must log in to post.