disParity

disParity » Announcements

1. Roland
   Roland

   Not sure if anyone is even reading this forum anymore, but I wanted to announce that I've had more free time lately so after a long hiatus I've restarted development of disParity.

   I started with a complete re-write of the core engine. Now my principle focus is a proper UI, with a smattering of other oft-requested features thrown in, like an "ignore" filter for certain file names. Command line use will still be supported, but the features available from the command line will probably be restricted to the basics (create, update, recover.)

   The new codebase is much more complex than the old, so I expect there will be an extended beta test before it becomes the new "official" version of disParity.

   Watch this space for more updates.

   Posted 3 years ago #

2. Phatty2x4
   Member

   good to hear.

   Posted 3 years ago #

3. freehand
   Member

   Looking forward to it!

   Posted 3 years ago #

4. Neostim
   Member

   Hot Diggity Dog! More updates to my favourite utility :)

   Posted 3 years ago #

5. rust0r
   Member

   Great to hear! I still can't believe with how many storage forums I read, NO ONE knows about this amazing utility! You should make a formal write up of it's features and perks and post it on main forums such as Hardforum, Overclockers, etc. I know there are tons of people running Unraid over there, this is better in my opinion, it takes the "guess" work out of recovering/rebuilding failed arrays which is a huge risk for the average user.

   Thanks again for all your hard work

   Posted 3 years ago #

6. Cybie257
   Member

   It's been a while since I've visited. I'm happy to see that you are working to improve such a useful, functional tool. I've yet to lose 1 kb of data with disParity, while I've lost gigabytes with RAID 5 failures (Corrupting my photos and videos from years past. UGH!) Anyhow, I look forward to seeing the future of disParity. :)

   -Cybie

   Posted 3 years ago #

7. Porcus
   Member

   New forum user, registered to weigh in on a few thoughts. Might make a fool out of myself, but taking the risk:

   - ignore filter is a good thing; please make sure we can ignore entire folders (right now I am applying the system flag ... not the best solution)

   - less approved from my side, is dropping features from the CLI.
   Likely, the better approach is to restrict GUI to basic usage. Point-and-click system administration is not a good thing at all, it isn't scriptable except using window handles (unnecessary and risky). And, if first you want advanced usage, then you should anyway be willing to read documentation. Better practice is for a GUI to be a wrapper for the real thing.

   - it seems to me that I cannot protect a full 2TB using a 2TB ... there is a little bit overhead. Any optimization that could be done?
   (One thing: compression? Likely to take prohibitively long time if applied in general, but what about an option to do so on files/folders with the compression attribute?)

   - some utilities have a _general_ 'dryrun' option -- that just simulates _whatever_ job you set it to, without writing anything to physical drive. Useful for example if you do not know whether your last changes will overfill the parity drive.

   - feature request: a 'recompress' feature that verifies and frees up space for deleted files. (Why Verification? I just suppose that one will do about the same job in the process anyway.)

   - And re the previous point ... and also for update: I have yet to have troubles with this, but is behind the scenes the update fully transactional, to the best of the file system's abilities? In the sense that new file is written side-by-side with old, and only then, change is committed? Would require a gigabyte free on the drive though.

   - is there any way to 'lock' a drive from being written to during the process? There could be such ways internally in Windows I think, used by chkdsk.

   Posted 3 years ago #

8. Roland
   Roland

   Lots of good ideas here, I'll try to respond to some of them:

   "dropping features from the CLI" - not really planning to *drop* any CLI features, except perhaps for some of the obscure "information only" commands like stats. It's the new features I'm planning to add (e.g. recovering individual folders or files) that would require complex, unwieldy command line syntax that I really don't feel like implementing, especially when 99% of the users would just use the UI for that anyway.

   "overhead" - the only way to eliminate the 32K-per-file overhead would be to change the parity algorithm to not be based on blocks anymore. That would pretty much be a complete rewrite, and would complicate the algorithm in many other ways. Also, the parity data is usually uncompressible. As the XOR'd contents of multiple files, it is effectively random (try compressing one of the parityXXX.dat files with a utility like WinRAR and see if you have any luck!)

   "dry run option" - yes, that would be nice

   "recompress" - I've thought about that. It's really more like defragmenting than recompressing. I would like to allow files to be stored non-continuously in the parity, which would solve most of this problem.

   "transactional update " - sort of. That's the purpose of the temporary parity. As each add or delete is processed, the modified parity is stored to a temporary buffer (in memory for smaller files, on disk for larger files). The modified parity is not copied to the real parity until the entire file has been processed. This minimizes risk to the parity if one of the data drives should fail during the update, however if something were to go wrong during the flush, that would still corrupt things.

   "locking entire drives" - there might be ways to do this, not sure. Sounds risky (it might cause other apps to crash), and I don't think it's necessary. I would rather tune the update algorithm to be able to handle changes to the drive while the update is running.

   Posted 3 years ago #

9. Klaatou
   Member

   Hello !
   As I don't "remember" each time not to touch the drives during update ... I like the idea to allow the update algorithm to be able to handle changes to the drive while the update is running :)

   A little change I could ask is something I saw during testing a disk : empty files are flagged as "...Verify FAILED!" not a big deal but I have some empty files used as "titles" or comment for some directories and it is always a "stress" to see a ...Verify FAILED! ;)

   Thanx

   Posted 3 years ago #

10. Porcus
    Member

    Oh, another feature request (if it can be done already apart from a spanned volume where I have no control over what's written where, I'd like to know):
    Start on a 2nd parity disk when the first is full. « parity = p:\ then q:\ » or something like that. Or, maybe one way to solve this: start using subfolders. If I use p:\ for parity, I can create an empty p:\_parity1\ folder and mount another drive there.

    Overhead: my two 2TB drives are now even at about 1 992 700 (decimal) megabytes. Making a fresh create, marking the .dat files and asking Windows for properties, I get about 1 997 000. So there's 6.something GB overhead ... somewhere.

    Yeah, parity files are not very compressible. That's why any compression has to be done by the software. But I guess that takes too long.

    I would rather tune the update algorithm to be able to handle changes to the drive while the update is running.

    That was the purpose of the question, so ... whatever works is welcome.

    Posted 3 years ago #

11. Roland
    Roland

    The 6+ GB overhead after a fresh create is strange, I can't explain that. How many files are on your 2TB drives?

    -Roland

    ps. I would like to be able to support multiple parity drives someday

    Posted 3 years ago #

12. Porcus
    Member

    The 6+ GB overhead after a fresh create is strange, I can't explain that. How many files are on your 2TB drives?

    100 000-ish. Slightly more than my c: disk.

    Well I found a possible issue though: one subfolder was a Windows compressed folder, and that seems to have saved 2 GB. At least that is the difference between "Size" and "Size on disk" under Windows "Properties" (Alt+Enter). But the maximum number is still 1 992 700, so if they are reliable, it is still six point something.

    There is nothing in the Recycle Bin, and the parity drive has no reserved space for it.

    Posted 3 years ago #

13. Cool1Net6
    Member

    As the creator of one of the GUI's for disParity, I guess my services will no longer be needed when this new version comes out. :( But I'm happy to see progress on this tool. I'd love to be apart of the beta testing group, if possible. :)

    Posted 3 years ago #

14. Roland
    Roland

    100K files * 32K overhead per file = a bit over 2GB, so that accounts for at least some of the overhead.

    Posted 3 years ago #

15. Porcus
    Member

    Here's something I think you should fix: I tried to overfill the parity drive (yeah, curious ... I would have to rebuild it anyway after all the fiddling around) and then the file1.dat was destroyed – likely because there was no space left to write to it after the last parity file expanded and filled the drive.

    I suppose that this move destroyed the entire protection, and that is certainly not a good thing. Maybe you should consider a backup folder for those files (even on another drive, for example C:\ ).

    Posted 3 years ago #

16. Roland
    Roland

    Yeah, losing a fileX.dat file is bad. It should have created a backup first though before writing out the new one...is there a files1.dat.BAK file on the parity drive?

    If there is one, the program won't automatically recover the .BAK file, but you can try renaming it back to files1.dat again.

    But yeah, in general, a full parity drive is probably not handled as well as it could be. It does try to prevent an update from starting if it thinks the parity drive doesn't have enough space, but that may not be 100% bulletproof.

    Posted 3 years ago #

17. cybrsage
    Member

    Any ETA on the release of the new version? I am about to try out disparity for the first time and will wait for this release if it is to be soon.

    Posted 3 years ago #

18. Roland
    Roland

    I would go ahead and use the existing version for now. It's stable and well tested. New version is still a ways off.

    Posted 2 years ago #

19. cybrsage
    Member

    One thing I noticed is that if part of the restore fails, the restore stops right there and does not continue. Will that be fixed in the next version? Also, is there a way to restore selected files?

    I did a test by creating a folder on one of my drives, putting a txt file in it, then refreshing the parity to add the new file and folder. The action went fast, as expected.

    I then deleted the folder, including the file, and went to restore it. The only thing I could figure out to do was to restore the entire drive. No biggie, I though, I am only missing a few bytes of data so it will be fast. Nope, hours later it failed:

    Recovering Wrath of the Titans\WRATH_OF_THE_TITANS.iso...Hash verified
    Recovering Christmas Movies\White Christmas\mymovies-back.jpg...Fatal error encountered during recover: Access to the path 'N:\Christmas Movies\White Christmas\mymovies-back.jpg' is denied.
    Stack trace: at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
    at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath)
    at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy)
    at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access)
    at Disparity.Recover.RecoverFile(FileRecord f, String dir, Boolean testOnly)
    at Disparity.Recover.RecoverDrive(Int32 drive, String dir, Boolean testOnly)
    at Disparity.Program.Main(String[] args)

    It stopped there due to the error. Unfortunately, that meant it did not recover my newly added file and folder - and I assume it would not have recovered anything past that point had I actually needed it to do so!

    Posted 2 years ago #

20. Porcus
    Member

    is there a files1.dat.BAK file on the parity drive?

    Realize now that I might have sorted files by type, and therefore overlooked it. Reformatted and rebuilt now, so I cannot know.

    Posted 2 years ago #

RSS feed for this topic

Reply »

You must log in to post.