disParity

disParity » General Discussion

21. cartman
    Member

    Results from extensive testing:

    1)Installed wine and mono(for .net compatibility) on linux.
    disparity runs ok, parity writing is done at 6-7MByte/sec when reading from 8 disks. Seems cpu limited, as the mono process takes 95% of cpu. overclocking the cpu improves write speeds. i have a hunch that i could be memory bandwidth limited too, as my cpu is a sempron 754@1.6GHz single channel ddr1 and small memory overclocks increased parity creation speed. Obviously mono is the major reason for slowdowns.

    Note: disparity throws a message about not being able to calculate free space on the parity drive, but it continues, so if you make any changes on this please leave a switch that overrides low free space warnings.

    2)I am now removing mono and installing .net2 from ms. Lets hope this runs disparity faster

    3)On the windows machine now, i ran disparity along with process monitor. Every write to the samba parity drive returned a FAST IO DISALLOWED message and was as always quite slow.

    4)Then i did the update command, and everything was as slow as before, with the FAST IO DISALLOWED messages showing up.
    BUT: i enabled 256kb write cache to samba and retried. then update speeds skyrocketed to 20-25mb/sec, and no FAST IO DISALLOWED messages appeared! when i tried a normal create, speeds were again slow, followed with the FAST IO DISALLOWED messages.

    So your slightly different algorithm for updates seems to work well with samba if write cache is enabled. Also the FAST IO DISALLOWED message seems connected to the slow speeds issue. If you want i can send you the process monitor logs for analysis.
    Obviously i will continue to tweak samba to see if i can improve performance

    For my situation:
    a)i am very happy that disparity runs under wine, and will try to see if dotnet from ms is faster than mono. even with current speeds @ 7mbyte/sec i am willing to use disparity under wine. I suspect that i wont be alone in that, especialy if write speeds get over 10-15Mbyte/sec.
    so please please please don't break compatibility with wine :D

    b)small feature request: a switch that disables logging, or makes it less verbose. my linux is installed on a flash drive, and i try to reduce logging etc as too many writes can kill a flash drive.

    c)a final small request: when improving the program in the future, please keep the command line functionality equal to a possible windowed version, so we text mode people can continue to enjoy your program :D

    Posted 6 years ago #

22. cartman
    Member

    At last, success! After installing dotnetfx20 on the linux machine, disparity now creates at 13.2Mbyte/sec, reading from 8 disks. It is now working ok for 8 hours.

    I seem to be limited by system bandwidth for this, as overclocking the cpu or memory does not yield any speed increases. Probably the sata controller is connected via pci to the chipset. When this finishes i will run a recovery test and an update to verify that everything runs ok. A final test would be to move everything to another motherboard with dual core and a better controller to see if i get any speed increases.

    Using dotnet disparity can check the free space of a disk, so you can ignore part 1 of my previous post. As it is now, if everything runs ok then i will keep disparity on the linux machine. if you want i can write a small guide to help others run this excellent program under linux.

    Posted 6 years ago #

23. cartman
    Member

    disparity create finished, at about 100mb/sec reported speed (18 hours for about 6TB of data, or 14MB/sec parity writing speed). update works ok, reading from the disk that has the new data and reading and writing to the parity disk at 25mb/sec. Deletes, renames, moves and changed files were handled correctly.

    I am now running a verify, which runs faster than the create, and if it turns out ok then i will be able to say that disparity runs perfectly under linux 2.6.27 and wine 1.0.9 , using .net 2.0.

    a small issue: when encountering files that have huge paths and names of over 256 characters, if disparity is on create mode they are effectively ignored. But on update mode they are counted as new files, and disparity stops throwing an error when reading them. I think that it would be better if that error was thrown when reading the file list from the disk, on all modes.

    and finally another feature request: resume mode. creates and big updates take especially long, and the ability to resume would be a huge timesaver.

    Posted 6 years ago #

24. Roland
    Roland

    Thanks for all the detailed testing Cartman this is great stuff.

    The 256+ character path issue is interesting, I'll have to test that. Paths that long are not valid under Windows so the .NET routines are probably doing unexpected things; the disParity code itself doesn't check path length or have any explicit 256 character limit. Actually Windows does support paths longer than 256 characters but they have to be specified differently by the application, so I think applications must be explicitly written to handle such paths. I'll look into that more. *Very* odd though that create ignores them but update picks them up: the file discovery code is the same for both.

    resume mode: could you elaborate on this a bit more, I'm not sure I understand. What are you resuming from exactly? How did the previous create or update get stopped?

    In other news, I added a new drive to my media server yesterday and did my first complete parity wipe/recreate last night in over a week. 3.12 TB in 6.8 hours (133 MB/sec.)

    Posted 6 years ago #

25. cartman
    Member

    more results from testing:

    disparity verify after working without problem for hours, gives a message that a parity block is problematic (cant remember the exact wording) at byte 0. This happens for 50-100 parity blocks, except for one that has the problem at byte 1. Then it goes out and verifies the hash for every file involved in the block, and the verification is successful every time. I checked my system logs and no bad sectors or read errors or any other error has occurred.

    This process takes a lot of time, because for every block that has a problem it checks the whole file that is involved in it - not just the part that is in the parity block. So the same file gets checked again and again and again, and in my case the file was 15GBytes, so after some time i stopped the process.

    Now i am running a disparity test for the first drive.

    Another feature request: a hash checking mode, that just checks every file against its stored hash value.

    about the resume mode: the create was stopped by an accidental kill of the disparity process at about 90%. also on the update, the process was stopped by the 256 char error. before that many files were added to the parity, and when i renamed the problematic files and restarted the update, the same files were added again. I hope i am wording this in a way that you can understand (English is not my first language), but the gist of it is that the same files are added to the parity every time the update process runs, until the process finishes without an error.

    more feature requests :D automatic naming of the log file by date or something, so that it does not get overwritten every time.

    Posted 6 years ago #

26. cartman
    Member

    My disparity test failed and gave me errors for 3 files, so the errors on disparity verify were not flukes. This could have been caused by me and my tinkering with the parity files, so I redid the disparity create and verify cycle, and everything went fine without errors. Tonight i will add files, do an update and then a verify to see if everything stays ok.

    Posted 6 years ago #

27. Roland
    Roland

    Cartman, I suspect the verify/test errors you were getting were due to the earlier update you mentioned that had aborted without finishing due to the 256+ character path. If an update dies for any reason before completing, the parity data is going to be in an indeterminate state. If you run the update again after that, it will almost certainly corrupt the parity (e.g. for any adds, it would add the same file again, a 2nd time, which would corrupt the parity.) So bottom line, "resume" after a halted create or update isn't supported, you must re-create the snapshot if this happens.

    I'd like to support resume, but it's really tricky, especially if it needs to support resuming from the process being killed externally. For example, what if it's killed when it's halfway through writing a parity block, and if so, how do you recover and resume from that? The entire application would need to be restructured such that every time the parity state is changed, the operation is atomic and can be recovered. That level of fault tolerance might be a little bit beyond the scope of a free utility like disParity.

    I'll add a fix to the next release so that it doesn't keep re-computing the hash for the same file over and over if there is a failure during verify.

    Posted 6 years ago #

28. cartman
    Member

    Maybe a transaction log could help you with the resume issue, with every block number touched during an update logged after the write is complete. Furthermore, stuff like problematic names and paths should be checked before parity is being written, so errors like that do not corrupt the parity. Of course i am not a programmer so all i can do is suggest stuff :D

    My update yesterday went well, i will run a new verify today.

    so, to sum up my feature requests:

    1) A mode that checks files on a disk by hash only.
    2) A list mode, that lists every file of a disk protected by disparity, possibly together with its hash and size
    3) Automatic naming of the log file by date or something, so that it does not get overwritten every time.
    4) Resume :D or at least protection from corrupting the snapshot
    5) BIG warnings that say "DO NOT CHANGE FILES WHILE DISPARITY IS RUNNING!!!"

    Posted 6 years ago #

29. Roland
    Roland

    I finally got around to testing the 256+ character file name problem you reported, and found that I can't reproduce this problem. It's hard because Windows won't even let you create a file with a name that long. I had to trick it by first creating a file with a very long name, then renaming the parent directory to have a really long name as well.

    What I found in this case is that disParity ignores the file on both the create and update pass. I couldn't repro the bug you reported where it ignored it on create but then picked it up on update. As I expected, the .NET file access routines behave strangely when I try to query info about the file. Some values return gibberish and others throw exceptions. The upshot is that the code skips the file in all cases that I could test.

    Is it possible that the bug you observed only happens with samba shares?

    Posted 6 years ago #

30. cartman
    Member

    That happened when disparity was running on wine under linux, using the official ms .net libraries. If you couldn't recreate the problem on windows that means that it is a linux problem, probably because of filesystem differences (linux filesystems like jfs do not have 256 character limits), so you probably shouldn't be too worried about it. Btw when i tried to access those files using samba windows was acting strangely, didn't let me rename the files and folders, had problems with file sizes etc.

    Yesterday a bigger problem showed up. My configuration has 9 1TB disks for data and 1 1TB disk for parity(they all have exactly the same capacity). The most filled up disk has about 2gb free space, so the parity drive has a few hundred megabytes less.

    So i copied many files to my data drives, deleted some, plus some moves on the same disk, and managed to get over 4gb of free space on the most filled up disk. That could be described as a normal day of adding movies and reorganizing stuff.

    When i ran an update, disparity asked for 25GB of free space on the parity drive, when it had only 2, and the result was that the update failed. So it asked for about 23GB more than the largest drive on the system. Is this something expected or a bug? I guess that this issue has to do with moves, as previous updates that did not have moves did fine.

    Posted 6 years ago #

31. Roland
    Roland

    Yes I'm guessing you probably ran into the "parity fragmentation" problem I think I briefly mentioned in another thread. As you delete files, it leaves "holes" in the parity data for that drive, where those files had been stored. As you add new files, it attempts to re-use the existing holes as best it can. But if the new files don't fit well into the old holes, it will wind up leaving gaps. If you are mostly dealing with very large files (e.g. multi-GB files) the unused gaps could be quite large, and add up to a lot of wasted space. So for example if you delete 3 files of sizes 4 GB, 5 GB, and 6 GB, and then add a new 7 GB file to the drive, disParity won't be able to re-use any of the existing holes and will have to tack it on to the end and use more space, even though the deleted files free'd up 15GB of parity.

    The only solutions I can see to this problem are either a "party defragmentation" mode where it actually moves files around in order to close all the gaps, or I could enhance disParity so that it can store files non-contiguously in the parity data.

    I should mention that moves (within the same drive) won't cause fragmentation, because a move only causes a change in the file meta data; no parity data changes as a result of a move. Moves across drives will cause it, because those are handled as a delete on one drive (which leaves a gap) followed by an add on the other drive.

    Posted 6 years ago #

32. cartman
    Member

    Then i guess i will need to recreate parity. Of course the non-contiguous storage of parity data would be better, but i don't know if that creates unexpected problems.

    Posted 6 years ago #

33. Roland
    Roland

    Yes re-creating is really your only option in this case.

    The only real problem caused by non-contiguous storage is increased code complexity and therefore increased possibility of bugs, etc. There's also a potential I suppose for decreased performance if your files get really jumbled up after a lot of deletes and adds...the same way your HD can slow down if it gets really fragmented. But I don't think this effect would be very noticeable.

    Posted 6 years ago #

RSS feed for this topic

Reply

You must log in to post.