disParity » General Discussion

Added 500GB of files, not all files are actually being added, some even removed?

(32 posts)
1 2

  1. rust0r
    Member

    First off, I just want to thank you Rolland. My good friend Neostim has been trying to get me to make the switch from my RAID5 setup over to disparity for quite some time now. I recently built a home file server for all of my data (mostly movies, bluray, tv shows, etc, about 6TB currently and soon to expand). For my server I had attempted numerous setups with RAID (freenas, ubuntu, etc), all of which were frustrating and proved error prone when simulating fails/recoveries.

    Disparity is by far much easier/reassuring in event of failure/recovery!
    I apologize for the topic title, I really was not sure how to describe it in such few words.

    My question related to a problem I am having with the addition of new files. All of my drives are 1TB, including the parity drive. From my understanding the parity drive is suppose to be the same size as the fullest drive (in terms of space used).

    Everything was completely fine, my parity drive had 204GB free, and my fullest drive also had 204GB free. I added approx 500GB of files (mostly TV shows, large files) and renamed some of my old TV shows, this meant that my fullest drive now had 92.5GB free out of a possible 931 (1TB). I then ran the update, it added my new files, and "moved" the renamed files. The weird thing was that after it was all said and done, there still about 100GB+ difference between the fullest drive (my tv shows) and the parity drive.

    I looked in the log file (which I can email if need be) and noticed that the files I had renamed had been "moved" and then later "removed".

    As a result I ran the update for a second time, at this point it ADDED all of the previous files in which it "removed" (which were always physically present on the drive).

    To give you an idea, between both updates, no files were added/deleted off the drives. After the 2nd update completed, it told me it added 200 GB to parity,
    my parity drive now appears to have 100GB free, while my fullest drive only has 92.5GB free.

    I am tempted to just format the parity drive and run "create" again, but wanted to get your input on this and see if it is something you have encountered before.

    Thanks again,
    Tony

    Posted 6 months ago #

  2. rust0r
    Member

    BTW: Running update for a 3rd time, results in 0moves, 0removed, 0adds; so it looks like it is finally satisfied with it's work

    Also, I don't think is relevant, I am running Windows 7 and using "Junction", which allows me to share 1 folder (C:\servershare), with all drives mapped within (ie: c:\servershare\tv shows refers to H: and so on). When a file is added on my personal computer through use of the shared folder "servershare" I can see it being added within the actual drive, so it doesn't appear that anything is getting mixed up due to junctioning the folders (all files are present on the physical respective drives)

    Posted 6 months ago #

  3. Phatty2x4
    Member

    rust0r,

    I had this "issue" as well.

    I make my data sets based upon drives (i.e. D;\ E:\ F:\ ...etc).
    The updates were taking an amazingly long time to run (6 hours instead of 30 to 40 minutes).

    That's when I noted that there were an incredible amount of removes and adds.

    It turned out that I accidentally made a circular connection to my data (e.g. data is on drive P: and S: I had set up a junction on P: representing all of S:). Disparity came through, noted the data changes, made updates accordingly, and then completed. Going through the logs, I was able to track down the improper junction that I made which was causing all the issues. Once I corrected the junction, Disparity started updating quickly again.

    My advise is to go through your logs and see if you have data that is being protected twice because of an improper junction direction.

    You may also make a batch script to check for junctions on your system:
    dir X:\ /A:L /s > TEMP.TXT

    This command makes it really easy to track down your junctions to verify that they are set up correctly.

    Posted 6 months ago #

  4. Roland
    Key Master

    I'm struggling to imagine how a file could be both renamed and also deleted in the same update. It just shouldn't be possible. I've never seen or heard of that happening before.

    Just to veryify: in the SAME log, you saw both "XXX moved to YYY" entries, and also "Removing XXX" entries? The only way I can possibly see this happening is if disParity there is a file called XXX on more than one drive, so the two entries aren't actually talking about the same physical file. But that wouldn't explain why the files were then added back again on the next update. It's so completely bizarre that I'm tempted to believe junctions must be the culprit somehow. I didn't consider the possibility of junctions when I designed disParity and they can definitely cause strange behavior, as Phatty notes.

    I assume your C:\ drive is not one of the drives being protected, correct?

    If you could make the complete log available for me somewhere I can take a closer look for clues, otherwise I'm stumped for the moment.

    Posted 6 months ago #

  5. rust0r
    Member

    Hmm, I will double check my junctions.

    Here is a sample of how it "moves it" and then "removes it"

    [Section 1]

    Scanning H:\
    TV Shows\24\24 - Season 8\24.S08E01.HDTV.XviD-2HD.[VTV].avi moved to TV Shows\24\24 - Season 8\24 - S08E01 - 400PM-500PM.avi

    [Section 2]

    Removing H:\TV Shows\24\24 - Season 8\24 - S08E01 - 400PM-500PM.avi...

    After I ran the update a second time, this is what appeared in the log:

    Adding H:\TV Shows\24\24 - Season 8\24 - S08E01 - 400PM-500PM.avi...

    Those 3 references are the ONLY references to that particular file, however picture this x500 large files, that is how it has occurred. This is the only instance of these files, they isn't duplicated across drives or anything of that nature.

    I understand the possibility for junction to throw a monkey wrench into the mix, but am not seeing how it would relate to this particular problem.

    I can provide you the full logs, I don't have anywhere online to store them, they are about 300k and 40k respectively if you would like I can email them to you.

    P.S. you are correct, my junction drive C:, IS NOT protected

    No circular junction references are present either

    Posted 6 months ago #

  6. rust0r
    Member

    To add to the above post, my understanding of the events are:

    1) it recognizes that the file has been renamed or moved, and takes action
    2) it removes the actual file from parity (??)

    thus at the end of update #1 it is not protected

    when running the update for a 2nd time:

    3) it adds and protects that file into parity

    Posted 6 months ago #

  7. Roland
    Key Master

    You can email the logs to: rolandv@gmail.com

    Thanks!

    Posted 6 months ago #

  8. rust0r
    Member

    Sent!

    Thanks!

    Posted 6 months ago #

  9. Neostim
    Member

    Hey Roland,

    I've experienced the same issue, and I'm not using Junctions. I upgraded to Windows 7 a couple of weeks ago, maybe it is a Windows 7 problem?

    I ran an update and get this in the logs:

    [First Section]
    Tv Shows\Transformers\Beast Machines\Season 1\Beast Machines 1x01 - The Reformatting.avi moved to Tv Shows\Beast Machines\Season 1\Beast Machines 1x01 - The Reformatting.avi

    [Second Section]
    Removing I:\Tv Shows\Beast Machines\Season 1\Beast Machines 1x01 - The Reformatting.avi...

    Then, ran disParity again, without making any changes (because it showed as only adding 40GB when I knew there was a lot more to be protected):

    Adding I:\Tv Shows\Beast Machines\Season 1\Beast Machines 1x01 - The Reformatting.avi...

    I too can send me logs if it helps, just let me know.

    Posted 6 months ago #

  10. Neostim
    Member

    Is anyone else running Windows 7 and disParity?

    Posted 6 months ago #

  11. Roland
    Key Master

    My main media server runs XP, but I have Windows 7 on my laptop and I've worked on disParity on the laptop. I've never noticed any odd behavior there, but then again the laptop isn't running a "production" configuration with terabytes worth of media data.

    Unfortunately I am completely swamped at work right now with a big new project. I just haven't had time to look at these logs or try to understand how this new problem could suddenly have appeared like this. It's completely baffling. Hopefully I'll get a break soon and be able to dig in. Thanks for your patience in the meantime.

    Posted 6 months ago #

  12. Neostim
    Member

    Thanks for keeping us up to date Roland. We understand work comes first, can't buy hard drives without a job ;)

    Just another note, I was running Vista prior to 7 and didn't have any problems, FYI.

    If I can find anything else out, or scenarios that do/don't work I'll post them up here, so you have all the information whenever you do get some time.

    Have a good one!

    Posted 6 months ago #

  13. rust0r
    Member

    Roland,

    Thanks for your reply, no worries, sometimes life gets in the way of play time; we appreciate all your hard work with making disparity what it is :)

    I am also running windows 7 (though I have never run disparity on anything else before, so I can attribute that to being the problem).

    I just ran "disparity list" in order to get all of the files it is protecting into a text file, at which point I also ran "ATTRIB /S > c:\OUTPUT.txt" on all of my individual drives.

    Once I had a list of the actual files on the drives, and the actual files disparity was protecting, I threw them all into excel. Through some data manipulation I managed to determine that EVERY file that windows says I have on the drives is being protected by disparity (based on the files in disparity's output list). This is something well into 115,000+ files.

    Neostim had directed me to a few threads about why I might have a discrepancy between the size of my parity drive vs my fullest data drive. Based upon the analysis I just did, it seems like they are all logical reason for the occurrence.

    The one thing this doesn't explain, the million dollar question: is why we both have had to run disparity twice in order for it to protect all of the files we have just added. It seems to get about 80-90% but there are always some that linger.

    I'm definitely more relieved now that I know that it is fact protecting all of the files, and running the update twice isn't much concern either, just something that I don't think you intended to happen.

    Thanks again, good luck with your project!

    Tony

    Posted 6 months ago #

  14. Phatty2x4
    Member

    For those running windows 7:

    How are you offering the data? By this I mean, is there a share or are you using libraries or public folders or home groups?

    Posted 6 months ago #

  15. Neostim
    Member

    I'm doing protection at the drive level, 6 of them, of which I have folders shared to the network, no junctions or anything else going on. Hope this answers your question

    Posted 6 months ago #

  16. rust0r
    Member

    I am using junctions, however it has no play when it comes to disparity updates, the updates are done at the drive level, the only time the junctions are used is in my day to day accessing of them (home network storage). If I monitor the server, I can see the files being copied/cut/deleted from the drives real time, so junction isn't at fault here; at least I don't see how it possibly could be.

    Posted 6 months ago #

  17. BlkKnight
    Member

    Windows 7 64bit, 4TB data protected across 5 drives, one protected root folder per drive, no junctions or symlinks within the folders covered by disParity. Folders on each drive are available to the network via normal sharing as well as a few via homegroup (access via HomeGroupUser$). I have never experienced any strange adds/deletes and have never had to run update multiple times to get all files.

    -BK

    Posted 6 months ago #

  18. rust0r
    Member

    BlkKnight, how far off is your parity vs the fullest drive in terms of size? (my party drive is about 18gb fuller than my fullest data drive for instance). Thank you

    Posted 6 months ago #

  19. BlkKnight
    Member

    Most full drive holds 1.32TB (1,457,686,337,254 bytes) and the parity drive is 1.32TB (1,457,725,492,544 bytes).

    -BK

    Posted 6 months ago #

  20. Phatty2x4
    Member

    I think I might have an idea of the issue here.

    I redid my parity set yesterday. I am using the most recent version of Disparity (0.20).

    New raid set created and all was nice. I added a few movies and moved one to a new location.

    When I kicked disparity off I ran into the following:
    During the scanning phase, Disparity registered that the file had moved locations.
    At the end of the scanning phase Disparity reported that the there were
    Adds: 16 Moves: 1 Deletes: 1
    Then Disparity Processes the Deletes.
    The Disparity Processes the adds.

    However, the Move is not processed.

    It only gets picked up when I ran disparity a second time and the file was detected as an add.

    At least for my set up, this appears that the version of Disparity does not process moves.

    rust0r, BlkKnight, Neostim - can you verify if this is happening to you as well?

    Neostim - from your example earlier up in this thread, what happened to me looks to be exactly happened to you.

    Roland - is there a command to dump the exact version of Disparity? I want to make sure what I have.

    Posted 6 months ago #

RSS feed for this topic

1 2

Reply »

You must log in to post.