disParity

disParity » General Discussion

1. rust0r
   Member

   I'm not sure if anyone saw my posts from a few days ago regarding my questions on whether or not "Drive Bender" would work in conjunction with disparity or not....but I decided to put it through the paces on another PC to see how it worked and if it was viable for disparity. I recently re-build my server which consists of 27TB (15 Data Drives), and thought I would check to what solutions were out there.

   Drive Bender: http://www.drivebender.com/

   Note: The combined drives and their mount point/drive letter are called a "Storage Pool" (I will be using that word below)

   It allows just that, what we've all been pestering Roland about for many years! I set it up using the "Merge" function and it is VERY simple to figure out once you've added one drive, you can add the rest without a hitch. There isn't any overhead calculation or additional time/slowness to the disk access. In fact you will find it operates similar to a raided drive. I was able to pull several hundred MB/s off the storage pool (of files stored on separate disks obviously) to other drives.

   In total, I added 14 of my drives (all media of some sort) to the storage pool and left only 1 drive out (the drive that stores personal files, picture, etc) out of it. I now have Z: drive, a 20.7TB pool!

   To anyone looking to set it up, I will provide a few quick tips that I found

   1) Before adding drives to the pool, sort out all the data you want, if you have for example 3 disks with "Home Movies" on them, make sure the folder path is simliar to D:\Home Movies\<files> E:\Home Movies\<files> and so forth. When you merge the drives into the pool, you will have (say the pooled drive is Z:) z:\Home Movies\<all files from 3 drives> in the same directory. I spent a good bit thinking out "how this drive will interact with the pool once it's added" to make sure I didn't have any conflicts/issues

   2) Use the MERGE feature, NOT Convert when adding a drive to the pool

   3) DO NOT place any files off the root directory prior to merging, unless you want a ton of loose files directly inside your pooled drive ie: Z:\<tons of loose files>. It may have been fine when you had individual drives, to have a drive laid out such as Home Movies -> D:\<home movie files> but when adding many drives like that too the pool, files will end up being clustered off the root and create a HUGE mess

   4) Once a drive is added to the pool, NEVER add/delete/access data from the standard drive path itself, always go through the pool.

   To be sure I didn't continue to go through the drive letters themselves, I used RegEdit to modify the "NoDrives" Dword value to hide all of the drives that were part of the pool, this way I never have to worry about breaking this key rule. See: http://www.wisdombay.com/hidedrive/ for info on this NoDrives setup

   5) READ and READ AGAIN, all prompts and questions before applying a setting in Drive Bender. There are a few separate sections that deal with "deleting all pool data", this WILL cause the program to erase ALL of your data across ALL of your drives. I read of 2 instances of people doing this and blaming the program. It is pretty obvious if you have any wits about you what it does, you can't blame a program like this for plain stupidity.

   Of course, you will need to recreate your parity drive once this is complete as Drive Bender, to work, changes the layout of each drive to sync itself when you merge it into the pool. Example: A drive that was D:\Home Movies may now be D:\{A3F6-GV53CX-533C}\Home Movies it would take forever to "update" all these changes with disparity, so simply recreate it. I had 20TB to protect and it took just over 26 hours, not bad.

   I did run into problems with 6 file names being too long ("Cannot query file attributes") due to the added folder at the root that Drive Bender makes, but I corrected that after creating parity, updated it, and it was all fine in seconds.

   If anyone has any questions before taking the leap, post them up and I'll try to clear them up!

   Disparity + Drive Bender, it really doesn't get any better IMO

   A BIG thank you to Roland for his continued support of this awesome application

   Posted 3 years ago #

2. Neostim
   Member

   Thanks for taking the time to document it all rust0r! I'm giving it a try now :D, just bought drive bender.

   Posted 3 years ago #

3. Cool1Net6
   Member

   I also thank you for this. I'm thinking of updating my server and I was looking at Drive Bender / DrivePool. Nice to know I have more options other than 'mirror' to protect my data.

   Posted 3 years ago #

4. Klaatou
   Member

   hello, after all I will perhaps give a try to drive bender but I don't understand your third point : "DO NOT place any files off the root directory prior to merging,......."
   I'm french, I understand the words but I'm not certain of the sentence meaning, does it mean we must have no file in root directories ?

   thanx/merci

   Posted 2 years ago #

5. Roland
   Roland

   I think he's just saying that if you have a lot of files in the root directories of the individual drives, they will all get mixed up together in the final merged drive and possibly make a mess.

   Posted 2 years ago #

6. Klaatou
   Member

   Yes that was what I understood, only lost by the sentence ;)

   I found something I didn't try yet but it seems to be a free replacement for Drive Bender :)
   http://liquesce.codeplex.com

   I think some one already talked about it, did some one tested it ?

   Posted 2 years ago #

7. SavageGizmo
   Member

   Hi rust0r, thanks a lot for the comprehensive write up, much appreciated! Just one question, probably obvious, but which drive/s do you add to disParity? Is it the pooled drive, or the individual drives? Many thanks!

   Posted 2 years ago #

8. Klaatou
   Member

   That cannot be the pool, in my case I have a 14GB pool drive, as disparity needs the same size .... better use a mirroring system ;-)
   I used the individual drives and then the {Disk letter}:\{106DF1D2-3563-4E6B-9065-8508EB9D6677} directories and have too long names paths in both cases :(

   Posted 2 years ago #

9. SavageGizmo
   Member

   Hi Klaatou, thanks for the reply! Will give disParity a try adding the individual drives and see how far I get. I'm using the Drive Pool addin from Stablebit, not sure if that's going to make a difference but I guess there's only one way to find out, fingers crossed it all goes smoothly.

   Posted 2 years ago #

10. Roland
    Roland

    Yes, as Klaatou says, you would definitely not want to add the pool to disParity, you would want to add each of the underlying physical drives.

    Posted 2 years ago #

11. SavageGizmo
    Member

    Hi Roland, thanks for the confirmation. Just tried it out but it displays a "Not a valid WIn32 FileTime" error. I assume this is because the file names are too long as Klaatou said above *massive sad face*. Thanks for all the prompt responses!

    Posted 2 years ago #

12. Roland
    Roland

    That's a strange error, I've never heard of that one before. Are you sure it was because the files names are too long (greater than 260 characters)? disParity should ignore those files completely, not generate weird error messages.

    Where in the application did you see that error message?

    Posted 2 years ago #

13. Klaatou
    Member

    Yes long file names never "told" me this message, "now" I only see the ignore message in the logs file and process is not stopped anymore.

    Posted 2 years ago #

14. SavageGizmo
    Member

    Looks like it might be a problem with the time attribute of certain files on my hard drive/s: http://social.microsoft.com/Forums/is/synctoy/thread/3169dc37-57a0-4939-9076-98746fffcd0f

    Will check out the application that's linked in that thread and try and clean up the time attributes and see if that remedies the error. Will post back...

    Oh and Roland, the error was displayed in the rectangular box which is specific to that hard drive(not a pop up error notification). Sorry, not a very good explanation but will upload a screen shot when I get home.

    Posted 2 years ago #

15. Roland
    Roland

    Actually that explanation is perfect, I know exactly what you mean, no need for a screen shot. Thanks!

    I'll poke around a bit, see if there is a way I can prevent this type of error from halting the scan.

    EDIT: If you could send me your log file, that would greatly help me pinpoint where in the code this error is tripping things up. Under Vista and later, your log file should be here:

    C:\Users\<username>\AppData\Local\disParity\logs\disParity.log

    Email it to: rolandv@gmail.com

    Thanks!

    Posted 2 years ago #

16. SavageGizmo
    Member

    Managed to get round the error by using this application: http://www.isdigital.com/FindInvalidFileTime.zip (nothing fancy but does the job). Now it's currently working through the update procedure.

    Hi Roland, saw your message after I started the update through disParity and the current log file and ".old" log file contain only data from today I'm afraid, sorry, got a bit eager - unless there's another way to get the older log files?

    Thanks to everyone that's helped and for the disParity application!

    Posted 2 years ago #

17. Roland
    Roland

    No problem, I think I've figured out where in the code this error was causing problems. Thanks again and I hope you find disParity useful!

    Posted 2 years ago #

18. rust0r
    Member

    Hello all, just checking in, sorry for missing some of the replies. I am glad the response has been positive :). I've been using it since my post without any problems whatsoever. It's a bit of a long post but there is some good information I've come to find in my 6 months of running the setup

    Klaatou: What I meant about not placing them in the root of the drive was this.

    When you add a folder to the pool, and merge it as I explained. If you had your D: kept for storing your Home Movies, and simply had the storage layout as D:\My 30 Birthday.avi, D:\Toams Birthday.avi, and have another drive E: for Music, stored in a simliar fashion E:\Led Zeppelin - Song.mp3, etc. When you merge all the drives into the Drive Bender Pool, they get all tossed under the root of your pool and unorganized. You will now have (say your pool is Z:):

    Z:\My 30 Birthday.avi
    Z:\Led Zeppelin - Song.mp3
    Z:\Tomas Birthday.avi

    So by placing folders for the drive type prior to merging, ie: D:\Home Movies, E:\MP3, and moving all files off the root into those folders, when you create the pool, it will look like this:

    Z:\Home Movies\My 30 Birthday.avi
    Z:\Home Movies\Tomas Birthday.avi
    Z:\MP3\Led Zeppelin - Song.mp3

    It will keep everything organized regardless of the number of files you had. It is something simple but I could see and have read of many people not planning properly and completely messing up their storage collection. I have around 21+TB currently so I had some thinking to do before acting on Drive Bender.

    SavageGizmo: I think Roland was able to answer your question, but I will confirm, it is the individual drives you have to use.

    I actually took the whole Drive Pooling one step further and HID all of my pooled drives using "NoDrives" registry entry. See the following link for an explanation: http://www.wisdombay.com/hidedrive/index.php

    Now when I go to My Computer, I only show my C: (windows) D: (day to day storage before moving files off to the pool drive) and Z: (Pool). It REALLY cleaned it up and looks very impressive.

    At this point, I've heard and investigated a few other options that have come out similar to disparity but I can't imagine switching from this setup as it really takes care of everything I want:

    1) It's secure

    2) It's simple and to manage my files/space

    3) It allows me to keep everything organized and I no longer have to shuffle data from one drive to another as they fill up

    4) It has saved me a TON of MONEY believe it or not. Prior to using drive bender with disparity, I had 20+ individual drives, all with varying free space some were full, some had 300gb and it was a NIGHTMARE to try and constantly shuffle things around to keep the free space where I needed it. 7 Drives for TV shows, 3 for Blurays, etc. At some points I would have to buy a new drive for one type of media, when I had other drives with 300gb free just due to how the storage ended up for them. When I moved to this setup, I ended up with about 3TB free on my pool and I no longer have to shuffle anything around. I can definitely say it PAYS to use this lol

    I really can't say enough how awesome Disparity + Drive Bender are together.

    Thank you again to Roland and all the users who support his efforts.

    Posted 2 years ago #

19. Klaatou
    Member

    hi
    Thank you rust0r, I understood the directories merging in DB it was your "DO NOT place any files off the root directory prior to merging" I didn't ;-)
    But I gave up, no more DB for me, first its way to move all HDD directories to another one with a very long name (pool uiid) increase all path lengths give me tons of too long name files ignored by disParity.
    Then its way to balanced datas thrue all drives is boring me, I don't speak of the REbalancing I found to disable but the recording on nearly each HDD files which are on a same directory, and we cannot managed that behavior.
    Result : for example a big directory (bluray rip or many images scan) will be on each HDD meaning running them all (not very green, we are close to the bad point of hard raids) and if you lose a drive (dispite of disparity) you will NEVER have any complete directory on remaining pool drives.
    I need a pool system keeping priority to directories we record, even if it means loosing some space, I think there are ways to lose not so much.
    Actually any way to see all HDD in one directory even in read only mode would be great for me. I tried to see with NTFS links but I didn't found how to merge directories with same names on differents HDD :(

    Posted 2 years ago #

20. Cool1Net6
    Member

    I have a question about how nicely Drive Bender (and StableBit DrivePool, a similar competing application) plays with disParity. I've done a few hours of testing in a virtual machine to check if this combo is something I want to deploy. The problem I keep running into is how to restore data in the event a drive is lost.

    On the disParity side, this is simple; Point disParity to the new drive, restore the data from the old drive, change the config file accordingly, and you should be all set.

    However, neither Drive Bender nor DrivePool handles drive failures in this way. Each program wants you to remove the damaged drive and rely on its internal duplication algorithm for redundancy. There is no way to add in a new drive as if it were the old one, meaning while you can restore the data on that drive, you can't restore the pool to as it was prior to drive failure.

    Every step I took with Drive Bender resulted in complete loss of data from the moment a drive was lost. I'm sure that there is user error somewhere in there, but even if only the data that remained could be assembled back into a working pool (as was expected), Drive Bender (and DrivePool) create unique folders that do not get folded back into the existing pool when merging a new drive.

    rust0r, have you tested any drive failure scinarios? Did you get the same results?

    Posted 2 years ago #

RSS feed for this topic

Reply »

You must log in to post.