disParity » General Discussion

Insufficient disk space available

(35 posts)
1 2

  1. Klaatou
    Member

    My free space is from 8 to 80 gb on each disk and of course it is mp3 disks where I have less space.

    I think if it was a question of 64k space rounded that compression would gains something but it doesn't so it must defenitly be the LARGE number of files with the sort of "table of content" stored with parity ...
    So dldummy I see what you think with zip : no compression matter but only one file instead of many.
    I will first see if I can do that with some folders only here for archive.

    Otherwise I will try to move some datas but I think I will start to protect only video disks waiting to have enough money for a larger disk.

    I'm aggree I will soon be short of space but as disparity is to protect datas not "moving" a lot I thought it was a good idea to use it on my full disks ;)

    And for long posts it must be hard if we have to read all of them but I must say it helps a lot to follow ideas, especialy when english is not our mother language :)

    Thanks again

    Posted 7 months ago #

  2. dldummy
    Member

    Can you tell me .. your MP3 .. whats the size of them ....
    and how many files are there ...

    My Metadatafile for my biggest drive 9100 Files (most of my Files are video) is 1.4 MB ... so possibly even the the ZIP method dont bring much space (in case of MetaData).
    Eventually the PaddingBytes (ExtraBytes) can help ...

    You can try to calculate Unzipped ParitySize (Bytes):
    The size of your MP3 Folders in Byte + (Number of Files * 32768)

    Zipped ParitySize (Bytes):
    Size Of your Mp3 Folders in Byte + 65536

    These are the sizes of Parity whitout Metadata if you want .. add 150 Byte Matadata for each file ... (thats the middle value of one of my drives)... or a little bit more if your paths are long

    If the difference isnt enough ... forget the ZIP method ....

    And thanks Roland for the easier way to calculate ...

    ... Short enough ? ... ;-)

    Posted 7 months ago #

  3. dldummy
    Member

    If you calculated the Zip thing and the difference isnt big enough,
    there is only the way i and Roland (but i missunderstodd him .. sorry) suggested.

    Calculat the parity size for every smallets Unit you can and want to move (without destroying the organization of your files) and try to rearrange them.

    What do i mean exactly .... Balancing of files ..

    Making parity work by Xoring the data of one drive with the data of the others.
    If you have 10 Bytes on each of your drives ... you get ??? ... 10 Bytes Parity
    But if one of them have 20 Bytes ... you get 20 Bytes ... the missing 10 Bytes on the other drives are Padded with 0 ...
    SO if you rearrange your Data so the biggest drive is as small as possibele you you can make your Parity smaller ...... BUT ....

    It is possible that your biggest (most filld) drive isnt the biggest ...... Hää ... why ... ?

    PaddingBytes ! ... example
    Drive one ... The biggest 90% filled but with 25GB BluRay ISO ...
    Drive two ... 85% filled but with mp3 ... it is smaller but if you add the paddingbytes its possible that the parity for this drive is much bigger than dive 1 ... its also possible that the parity is bigger than the total capacity of the drive ...

    So calculate the Real Parity size for as example
    The MP3/ROck FOlder on drive 1
    The MP3/Pop FOlder on drive 1
    The SeriesFolder on drive 2
    The ... whatever your system is and your smallest unit you are willing to move is.

    The size of your Unit in Byte + (Number of Files * 32768)
    Ignore The metadata ... the size of metadata dont change, it depends on the Count of files.

    Now you can try (on paper) to rearrange them until the biggest RealParitySumm is small enough to write it on your Parity drive.

    If this even dont work ... then nobody can help you ... there is no way ...

    Sorry ... much to explain ... so its longer ...

    And ... all the calculation is statistical not the exact value ... but if calculation say it dont work ... the chance that it work in real is very small ...

    And to correct what you wrote ...
    >>LARGE number of files with the sort of "table of content"
    Large number of files is correct .... but
    Not the Table of content (MetaData) is the problem ... as wrote before it is someting like 100-300 Bytes depending on Pathlength per file ... so for 100.000 Files its only 15MB (i calculated with 150 Bytes per file)

    But there is up to 64KB PaddingBytes per files ... which is 6GB in worst case for the same 100.000 Files.

    Posted 7 months ago #

  4. Klaatou
    Member

    I'm back ! ;-)

    So, I have far too much files and rearranging them was like hell ...

    And now I have a brend new 1.5 TB and of course everything is much easier (but very long : more than 2 days ;)
    Parity is now 1.08 TB for 1119 files (740 MB for files0 to 6.dat) but the strange thing is in the disParity Center's logs window : it ends on a "Reading L:\Audio\_Various......."
    I was remembering a clear ending message ... it may be ok, I will try a verify.

    Thanks again for helping, I think solution was here but the bigger disk is the best solution for lazy geeks :) (even if it is not good when €uro is going down ......)

    Posted 6 months ago #

  5. Roland
    Key Master

    The last line of the log after a create has completed should not be a "Reading..." line. It should be something like this:

    5.88 TB protected in 40,851.94s (150.97 MB/sec)

    Check the log file in disParity's folder that was generated by the create command and make sure it ends with a line like that. Otherwise the create may not have completed successfully for some reason.

    Posted 6 months ago #

  6. Klaatou
    Member

    yes that's what I saw with my test without mp3 disks, this time I forgot the logs rep sorry ;)
    the log is 575 MB and was not easy to read ... here is its end :
    4273089 records saved in 17,40 sec
    6,12 TB protected in 96 890,02s (66,26 MB/sec)
    so it is good ! :-)

    Posted 6 months ago #

  7. Roland
    Key Master

    575MB!?! Wow...you really must have a huge number of small files. My create log is 2MB for 6TB worth of data. Apparently just one of your drives has over 4 million files on it...all of my drives combined top out at just over 28,000 files. You are really pushing disParity way beyond what I designed it for. No wonder you were having trouble with free space on the parity drive earlier!

    Posted 6 months ago #

  8. Klaatou
    Member

    not millions but a lot :
    first mp3 disk : 329 030 files for 921GB
    second mp3 disk: 204 534 files for 914GB

    Posted 6 months ago #

  9. Roland
    Key Master

    In your post before, you said the end of your log said "4273089 records saved in 17,40 sec". There is one record per file, so it sounds like at least one of your drives has 4.27 million files on it. The "stats" command will list the number of files on every disk, if you want to check.

    Posted 6 months ago #

  10. Klaatou
    Member

    Ho yes you are right ...
    I forgot: now I have a lot of parity space I added a disk with a lot of little files (picts, ebooks, downloads archives ....) 4 275 311 files for 899GB

    by the way: files.dat are very well compressed : 739MB size and only 285MB on the disk (and no compression at all for parity files)

    Posted 6 months ago #

  11. Roland
    Key Master

    OK, just keep in mind that disParity wasn't really designed for disks filled with millions of tiny files. You aren't going to be getting ideal performance. For one thing, it loads all the file lists from all drives into memory before each update. In your case, that's going to take up hundreds of megabytes just to store the file table. It will work, just not quickly!

    Posted 6 months ago #

  12. Phatty2x4
    Member

    Klaatou,

    Just an idea to help, you may actually want to break up your parity sets.

    For example, back up movies in one data-set, mp3's in another data-set, and pic's in yet another.

    Yes you will need additional drives to hold the parity, but that should significantly help to cut down on some of the updating speed.

    Posted 6 months ago #

  13. Klaatou
    Member

    Yes I saw it is very long ;-) even with 8gb Ram...
    And I think I will see how to organise all that in another way, maybe with 2 parity sets ....

    Posted 6 months ago #

  14. dldummy
    Member

    Hi, im back again ...

    I was a little busy last weeks ...

    Klaatou>> Yes you will need additional drives ...

    Possibly not ....
    He can make Parity for more than one set on the same Paritydrive ....
    It makes no difference ... no lowering of security ...

    If he loose one drive ther are Parity ...
    If he loose Parity ... he have Data ...

    Make 2 ParityFolders on Parity Drive ... and ... 2 Configs and logfolders ...
    Make 2 batch files which rename Config and log and start update ...

    ... and WOW ... 4 Million Files is a lot ....

    Posted 6 months ago #

  15. Klaatou
    Member

    I tried to have 2 parity sets, I had to move a lot of datas to some rep or disks and back. (of course I removed the 4 million files disk ;)
    It finished to work (I deleted some reps too), it worked so fine I even retried to have only one parity set and it works, we are exactly in what you said, we have to organized the files and not keep a too big number of files (especialy on the same disk).
    Thank you again for your time and ideas.

    Posted 4 months ago #

RSS feed for this topic

1 2

Reply

You must log in to post.