disParity » General Discussion

64 bit

(11 posts)
  1. Roland
    Roland

    I recently upgraded my media server to a 64 bit OS, and also increased the RAM to 4GB (the most the aging mobo will support.)

    An area where disParity can benefit from 64 bits and the extra virtual memory available is the temp file it generates while processing a file. In order to minimize the risk of parity corruption due to an unexpected failure, parity changes due to an add or delete are written to a temp file first, and then flushed to the parity set after the add or delete is finished. For small files (typically less than 1GB) disParity currently uses a memory buffer for this temp file, but for larger files, it has no choice but to write it to disk, and then read it back from disk and write it to parity, which significantly slows down parity updates for large files.

    I've tinkered with the code a bit, and tried switching the "memory buffer" implementation to one using memory mapped files instead. It seems to provide a dramatic improvement to parity updates for the cases where it would have previously used an on-disk temp file.

    Is this something any current disParity users on 64 bit out there would find useful? There are a couple of drawbacks I know of:

    1) The change requires updating to version 4 of the .NET runtime, which is where MS added support for memory mapped files. Not a huge deal, but I liked the fact that previously disPariity only required version 2, which just about everyone has these days.

    2) It's hard for disParity to know how big a memory mapped file it can get away with. This is a function of the amount of RAM in the machine and how much of it is currently available for the file. If disParity tries to create a memory mapped file that is too large, it will work, but system performance slows to an absolute crawl. Ultimately, the "max" size for the temp file is something that the user will have to tune, which is not very user friendly, unless I can come up with some other way to figure this out automatically.

    -Roland

    Posted 3 years ago #
  2. Satsuki
    Member

    i'm realy waiting for it.
    as for the 32bits version, it would be cool to be able to set a memory size for disparity to use to improve snapshot creation ^^

    thanks for this great software

    Posted 3 years ago #
  3. Roland
    Roland

    I'm still trying to work out how to make this change available without making it too easy for people to accidentally crush their servers with too much VM use.

    As for memory usage during snapshot creation, I'm not sure how a setting there would help. Snapshot creation (i.e. the "create" command) performance is limited by disk speed. Having more RAM available wouldn't help in any way that I can see.

    Posted 3 years ago #
  4. freehand
    Member

    I would find this feature really useful. Going to .NET 4.0 is fine with me. The install is pretty small.

    Posted 3 years ago #
  5. Neostim
    Member

    I too would like this feature :)

    Posted 3 years ago #
  6. Roland
    Roland

    OK, I've decided to make this change available as an optional, experimental version of disParity for those of you with 64-bit systems and lots of RAM who want to try it out. Here is the download link:

    http://www.vilett.com/disParity/disParity0.22beta.zip

    The readme has details under the "CHANGE HISTORY" section.

    Please note that .NET 4.0 is required to run this version of disParity (download link is in the readme.)

    How much tempram is too much? Hard to say. On my Windows 7 64-bit system with 4GB of ram, I have my tempram set to 3000 (~3GB) and it seems to work well.

    Posted 3 years ago #
  7. Cybie257
    Member

    Nice. I will try it out. I am running Windows 7 64 with 24 Gigs of RAM. If possible, I will set it really high, like 12gigs?? and try it. I have yet to see the details on the settings/config, but I will report back what I find out.

    Maybe there is no advantage to going that high?? But if allowed, I will give you the results so you can use them in your description/help/etc.

    -Cybie

    Posted 3 years ago #
  8. Roland
    Roland

    Hi Cybie,

    You could see a performance boost by setting it that high, if your updates are processing files which are that large. For example if you've added a 10GB movie file, with tempram set to 12GB disParity can skip the on-disk temp file and use RAM instead, which should make the update go much faster.

    There's no point in setting tempram higher than the largest files you have, though, since it never uses more RAM than the size of the file it is currently processing.

    Posted 3 years ago #
  9. rust0r
    Member

    Damn you Roland! You just gave me a reason to have more ram in my server! Haha! I am only running 2GB at the moment and an ageing 320GB OS drive so I feel everyone on how slow some of the large file additions can be. I will definitely be keeping this in mind for the future if I choose to upgrade my server hardware.

    Thanks for the continued support Roland, I have a question for you but I'll start a new thread as to not clutter this one.

    Posted 3 years ago #
  10. cybrsage
    Member

    Hmmm...since the average BluRay disc I have is around 35GB large, that means I would need to have 35GB of free memory to use this feature if working with BluRay ISO files, right?

    Posted 3 years ago #
  11. Roland
    Roland

    That's right. Since most people don't have nearly that much RAM, a 35GB single file update will require 35GB free on your HD for the temp parity file.

    Posted 2 years ago #

RSS feed for this topic

Reply

You must log in to post.