I recently converted all the BluRay ISOs on one of my hard drives to sparse files, saving me a good bit of space (roughly 200GB of space). However, now I am running into an issue.
When I run the check on my drives, the one with all the sparse files takes a LONG LONG time to scan the drive and then comes up with no changes. I did this several times and each time it is the same, no changes.
I am just curious how Disparity handles sparse files.
There's no special handling in place for sparse files. They will be treated like any other file.
I can't think of why they would cause scans to take a long time, though, since files are not read during a scan. It's possible something about them causes disParity to think they may have been edited or moved every scan, causing it to have to re-calculate the checksums to be sure. I'll have to test it myself on a sparse file to see.
How long are we talking about here? Minutes? Hours?
Hours for about 30 BluRays. Here is a snippet from the logs:
4/22/2013 12:03:33 PM Scanning N:\...
4/22/2013 12:03:33 PM Scanning P:\...
4/22/2013 12:03:33 PM Scanning Q:\...
4/22/2013 12:03:33 PM Scanning O:\...
4/22/2013 12:03:34 PM Scanning R:\...
4/22/2013 12:03:35 PM Scan of Q:\ complete. Found 2766 files (1.3 TB total) Adds: 0 Deletes: 0 Moves: 0 Edits: 0 Ignored: 0
4/22/2013 12:03:35 PM Scanning S:\...
4/22/2013 12:03:35 PM Scan of S:\ complete. Found 0 files (0 bytes total) Adds: 0 Deletes: 0 Moves: 0 Edits: 0 Ignored: 0
4/22/2013 12:03:36 PM Scan of P:\ complete. Found 3652 files (3.1 TB total) Adds: 0 Deletes: 0 Moves: 0 Edits: 0 Ignored: 0
4/22/2013 12:03:37 PM Scan of R:\ complete. Found 2427 files (1.4 TB total) Adds: 0 Deletes: 0 Moves: 0 Edits: 0 Ignored: 0
4/22/2013 12:22:44 PM Scan of O:\ complete. Found 2211 files (1.5 TB total) Adds: 0 Deletes: 0 Moves: 0 Edits: 0 Ignored: 0
4/22/2013 2:45:36 PM Scan of N:\ complete. Found 3118 files (1.5 TB total) Adds: 0 Deletes: 0 Moves: 0 Edits: 0 Ignored: 0
4/22/2013 2:45:36 PM Beginning Update
4/22/2013 2:45:36 PM Update complete (operation took 2 hours 42 minutes 2 seconds)
I have two sparse ISO files on O and about 30 on N. All the other drives take seconds, the remaining time is on those few files.
You can download a program called Sparse Checker, which will check the file to see if it is sparse and if it can be made sparse if it is not. If you run it on a bluray ISO you can save a lot of space. It gives you the option to check the file, then after the check you can make it a sparse file.
http://opalapps.com/sparse_checker/sparse_checker.html
I also wrote 200GB of saved space and I meant to write 20GB of saved space total.
Oh, and I should say, these files in question started as normal files and I changed them to sparse files. I ripped the entire Stargate:Atlantis series BluRays to sparse ISOs and it has no problems at all scanning that drive (P drive).
The only drives that are SLOW to scan are the ones where I converted the already existing files to sparse files.
OK, pretty sure I know what the problem is. It doesn't have anything to do with the files being sparse per se. The problem is that when you converted the files to sparse, it changed the "last modified time" for each file, but didn't change them in any other way (i.e. the contents of the files are still the same.) This exposed a bug in disParity.
The bug is that when the "last modified time" of a file changes, disParity notices that fact during a scan and re-reads it to compute the checksum to see if the contents actually changed. If the contents changed, it removes it and re-adds it during the Update and everything is fine. The bug is that if the file hasn't actually changed, it doesn't update its saved value for the last modified time. So on each subsequent scan, it keeps thinking the last modified time has changed, so it keeps re-checking the checksum on all those files every time, which is why the scans are now taking so long.
I'm hesitant to put out a new version to everyone right now to fix this tiny bug, but it will certainly be fixed in the next release. In the meantime, you could fix it by removing all the affected files off of the drive, doing an update, then adding them back and doing another update. Or you could remove/re-add the entire drive if that would be easier.
Any idea when the next release is to hit the streets? It will take many hours to remove and re-add the drive to the parity set.
It's gonna be a little while, sorry. I've got a lot of things on my plate. While it's true the remove/re-add process will take a long time, you can just start it up and walk away, right?
Actually, on second thought, if you want I can email you a patch with a fix for this issue.
Sure, email me the patch and I will test it out for you. You are right, I could move the files away, run the program, move them back, run it again. :)
email sent
You must log in to post.