Several users have requested a feature where disParity periodically updates the backup automatically. My question is which would people prefer:
1) A simple setting where an update runs automatically on an hourly or daily basis
- or -
2) A "smart" update mode which monitors all data drives for changes and updates the backup automatically whenever a change is detected.
hello
I would say a mix of both ;)
monitoring is really fine but can occured when we work on something so it could be only a little warning (a sort of smiley for telling it would be a nice idea to run it)
Plus an auto run at a recorded hour if monitoring said to.
But wouldn't it be a big work too ? How do you plan to monitor the datas changes ? Are you about to think to come close to a kind of real time parity ?
A mix of both doesn't really make sense to me. If you want to run the update at a preset hour, it might as well just run the update regardless. An update when no changes have occurred is a quick & painless operation that does nothing other than scan the drives. I don't see the point of making it more complicated by only running the hourly update if changes have been detected in the last hour.
But anyway, yes, the goal would be to approximate "real time" parity as close as possible using a strictly software-only approach.
Monitoring the drives is not difficult. Windows provides an easy mechanism for notifying an application if any changes have occurred on a drive, or within a certain folder, etc.
And yes one tricky thing about "real time" monitoring is that it could try to run an update when the user is still modifying the drive, which may not be desirable. My thoughts so far on how it work work:
* Establish a "FileSystemWatcher" for each data drive (that's what .NET calls it)
* Whenever a change is detected, start a timer, say for 1 minute. If multiple changes occur quickly, just keep restarting the 1 minute timer.
* If no changes have occurred after 1 minute, scan the drive. If changes have occurred that require an update, update the UI to indicate this fact (the same as if the user had scanned the drive manually.) Start another timer, this one for 5 minutes.
* If, after 5 minutes, no further changes have occurred, run an Update automatically.
The 1 minute and 5 minute delays could potentially be configurable by the user.
One downside with "real time" monitoring like this is that it basically renders the "undelete" feature useless. Undelete only works if a file has been deleted but the backup has yet to be updated. With real time monitoring, a delete would trigger an update within just a few minutes. So if you delete a file by mistake, you'd better realize it right away, before the automatic update kicks in.
Anyway, that's the basic idea. Lots of tricky details involved whenever you have an interactive application doing things automatically, because what if the user is trying to use the app at the same time? Things like that.
If I talk about ways to limit the running of even a scan is because one of the "bad thing" of "real" raid 5 solutions is that all disks are always online and working together getting them older before the age.
By the way, my last aborted parity creation test on my 10 2TB revealed bad sectors warning in SMART for one of them, I wonder what is the impact of complete scanning on a disk pool.
For the "real time" protection you are right to give time before running it, especialy because this protection is for datas and datas now could be very big and long to copy (a complete spring shooting 10GB.mkv ;)
PS: with FileSystemWatcher can you give a warning when we create a too long name path ? ;)
Good point about a scan requiring a disk to be spun up. If you don't want to spin up all of your disks every hour, then making it conditional on detected changes might make good sense. Still, once you are making updates always conditional on detected changes, I'm not sure why you would want to still force updates to be on a fixed hourly or daily schedule vs. X # of minutes after the last change.
I don't know what FileSystemWatcher will do with paths that are too long, I'd have to test that case.
Hello,
For me neither. If you implement this, please give an option to turn it off.
Having said that, #1 will be better. #2 can render the "Undelete" feature useless. I certainly prefer to have the Undelete feature to recover from accidental deletes.
Personally I cannot understand why monitoring (aka #2) is even needed. By definition, the pc running disparity is a pseudo NAS, serving up media files to another HTPC or media devices. In this day and age, such an "NAS" shouldn't be running 24/7. Mine is in hibernation most of the time, waking up on demand via WOL to serve the files to the HTPC and media devices in other parts of the house. When there is no demand from the media playing devices, the disparity pc goes back into hibernation. Therefore, disparity can't monitor, since its PC is not awake.
When adding or deleting files, 2 scenarios apply.
a. One is adding/deleting manually in person (either sitting in front of the pc or remotely). In other words, one KNOWS a change has been made. One can start disparity right after the change if one is paronoid. Otherwise just wait for the scheduled dispartiy run.
b. One has some automatic process to add/delete files unattended. If one is capable of setting up such a process, then it should be trivial for such a person to add dispartiy to the end of that chain. In this scenario, both the #1 or #2 disparity options can be disastrous if disparity updating is started independantly of the other automatic add/delete process. You will have to made at least one of them aware of the other's state.
Instead of disparity options #1 or #2, my humble suggestion is:
i) create a command option that starts dispartiy scan/update automatically (much the same as the previous commnand line disparity). Users who need a scheduled run can then just add this command to Windows buildin task scheduler. (Why reinvent the scheduling wheel?) Those who uses automatic process (aka #b above) can then easily add this command to their process chain. Kill two birds with one stone.
ii) output some easily checked flags while disparity is updating (whether started manually or automatically). Example create a temporary text file named "updating" in disparity's home folder. Delete said file when updating is over. Or throw up a status window. Other scripts, processes etc can then check for this status before touching the protected drives.
Thanks for mentioning the NAS scenario, sub0. It's a good reminder that people use disParity in a lot of different configurations. Personally, my media server also functions as a server for several other things, so it never hibernates.
Of course any automatic update feature would be optional. And I don't see it as an either-or vs. a command line update option like you describe. A command line update seems like a useful thing for advanced users who like to heavily script and automate their servers, but it's not for everyone. Other people might just want to install disParity on their servers and forget about it.
Even in the hibernation scenario though, wouldn't it be nice if disParity automatically updated the backup after you woke up the machine to add a bunch of new files, without you having to do any extra configuration or scripting? Especially if you are accessing the NAS purely as a file server, in which case you don't have the desktop in front of you in order to manually trigger an update. This seems like it would be a nice feature to have, but maybe I'm missing something.
I'm not sure that the 2 scenarios you list are the only 2 that exist. For example, in my case, my girlfriend usually deletes video files off the server when she's done watching them. I may or may not be around when that happens. Since deleted files put the backup integrity in jeopardy, I would really like disParity to automatically update a few minutes after she does that. Yes, I would lose the ability to undelete those files, but that's a choice I would make as a user based on my personal preferences (i.e. the ability to undelete a file is less important to me than the risk of losing other important data if a disk crashed before the backup was updated.)
Anyway, please keep the suggestions and scenarios coming. The more people that chime in on this, the better the final result will be.
Hello,
Y es, of course there are many scenarios. The 2 I mentioned are from 25000 ft up looking down ie manual vs unattended file changes. Below each there are tons of variations, including hibernate or not, headless or not, etc.
For me, the primary function of your excellent code is data protection. Besides media, I also use it to protect other large, unchanging data such as boot disk images etc. My own rule is nothing except disparity itself is allowed to touch the parity drive.
For a headless useage, I think it would be dangerous for disparity to update automatically.
Example: I start remotely to add/delete files to/from data drive B while disparity is just starting to update the parity drive. That in itself would already be bad, right? To make matters worse, say a power failure occurs then. Data drive B may be corrupted by my writing, while the parity drive may be corrupted by disparity's writing. I would have no way to recover data drive B. For headless use, disparity should not be allowed to write to the parity drive unless it can prevent other software/human from writing to the data drives at the same time. Same goes for unattended use (meaning automatic file add/delete situation).
The case of deleting files after watching, you're right. It is up to the user to make a choice. My choice is protection of important data, just like you. So I chose not to put unimportant data into disparity. My wife's daily soap recordings go into an unprotected folder. So do my stuff that I don't intend to keep. Why even involve disparity with unimportant data that's going to be deleted soon anyways? That only makes fragmentation of the parity drive worse. That is why the Undelete feature becomes important to me; because only important data gets into my disparity data-set ;)
Anyway, I hope my comments will help you make your excellent software even better. Ultimately, you are the master who makes it happen, which ever direction you decide on.
If you are going to have an automated update (I'm against that) you might want to consider having some type of delta backup like windows previous version.
Automation (real time or timed) adds a level of "shot my own foot off" complexity. Your example is a great indicator of problems. What if your girlfriend went through your pictures and accidentally deleted a few hundred very special pics? Ones that you haven't had a chance to properly back up to an outside source or DVD or what ever? Those pics are gone forever. Worse, what if you are using something like McAfee VirusScan and it goes off its nut and decides to delete anything that's larger than 1 gig in size (if you don't think that's possible, you haven't used McAfee products before - I've HAD to use McAfee for the last 15 years in my job and I have seen every possible mess up you can think of)you could be out 100's of files from a media collection or personally created videos.
For me, I use disparity in a multistage layering of protection.
1) Protect at disk level (in case drive fail)
2) Protect at file level (In case i get stupid and go delete crazy)
3) used as an overall safety net until my off-site back up routine kicks off
I'm still loving the idea of an automated update, of course with taking care of possible human writing interfering with it, but I trust in Roland to find a way or he will not think to implement it.
For deleted files, as they are on disks in pure ntfs and usually on a known one, if really you want to undelete them you have a lot of softwares to do that (or even the trash basket ;) beside unedeletion via disparity and auto update I vote twice for auto ubdate, for me real deleting by mistake are rare.
For antivirus , what a strange crazy mcAfee I'm glad not to use it, but for my data pool under disparity I'm use to put all its drives on exception for antivirus, not for such crazy behavior (but now I know it is possible) but also for less antivir computing time (and some "bad" tools I can store ...)
I even banned ms media player wich dare to change mp2 tags, and would like to find a way to avoid time stamps changing as we can in linux zfs for example ...
of course all that is not conficting with others ideas as long averything is optional and manageable, if Roland can find time to do all that we will be a lot to find our happyness :)
Klaatou does present an interesting point. in case of deletes, there are more than enough recovery programs that can help.
After thinking about this for a while - perhaps I was a bit hasty. There really isn't much difference from putting a scheduled task to update and having disparity update automatically.
One idea to add - if you do make it real time - mark in the log file how a file is added/deleted - i.e. mark it as added/delete through the manual update or through the automatic update.
Lots of great points and ideas here. Hard to respond to them all. Here are a few of my thoughts on what has been said:
First of all, everyone should keep in mind that however the feature is done, it will always be optional. If for your particular scenario it doesn't make sense, just leave it off.
The undelete feature has always been a sort of freebie. It was never intended to be a core function of disParity, rather it was just one of those things I realized along the way that disParity could do easily using the code already written, so I added it. Personally, I've never accidentally deleted a file and am not particular worried about that case vs. the far more worrying case of a deleted file compromising the entire backup because an update hasn't been run. Again, this is my personal preference, so if the option existed for automatic backup, I would use it and willingly give up the ability to undelete. Other folks may choose differently and decide not to use the automatic update feature.
Also, keep in mind that there are far more people out there using hardware solutions like unRAID to protect their media servers, and as far as I know all those solutions have instant-update behavior with no ability to recover a deleted file via the RAID system.
Regarding the danger of a power outage, well, all I can say to that is that if you do not have your server containing precious data plugged in to a UPS, disParity can't save you. Most RAID systems, in particular RAID5, are vulnerable to data loss and/or corruption if power is interrupted during a write.
If you do happen to lose power while disParity is running an update, I would strongly recommend running a verify after power is restored. It will have a good chance of finding and fixing any corruption in the parity.
There seems to be some concern about updates "interfering" with changes being made at the same time by the user. This is a valid concern, but perhaps not as bad as you would think. The only way the update process would interfere is if the user tried to write to a file at the same moment disParity is reading it, because disParity locks files for writing while it is reading them. I don't like locking files, but I feel this is preferable to allowing another application to write to the file unnoticed during the read, in which case the backup would be compromised. Meanwhile, if another application has a file locked preventing disParity from reading, disParity will just skip it.
If another application reads or writes other files elsewhere on the disk during an update, that should also be fine. An update only processes those changes noticed during the scan. I've tried to imagine all the various ways files could change out from under disParity during an update, and I've written the code to handle all the cases I can think of.
sub0 makes a good point about why bothering to protect with disParity stuff that is going to be deleted anyway? Well in my case, it's just simpler. All the TV stuff goes in one place. Some of it might sit there for months before being watched. I want it to be protected as long as it hasn't been watched yet. If it were on a unprotected drive that failed, that would be a pain because I wouldn't know what has been watched and what hasn't, and I may not even be able to get the unwatched shows again.
In my opinion what would be perfect is an option like:
Update parity data "5" "mins" after last detected file change.
Force parity update every "2" "hours".
Verify/Maintenance every "24" "hours"
If all statements were checkboxes, and it would be up to the user to decide on the options between the quotes, this would provide every user with all combinations of options. It may seem pointless to have both options active at the same time, but at least users have the choice if there is some obscure situation that requires it!
Interesting idea to support operations other than update from the schedule settings. That hadn't occurred to me.
Although verify might need to be on a weekly schedule...on some systems it can take more than 24 hours to complete!
A few considerations:
disParity isn't well suited for drives written to all the time, and not really for realtime either; compared to a RAID5 type solution with three drives including parity: give it a write operation of two blocks in one file, and it will compute a third and write three blocks to three drives more or less simultaneously. With a snapshot solution like disParity with each file residing on one drive, then the file system will write two blocks on one drive, then disParity must read two blocks on another, compute, and then write 1 gigabyte to a third. Even in the least time-wasting case of one GB to be written, then there will have to be written 1 GB by the file system, then disParity will have to read read 1 GB, compute 1 GB and write 1 GB. That isn't much faster than half speed of a RAID5. So does that mean that realtimeness shouldn't be a priority because those who need it wouldn't choose disParity in the first place, or does it mean that those who choose disParity would do so few overwrites in total that they don't mind being slowed a bit down? (I don't know, but it is maybe worth a thought.)
I do not know whether there is any way for disParity to (I) monitor files system changes in real-time, and/or (II) to actually interfere with i/o. If there is no way for disParity to interfere in real-time (or it isn't desirable, for the above considerations of speed), then my gut feeling is that disParity shouldn't start slowing down my reads/writes significantly until idle. However, if there are ways to get real-time reports from Windows of what files are changed (without scanning entire drives all the time), then what about the following: once I overwrite a filefoo.bar, then disParity can start immediately to process the remove from the parity drive in order to close the timespan where my change puts other files at risk. (Then later when the system has been idle for some, time, it can add, as that requires read operations on my working drives.)
Is there BTW any way you can monitor the Recycle bin? If I want to delete a file, then again I'd prefer that the operation does not put other files at risk. If I bin the file, then it is there, but with a clear hint that it is going to disappear soon. The user could wish an option to remove it silently from parity. (Or maybe have it moved to a separate trash.dat on the parity drive .)
Just a few randon thoughts ... hope they were worth the time taken to read them.
then the file system will write two blocks on one drive, then disParity must read two blocks on another, compute, and then write 1 gigabyte to a third.
Whoa, whoa, wait...what? You lost me there. How did you go from two blocks (128K) to 1GB?
Is there BTW any way you can monitor the Recycle bin?
There might be, but Windows makes it difficult for apps to look at the Recycle Bin for some reason. If the point is to just detect deletes, the same thing can be achieved by monitoring the actual data drive for deletes using FileSystemWatcher, without messing around in the Recycle Bin.
FileSystemWatcher actually provides a lot of detail about what is happening on a drive in the form of files being created, deleted, changed, etc. all as individual events. So in theory it's possible to figure out exactly what happened on the drive simply from those events without even doing a scan. However I've fiddled with that a lot in the past and I've found that it's damn tricky to reliably convert the FileSystemWatcher event stream into a clear list of the individual adds, deletes, moves, and edits that comprise an update. It turned out to be far more reliable (and far easier) just to do another scan once I know *something* happened.
How did you go from two blocks (128K) to 1GB?
That's the size of the parity*.dat file, which has to be overwritten.
... or do you overwrite that in merely blocks like, say, a tagger writes merely the metadata part to an mp3?
Windows makes it difficult for apps to look at the Recycle Bin for some reason
What about a “trash folder” on the drive itself? Move a file there, and disParity will know that it's going to disappear soon? (It is maybe not worth it compared to a full scan, except for us who protect remote drives which take ten minutes to scan over ... and you should maybe not prioritize us.)
BTW, what happens if I merely move a file and not run an update; will I still be protected, or have I then put files on the other drive(s) at risk? In detail: I move X:\folder0\file0.txt to X:\folder1\file0.txt or to X:\folder1\file1.txt (dunno if that matters), and drive Y crashes before update – will disParity be able to find the new location and recover as if the move didn't happen?
... or do you overwrite that in merely blocks like, say, a tagger writes merely the metadata part to an mp3?
Yes, that's right. Only the blocks that changed get overwritten, the rest of the file is not re-written.
It is maybe not worth it compared to a full scan, except for us who protect remote drives which take ten minutes to scan over
When you say "remote" do you mean network drives? Yeah, those are going to be pretty slow to scan.
what happens if I merely move a file and not run an update; will I still be protected, or have I then put files on the other drive(s) at risk?
Sort of, but not really. Moves are actually processed as part of the scan, not the update. So, in this case all you would need to do is scan the drive with moved files first before the recover, and all would be good. If you don't scan it, then disParity would not be able to recover the lost drive because it wouldn't be able to find the file in its old location (and it's not smart enough to go hunting around for it by itself.)
One additional item to think about - what do you want disparity to be used for. Is it a drive protection program (if a drive dies, you can get it back). Is it a safety net program (oops, I just deleted a bunch of picture by accident and need them back quickly). Is it a layer in a back up process (I'll first ensure my data is safe with disparity then I'll back up to some remote storage facility). Are you going to take it to a level that mirrors some of the functionality of ZFS (automatic file health monitoring and repair in case of corruption).
Roland, you have a ton of possibilities open to you at the moment. You need to make sure to ask "just because I can add this feature, does disparity need this feature" Don't take a great product and make it gimicky by adding a lot of "neat" features. The greatness of disparity is its simplicity. I see data, I protect that data, I protect that data well.
Also - verify should be on a weekly basis at the least. You shouldn't have to scrub your data more than once a week.
Ah yes, the dreaded "feature creep", the scourge of software projects everywhere. I hope disParity never falls victim to that. Right now all I want it to be is a solid backup solution that hits all the basic needs of someone wanting to safely backup their media server. I think that wanting the backup to operate in an unattended mode is a reasonable expectation of most backup software.
So I'm not sure whether you are suggesting that an automatic backup feature is "gimicky." It's something that several users have already asked for, and something I would personally want, so I don't think it is. It almost seems like it's the one significant feature left that would make disParity a complete basic backup solution.
You must log in to post.