disParity » General Discussion

System.IO.FileSystemWatcher

(7 posts)

  1. dldummy
    Member

    Hi Roland ..

    At weekend i was a little bit curious ... so i taked a look into Source of Disparity ...
    >Monitor.cs ... As i see ... you experiment with System.IO.FileSystemWatcher

    How does your experiemnts go !?

    Somewhere in the past i experimented .. not with NET but with WMI-EventSinks, there is one Eventsink for Filechanges ... (fine is .. you can watch changes on another machine)
    Its another API but i think it use the same feature of windows ...
    My results where, if i remember right, that my Eventhandling (only VBA) was to slow ...
    If i copied a small amount of files it worked ...
    But Moving/copying 100 ore more files, resulted in massive loss of changes ....

    I now overvied MSDN ... There is an internalBuffer for the events .. it can overflow and this result in loss of events ... an OnError is then raised ... and you can set the BufferSize in 4K steps but it use NonPagedMemory (its not swapped) so its better to keep it as small as possible.

    I dont now of which Stadium (how old) the codefragments in v0.2 are but i missed the capturing if errors (BufferOverflow) and IncludeSubdirectories isnt set.

    The handling of events is TimeCritical ... how do you plan to implement ...
    Write changes to log .. and make parity asynchron by another thread, activated by PerfCounter/CpuUsage ore somthing simmilar ?

    And how can we be shure that we dont miss events ... or to be exact ... what to do if Bufferoverflow occures ... We know that we loosed filechanges but not for which files ?
    So we need manual scan ... so its not LIVE ...

    How much live do you think should live be ?
    I have the feeling that safe Real-Live can only be done by Hooks to filesystem like Rootkits or AV-Software do.

    And if live means real live ... we need VolumeShadowCopy ...
    But VolumeShadowCopy of 6 ore more drives at same time .... hmmmm !?
    And for me and some others then it start to be complicated ...
    I dont know how VolumeShadowCopy works wit As FolderMountedVolumes and CrossDisk hardlinks ... !

    I wrote some posts before that i dont need the Life Feature ... but that dont mean that its not worth to think about ways, how it can be realized ...

    Posted 7 months ago #

  2. Roland
    Key Master

    > so i taked a look into Source of Disparity

    the source? Do you mean the assembly info embedded into the .exe? Or did I accidentally publish the source code somewhere?

    Posted 7 months ago #

  3. dldummy
    Member

    I hacked your PC and stole the Source ...... ;-)

    No its ... DeCompile ....

    Example:

    {
    byte* bPtr1, bPtr2;

    byte[] bArr1 = buf1;
    if ((buf1 == null) || (bArr1.Length == 0))
    bPtr1 = 0;
    else
    bPtr1 = bArr1[0];
    byte[] bArr2 = buf2;
    if ((buf2 == null) || (bArr2.Length == 0))
    bPtr2 = 0;
    else
    bPtr2 = bArr2[0];
    long* lPtr1 = bPtr1;
    long* lPtr2 = bPtr2;
    for (int i = 0; i < 8192; i++)
    {
    lPtr1 ^= lPtr2;
    lPtr1 += 8;
    lPtr2 += 8;
    }
    bPtr2 = bPtr1 = 0;
    }

    .NET is not real Compiled, only translated into IntermediateCode ...

    Posted 7 months ago #

  4. Roland
    Key Master

    > .NET is not real Compiled, only translated into IntermediateCode ...

    Yes, I'm familiar with that, and with decompilers. But you referred to a source file by name: "Monitor.cs"... where did that come from? I'm just curious, I know .NET assemblies expose their public interfaces and dependencies but I didn't know that individual source file names are exposed in release builds.

    Anyway, yes, live mode has all of the issues you mentioned. Error conditions like buffer overflow, etc. would have to handled properly before I would be comfortable releasing anything. And a true "live mode" that would be comparable to something like hardware RAID, or even unRAID, would require a driver-level implementation that I'm not going to be pursuing any time in the foreseeable future. The best to hope for with disParity would be a "lazy" live mode, where it notices changes to the file system and updates parity a few seconds or minutes later.

    Because of the way FileSystemWatcher sends events to the application, it's turning out to be fairly complicated. There are a lot of cases that have to be handled correctly. For example, say a file is created, and then renamed to something else before disParity can respond. When disParity goes to process the create event, it finds that the file doesn't exist anymore. Then it goes to process the rename event and finds it's for a file that's not even in parity. There are many other cases like this that can get tricky if not handled correctly.

    Honestly, I'm not really comfortable discussing the intimate details of this feature yet. For all I know it may never see the light of day. I need to spend a lot more time on it first.

    Posted 7 months ago #

  5. dldummy
    Member

    Sorry for the delay in response, im a little bit busy at moment ...

    I dont understand your position to feel not comfortable discussing the intimate details.
    Do you fear that someone catch up the ideas and do it before you !?
    Or, are you a lonesom Cowboy, who fight alone against tha bad boys ? ;-)

    >>For example....
    We need VolumeSHadowCopy to freeze the system to a defined state ....

    Did you checkd the project of the other guy, who work on joining drives together.
    Its based on a library that looks interesting ....
    That could be a possible way to do it ....
    You dont have tho mess with Kernlmode drivesr, it provide Callback for the Filesystem Functions. ANd you get new Drives ... wich are completly under our control ...

    What do i mean with this .... our problem is now ... we are only informd whats happend, but we have no control over FS-Operation (FileSystem).

    WIth the other library we get new drives, the user has to use this one for his operations, all file operations to Paritydrives go thruw our control.
    If i did unerstand the principle right its a virtual drive, user perform operations on virtual drive, and we do it when and how we want on the real.

    WHitout this (only by get informations what happend) i dont see a point in timeline, where we can be shure that parity we make consits of actual and valid dataset.

    First at boot time we have a safe state, the data on all DataDives drives are consistente, we need somthing like an inactivity (modification not read) window, if ther are no operations on dataDrives for as example 5 seconds we switch our Virtual FIlesystem to Cachemode, all operations are cached but on the virtual FS for the user its seem like its done.
    We start Parity creation on real FS, if we are done, we work down the cached operations, and start to wait for the next inactivity window.

    ANother idea ... with VolumeSHadowCOpy, without virtual FS ... we wait for our inactivity window, but only until halfe time of the window, there we start volumeshadowcopy and wait the rest of the window if ther arent evets happend during creation of VSC (VolumeSHadowCopy).
    If not start Parity, if yes disable VSC and wait for the next window.
    But even with this scheme ... i dont now how paralell we can start VSC on all drives, i mean they have to start exactly at sane time, our our data we parity together have different ages.

    Posted 7 months ago #

  6. dldummy
    Member

    Supplement to DeCompile ...

    Just for fun i tried to ReCompile the Source i get ...
    But the tool did a bad job, there where a lot of errors ....
    They are easy to fix, but i have no intention to do that.
    I can see if this or that is done like i thinked, and thats all i wanted ..

    Posted 7 months ago #

  7. Roland
    Key Master

    For me, the point of a "live mode" isn't to guarantee a perfect parity backup at all times of a file system with arbitrarily changing data. The purpose is to make disParity easier to use. Right now, you have to keep manually running update, or set up an automated task, or something like that for disParity to be useful. The point of live mode would be for disParity to effectively run update itself when things change, and to do so in the least intrusive manner possible.

    Posted 7 months ago #

RSS feed for this topic

Reply

You must log in to post.