Hello
I think disParity is exactly what I was looking for to protect my movies and mp3 ;)
I have 7 one tera disks and wanted to protect 6 of them on the 7th.
they are all identical but I have this error trying to create the parity :
Disk space required for parity: 933,55 GB
Disk space available on P:\: 931,42 GB
Insufficient disk space available.
is it a way to use my disks or am I forced to have a bigger one ?
thank you
Are there any files at all on the P: drive? What does Windows say for P: if you right-click on the drive and choose Properties, for Used space, Free space, and Capacity?
I just checked all of my 1TB drives, Windows says 931GB capacity for all of them. So the question is, why does disParity think that one of your drives has 933GB worth of data to protect?
One user recently saw this problem when one drive had a hard link on it to a folder on a different drive. DisParity isn't aware of hard links so this makes it looks like all the data is on the first drive.
Try doing a right-click -> Properties on all of your 1TB drives, see if Windows is reporting a value larger than 931GB for any of them.
Another possibility is that one of your drives (maybe your music drive?) has lots and lots of small files. Large numbers of small files can introduce inefficiencies into disParity because it rounds each file up to the next 64K when storing it in the parity. If the drive is completely crammed full (no space left) I can imagine a situation where this would cause it to require more parity space than what's available. If this is happening, your only real choice is to move some of those files to another drive that has more space available.
Drives are all the same type and I've just did a del and create full partition on P: to be certain there where nothing hidden on it ...
used space 93.5 mo (don't know where ;)
free space 931 go
capacity 931 go
all same for the 7 disks except the 6 others are not empty
Sorry, I just edited my first reply after you responded. What do you think of the many small files theory?
I was editing mine as well ;)
I didn't used any windows link directories features so it must be my many many mp3 ....
thanks, I will try not insert the mp3 disks to test !
Conclusion : it is the large number of little files wich needs a larger parity disk.
I will see how I can add another disk to span with the one tera parity one.
Thanks for the help ! :)
Do you mean by span to Join your existing TB parity wizh another, to build a bigger one (RAID 0 or JBOD)?
It should work, but first you loose your Parity, i know no way to do it without the need of reformat.
And ... you lower your security ... it sounds a little bit paranoic but if one of the two parity drives fail the whole parity drive fails (and you doubled the possiblity of parity failure by usind 2 Drives) so if then another Data drive fails you loose Data.
Better is to make shure that your Parity drive is always bigger than your Data drives.
Buy an 1,5 TB drive ... and reuse the old parity as Data drive.
I do the same ... my Parity is now 1,5 TB ... my Datas are 1TB ....
The next step is to by an 2TB Drive as Parity and reuse the 1,5 TB as Data ...
Another argument for this method is Lifetime of drive ...
I think in most scenarios the Parity drive has the most diskwrite access, so its aging faster then the others, so changing it more frequently is better, ... thats the way i see ist.
Yes I plan to span 2 disks with dynamics windows' disks and I'm aware I will have more "dangers" doing that but for now I must "digest" the cost of my one tera diskS before to have a bigger one ...
And as I cannot create any parity for now I will lose nothing ;)
But I don't see why the parity disk should be "overused", it is used "only" when creating or updating the parity so not on a dayly basis and in these cases all disks are used as the same time doing the XOR, am I wrong ?
Another thing I think about : as the parity asks me a larger disk for only 3 go, I can activate the disk compression on the parity disk to have this 3go more, but how can I fool disparity ? ......
>> Parity Disk "overused"
Thats why i wrote "... thats the way i see it. "
There is no proofe that the way i do it is better .....
>>compression on the parity
I didnt use the Windows Compression much.
If FreeSpace of the drive increase then it works ...
I dont know exactly but if you activate compression arent only new files compressed ?
If thats so then possibly delete Parity data ... NO it dont help ...
There is no way windows know how much space he earn by compression before he do it, and so he cant show more free space as there in real is.
By Compression you dont get MORE space, you only LOOSE less space by DATA.
Free Space and size of the Disk cant change by compression.
But you can have more Data on the Disk as the physical size of the Disk.
Dont put too much hope in WinCompression, the Compression Rate is verry small.
If Roland implements the "Disabele Parity Free Space Check" Option i suggested then it works ... for now i cant figure a way to fool disparity.
What do you mean with 3go ? I know KB MB GB TB ... but go ?
OK ... after rethinking ther IS a way to fool disparity .... maybe.
But is a verry verry weird way.
You have to part your Datadisks in to two halves and make 2 Parity Sets.
As Example:
DATADISK1 900 GB - SET1= 450GB Set2=450GB
DATADISK1 800 GB - SET1= 400GB Set2=400GB
.....
This can be don Physically by moving files to two seperate folders, or by Hardlinks
And on the Parity Drive you have 2 seperate Folders for ParitySet1 and ParitySet2
You have to delete the old Parity from the ParityDisk.
You need 2 seperate Config.txt files one for each Set.
OK ... Now you Buld your Parity for the First Set/ first halve of your Disks.
This result in the halve amount of Parity Data, but it dont use the same amount of space it would use ... yes the Compression.
So you result in more free space for the second halve of your data, and hope that its enough to do the second parity set.
Fot updates you have to do the updates for booth sets.
For recovery its the same ....
There is no higher risk of loosing Data, or am i wrong !?
But i dont think you want to do it this way, i wrote it only couse i wrote before that there is NO WAY, and that is not correct, ther is NO GOOD WAY to do it.
EDIT:
By ReRe-thinking there is a better way ....
You result in only 1 parity set.
You Do parity step by step.
Delete parity, activate Compression ... OK DONE
If you use the Root Folder in your Config.txt for dataX= then you have to make a subfolder laike Mydata or what else ... and change the Config like data1=G:\Mydata
Move halfe of your data to this folder ....
If your current config is already set to a subfolder of the Datadrives than leave your config unchanged and move halve of your data to a new folder at root level.
Repeat this for all your Datadrives.
The clue is to make at first step the parity only for the halve of data.
Do update ...
This leads in Compressd Parity Data ... halve of your data but hofully smaller so ther is more free space to do the rest.
Now you move back again the halfe of the left over data and do update.
You can repeat this 2-3 time or you can move back the whole rest of data if you think there is now enoughe parits space.
At End you have More Data on the drive as the Size of thePhysical drive.
I hope you understand what i mean ....
I diddnt coded disparity, but i think this way disparity can be fooled ... but only if Compression is strong enough.
When you activate compression on an existing partition you can ask to apply it on all current data on the disk or only to new data, but you can also activate compression when you create the partition so everything on this partition will be compressed.
BUT ! Yes windows compression can't predict the size of data space available as it depends on the possible compression rate of recorded datas and windows will always report the uncompressed physical free size of the disk when it is empty ...
Anyway, as the amount required for parity is 933,55 GB and Disk space available is seen as 931,42 GB the extra needed space is so small regarding the size of the disk I think compression will be enought ...
(my 3go was for the very little difference between this two numbers and for go= giga octets = giga byte = GB ;)
I understood your ideas : as long as the size checking is not a warning but stop the parity creation I can't start this creation even with compression and you probably found the solution with the last idea to move less datas in datadrives only for the first creation.
If I have the luck reported free space is large enought for the "last" update for all my datas I will be saved :-)
It will be a long process but thank you for your help, I think I will be able to use my disks !
Yessss ....
Thats exactly what i mean ...
I hope you succeed ... if not you spend a lot of time to be again there where you started ..
I'm not 100% sure how disParity will interact with drive compression, I've never used it. Since most media data (.mp3, mpeg, jpeg) is already compressed, I don't think drive compression will help much. But since drive compression is "transparent", I don't think it will make any difference. At the level that disParity interacts with the file system (on a per-file basis) the fact that the drive is using compression underneath won't change anything about the file sizes or their contents as disParity builds a list of what needs to be added to parity. I also suspect using compression on the parity drive will provide little or no improvement. But it will definitely impact performance.
There's a good reason why disParity stopped the create in this case. The create would have failed right near the end when the parity disk ran out of space. An option to bypass the check would not have helped, you would still not have had a valid snapshot at the end.
You can approach this problem a variety of ways, but personally I think by far the easiest would be to identify a couple of GB of data on the problem drive and just move those files to another drive. Problem solved in 30 seconds.
Roland, your wrong and right ...
To move some GIG of data out of the parity set work and sovle the problem ...
But that wasnt the question of Klaatou.
Drive compression is transparent, the only difference is that the space used by parity date is after completing the job, less than calculated at start.
And we talked about compressing not the DATA ... thats nonsens ... exatly like written by you ...
But if you miss only a little amount of space on your Parity drive, and if the compression of parity is enough disparity can complete the job.
In his firs post he talk abaut missing 3 GIG, if the compression do a bad job and compess only by 1% he get 1% of 900GB is 9GB extra free space ... thats more than the missing 3 GB.
And yes your right and wrong again, the option to bypass the chek helps not a little bit to have more Parity pace.... thats right ... if you start with an empty Parity Drive you get the REAL physical free space, even if Compression is on and disparity complain the lack of space, not knowing that he need 1-2% less space spared by the compression. Thats the situation the option wuld help.
So i told him to do it in more steps, if start with only the halfe of his data he get parityData of example 500GIG which are compressd by 1% only 495 GIG. So he win 5 GIG of space. diparity now see 5 GIG more freespace to complete the update for the rest of Data.
The sense of the whole thing is another question.... even if this works ... he has not enough parity space and this solution has no future ... but thats something Klaatou has to think about.
OK, but as I said, I also strongly suspect that compression of the parity will produce little or no improvement. The parity data is XOR'd data from multiple already compressed files (i.e. media data.) The resulting parity will be effectively the same as random data to a compression algorithm, and therefore uncompressable, not even by 1%. I just tried compressing a parity file with WinRAR, and it couldn't make it any smaller.
I'm not saying it's impossible, just unlikely. It really comes down to what types of data files are being protected.
Thanks god ... you understand me .... ;-)
>> I also strongly suspect that compression of the parity
Not only you ....
If you read my posts carefully you read :
"Dont put too much hope in WinCompression, the Compression Rate is verry small."
The 1% thing was an example to explain it to you ... you can read .. as Example ... i dont know how mutch it really is without trying ...
And to compress the parity drive wasnt my idea ....
Klaatou asked if ther is a way to do it by compression ...
The best solution is to buy a bigger parity drive ... and thats what i first wrote ...
Your suggestion to move files out of the ParitySet is also a solution but only the second best, cous the Data i moved out is not secured by Parity ....
And i know how Compression algorithms work ....
It least i know how LempelZif works ....
Im experimented 15 or so Years before, a little bit with them ...
And somtimes i uses the first stage of compression ... replacing longer Bitsequences by shorter one, in my DBs
-----
Am i a little bit sensitive, or do you not want that OTHERS than you give support or help ?
If its so, i stop it, its your Forum ...
But on one side you complain that have a break, and spend less time, on the other side it sounds like your disappionted if somebody else try to help.
I have the feeling you dont even read completly what ive wrote ...
Couse, if you do, we dont have to discuss things ... even if we have the same opinion ...
I think another point everyone keep forgetting - Disparity does not protect at the file system level (such as how UNRaid operates). Disparity protects data sets. Even though you are setting up a drive to be protected, it is still a data set. There is an additional tiny amount of MetaData that is captured with the protection.
In a very simplifies example think of disparity protecting as follows:
{Desired data + metadata} < {parity set} Where MetaData size is based upon amount of data being protected. So if you want to protect you 1 TB drive, you are actually protecting 1 TB of data plus the metadata disparity creates with the protection.
Roland, this seems to be a common issue. Perhaps adding an file to your release that provides a very simple example of how disparity protects might clear some confusion.
so I can confirm parity files are not compressed .... :(
Im sorry to hear that it was only wasted time .....
What else can you do .... not much ....
Hmmm. there are 2 reasons why parity is bigger than data ...
The paddingbytes added if a file is not exatly a multiplie of 64k,
so the paritysize for this file increase by 64K
it not really 64k, its 64k - the rest of (filesize/64k).
The other thing is the managementdata ... if you have a large amount of files
then there is more management data to store.
We can minimize the amount of paddingbytes and/or ParitySize ...
But only if not all of your 6 Drives are full.
if you have as example 3 Drives
1 Drive with 800GB and the second has 200GB, the third with 500GB data
you get something like 810GB parity
If you move 300GB from drive 1 to drive 2 you get something like 510GB parity
So, by balancing your drives (moving files from drive to drive) so that every drive has nearly the same amount of data, or at least your resulting biggest datadrive is smaller then the biggest one at beginnig, you can decrease the Paritysize.
Thats the theory, but ...
Have you the space to do it !?
And you destruct the arrangment of your Video and MP3 ...
Another possiblity is to ZIP the large amount of small files to one,
or a small number of Archive(s) ....
This result in less paddingbytes for parity and less managementdata...
But now its a little bit hard to hear the MP3s ... and you need space to do the compession ...
And NO ... its not compession again ... the reason is to build a small amount of Big files in place of the big amount of small files ... best without compression, let zip only join them ...
But now im really out of ways to try to solve it ...
You can try it if you want, but dont blame it to me if its again only waste of time.
> Your suggestion to move files out of the ParitySet is also a solution but only the second best, cous the Data i moved out is not secured by Parity ....
Actually what I meant was, move some of the files to another one of the protected drives that has a little more space left. Unless he can't because *every* drive is completely full, but in that case, the whole server config is almost useless because no new data can be added to it anywhere. If moving files around is possible, it seems like a superior solution to buying a whole new drive. But if all of his drives are completely full, then it seems like a new drive is in his immediate future anyway....
> Am i a little bit sensitive, or do you not want that OTHERS than you give support or help ?
Yes, I think maybe you are. :) Or maybe I'm coming across too strong. I'm glad you are here offering help. I'm not sure where you got the idea that I don't approve. All I did was point out that I don't think the compression idea will work.
> If its so, i stop it, its your Forum ...
No, please don't.
> But on one side you complain that have a break, and spend less time, on the other side it sounds like your disappionted if somebody else try to help.
I wasn't complaining, again, I was just setting expectations. And no I'm not disappointed!
> I have the feeling you dont even read completly what ive wrote ...
That may be true. You've made a lot of posts and some of them are a little long and I might be skimming them a bit too quickly. For that I apologize.
To Roland:
Ok then everything is fine ....
>Actually what I meant was, move some of the files to another one of the protected drives
Then i missunderstood you completly.
If you look back to the last post i directed to Klaatou, thats exactly one of the things i suggested in my last post. I named it balancing the drives.
Im sorry that my posts are so long, i try to make them shorter.
You must log in to post.