disParity » General Discussion

Update speed USB 2.0 vs 3.0

(6 posts)

Tags:

  1. gworrel
    Member

    I just started using disparity a couple of days ago. I have 25TB of data spread over 11 drives. I connected a new 5TB drive as the parity drive, added all the drives and started the update. It looks like it will take at least 7 days to finish. It is creating about 400GB a day of parity data. Does this speed sound right?

    Most of the drives never change, but I am adding data each week to one of the drives. Will future updates as new data is added go faster?

    I have some internal drives but also several external USB 2.0 drives. (Most are 3.0 compatible but my motherboard does not support 3.0.) I plan to replace all the smaller drives with 5TB drives whenever I can get the new drives at a good price. Would upgrading to 3.0 USB ports speed things up? Thanks.

    Posted 9 months ago #
  2. Roland
    Roland

    The time it takes to perform the initial update is mostly drive I/O dependent. 25TB is a huge amount of data. If you assume, generously, that a USB 2.0 drive can read data at, say, 50 MB/sec, then if you were to read each drive in sequence, 25TB would take about 139 hours, or 5.8 days to read it all. Now in the initial update it reads from all drives simultaneously, which should be faster than reading them in sequence, but you are also writing to the parity drive, which will compete with reading to slow things down. Overall, 7 days doesn't sound too bad for USB drives. You are almost certainly running up against the limits of your system's USB bus.

    On my system with 14TB spread across 10 SATA drives, the initial update takes about 20 hours. Keep in mind that the more drives you have, the slower parity builds because the more drives it has to read from simultaneously (i.e. each 1 GB of parity contains 1 GB of data from each drive, so if you have 12 drives, it has to read 12 GB of data to generate 1 GB of parity.) If some of your drives are smaller than others, you may find the parity build rate will increase over time, since it has to read from fewer and fewer drives the farther along it gets.

    I can't predict whether switching to USB 3.0 ports will help. Maybe a little, but probably not that much.

    Anyway, the good news is, once your initial update is done, subsequent updates will be fast. When you add new data, it only has to read the new files you added to update the parity. It doesn't have to start over from scratch each time.

    Posted 9 months ago #
  3. gworrel
    Member

    Thanks for the quick reply. So far I am thrilled with the program. Very easy to set up. I was concerned that if future updates took just as long that it might be a problem. But this should work fine for me. Thanks.

    Posted 9 months ago #
  4. gworrel
    Member

    I took a look at the progress this morning and had bad news. I had the following message on the screen: "Sorry, a fatal error interrupted the update. The request could not be performed because of an i/o device error. The update could not be completed."

    I restarted the update, but now I am thinking that it might be better to cancel it and restart it with only one or perhaps several drives instead of the whole group. The error doesn't tell me which drive had the error. What is the best approach?

    Posted 9 months ago #
  5. gworrel
    Member

    I see that the new update I started after the i/o error is only adding the first drive in the list rather than reading all the drives simultaneously. Perhaps this is by design?

    Posted 9 months ago #
  6. Roland
    Roland

    Yes, if the initial update was interrupted by an I/O error, you should start over. There's a good chance the parity snapshot is now in a bad state.

    You should be able to see which drive had the I/O error in the log file. The beta version has a button in the UI to view the log file. If you aren't running the beta, you can usually find the log file here: C:\Users\(username)\AppData\Local\disParity\logs\

    Once you figure out which drive had the error, I would start over but leave that drive out of the initial update. You can try adding it later once the initial update is complete.

    Posted 9 months ago #

RSS feed for this topic

Reply

You must log in to post.