The support forum

Issues with Flushing Write Cache (Local External USB 3.0 --> Network Mounted Drive)

User910590 :

Jul 09, 2017

Hello,

I just downloaded Bvckup 2 trial after hearing about it  potentially being a replacement to my trusty Microsoft Synctoy 2.1. I'm trying to create a weekly backup of my external drive to a Network Mounted Google Drive (using NetDrive) for large files (videos, sometimes in excess of 10GB each).

Synctoy was having issues writing fast enough to my Network Drive. With Bvckup, I notice that the file copies very fast, but then it gets stuck on Flushing Write Cache after every single file. This can last anywhere from 10sec for 100mb files to upwards of a a few minutes for large files. This is obviously very slow compared to just straight copying from Explorer to my Network drive.

Is there anyway you can think of that could help me speed this up? I have tried opening the individual settings file for my backup (after app is closed) and altering the below 4 lines, but whenever I reopen Bvckup, it reverts the Flushing line items to "2". The max_io_buffers do seem to stick though.

        conf.copying.async.max_io_buffers  2
        conf.copying.delta.max_io_buffers   2
        conf.copying.async.flushing  0
        conf.copying.delta.flushing  0

I am not sure what is faster - delta copying or full copies as I'm still doing my initial copy. I have it on Delta for now but can switch to full copy if you guys advise for speed sake when doing Local --> Network.

Any help really with the flushing write cache issue would be greatly appreciated. This is the only thing holding me back from purchasing the pro edition right this moment.

Alex Pankratov :

Jul 09, 2017

It should be

        conf.copying.async.flushing  1
        conf.copying.delta.flushing  1

1 is "Flushing is Off"
2 is "Flushing is Off unless it's a removable drive"
3 is "Flushing is On"

I am not sure what is faster - delta copying or full copies as I'm still doing my initial copy. I have it on Delta for now but can switch to full copy if you guys advise for speed sake when doing Local --> Network.


Depends on whether you have a fair amount of larger files getting _updated_ between the runs. Conversely, if you are just adding new files to the backup with each run, there's no point in delta copying.

PS. I hid your other post in "Release 72" thread.

User910590 :

Jul 10, 2017

Thanks Alex. I guess my issue was I was using "0" instead of "1" haha. I purchased the software regardless, as its awesome.

Two questions:

1. Is there any risk to turning off flushing? I don't truly understand the reason for it or what it does, but if there is no risk, I will turn it off to speed up my backups.

2. I have a large amount of 5GB+ files (~1000 files) and a lot of 200-1000MB files (~10,000), plus many thousands of very small files (less than 2mb). With that said, I do update a good mix of the 5GB+ and 200MB-1000MB files. Keep in mind these are all video files. Would you recommend I still use delta copying or full copying in this case? I know there is no difference between delta or full when adding files, so it is only relevant to my updates of these large media files.

Alex Pankratov :

Jul 10, 2017

1. Is there any risk to turning off flushing?


Flushing ensures that all data that the is _queued_ to be written is actually written out.

Normally, Windows would accept write requests and actually put them through to the device a bit later on - this is called "write caching". This presents a problem if the device is removed or disconnected right after a program (like Bvckup 2) reports that the writing is completed, whereby in reality Windows is still busy writing stuff out to the device.

2. Would you recommend I still use delta copying or full copying in this case?


Generally, yes, this sounds like a good case for delta copying, but you can evaluate it yourself - run the backup, look at the last entry in "Running the backup" section of the log and compare "read" and "wrote" counters:

        Completed in 52 min 33 sec with no errors
                Read 256 GB, wrote 1.08 GB, throughput ...

If they are substantially different, then delta copying is beneficial. But if they are nearly the same, then it's not.

User910590 :

Jul 10, 2017

great, so i guess the only issue with turning off flushing is as follows:

I turn it off, and say there is something still in the writing cache and the network drive (destination drive) disconnects from a network error, or my source (external USB) disconnects from accidentally bumping the USB cord.

Bvckup will report that the write has been done, but Windows may have actually not written it as its in the cache. This would cause issues ONLY at the destination (network drive) where the backup is being stored.

Would just rerunning the backup detect this failure and just overwrite it? Therefore there is no real risk if you are running a consistent backup schedule?

Alex, this is incredible software btw. I used Hamachi dearly back in the day. People like you make real differences in the world.

User910590 :

Jul 10, 2017

Another issue I'm having - I have set the two commands below to "1" for one of my two backups.

        conf.copying.async.flushing  1
        conf.copying.delta.flushing  1

I made sure to close app, change settings then reopen app, and ran the backup, but it still is adding a large amount of time for flushing the write cache. Any idea what could be happening?

Alex Pankratov :

Jul 11, 2017

Would just rerunning the backup detect this failure and just overwrite it?


It will.

Therefore there is no real risk if you are running a consistent backup schedule?


The risk is with potentially having a backup in inconsistent state after an aborted run and before the next one.

I made sure to close app, change settings then reopen app, and ran the backup, but it still is adding a large amount of time for flushing the write cache.


If you don't see xxx.flushing entries reverting to 2 after the app is restarted, then there are in effect and there should be no explicit "flushing" state.

Also, if you've switched bvckup2 into service mode, you will need to stop bvckup2 service as well before changing the INIs.

... but it still is adding a large amount of time for flushing the write cache.


What are the exact symptoms that make you say this?

galileo :

Jul 12, 2017

Alex:

How are NAS and network (LAN) drives treated with respect to this setting - i.e. removable or fixed?

galileo

Alex Pankratov :

Jul 14, 2017



They should be recognized as fixed. You can check this by looking in bvckup2.log:

2017.05.18 ... en | Enumerating drives ...
2017.05.18 ... en | Drive C -> type 3, rem n, bitlocked 0, tc 0, [], ...

It's the "rem(ovable) n(o)" part.

galileo :

Jul 16, 2017

Ahh yes...found and verified...thx

The reason for my question has to do with an ongoing issue that I have not been able to resolve:  

After a reboot and before manually "accessing" any mapped NAS shares, bvckup2 will always generate a "destination not accessible" error when attempting to backup to the share on the NAS device.  I have tried this both with and without adding the logon credentials within bvckup2.  The shares are persistently mapped.  If I have manually accessed any of the network shares on the NAS prior to bvckup2 attempting to run against the share, then bvckup2 finds the share.  The NAS is a QNAP device with the latest firmware.  My OS environment is Win 10 Pro with CU (1703) fully updated.

Basically, the issue is that if the system performs an overnight reboot, then bvckup2 will not find the network share when trying to run the next morning...at least until I have manually accessed the NAS device after the reboot.

Alex Pankratov :

Jul 16, 2017



I believe this has to do with how Windows caches/stores share credentials. In recent Windows versions (Vista+) there appears to be some logic that requires an interactive access to a share to "unlock" its credentials from the protected storage and establish an actual SMB connection.

That said, Bvckup2 has a piece of code that attempts to resuscitate mapped drives using poorly documented Wnet function. It *used* to work fine, when I last checked it a couple of years ago, but Microsoft might've patched thing up since then.

Take a look in bvckup2.log for entries starting with "Netmon". These are added by the "network monitoring" module, which is also responsible for the drive resuscitation. This module kicks in when the app needs to check the status of a network share or a mapped drive referenced in a backup job. See if you have anything interesting logged, in particular any errors for WNetRestoreSingleConnection() and WNetAddConnection2() functions.

Finally, the ultimate fix for this is to switch from using mapped drives to using explicit network paths + specifying logon credentials in the app itself - https://bvckup2.com/support/forum/topic/413

galileo :

Jul 27, 2017

Thanks for the reply...sorry for my delay in "returning serve"...

Regarding explicit network credentials, that is what I have been doing and the problem still persists.  

The "backup to" path is:       \\QNAP\Storage\Backups\ATL\Users\

...don't waste too much time on this...my fix is just to make sure that I have accessed the network shares at least once after a reboot...not optimal but functional nonetheless...

New topic

Create
Made by Pipemetrics in Switzerland
Support


Follow
Twitter
Dev blog
Miscellanea Press resources
Testimonials
On robocopy
Company
Imprint

Legal Terms
Privacy