The support forum

Remote Agent for Destination File Hash Computing, etc.

justbackmeup :

Aug 24, 2017

Greetings,

I'm a potential new user, and am really liking everything I'm seeing/reading about Bvckup! :) I so appreciate your focus on quality of quantity, and that your software doesn't lock the user into any proprietary storage formats.

I have a few questions/requests, but first some context/background: I will be primarily using Bvckup on a computer to backup data residing on my local Synology NAS to several NAS' remotely located (which will be accessed via my router-to-router VPN arrangement). I can access the remote shares locally (because of how my VPN is configured) via UNC paths (and the same UNC paths can also access those remote shares via a Windows computer at the remote location). I will be backing up a mix of files, but ~10TB (and growing) are large video files of varying sizes (ranging from 300MB to 40GB), which will rarely (and in most cases never) change. I am aware (and very grateful!) that I can seed my initial backups, so that is not a concern.

But here are my questions:

1) Integrity Checking: I've seen the following posts indicating that the ability to schedule periodic integrity verifications of the backed up files is a forthcoming feature.
https://bvckup2.com/support/forum/topic/605
https://bvckup2.com/support/forum/topic/713
I also gather (and appreciate) the ways in which Bvckup works quite well without having a 2nd piece of software at the destination. However, I am wondering if you would be willing to consider providing the option for installing a 2nd copy (or just a utility version of the software) at the remote location for the purposes of performing expensive network tasks (when going over WAN via VPN, as I am doing) such as re-reading
destination files for purposes of integrity checking (or any other reason that the destination files might need to be re-read).

Syncovery (which I've used in the past) is one example of this sort of thing (though it does not currently support the above kind of file integrity checking I'm wanting): https://www.syncovery.com/remoteservice/

I have toyed with trying to achieve remote integrity checking totally separate from backup software by using something like Checksum ( http://corz.org/windows/software/checksum/ ), but I'd much rather have this built into/with something like Bvckup.

2) Delta Copying: I think this feature is great from what I've read ( https://bvckup2.com/support/forum/topic/739 ). But I am wondering about any possible enhancements for handling "abortive cancellation" scenarios? In particular, if (1) above were implemented, would it be possible to write the changed blocks to a temp file(s) on the destination first, and then let the remote Bvckup utility take care of integrating the blocks into the destination file (and there would be the added benefit that the remote utility could do an integrity check of the whole assembled file at the end to be sure everything matches whatever hash data was sent)? Basically, I'm trying to avoid any possible corruption of an interrupted transfer as well as the possibility of a full re-copy being required after an interrupted transfer (which would be problematic for a 30GB file over WAN via my VPN).

Thanks again for providing such excellent software -- it's a real pleasure coming across folks like yourself doing such quality work, and clearly taking pride in your work. :)

Alex Pankratov :

Aug 24, 2017

I am wondering if you would be willing to consider providing the option for installing a 2nd copy (snip) at the remote location.


Yes, this is in the plans.

This will allow for several very interesting things, including rsync-style two-process copying and file comparison (without a need to store block hashes between the runs), potentially faster custom network protocol for bulk transfer and few other things.

No ETA on this yet though.

Would it be possible to write the changed blocks to a temp file(s) on the destination first ...


It's an option for sure. However a simpler option would be to mimic what rsync does and simply make the process on the backup side create a temporary copy of a file upon detecting a change and then build this file from unchanged local blocks and new remote blocks. Once the temp file is done, move it on top of the previous version.

PS. Re: aborted transfers - what happens in this case is that bvckup2 remembers which part of destination file it processed last and on the next run it will pick from that point (assuming that timestamps on destination file didn't change between the runs).

justbackmeup :

Aug 25, 2017

Yes, this is in the plans.

This will allow for several very interesting things, including rsync-style two-process copying and file comparison (without a need to store block hashes between the runs), potentially faster custom network protocol for bulk transfer and few other things.

No ETA on this yet though.

Fantastic! I very much look forward to all of that! :)

However a simpler option would be to mimic what rsync does and simply make the process on the backup side create a temporary copy of a file upon detecting a change and then build this file from unchanged local blocks and new remote blocks. Once the temp file is done, move it on top of the previous version.

That all sounds good to me.

Re: aborted transfers: I was interpreting that the following meant that under certain (perhaps extreme or edge cases) a file would have to be re-copied in it's entirety (ie, not just re-copy the changed blocks), which would be costly in the case of a large file.

For abortive cancellations the size-timestamp caching provision from #2 above will ensure that the file is re-copied in full on the next run.
https://bvckup2.com/support/forum/topic/739

Am I misunderstanding the above?

Another question: Much of my backup data (multiple TB's) is already at the remote destination. If I absolutely have to I can bring those NAS devices to the local source (which is what I was interpreting would be the normal way to do seeding), but I'm wondering if there are any other options/ways that I can leave the NAS devices at the remote location and setup/configure Bvckup to incrementally backup to those without it having to fully scan (as in, read every byte) all of the files (which would be problematic over WAN VPN)? This is assuming that I would have directory structures on the remote NAS' that matched up with the local source directory structures which I would be intending to backup.

justbackmeup :

Aug 25, 2017

but I'm wondering if there are any other options/ways that I can leave the NAS devices at the remote location and setup/configure Bvckup to incrementally backup to those without it having to fully scan (as in, read every byte) all of the files (which would be problematic over WAN VPN)?

I think maybe the following answers that particular question (though I welcome any additional suggestions, thoughts, etc.):
https://bvckup2.com/support/forum/topic/739/5239

I'm wondering though, would you be willing to add a hidden, advanced option (perhaps only available via an .ini flag or something) to just create/compute delta states for a job, without having to actually copy the files? I'm dealing with many TB, so it's a bit of a pain to have to arrange for that much temporary space for the temp copy operation (though that's still probably easier than bring the NAS' to the source location).

As a side note, I'm coming from CrashPlan, which I'm going to continue to use for backing up to their servers. But like everyone else I'm losing their excellent computer-to-computer backup (with their recent policy change), which is what is prompting me to explore other options. So my remote NAS backup data is stored in CrashPlan archives which I'll then be extracting there at the remote location (and thus I have high confidence the files should all be fine, as CrashPlan has built-in error checking, etc.).

Alex Pankratov :

Aug 28, 2017

Am I misunderstanding the above?


No, but I should explain a bit more how cancellations/aborts are handled.

When copy is terminated part way through, it may happen either due to a copying error or due to an explicit cancellation by the user.

In both cases the cleanup sequence is the same. The program will remember where the last completed write was and then it will try and get current timestamps from the backup copy.

If the copy is gracefully cancelled *or* if it was an error that did not render the backup file inaccessible (e.g. "out of disk space"), then Bvckup 2 will be able to get the timestamps and this will enable it to resume delta copying on the next attempt.

However if the backup file is not accessible anymore (e.g. "network path is no longer valid"), then it won't be able to get the timestamps, will record them as zeroes and this will cause the file to be recopied in full on the next run.

Much of my backup data (multiple TB's) is already at the remote destination.


Bvckup 2 looks at a triplet of "created time", "last modified time", "file size" to determine if a backup copy is out of sync with the original. Any difference in these will trigger a file update.

HOWEVER on the very first run it will use a weaker criteria - it will only require "last modified" and "file size" to match and it will allow "created" times to be different. This is done to allow taking over backups that are created by conventional copying (which replicates the "last modified" time, but set "created" to the time of copying).

Bvckup 2 will still process all files that have "created" out of sync with the source, but it will merely reset this timestamp and that's it. This is a very fast operation.

---

Now, what you can do is use the "Simulate" option from the right-click menu to check what exactly the program is planning to do if you are to run the backup.

That  is, you'd create a backup job, set it to manual (to prevent from auto-running after it is created), simulate the run and then look at the log to check if it's going to copy file or merely update their timestamps.

I'm wondering though, would you be willing to add a hidden, advanced option (perhaps only available via an .ini flag or something) to just create/compute delta states for a job, without having to actually copy the files?


This is a *very* good idea. Let me see what we can do.

CrashPlan


Yep, I've been watching this development with interest.

My guess is that they dropped the personal option in part (if not exactly) because they had to maintain the backend for the computer-to-computer option, which didn't earn them a dime and probably didn't convert that well to their paid personal cloud option.

justbackmeup :

Aug 28, 2017

Thanks for those details -- very helpful!

Bvckup 2 looks at a triplet of "created time", "last modified time", "file size" to determine if a backup copy is out of sync with the original. Any difference in these will trigger a file update.

HOWEVER on the very first run it will use a weaker criteria - it will only require "last modified" and "file size" to match and it will allow "created" times to be different. This is done to allow taking over backups that are created by conventional copying (which replicates the "last modified" time, but set "created" to the time of copying).

Bvckup 2 will still process all files that have "created" out of sync with the source, but it will merely reset this timestamp and that's it. This is a very fast operation.

What do you think about another advanced/hidden option to temporarily enable this weaker criteria mode? This is probably quite an edge case, but I could see this being useful in circumstances where I've added some files to a location at the remote destination (perhaps I've copied them from one device to another [both at the remote location], and so the "created" time is out of sync with the source, but everything else is good) and they are to be part of already existing backup job. Although, I guess in that case I could just create a new job if I had too and disable the old one.

I'm wondering though, would you be willing to add a hidden, advanced option (perhaps only available via an .ini flag or something) to just create/compute delta states for a job, without having to actually copy the files?


This is a *very* good idea. Let me see what we can do.

Awesome -- very glad to hear that! :) And if you're thoughtful about how you write it (which I'm sure you will be) you can later (some day) fold this code into the above remote agent functionality ( https://bvckup2.com/support/forum/topic/965/5330 ). :)

Alex Pankratov :

Aug 28, 2017

What do you think about another advanced/hidden option to temporarily enable this weaker criteria mode?


It's already supported - https://bvckup2.com/support/forum/topic/139

justbackmeup :

Aug 29, 2017

It's already supported

Excellent -- thank you!

justbackmeup :

Sep 01, 2017

I'm wondering though, would you be willing to add a hidden, advanced option (perhaps only available via an .ini flag or something) to just create/compute delta states for a job, without having to actually copy the files?

I'm wondering if you have a rough sense of when you might get to work on this? Not expecting any commitment to a time frame, but your current guess of the timing would give me something to work with as I considering the timing of things on my end (ie, my migration from CrashPlan to Bvckup for certain backups).

Also, if/when you implement Remote Agent functionality (someday), I would find it useful if the Remote Agent had the ability to initiate a Pause of inbound backups. This was a really useful feature in CrashPlan for my peer-to-peer backups as I do some video conferencing on the computer that is at the remote destination (where my offsite backups are stored) and it is handy to be able to pause the inbound backups if I'm needing the extra bandwidth.

Alex Pankratov :

Sep 01, 2017

I'm wondering if you have a rough sense of when you might get to work on this?


Let me try and hack something together to see the scope of the changes this would require. If it's too much work, then it might be possible to tack this on top of the backup verification when it comes out. If it's not, then we can probably get it into the next major release.

Personally I really like this sort of feature. It's simple, it has great impact when used properly. And the fact that it requires some extra intelligence to use is what makes it interesting :)

a Pause of inbound backups.


Noted.

justbackmeup :

Sep 01, 2017

Personally I really like this sort of feature. It's simple, it has great impact when used properly. And the fact that it requires some extra intelligence to use is what makes it interesting :)

You and I are of similar mind. :)

And thanks for being willing to give it a look.

justbackmeup :

May 05, 2018

I'm loving all the great new features in Release 79!!

Just checking in as to your thoughts on where in the timeline Remote Agent, etc. functionality might get implemented (based on your current rough approximation). :)

New topic

Create
Made by Pipemetrics in Switzerland
Support


Follow
Twitter
Blog / RSS
Miscellanea Press resources
Testimonials
On robocopy
Company
Imprint

Legal Terms
Privacy