The support forum

Unselecting/exclude sub-folders included when starting with empty list

ScottD-KMHA :

Apr 07, 2017

Here is a feature request I may have brought up, out of context or before it was possible, in another topic.  Now that "conf.dst.filters.as_source" feature is in place, perhaps this request would be feasible?

It would be very helpful if we were able to use the "What to backup" -> Details button -> Start with an empty list -> include -> and then click a top level folder (check box it), but then be able to unselect/uncheck/exclude a sub-folder below that.

Currently, you can only select to include the top level folder with no choice to then exclude the sub-folders or sub-files below that.  A lot of times we have continuously changing log files, cache files, or temp files that we don't need to waste time or resources syncing.  It would be very beneficial if we could unselect to ignore these sub-folders/files but still be able to target the remaining folders (at that same level) or any new folders/files added under the top-level folder that is check marked, etc.

Alex Pankratov :

Apr 10, 2017

This is already on a ToDo list, but it's not an easy change to make under app's existing design. This is due to the fact that the filtering system is organized as a set of 3 settings -

    1. Initial action - include all, exclude all
    2. A list of items with reverse action
    3. A general list of rules

It's exactly what you see in the UI. Allowing to exclude items of explicitly included folders will require changing the semantics of #2 and it's not a simple change.

That said, it's a very natural feature to have, so we'll get it done.

Meanwhile, the workaround is to use generic rules (in 3rd section) to exclude what you don't need. Not as convenient, of course, but should make do as a crutch.

ScottD-KMHA :

Apr 10, 2017

Fair enough.  Thank you.

ScottD-KMHA :

May 25, 2017

Just wanted to check back for any progress made on this request.  I've been struggling with this more and more lately with many of our newer systems.  My goal is to have a Bvckup2 set that is pointed to the F: drive of primary server as Source and F: drive of backup server as Destination, using "Start with an empty list" and then selecting the folders I need, BUT I need to ensure that any new folders added into the second level of the directory are automatically included.  

For instance, I select to include "F:\Exa\Images\2017\" and I DO NOT select to include "F:\Exa\Images\2016\"...but when a "F:\Exa\Images\2018\" folder is newly created into the "F:\Exa\Images\" folder, I need it to be automatically included.  The easiest way to do this is to simply select to include "F:\Exa\Images\" and then unselect "F:\Exa\Images\2016\".  As you know, this is currently not possible.

I've been experimenting with selecting to include "F:\Exa\Images\" (leaving every sub-folder selected) and then adding an exclude rule of "*\2016\".  I think it will work (testing now), but I foresee the more exclude rules that need to be added...the harder it will be to teach this to other installers or explain this in my instructions.

All that said...just checking on the status of having a hope this feature will be added to Bvckup2.  I've made some strides with the workaround, but would much prefer we be able to deselect the sub-directory of a selected top directory when starting with an empty list.

Alex Pankratov :

May 26, 2017

We didn't get to this yet, sorry.

This is one of the changes that will break a fundamental assumption that existing code is built upon, meaning that we'll need to redo (and retest) quite a bit.

I will have a bit of spare time next week. Let me have a look and I will be able to give you an idea when this might get done.

PS. The exclude rule alternative *will* work. You can also fix the exact level of the subfolders to be excluded using .\*\*\*\Xyz syntax. The leading . anchors the patter to the top of the source directory, so this particular rule will match Xyz folders in sub-sub-sub-folder at the source location.

ScottD-KMHA :

May 30, 2017

Very helpful.  Thank you.

Alex Pankratov :

Jun 14, 2017

Let me have a look and I will be able to give you an idea when this might get done.


Took a bit longer to follow-up on this, but the good news is that this is tentatively scheduled for the next maintenance release (77.10), so it shouldn't be long now.

ScottD-KMHA :

Jun 29, 2017

Fantastic.  This will give us greater flexibility, save setup time, and help us to make less exclusion rule mistakes (accidentally making exclusion wild cards too broad).  Looking forward to this release.

Alex Pankratov :

Jun 30, 2017

Scott, working on this right now. Rather than folding it into a smaller maintenance release, it will be one of principal new features of release 78. Please bear with us a bit longer.

Alex Pankratov :

Jul 12, 2017

An update - https://bvckup2.com/wip/12072017 - almost there.

ScottD-KMHA :

Jul 15, 2017

That looks perfect.  Any ideas on ballpark ETA for this 78 release?

Alex Pankratov :

Jul 15, 2017

Aiming for the second week of August. This is not the only large change in the release, hence the timing.

ScottD-KMHA :

Jul 18, 2017

Understood.  Looking forward to this one.

ScottD :

Aug 21, 2017

Just checking in on this one (78 release) for an updated ETA. I'm working on some documentation to share with other departments here, but was waiting for this feature before I finalize the instructions.

Alex Pankratov :

Aug 21, 2017

Got delayed till next week. I should have an exact release date by the end of this week, next Monday the latest.

Alex Pankratov :

Aug 25, 2017

OK, the next release is planned for Tuesday, Aug 29th.

Alex Pankratov :

Aug 30, 2017

Done and out - https://bvckup2.com/support/forum/topic/973/5363

ScottD :

Aug 31, 2017

Thank you.  Testing and then rolling out.

ScottD :

Sep 05, 2017

Bad news, it would seem about the new 78 release.  We tested and it appears to be playing not so nice with us again, by deleting or moving the folders/file we did not select  For example...

Backup from:  \\192.168.1.4\c$\
Backup to: C:\
What to backup: "Start with an empty list"
Then, include/exclude the following items:  

check mark "C:\KMMI Files"

First Ctrl+Go attempt, it appears to target almost all folders outside of the "C:\KMMI Files" folder we check marked (except "C:\Windows", "C:\Users", "C:\Program Files", and "C:\inetpub") and it moves them to the "C:\$Archive (Bvckup2)" folder.

And then another oddity, it seems to delete/move the folder you select (check mark) when the folder is checked but the parent folder is not checked (but does have small check on parent folder icon).  For example, I check mark the "C:\KMMI Data Management\Service" folder, but don't directly check mark the "C:\KMMI Data Management" folder.

Ctrl+Go and then find that "C:\KMMI Data Management\ Service" folder is moved to the "C:\$Archive (Bvckup2)" folder.  Actually, it looks like it moves that folder I checked and every other folder/file under the small check boxed parent folder as well.

Alex Pankratov :

Sep 06, 2017



Scott,

Tried reproducing this (including the "sync filter" setting) and we couldn't. It "works on my machine".  So I will need more details.

If you are OK with sharing your settings.ini file for the job (over email to support@pipemetrics.com), that would work the best.

If not - can I get exhaustively detailed step-by-step sequence?

I.e.
* Did you reuse the existing job or create a new one?
* If created, did you manually patch it?
...

ScottD :

Nov 24, 2017

Been a long time, but I'm back and I think we're still having issues with Bvckup2 reaching beyond the scope of what we assign it to, when it chooses which folders/files to delete or archive.  I'm working with one now in which I'm only selecting to sycn a single folder on the drive, but it is archiving/deleting everything else on the drive outside that single folder I selected.  

Here are the steps and logs...

Bvckup2 version is 78.11.

1) right click > Add New Backup
2) Backup from: \\172.16.22.99\H$\
3) Backup to: F:\
4) What to backup:  Start with an empty list
     File selection:  \\172.16.22.99\H$\DB_Backups\
5) Create
6) Click Go button.
7) Source scan looks to correctly only scan the selected folder (42GB).
8) Destination scan incorrectly scans the entire destination drive (48TB)

Instead of only 2 steps it shows it has16,590,835 Steps, in which it proceeds to archive seemingly every folder outside of the folder I selected (\DB_Backups\).

Do you know what we may be doing wrong here?

ScottD :

Nov 24, 2017

I've captured below part of the backup.log for this set, to show the issue...

2017.11.24 07:16:52.805 (UTC-7) 2 0 Created
2017.11.24 07:16:52.805 (UTC-7) 3 1     Configuration: C:\ProgramData\Bvckup2\engine\backup-0004\settings.ini
2017.11.24 07:16:52.805 (UTC-7) 2 0 Device status
2017.11.24 07:16:52.805 (UTC-7) 3 1     Source: network share / accessible
2017.11.24 07:16:52.805 (UTC-7) 3 1     Destination: local drive / present
2017.11.24 07:16:54.839 (UTC-7) 2 0 Running the backup ...
2017.11.24 07:16:54.839 (UTC-7) 2 1     Preparing ...
2017.11.24 07:16:54.839 (UTC-7) 3 2         Run number: 1
2017.11.24 07:16:54.839 (UTC-7) 3 3             App: 78.11
2017.11.24 07:16:54.839 (UTC-7) 3 3             Source: \\172.16.22.99\H$\
2017.11.24 07:16:54.839 (UTC-7) 3 3             Destination: F:\
2017.11.24 07:16:54.839 (UTC-7) 3 3             Copying: contents, timestamps (modified, created, accessed), attributes
2017.11.24 07:16:54.839 (UTC-7) 3 4                 Using delta copier
2017.11.24 07:16:54.839 (UTC-7) 2 2         Verifying configuration ...
2017.11.24 07:16:54.839 (UTC-7) 3 3             Normalized src path: \\?\UNC\172.16.22.99\H$\
2017.11.24 07:16:54.839 (UTC-7) 3 3             Normalized dst path: \\?\F:\
2017.11.24 07:16:54.839 (UTC-7) 2 2         Checking source location ...
2017.11.24 07:16:54.839 (UTC-7) 3 3             OK
2017.11.24 07:16:54.839 (UTC-7) 2 2         Checking destination location ...
2017.11.24 07:16:54.839 (UTC-7) 3 3             OK
2017.11.24 07:16:54.839 (UTC-7) 2 2         Scanning source ...
2017.11.24 07:16:54.839 (UTC-7) 3 3             Included: DB_Backups
2017.11.24 07:16:54.839 (UTC-7) 3 3             Completed in 1 ms
2017.11.24 07:16:54.839 (UTC-7) 3 4                 8 folders, 2 files, 42.05 GB
2017.11.24 07:16:54.839 (UTC-7) 2 2         Scanning destination ...
2017.11.24 07:20:06.051 (UTC-7) 3 3             Completed in 3 min 11 sec
2017.11.24 07:20:06.051 (UTC-7) 3 4                 1795294 folders, 14795547 files, 48.71 TB
2017.11.24 07:21:12.060 (UTC-7) 2 2         Running archive maintenance ...

2017.11.24 07:21:12.060 (UTC-7) 3 3             Location: F:\$Archive (Bvckup 2)\
2017.11.24 07:21:12.060 (UTC-7) 2 3             Scanning ...
2017.11.24 07:21:12.060 (UTC-7) 3 4                 Configuration
2017.11.24 07:21:12.060 (UTC-7) 3 5                     Flags 0x00, threads 31, priority -1, retry count 1046112
2017.11.24 07:21:12.060 (UTC-7) 3 4                 Completed in 1 ms
2017.11.24 07:21:12.060 (UTC-7) 3 4                 Found 30 folders, 163 files, 120.65 GB
2017.11.24 07:21:12.060 (UTC-7) 2 3             No items older than 2 weeks
2017.11.24 07:21:12.060 (UTC-7) 3 2         Next maintenance in 1 hour
2017.11.24 07:21:12.060 (UTC-7) 2 2         Deducing changes ...
2017.11.24 07:21:12.060 (UTC-7) 3 3             File system information
2017.11.24 07:21:12.076 (UTC-7) 2 4                 Source
2017.11.24 07:21:12.076 (UTC-7) 2 5                     Device : remote, over SMB 3.0.2
2017.11.24 07:21:12.076 (UTC-7) 2 5                     Type : NTFS
2017.11.24 07:21:12.076 (UTC-7) 2 5                     Created timestamps : supported
2017.11.24 07:21:12.076 (UTC-7) 2 5                     Created granularity : 100 ns  ( 100 ns probed, 100 ns deduced )
2017.11.24 07:21:12.076 (UTC-7) 2 5                     Modified granularity : 100 ns  ( 100 ns probed, 100 ns deduced )
2017.11.24 07:21:12.076 (UTC-7) 2 5                     Supported attributes : read-only, hidden, system, archive
2017.11.24 07:21:12.076 (UTC-7) 2 5                     Supported API mask: 0x0d

ScottD :

Nov 24, 2017

2017.11.24 07:21:12.076 (UTC-7) 2 5                     Volume : flags 00C700FF, serial 0243E583
2017.11.24 07:21:12.076 (UTC-7) 3 6                         Encryption: supported
2017.11.24 07:21:12.076 (UTC-7) 3 6                         Compression: supported
2017.11.24 07:21:12.076 (UTC-7) 3 6                         NTFS streams: supported
2017.11.24 07:21:12.076 (UTC-7) 2 4                 Destination
2017.11.24 07:21:12.076 (UTC-7) 2 5                     Device : local, non-removable, physical disk 1
2017.11.24 07:21:12.076 (UTC-7) 2 5                     Type : NTFS
2017.11.24 07:21:12.076 (UTC-7) 2 5                     Created timestamps : supported
2017.11.24 07:21:12.076 (UTC-7) 2 5                     Created granularity : 100 ns  ( 100 ns probed, 100 ns deduced )
2017.11.24 07:21:12.076 (UTC-7) 2 5                     Modified granularity : 100 ns  ( 100 ns probed, 100 ns deduced )
2017.11.24 07:21:12.076 (UTC-7) 2 5                     Supported attributes : read-only, hidden, system, archive
2017.11.24 07:21:12.076 (UTC-7) 2 5                     Supported API mask: 0x0d
2017.11.24 07:21:12.076 (UTC-7) 2 5                     Volume : flags 03E700EF, serial 92EA0F87
2017.11.24 07:21:12.076 (UTC-7) 3 6                         Encryption: supported
2017.11.24 07:21:12.076 (UTC-7) 3 6                         Compression: NOT supported
2017.11.24 07:21:12.076 (UTC-7) 3 6                         NTFS streams: supported
2017.11.24 07:21:12.076 (UTC-7) 3 3             Timestamp granularity
2017.11.24 07:21:12.076 (UTC-7) 3 4                 Created: 100 ns
2017.11.24 07:21:12.076 (UTC-7) 3 4                 Modified: 100 ns
2017.11.24 07:21:12.076 (UTC-7) 3 3             Preparing backup plan ...
2017.11.24 07:21:12.076 (UTC-7) 3 4                 Planner setup
2017.11.24 07:21:12.076 (UTC-7) 3 5                     Created timestamps: don't compare, but update
2017.11.24 07:21:12.076 (UTC-7) 3 5                     Modified timestamps: compare and update
2017.11.24 07:21:12.076 (UTC-7) 3 5                     1-hour offset: disabled
2017.11.24 07:21:12.076 (UTC-7) 3 5                     Attributes: read-only hidden system archived
2017.11.24 07:21:12.076 (UTC-7) 3 5                     Security info: don't compare, don't update
2017.11.24 07:21:12.076 (UTC-7) 3 5                     Alternate streams: ignore
2017.11.24 07:21:12.076 (UTC-7) 3 5                     Clone top folder info: no
2017.11.24 07:21:12.076 (UTC-7) 3 5                     Files to ID: all
2017.11.24 07:21:12.076 (UTC-7) 3 5                     Move detection: folders, files (by name/size/times, by file ID, by fuzzy matching)
2017.11.24 07:21:12.076 (UTC-7) 3 5                     Disable deletes: no
2017.11.24 07:21:12.076 (UTC-7) 3 5                     Archive deleted: yes
2017.11.24 07:21:12.076 (UTC-7) 3 5                     Archive modified: no
2017.11.24 07:23:00.341 (UTC-7) 3 4                 Retrieving IDs of 2 new files ...
2017.11.24 07:23:00.357 (UTC-7) 3 5                     Completed in 16 ms
2017.11.24 07:28:20.278 (UTC-7) 3 4                 Completed in 7 min 8 sec
2017.11.24 07:28:20.278 (UTC-7) 2 2         Checking disk space ...
2017.11.24 07:28:20.278 (UTC-7) 2 3             Space needed : 21.03 GB
2017.11.24 07:28:20.278 (UTC-7) 3 4                 Exactly 22578178845 bytes
2017.11.24 07:28:20.278 (UTC-7) 2 3             Space available : 11.06 TB
2017.11.24 07:28:20.278 (UTC-7) 3 4                 Exactly 12160262864896 bytes
2017.11.24 07:28:34.554 (UTC-7) 2 1     Processing ...
2017.11.24 07:28:34.554 (UTC-7) 3 2         A total of 16590835 steps
2017.11.24 07:28:34.554 (UTC-7) 2 2         1. Preparing file for archiving EXA\DATA\PG_VERSION
2017.11.24 07:28:34.554 (UTC-7) 2 2         2. Preparing file for archiving EXA\DATA\pg_hba.conf
2017.11.24 07:28:34.554 (UTC-7) 2 2         3. Preparing file for archiving EXA\DATA\pg_ident.conf

Alex Pankratov :

Nov 25, 2017



It looks to me that you don't have "conf.dst.filters.as_source" override set, because if it were set, you'd have

    Note: using source scan selectors for destination scan

logged in the "Preparing" > "Verifying configuration" part of the log.

PS. In recent releases "conf.dst.filters.as_source" was deprecated in favor of "conf.filters.dst.as_source" (that is, with "dst" and "filters" parts swapped). However, this shouldn't matter for your case as the program will still read both when loading the INI, only electing to _write_ the new key when the INI is save.

ScottD :

Nov 25, 2017

Alex, thank you for that.  I had a feeling it was something along those lines, but my memory was a bit hazy on that feature.  I'll have to search real quick to find our old conversation in which you told me where to put that override and such.

As always, I'm trying to streamline the setup of this application for other installers in our department...so, I was wondering, would you be able to make the "conf.filters.dst.as_source" override the actual default behavior of the application (rather than a manual override that has to be setup or else everything outside of your file selection will be deleted from the drive)?  It seems like such a big risk factor to be something you always have to remember to set manually.  

I can't think of any reason a user would purposely like the application to behave in that way by default.  I would think most users would like the source and destination filters fully linked in this way by default.  Why would I ever want the application to go outside of the scope of the file selections I chose when doing any given job?  

I'm sure you know of some reasons why someone would want it to, but I would think it would be rare compared to the overall user base who don't feel very comfortable about risking the deletion or archiving of all of the data on their drive (except what has been selected) simply because the installer happened to forget to manually set this override.

Is this a viable feature request or have I gone off the reservation on this one?

Alex Pankratov :

Nov 27, 2017



Scott, it's a reasonable request, but, to put it bluntly, it is few years too late. Changing the defaults is an intrusive thing to do, especially when it concerns changing the principal operating mode.

The program has always been of a file _mirroring_ variety. It recreated the exact copy of A at B. The "as_source" toggle effectively changes it into the "inject A into B" kind, which is different.

Why would I ever want the application to go outside of the scope of the file selections I chose when doing any given job?  


Consider the case when I exclude a file from an existing backup.

In default config bvckup2 will remove/archive a backup copy of this file, which is likely what I wanted when I excluded the source. However, in the "as_source" mode, the backup copy will remain and there's no obvious way to trim it from the backup using just the app itself.

ScottD :

Nov 27, 2017

Your recommendation to set "conf.filters.dst.as_source" to 1 worked for this set and so it is only scanning the files selected on the source and destination.  Forgot that item needed to be added per new set created.

I get what you're saying about what I'm asking being such a radical change to the default behavior of the application.  Make sense.  Yet, it is going to be difficult for me to instruct our installers to manually go in and modify this setup file for each new set they create or even it there is a one time override that they would only have to do once more install.  

What I would much more prefer is if you could perhaps add a radio button or check box or something along those lines into the UI (such as in the "What to backup" window where there is plenty of space to the right of the "Include everything" and "Start with an empty list" area) and this new option would automatically switch the mode "conf.filters.dst.as_source" to 1.  That would flow well with windows they will already be navigating through while doing their setups and so will be an option I can easily build into the install instructions.  Additionally, it would be nice (if there is a way or application level override to set "conf.filters.dst.as_source" to 1 for each new set that is created) if then another option could be available in the Options > Preferences as well.

Having one or both of these options added into the UI would make it so much easier to teach others how to properly setup the application each time, without having them wade through lines of text in the setup files themselves.

Alex Pankratov :

Nov 29, 2017

Yet, it is going to be difficult for me to instruct our installers to manually go in and modify this setup file for each new set they create or even it there is a one time override that they would only have to do once more install.


Certainly a hassle, but one option would be to provide them with an installer customized to your requirements, as per https://bvckup2.com/support/forum/topic/801.

For example, a simple .bat can launch the installer with command-line arguments and the installer will drop a template for all new jobs with "as_source" set to 1 (see below).

Additionally, it would be nice (if there is a way or application level override to set "conf.filters.dst.as_source" to 1 for each new set that is created) if then another option could be available in the Options > Preferences as well.


This is already supported, see the last item in this post -
https://bvckup2.com/support/forum/topic/480/2839

Having one or both of these options added into the UI would make it so much easier to teach others how to properly setup the application each time, without having them wade through lines of text in the setup files themselves.


This will be done. The way it exists now the UI covers probably 10-15% of all options supported, so we will be adding a way to access the rest from the UI as well. Most likely this will end up being a registry-like list of options grouped by the backup phase (scanning, planning, executing, etc.), but we'll see.

ScottD :

Nov 29, 2017

I already use a batch for install as it is.  Each set is already configured for "conf.dst.filters.as_source", but the problem comes when someone then creates a new set without utilizing the clone option.  At that point, my defaults for "conf.dst.filters.as_source" get thrown out the window and catastrophe becomes a real possibility.

But, if I'm understanding correctly, you're saying I can create a file "\Bvckup2\engine\bvckup2-engine-override.ini" and then add the following line into that file...

conf.filters.dst.as_source 1

And then make that part of my batch file install...that it will force the "conf.filters.dst.as_source" to be 1 for each set created via clone or manually?  If so, that at least should solve the problem if they use my batch file installer each time.  Is that what you're saying?

For those who find themselves in a position that they feel like they need to install Bvckup2 without the batch file, they would still be in danger.  For that possibility, I would still think it would be a great feature to add these options into the UI.

Alex Pankratov :

Nov 29, 2017

\Bvckup2\engine\bvckup2-engine-override.ini


Not this file, but this one -

        \Bvckup2\ui\backup-template.ini

It's a template of backup job's settings.ini.

When a new backup is added through the UI, the UI will initialize the config to the hardcoded defaults and then patch it up with any overrides it finds in backup-template.ini

So you need to put "conf.filters.dst.as_source 1" in this INI and then have this INI deployed with your installer script. After that when someone goes Menu > File > "Add new backup", they will have "as_source" preset to 1 in the backup's config.

ScottD :

Nov 30, 2017

Got it.  Thank you.

New topic

Create
Made by Pipemetrics in Switzerland
Support


Follow
Twitter
Blog / RSS
Miscellanea Press resources
Testimonials
On robocopy
Company
Imprint

Legal Terms
Privacy