The support forum

Exclude/include pattern specification

Hooch :

Dec 17, 2014

Hello.
I would like to know exactly (100%) how pattern for includes/excludes works?

Is it exactly like in C#?

Nice thing would be to see live previews of patterns in that window where you select folders and files.

Alex Pankratov :

Dec 18, 2014

Perhaps it'd be the simplest to start with an example. Consider the following pattern:

        foo/bar*/*/file.e?e

It consists of 4 parts, split by the slashes (/) -

        foo
        bar*
        *
        file.e?e

For this pattern to match a file, each part needs to match a part of the file path. By default, only relative path from the configured location to the file is used for comparison. That is, if the source location is

        C:\Source

and the file is

        C:\Source\abc\xyz\file.ext

then the app will only be matching

        abc\xyz\file.ext

to the pattern.

The file path is split the same way as the pattern, at slashes. Then both sets of parts are compared *starting from the last part*. In the above case the first comparison will be between "file.e?e" and "file.ext".

If all parts *of the pattern* are matched, it's a match.

Since there might be more parts in the file path, this is referred to as a "tail match" pattern.

--

To require a complete match between the pattern and the relative file path, the pattern needs to start with ".\" (dot slash). For example,

        .\xyz\file.ext

will match

        C:\Source\xyz\file.ext

but not

        C:\Source\abc\xyz\file.ext

because the "abc" in latter won't have a matching part in the pattern.

This is a "complete relative match" pattern. In a sense, the leading .\ simply anchors the pattern to the top of the configured location.

--

If a pattern starts with a slash alone, it will be compared to the absolute path of a file. For example,

        \Source\abc\xyz\*.ext

will match

        C:\Source\abc\xyz\file.ext

but won't match

       C:\Source\Source\abc\xyz\file.ext

This is a "complete absolute match" pattern. It anchors the pattern to the root of the drive (or network share) of the configured location.

--

And, finally, there's a "complete match" form which includes either a drive letter or the \\server\share -

        X:\Source\abc\xyz\*.ext
        \\StorageBox\Space\Source\abc\xyz\*.ext

These are, obviously, the most specific anchors.

--

Apart from the slash, there are two more special symbols - the asterisk (*) and the question mark (?). These are your standard wildcards that work exactly as you'd expect - the asterisk matches zero or more symbols and the question mark matches exactly one symbol.

Do note that since both the pattern and file path are split first and compared second, the wildcards cannot be used to match slashes as they are removed during the splitting step.

--

... and that's about it. Is there anything I missed?

Hooch :

Dec 18, 2014

Thanks that was good read.
I would propose to add this description with some examples to application help.

If you have time it would be awesome to see your patterns working in real time in window where you select files/folders to include (Under "Then include the following items). Something like small icon next to folder/file showing if it is included/excluded by filter.

http://i.imgur.com/cT9okYf.png

Alex Pankratov :

Dec 22, 2014

Aye, noted.

This might not be an easy thing to implement *efficiently*, i.e. so that it won't lag on million-item trees, but let me see what I can do.

Hooch :

Jan 01, 2015

@Alex Pankratov.

My suggestion is to show it only on visible elements. It shouldn't lag that much or even at all than.

Alex Pankratov :

Jan 02, 2015

It won't work in all cases, e.g. if you start by excluding everything and then add "include *.exe" filter.

Hooch :

Feb 11, 2015

Hello. Small request. Can you add this topic to "FAQ / Features" Post before its gets lost? I think this is very helpful for new users.

Alex Pankratov :

Feb 12, 2015

Good idea. Done.

JohnO :

Jan 31, 2016

Might there possibly be a way around the "wildcards cannot be used to match slashes" in the future? It would be really great if it was possible to exclude e.g. all files with extension .xyz that are contained in directory abc/ and all subdirectories. Right now I have to do something like:
abc/*.xyz
abc/*/*.xyz
abc/*/*/*.xyz
abc/*/*/*/*.xyz
abc/*/*/*/*/*.xyz

Alex Pankratov :

Feb 01, 2016

Noted. The way to do this would be to add support for **, which in (some) Unix shells does exactly what you are asking for. Can't give you the timeframe, but this will be done at some point.

mtc :

Feb 23, 2016

Any chance of changing this topic heading to "Full pattern specification" instead of "Full patter specification"?   It's a worthwhile thread, and it'll be easier to find with the right spelling.

Alex Pankratov :

Feb 24, 2016

Good idea, done.

aweber :

Feb 07, 2018

So I read this a few times.  It has good examples, but it never really lays-out the actual syntax rules.  It appears to be a very small subset of regexp, of course, but I'm not entirely sure.

Can we used more advanced regexp matching, or can you please post a list of all matching symbols in the latest versions?

Thanks!

Alex Pankratov :

Feb 07, 2018

Patterns are not regexps.

Patterns and paths are first split into parts at slashes and these parts are then matched using basic wildcard rules.

  *  matches 0 or more symbols
  ?  matches exactly one symbol

Patterns starting with \ and .\ have additional matching requirements (see top post for details).

That's it.

aweber :

Feb 07, 2018

So it's not easy to match directories named "log" and "logs", for example...which would be easy using regexp.

Alex Pankratov :

Feb 07, 2018

I feel your pain, but there are no current plans for adding regexp support.

rfgamaral :

Jun 28, 2018

Hey Alex,

You didn't found an efficent way to implement something like this as posted above by Hooch, did you?

http://i.imgur.com/cT9okYf.png

I come from SyncBackPro and they had this feature that I found nice to have. I don't know if they have performance issues with large tress, but they only apply the visual difference (if filters were changed) by asking you if you want to apply the new exclude filters. Maybe you could take a look?

highend :

Jun 20, 2019

How do I exclude e.g. all .git folders, regardless of their depth and location?

E.g. src root:
D:\

.git folders:
D:\Development\Language 1\Project 1\.git
D:\Development\Language 2\Project 1\.git
D:\Scripts\Project 1\.git

In other words, they can be anywhere and at any level

Alex Pankratov :

Jun 20, 2019

Just Exclude + Folder + ".git"
https://bvckup2.com/support/data/exclude-git.png

primate :

Sep 20, 2019

Hi Alex,

How can I exclude folders at the top level only with folder names beginning with T-Z?

Thanks,
Simon

Alex Pankratov :

Sep 20, 2019

You will need 7 separate rules, one per letter.

Exclude, folder, matching .\T*
Exclude, folder, matching .\U*
...

primate :

Sep 20, 2019

Thanks

Arbie :

Oct 07, 2019

FYI here's one way to backup only the files in the top level of a folder tree i.e. excluding subfolders:

•  Set "What to backup" to be "Everything with some exceptions".
•  In Details select "Include everything".
•  (Will see a long default list in "Then, apply the following rules...")
•  Add to that list a filter to exclude:   .\*

I have a feeling that there's a simpler way but couldn't find one.

Alex Pankratov :

Oct 07, 2019

Yep, that's the right way to do it. You can also do it in reverse -

1. Start with an empty list
2. Include .\*    (for Files)

Arbie :

Oct 08, 2019

Thanks, yes, that's the more elegant method I thought probably existed.

Arbie :

Oct 09, 2019

[ Moved to https://bvckup2.com/support/forum/topic/1248 ]

Eproxus :

Nov 12, 2019

What about case sensitivity?

Alex Pankratov :

Nov 12, 2019

Patterns are matched in a case-insensitive way.

highend :

Jul 29, 2020

Two questions:

1. A test backup for checking src filters.
Src: D:\Tools\Vivaldi_x64\
Dst: R:\bck\

Include file: user data/first run
Include file: user data/default/bookmarks
Exclude folder: user data
user data is one of the folders directly under the src root.

This example above doesn't work, user data is excluded fully, the files aren't backuped.
But when I add a ./ in front of both inclusions, it does work as expected.
I know that ./ defines a relative path but I don't understand why the includes need the trailing ./ and the exclude doesn't.

2.
conf.filters.src.generic.rule                      1 00000010 00000000 ./user data/first run
conf.filters.src.generic.rule                      1 00000010 00000000 ./user data/default/bookmarks
conf.filters.src.generic.rule                      0 00000010 00000010 user data

First value = file (1) / folder (0)
What exact meaning do the other two values have?

I'm asking because I want to be able to easily change filters in the .ini file(s) directly.

highend :

Jul 29, 2020

Em... sorry, another example

3.
Src: D:\Tools\
Dst: E:\bck\Tools\

The only folder in Tools is Vivaldi_x64

Include file: vivaldi_x64/user data/first run
Include file: ./vivaldi_x64/user data/default/bookmarks
Include folder: vivaldi_x64/user data/WidevineCdm
Exclude folder: vivaldi_x64/user data

Notice that the bookmarks include file uses the trailing ./

This works as expected. Both files are copied and the folder while everything else is excluded.

When I now remove the trailing ./ for the bookmarks file, user data will get deleted completely...

Alex Pankratov :

Jul 29, 2020

Q1 - I cannot reproduce this, not based on your description.

https://i.imgur.com/uTgy69a.png - the setup
https://i.imgur.com/1fY9zaR.png - the result

Q2 - leading 1 or 0 is include/exclude, followed by an attribute mask, followed by an attribute value that we want to match after applying the mask. Folders will have the Directory attribute set, hence the values in your rules.

That said, this is an older format and I would advise against using it. New format is described here - https://bvckup2.com/support/forum/topic/632, and further elaborated here - https://bvckup2.com/support/forum/topic/972/5360

Q3 - again, I can't readily reproduce this, because I don't have full details of your excl/incl setup. However the behavior that you describe doesn't look right, at least at the first glance. Let me take a look at a full copy of conf.filters.src.* section of settings.ini and we can go from there. Ideally via email to support@iobureau.com.

highend :

Jul 29, 2020

Ok, let's try to solve Q1.

Btw, it's the same problem as with Q3.

I really only use two file inclusions and one folder exclusion.
I've send you an e-mail that contains the requested lines from the .ini file for that profile.

When there is no leading ./ for any of the two files, the whole user data folder won't be backuped. It isn't necessary to add both leading ./ for the files but sufficient to add it for only one of them and after that, the files are backuped and the rest of user data is excluded (as expected).

It's the same as in the Q3 so I guess if Q1 is solved, Q3 will be as well.

highend :

Jul 29, 2020

Addendum: It seems that to backup both files, the deeper nested one needs to have the leading ./.

Include file: user data/first run
Include file: user data/default/bookmarks

If (only) the first one has it, only this will be backuped, if (only) the second one (or both) has / have it, both will be backuped.

Alex Pankratov :

Jul 29, 2020

Alright, for the public record - Q1 and Q3 are due to a bug. To quote from my email reply -

It's caused by the scanning module erroneously suppressing the
descend into excluded folders even if it has tail-matching rules
that may potentially match something in there.

This happens when the default is to "include everything" and the
"verbose exclude actions" are set to final. This condition was
meant to cut down on unnecessary scanning, but in retrospect it
is too broad.

Ironically enough I have a "$fixme?" remark right next to this
condition, literally the only one of this kind in the whole code
base.

Prepending ./ to the pattern converts it from tail-matching to
full-matching type and this triggers a descend into all its
parent subfolders even if they are excluded.


The patch will be in 81.6.

New topic

Create
Made by IO Bureau in Switzerland
Support

Updates Newsletter
Blog & RSS
Follow Twitter
Reddit
Miscellanea Press kit
Testimonials
Company Imprint

Legal Terms
Privacy