description of exclude wildcards

[joolswills]

I've not checked the code on this one, but in my backup line I have

--exclude "'/home/*/temp'"

this seems to exclude all temp folders at any depth under /home/

what would be the correct exclude line in the same way as this would be treated from a shell (to just exclude temp folders in /home/something/temp. rdiff-backup which I used previously used extended shell globbing, so you would need to specify ** to match any depth

Would be good to have some more detailed documentation on this. Thanks.

[jborg]

"attic help patterns" is probably the documentation you're looking for.

Switching to extended shell globbing would be a nice improvement but would also be a backwards incompatible change. But as long as the change is clearly visible in the release notes it should be ok.

[joolswills]

Looks like others would prefer more flexible wildcards also - adding it here for reference

http://librelist.com/browser//attic/2015/1/6/exclude-patterns/#e24a3f3abb26b3f2c5666177048a1d35

[joolswills]

I have a patch which changes * matching to not match "/" and includes * (which works like previous "") which better matches shell extended globbing and makes it more usable. However the code is a modification to the python3 "fnmatch.translate" function - so including it might be a licence issue - not sure if we can include it or not - I think it may be ok if we include some PSF licence stuff as well ?

Alternatively I think another option would be to "post process" the regular expression, which I will look at doing now.

[joolswills]

Reading though https://docs.python.org/3/license.html it looks as though we can include it, so long as we include their licence files, and a mention as to what we used/changed. Would that be agreeable, or would you prefer another solution ?

[joolswills]

Here's a version that uses code from the python3 fnmatch library

diff --git a/attic/helpers.py b/attic/helpers.py
index 2ad2806..ad03e6d 100644
--- a/attic/helpers.py
+++ b/attic/helpers.py
@@ -9,7 +9,6 @@ import stat
 import sys
 import time
 from datetime import datetime, timezone, timedelta
-from fnmatch import translate
 from operator import attrgetter
 import fcntl

@@ -243,12 +242,12 @@ class ExcludePattern(IncludePattern):
     """
     def __init__(self, pattern):
         if pattern.endswith(os.path.sep):
-            self.pattern = pattern+'*'+os.path.sep
+            self.pattern = pattern+'**'+os.path.sep
         else:
-            self.pattern = pattern+os.path.sep+'*'
+            self.pattern = pattern+os.path.sep+'**'
         # fnmatch and re.match both cache compiled regular expressions.
         # Nevertheless, this is about 10 times faster.
-        self.regex = re.compile(translate(self.pattern))
+        self.regex = re.compile(self.translate())

     def match(self, path):
         return self.regex.match(path+os.path.sep) is not None
@@ -256,6 +255,43 @@ class ExcludePattern(IncludePattern):
     def __repr__(self):
         return '%s(%s)' % (type(self), self.pattern)

+    def translate(self):
+        pat=self.pattern
+        i, n = 0, len(pat)
+        res = ''
+        while i < n:
+            c = pat[i]
+            i = i+1
+            if c == '*':
+                if i < n and pat[i] == '*':
+                    res = res + '.*'
+                    i = i+1
+                else:
+                    res = res + '[^\\' + os.path.sep + ']*'
+            elif c == '?':
+                res = res + '.'
+            elif c == '[':
+                j = i
+                if j < n and pat[j] == '!':
+                    j = j+1
+                if j < n and pat[j] == ']':
+                    j = j+1
+                while j < n and pat[j] != ']':
+                    j = j+1
+                if j >= n:
+                    res = res + '\\['
+                else:
+                    stuff = pat[i:j].replace('\\','\\\\')
+                    i = j+1
+                    if stuff[0] == '!':
+                        stuff = '^' + stuff[1:]
+                    elif stuff[0] == '^':
+                        stuff = '\\' + stuff
+                    res = '%s[%s]' % (res, stuff)
+            else:
+                res = res + re.escape(c)
+        return res + '\Z(?ms)'
+

 def is_cachedir(path):
     """Determines whether the specified path is a cache directory (and

here's a version that rewrites the regular expression to have the same effect.

diff --git a/attic/helpers.py b/attic/helpers.py
index 2ad2806..926a0b6 100644
--- a/attic/helpers.py
+++ b/attic/helpers.py
@@ -243,12 +243,19 @@ class ExcludePattern(IncludePattern):
     """
     def __init__(self, pattern):
         if pattern.endswith(os.path.sep):
-            self.pattern = pattern+'*'+os.path.sep
+            self.pattern = pattern+'**'+os.path.sep
         else:
-            self.pattern = pattern+os.path.sep+'*'
+            self.pattern = pattern+os.path.sep+'**'
         # fnmatch and re.match both cache compiled regular expressions.
         # Nevertheless, this is about 10 times faster.
-        self.regex = re.compile(translate(self.pattern))
+        pattern = translate(self.pattern)
+        # rework the regular expression so that it is the equivalent of doing
+        # ** -> .* and * -> [^\/]* so ** matches any sequence of characters,
+        # and * matches any except a path separator.
+        pattern = pattern.replace('.*.*','**')
+        pattern = pattern.replace('.*','[^\\'+os.path.sep+']*')
+        pattern = pattern.replace('**','.*')
+        self.regex = re.compile(pattern)

     def match(self, path):
         return self.regex.match(path+os.path.sep) is not None

Let me know if you would consider either of these, and I will create a pull request. Thanks.

[Ernest0x]

I would like to clarify that the */** thing is not part of the extended shell globbing (set with 'extglob' option). It is a separate extension (set with 'globstar' option). If only 'extglob' is implemented, then one can write a pattern like this: /home/+([^/])/temp which is the same with /home/*/temp and 'globstar' enabled.

So, since it would also break current configurations, 'globstar' is not really needed, unless we need it for cosmetic reasons and/or gaining familiarity with other tools. On the other hand, 'extglob' gives all the power and it is unlikely to break current configurations.

[joolswills]

we don't have a patch for that though and the fix above adds support for something some users have been asking for. I would also argue that people have been thinking * acts like it does in a shell, so the changed functionality is still closer to shell behaviour than the current.

[joolswills]

@ThomasWaldmann - Would you consider one of the patches from #97 (comment) for your fork ?

[ThomasWaldmann]

@joolswills sounds useful.

I am just asking myself if it can use some existing library function instead of writing a new one.

[joolswills]

@ThomasWaldmann the second patch modifies the behaviour of the existing function in python - I did search around, but couldn't find anything else to do this correctly.

Main reason for the change is I believe currently it is misleading where you have a situation like

--exclude /home/*/somefolder

I would think to most people this would exclude just the somefolder within home dirs, yet currently it would exclude it at any depth below. I'm basically trying to make the behaviour like rdiff-backup which is more intuitive imho.

[ThomasWaldmann]

I've understood your motiviation (and I share your POV).

Just wondering that this somehow "standard thing" does not exist in some ready-to-use form.

[joolswills]

There is https://pypi.python.org/pypi/glob2 - but as I remember the translate function is virtually the same as python3 - and does not include the additional functionality.

[ThomasWaldmann]

Maybe try it and if it fails point its author to this issue? The description of the package sounded like he wanted to support that.

[joolswills]

I will submit my changeset for the translate function there as a starting point then. thanks for the feedback.

[joolswills]

miracle2k/python-glob2#6