Installation problems on CentOS 6.7

[otraupe]

Hi,
I have problems with the installation and would like to post to the list "attic@librelist.com". I have sent multiple messages to it, but I don't get a response.
Ole

[ThomasWaldmann]

librelist seems to have problems now and then (I recently moved another mailing list away from there due to that).

So, try again later, contact support there, or just create a ticket about it here.

[otraupe]

Thank you for clarifying this, Thomas. So I will try here.

I have installed Attic on CentOS 6.7. CentOS requires Python 2.6 for its packet manager "yum". So I installed Python 3.4.3 with "make altinstall". Further I installed it as a shared library and put it in /usr/local/Python3/bin. Python 3.4.3 is on the path as python3 and python3.4. I took some of this advice from here:
http://stackoverflow.com/questions/8087184/installing-python3-on-rhel

Further, Selinux is off, libacl-devel is installed, OpenSSL is > 1.0, python-msgpack is installed via "pip3.4 install msgpack-python", and finally Attic via "pip3.4 install Attic".

Initially I had forgotten about libacl-devel and selinux. I might have done those after the installation of Attic.

Then Attic hung on the initialization of my first backup repo. I then ran the test suite by "python3 -m attic.testsuite.run", which gave me the below output.

Did I do something wrong? Is the coexistance of python (Python 2.6) and python3 (Python 3.4) on the path a problem for attic?

Some of the errors below are supposedly due to permission problems, but this makes no sense to me, as I run the test suite as root (in the temp dir, obviously). In particular:

PermissionError: [Errno 1] b'/tmp/tmp3ml_3ov1/input/link1'

The dir "/tmp/tmp3ml_3ov1" doesn't exist, but other similar ones are there.

Thanks for any ideas!
Ole

[root@backuper ole]# python3 -m attic.testsuite.run
...........EEE.....ss.....EE..EEE.Error reading segment 2
Index object count mismatch. 87 != 0
....ss......EE........................sKey file "/tmp/tmplt2gfk6i/tmp_tmpmnlwzgia" created.
Keep this file safe. Your data will be inaccessible without it.
..Remember your passphrase. Your data will be inaccessible without it.
.Encryption NOT enabled.
Use the "--encryption=passphrase|keyfile" to enable encryption.
...s....Error reading segment 1
Index object count mismatch. 6 != 3
Error reading segment 1
Index object count mismatch. 6 != 3
Error reading segment 1
attempting to recover /tmp/tmpllpnu5fj/repository/data/0/1
.....Error reading segment 0
Error reading segment 0
Error reading segment 0
attempting to recover /tmp/tmp3t4ltrwa/repository/data/0/0
Adding commit tag to segment 0
..........Error reading segment 1
Index object count mismatch. 6 != 3
Error reading segment 1
Index object count mismatch. 6 != 3
Error reading segment 1
attempting to recover /tmp/tmp8sfnusbq/repository/data/0/1
.....Error reading segment 0
Error reading segment 0
Error reading segment 0
attempting to recover /tmp/tmpyc3nm4pp/repository/data/0/0
Adding commit tag to segment 0

..............

ERROR: test_aes_counter_uniqueness_keyfile (attic.testsuite.archiver.ArchiverTestCase)

Traceback (most recent call last):
File "/usr/local/Python3/lib/python3.4/site-packages/attic/testsuite/archiver.py", line 435, in test_aes_counter_uniqueness_keyfile
self.verify_aes_counter_uniqueness('keyfile')
File "/usr/local/Python3/lib/python3.4/site-packages/attic/testsuite/archiver.py", line 422, in verify_aes_counter_uniqueness
self.create_test_files()
File "/usr/local/Python3/lib/python3.4/site-packages/attic/testsuite/archiver.py", line 165, in create_test_files
xattr.setxattr(os.path.join(self.input_path, 'link1'), 'user.foo_symlink', b'bar_symlink', follow_symlinks=False)
File "/usr/local/Python3/lib/python3.4/site-packages/attic/xattr.py", line 101, in setxattr
_check(func(path, name, value, len(value) if value else 0, 0), path)
File "/usr/local/Python3/lib/python3.4/site-packages/attic/xattr.py", line 36, in _check
raise OSError(get_errno(), path)
PermissionError: [Errno 1] b'/tmp/tmp2hlwh3ja/input/link1'

ERROR: test_aes_counter_uniqueness_passphrase (attic.testsuite.archiver.ArchiverTestCase)

Traceback (most recent call last):
File "/usr/local/Python3/lib/python3.4/site-packages/attic/testsuite/archiver.py", line 438, in test_aes_counter_uniqueness_passphrase
self.verify_aes_counter_uniqueness('passphrase')
File "/usr/local/Python3/lib/python3.4/site-packages/attic/testsuite/archiver.py", line 422, in verify_aes_counter_uniqueness
self.create_test_files()
File "/usr/local/Python3/lib/python3.4/site-packages/attic/testsuite/archiver.py", line 165, in create_test_files
xattr.setxattr(os.path.join(self.input_path, 'link1'), 'user.foo_symlink', b'bar_symlink', follow_symlinks=False)
File "/usr/local/Python3/lib/python3.4/site-packages/attic/xattr.py", line 101, in setxattr
_check(func(path, name, value, len(value) if value else 0, 0), path)
File "/usr/local/Python3/lib/python3.4/site-packages/attic/xattr.py", line 36, in _check
raise OSError(get_errno(), path)
PermissionError: [Errno 1] b'/tmp/tmp125a_5wt/input/link1'

ERROR: test_basic_functionality (attic.testsuite.archiver.ArchiverTestCase)

Traceback (most recent call last):
File "/usr/local/Python3/lib/python3.4/site-packages/attic/testsuite/archiver.py", line 172, in test_basic_functionality
self.create_test_files()
File "/usr/local/Python3/lib/python3.4/site-packages/attic/testsuite/archiver.py", line 165, in create_test_files
xattr.setxattr(os.path.join(self.input_path, 'link1'), 'user.foo_symlink', b'bar_symlink', follow_symlinks=False)
File "/usr/local/Python3/lib/python3.4/site-packages/attic/xattr.py", line 101, in setxattr
_check(func(path, name, value, len(value) if value else 0, 0), path)
File "/usr/local/Python3/lib/python3.4/site-packages/attic/xattr.py", line 36, in _check
raise OSError(get_errno(), path)
PermissionError: [Errno 1] b'/tmp/tmp0x7tqa35/input/link1'

ERROR: test_repository_swap_detection (attic.testsuite.archiver.ArchiverTestCase)

Traceback (most recent call last):
File "/usr/local/Python3/lib/python3.4/site-packages/attic/testsuite/archiver.py", line 202, in test_repository_swap_detection
self.create_test_files()
File "/usr/local/Python3/lib/python3.4/site-packages/attic/testsuite/archiver.py", line 165, in create_test_files
xattr.setxattr(os.path.join(self.input_path, 'link1'), 'user.foo_symlink', b'bar_symlink', follow_symlinks=False)
File "/usr/local/Python3/lib/python3.4/site-packages/attic/xattr.py", line 101, in setxattr
_check(func(path, name, value, len(value) if value else 0, 0), path)
File "/usr/local/Python3/lib/python3.4/site-packages/attic/xattr.py", line 36, in _check
raise OSError(get_errno(), path)
PermissionError: [Errno 1] b'/tmp/tmpe1f8m7bp/input/link1'

ERROR: test_repository_swap_detection2 (attic.testsuite.archiver.ArchiverTestCase)

Traceback (most recent call last):
File "/usr/local/Python3/lib/python3.4/site-packages/attic/testsuite/archiver.py", line 214, in test_repository_swap_detection2
self.create_test_files()
File "/usr/local/Python3/lib/python3.4/site-packages/attic/testsuite/archiver.py", line 165, in create_test_files
xattr.setxattr(os.path.join(self.input_path, 'link1'), 'user.foo_symlink', b'bar_symlink', follow_symlinks=False)
File "/usr/local/Python3/lib/python3.4/site-packages/attic/xattr.py", line 101, in setxattr
_check(func(path, name, value, len(value) if value else 0, 0), path)
File "/usr/local/Python3/lib/python3.4/site-packages/attic/xattr.py", line 36, in _check
raise OSError(get_errno(), path)
PermissionError: [Errno 1] b'/tmp/tmp52df34rs/input/link1'

ERROR: test_aes_counter_uniqueness_keyfile (attic.testsuite.archiver.RemoteArchiverTestCase)

Traceback (most recent call last):
File "/usr/local/Python3/lib/python3.4/site-packages/attic/testsuite/archiver.py", line 435, in test_aes_counter_uniqueness_keyfile
self.verify_aes_counter_uniqueness('keyfile')
File "/usr/local/Python3/lib/python3.4/site-packages/attic/testsuite/archiver.py", line 422, in verify_aes_counter_uniqueness
self.create_test_files()
File "/usr/local/Python3/lib/python3.4/site-packages/attic/testsuite/archiver.py", line 165, in create_test_files
xattr.setxattr(os.path.join(self.input_path, 'link1'), 'user.foo_symlink', b'bar_symlink', follow_symlinks=False)
File "/usr/local/Python3/lib/python3.4/site-packages/attic/xattr.py", line 101, in setxattr
_check(func(path, name, value, len(value) if value else 0, 0), path)
File "/usr/local/Python3/lib/python3.4/site-packages/attic/xattr.py", line 36, in _check
raise OSError(get_errno(), path)
PermissionError: [Errno 1] b'/tmp/tmp0zfss0ve/input/link1'

ERROR: test_aes_counter_uniqueness_passphrase (attic.testsuite.archiver.RemoteArchiverTestCase)

Traceback (most recent call last):
File "/usr/local/Python3/lib/python3.4/site-packages/attic/testsuite/archiver.py", line 438, in test_aes_counter_uniqueness_passphrase
self.verify_aes_counter_uniqueness('passphrase')
File "/usr/local/Python3/lib/python3.4/site-packages/attic/testsuite/archiver.py", line 422, in verify_aes_counter_uniqueness
self.create_test_files()
File "/usr/local/Python3/lib/python3.4/site-packages/attic/testsuite/archiver.py", line 165, in create_test_files
xattr.setxattr(os.path.join(self.input_path, 'link1'), 'user.foo_symlink', b'bar_symlink', follow_symlinks=False)
File "/usr/local/Python3/lib/python3.4/site-packages/attic/xattr.py", line 101, in setxattr
_check(func(path, name, value, len(value) if value else 0, 0), path)
File "/usr/local/Python3/lib/python3.4/site-packages/attic/xattr.py", line 36, in _check
raise OSError(get_errno(), path)
PermissionError: [Errno 1] b'/tmp/tmpx4aq34j5/input/link1'

ERROR: test_basic_functionality (attic.testsuite.archiver.RemoteArchiverTestCase)

Traceback (most recent call last):
File "/usr/local/Python3/lib/python3.4/site-packages/attic/testsuite/archiver.py", line 172, in test_basic_functionality
self.create_test_files()
File "/usr/local/Python3/lib/python3.4/site-packages/attic/testsuite/archiver.py", line 165, in create_test_files
xattr.setxattr(os.path.join(self.input_path, 'link1'), 'user.foo_symlink', b'bar_symlink', follow_symlinks=False)
File "/usr/local/Python3/lib/python3.4/site-packages/attic/xattr.py", line 101, in setxattr
_check(func(path, name, value, len(value) if value else 0, 0), path)
File "/usr/local/Python3/lib/python3.4/site-packages/attic/xattr.py", line 36, in _check
raise OSError(get_errno(), path)
PermissionError: [Errno 1] b'/tmp/tmp7m7dpqc1/input/link1'

ERROR: test_repository_swap_detection (attic.testsuite.archiver.RemoteArchiverTestCase)

Traceback (most recent call last):
File "/usr/local/Python3/lib/python3.4/site-packages/attic/testsuite/archiver.py", line 202, in test_repository_swap_detection
self.create_test_files()
File "/usr/local/Python3/lib/python3.4/site-packages/attic/testsuite/archiver.py", line 165, in create_test_files
xattr.setxattr(os.path.join(self.input_path, 'link1'), 'user.foo_symlink', b'bar_symlink', follow_symlinks=False)
File "/usr/local/Python3/lib/python3.4/site-packages/attic/xattr.py", line 101, in setxattr
_check(func(path, name, value, len(value) if value else 0, 0), path)
File "/usr/local/Python3/lib/python3.4/site-packages/attic/xattr.py", line 36, in _check
raise OSError(get_errno(), path)
PermissionError: [Errno 1] b'/tmp/tmpur53mpbb/input/link1'

ERROR: test_repository_swap_detection2 (attic.testsuite.archiver.RemoteArchiverTestCase)

Traceback (most recent call last):
File "/usr/local/Python3/lib/python3.4/site-packages/attic/testsuite/archiver.py", line 214, in test_repository_swap_detection2
self.create_test_files()
File "/usr/local/Python3/lib/python3.4/site-packages/attic/testsuite/archiver.py", line 165, in create_test_files
xattr.setxattr(os.path.join(self.input_path, 'link1'), 'user.foo_symlink', b'bar_symlink', follow_symlinks=False)
File "/usr/local/Python3/lib/python3.4/site-packages/attic/xattr.py", line 101, in setxattr
_check(func(path, name, value, len(value) if value else 0, 0), path)
File "/usr/local/Python3/lib/python3.4/site-packages/attic/xattr.py", line 36, in _check
raise OSError(get_errno(), path)
PermissionError: [Errno 1] b'/tmp/tmp3ml_3ov1/input/link1'

---

Ran 118 tests in 27.728s

FAILED (errors=10, skipped=6)

[otraupe]

Found this:
#201
The init is working now, the repo is on a cifs mount.

Backups seem to work, too. A lot of 5 M size files are created.

[ThomasWaldmann]

The tests should be run with fakeroot -u. Not sure if that has to do with your issue though.

It is no problem to have both python 2.x and 3.x installed, you just need to use python 3.x for attic.

[otraupe]

With fakeroot, it takes much longer, but the result seems to be the same. Very similar to what Whissi posted two weeks ago. Makes me worry, honestly.

use python 3.x for attic

This should be ensured by "pip3.4 install attic", right? The "attic" executable resides in "/usr/local/Python3/bin/".

[otraupe]

All detailed error reports seem to be related to the "xattr.py". Another dependency missing?

[ThomasWaldmann]

I assume you have libattr and -dev?

[otraupe]

[root@backuper ole]# yum list installed | grep attr
attr.x86_64 2.4.44-7.el6 @anaconda-CentOS-201508042137.x86_64/6.7
libattr.x86_64 2.4.44-7.el6 @anaconda-CentOS-201508042137.x86_64/6.7
libattr-devel.x86_64 2.4.44-7.el6 @base

[otraupe]

I would like to ask the following questions:

Which tests are failing exactly, what do they do?
Is this critical, and in what way?
What might be the reason?
What can I do?

My first backup attempt over night finally "failed", although it seems that a lot of data was backuped:
"attic: Exiting with failure status due to previous errors"
Does it have to do with the failed tests?

[otraupe]

And why is it so slow? Ok, that might have many reasons, but 16 GB took about 45 minutes. I intend to backup multiple TB of data. That's probably not working in the long run.

[ThomasWaldmann]

Tests: it seems like they can not set extended attributes (xattrs) on the files in /tmp. Maybe this is because your /tmp filesystem does not support xattrs or is mounted without enabling that support.
It is not the test itself failing, but the test setup.

"attic: Exiting with failure status due to previous errors" is a warning, you have to read the log output above that to see what exactly did not work. Likely it could not access some file(s) due to permissions or so. That is often not a fatal error, your backup has reached its normal end.

[ThomasWaldmann]

Aside from all hardware, setup and data related factors, some of the slowness may be caused by attic's usage of zlib compression at level 6, by encryption (if you do not have hw accelerated AES) and by internal scheduling. The first two can slow down if your CPU is relatively slow and/or old.

If you intend to go multiple TBs, you likely will run into the issue that the target chunksize of attic is ~64KB, so multiple TBs create a lot of chunks, a huge chunk index and huge RAM consumption. I made the chunker configurable in my fork "borgbackup".

[otraupe]

Thanks for your help, Thomas!

@xattrs: that's what I thought first. But the respective dirs in /tmp don't even exist - probably DUE to the failed tests. Also, I tested for xattr support following this procedure:
https://wiki.samba.org/index.php/Operating_system_requirements/File_system_support#Testing_your_filesystem
No problems here.

@the log file: where do I find it? The output in the console is cut to several hundreds (?) of lines. No errors there as far as I can see.

[ThomasWaldmann]

Well, it seems rather clear after analyzing the traceback that there are xattr problems, because the setxattr call in the test setup fails all the time. So, did you check xattr support inside /tmp/?

About the log: redirect stdout and stderr to files so you are not limited by the scrollback buffer. The warning/error msg will be in there then.