Copy repository to secondary location

[X1M7]

A way to make a copy of the repository to another location for a secondary repository backup. Would be a nice feature for Enterprise backup.

I am using storeBackup right now and it have this future 'LinkToDirs.pl', link: http://www.nongnu.org/storebackup/en/node41.html

[Ernest0x]

I don't think that repository replication is something that attic should have built-in, since there are other tools that can do it effectively. For example, you can replicate a repository with rsync. Just make sure that the repository is not being written at the time of the transfer. E.g. arrange one schedule for taking backups on that repository and a different schedule for replicating that repository to another location. Obviously, the replication should start enough time after the time you estimate that the backup process would have finished. The replication should also have finished before the next backup process starts.

If you want to be absolutely sure that these tasks always run in sequence you can try to acquire a shared (read) lock on '/path/to/repository/config'. Another option would be to put repositories on LVM volumes, take snapshots when no backup process is running and then replicate from these snapshots.

In addition, to be more confident, you can run 'attic check --repository-only /path/to/repository' before replicating.

[X1M7]

If it is as simple as just doing an 'rsync' i agree that it is not something that is vital for Attic. I am used to storeBackup that base dedup on hardlinks, this makes it difficult to replicate the repository.

So if I undertand you right, rsync would do the job when used outside backup time of course?

[Ernest0x]

So if I undertand you right, rsync would do the job when used outside backup time of course?

Yes

[jborg]

Yes, rsync would do the job just fine.

Since locking is a bit tricky to get right, an alternative approach would be to provide a way to acquire a read lock on a repository and then run a custom command. Something like this:

$ attic lock-and-run <command> 

$ attic lock-and-run /path/repository.attic rsync -av $REPO somewhere:/data 

[PiQuer]

I have a setup where a local repository is synced daily to an off-site location. How robust is this against failure in the sync process?

For example: let's assume we start with a consistent remote repository and a local repository where (compared to the remote version) some archives have been added, some others deleted. Now we sync from the local to the remote repository, but the sync process gets interrupted midway, leaving the remote repository in an inconsistent state (I assume). If disaster strikes and everything is lost locally, could it be that the remote repository was rendered totally useless by the failed sync? Or would it be possible to restore at least those older archives which were untouched by the last change?

[Ernest0x]

You can make your sync process atomic. E.g. rsync has a '--delay-updates' option which updates files in the receiving side after the transfer has finished (successfully). Keep in mind though that this requires on the receiving side twice as much space as the size of your repositories.

[PiQuer]

Thanks for pointing me to this useful feature of rsync, I didn't know it. In my case the synchronization is to Amazon S3, so rsync cannot be used (I know there is s3fs, but this turned out to be impractical for me).

So my question still stands and is more directed towards the details of how the repository changes between new snapshots. Most importantly, could attic recover from a half-finished synchronization with attic check --repair?

[Ernest0x]

I think it's not predictable how successful (if any) would an attic check --repair operation be. It depends on how much inconsistency has the partial transfer introduced, which could vary a lot depending on a number of factors (what repositories files were left inconsistent, how many 'real' files share chunks on these, etc.). The question is why do you want to know if you can trust the attic check --repair for recovering from failing to sync to off-site, when you can arrange for not being left with only an inconsistent copy on such failure?

Since, you are using Amazon S3, you may be interested in S3QL, which would allow you to create a snapshot of the filesystem on the receiving side before running the sync process, effectively allowing you to roleback to a previous version in case of a sync failure.

[PiQuer]

S3QL looks promising. I will have a look at this, thank you. I still have to learn a lot about these things and find a perfect setup.

Of course you are right, the absolute priority is to make the synchronization fail proof and to recover from a failed synchronization as soon as possible, and this is definitely also what I will focus on. But then again you cannot prepare for all eventualities.

I was thinking along the lines: what good is 99.9999999 % data durability of S3 if your repository was rendered useless just before you need it by a failed synchronization (which we agree should be prevented, but for the sake of the argument let's assume it happened...). So I was looking for some kind of statement: "At least the files which have data chunks exclusively in files untouched by the last sync can be recovered (through the attic client or otherwise)". Maybe it's not that statement but a weaker one, but you get the idea :)

I really know nothing about how attic works internally. If the answer is that I have to live with the remaining risk after making the sync as reliable as possible, that's also fine.

[jewel]

I want to copy my backup repository offsite regularly. In my case I can use rsync, but I'm worried that rsync might perpetuate corruption that is present in the master repository. While I can run check occasionally on both master and slave copies, it might not catch a problem until it is too late.

For that reason it'd be nice if attic had a built-in way to copy archives from one repository to another (possibly remote) repository.

In lieu of that, would it work to use the FUSE mount of the master repository as a source to a slave backup? I imagine I'd have to write a script to walk through each archive and check if it is present in the slave, and if it is not, run attic create to copy it to the slave.