For long, I was looking for an ideal backup setup. Expectations were minimal;periodically backup set of folders from the main disk to secondary disk, compressing the files to save disk space, retaining the earlier sets and easy to learn, deploy and maintain. I tried and dropped many backup solutions because of cumbersome setup and steep learning curve.
Finally, I chose Borg Backup for its sophistication and ease of use; thereupon, using it for more than a year now. This post covers the basics of Borg Backup and the next post, Automate Borg Backup, covers steps to automate the whole setup.
BorgBackup is a de-duplicating backup program which provides an efficient and secure way to backup data. The data de-duplication makes it suitable for daily backups since only changes (similar to GIT commits) are stored.
Borg gives you:
- space efficient storage of backups.
- compression using LZ4, zlib and LZMA.
- secure, authenticated encryption.
- easy installation on multiple platforms.
In Ubuntu, it is as simple as this
$ sudo apt install borgbackup
Refer Borg Documentation for other systems.
Let’s create a Borg Repository and go through frequently used operations.
Borg repository holds set of Borg backups. To create repository use borg init command.
$ borg init --encryption=none $HOME/borgrepo
It initializes repository in $HOME/borgrepo directory. For sake of simplicity, we are creating repository in home dir, but for production backup we need to create the repository in a file system on fallback disk or on a remote system.
Backup some folders
In Borg, the backup is called as archive. The following command backup $HOME/Documents folder to a archive named Monday.
$ borg create $HOME/borgrepo::Monday $HOME/Documents
The archive referred with
path/to/repo::archive-name. We can’t create another backup using same name, so to backup Documents folder, say next day, we need to use new archive name.
$ borg create $HOME/borgrepo::Tuesday $HOME/Documents
The first backup is a full backup, and Borg de-duplicates data during subsequent backup of same folder and stores only the changed chunks to save space and backup time.
Use borg list command to list all archives in borg repository.
$ borg list $HOME/borgrepo Monday Mon, 2019-07-29 10:17:12 Tuesday Tue, 2019-07-30 10:45:14
We can obtain info about the repository using borg info command. The info from my borg repository is shown below:
$ borg info /cherry/borgrepo/ Repository ID: 0307c0296168c2e75c357334b8ae Location: /cherry/borgrepo Encrypted: No Cache: /home/m/.cache/borg/0307c0296168c2e75c357334b8ae Security dir: /home/m/.config/borg/security/0307c0296168c2e75c357334b8ae ------------------------------------------------------------------------------ Original size Compressed size Deduplicated size All archives: 18.55 GB 17.52 GB 1.93 GB Unique chunks Total chunks Chunk index: 27125 353185
It is info of my backup; the backup size is roughly about 1.6 GB. The repository contains about 18 archive and the total size should have been about 17 GB, but because of de-duplicating feature the repository size is still 1.93 GB.
Use extract command to retrieve data from the repository.
$ mkdir tem $ cd tem $ borg extract $HOME/borgrepo::Monday
Extract always writes into the current working directory, so make sure you cd to the right place before calling borg extract!
You can pass –exclude option to exclude specific path or paths. We can also pass specific path to extract as below:
$ cd tem $ borg extract $HOME/borgrepo::Monday **/Documents/somefolder
Suppose, we have automated the borg for daily backup and after an year, repository will have hundreds of archive. They may not take lot of space because of data de-duplication, but we soon realise that many of them may be purged. For example, we may wish to retain only one backup per week or even less, one for each month. The prune command allows to sparse out unwanted archives.
$ borg prune -n --list --keep-weekly -1 $HOME/borgrepo Keeping archive: Tuesday Fri, 2019-08-02 16:07:28 Would prune: Monday Fri, 2019-08-02 16:07:16
The command uses -n for dry run and –list shows which archives are pruned or retained. The option –keep-weekly tells borg to retain the last archive in each week. The limit -1 indicates that there is no limit on number of archives to retain. If limit is greater than zero, then that many archive are retained. Say, keep-weekly prunes 25 archives and retains 10; if limit is set to 3, then the latest three, from the retained set of 10, are retained and rest seven are pruned. For actual pruning, remove dry run option
-n from the above command.
The frequently used keep options are:
- –keep-daily DAILY, number of daily archives to keep
- –keep-weekly WEEKLY, number of weekly archives to keep
- –keep-monthly MONTHLY, number of monthly archives to keep
- –keep-yearly YEARLY, number of yearly archives to keep
The prune doesn’t distinguishes between different data set. For example, let’s say we have archives as follows
$ borg create $HOME/borgrepo::Docs-Sunday $HOME/Documents $ borg create $HOME/borgrepo::Picts-Sunday $HOME/Pictures $ borg create $HOME/borgrepo::Docs-Monday $HOME/Documents $ borg create $HOME/borgrepo::Picts-Monday $HOME/Pictures
Then –keep-weekly prunes the first three and retains only the last archive, and we stands to loose Documents folder backup. We can avoid this by using prefix option
$ borg prune -n --list -P Docs --keep-weekly -1 $HOME/borgrepo Keeping archive: Docs-Monday Fri, 2019-08-02 16:48:47 Would prune: Docs-Sunday Fri, 2019-08-02 16:48:32
-P Docs, tells borg to prune only the archives that start with Docs and ignore others.