3rd

A script to automate the "off-site copy" in the 3-2-1 Backup strategy with encryption, uploading to AWS S3, and independent definition of compression method and [temporary] archive storage locations for uploading large archives to S3, with support for independent configurations for subdirectories as well.

This script is a wrapper for the AWS CLI aws and the 7zip CLI 7z and is meant to be run on Linux. Other operating systems are untested.

Key features

• Archive encryption before uploading to AWS S3.
• Independent compression level, archive location, S3 storage location, for directories and subdirectories.

Installation

Make sure you have the following prerequisites before starting:

• Python 3 installed.
• The 7zip CLI installed.
• The AWS CLI installed and configured with write access to your target bucket.

1. Clone this repository

git clone https://codeberg.org/vlw/3rd

2. Copy .example.config.json to .config.json

cp -p .example.config.json .config.json

3. Set environment variables in .config.json

See the config file documentation for more information.

4. Run run.py in autorun mode

python3 run.py -a

See the CLI section for a list of all available arguments.

Optional cron

Schedule this backup script to run with a crontab entry, for example:

30 2 * * 3 cd /opt/3rd && /usr/bin/python3 run.py -a

Which will run at 2:30 each Wednesday.

Config

The config file .config.json is used to define parameters and which directories to archive (in autorun mode).

{
	"config": {
		"cloud": {
            // Name of the target AWS S3 bucket
			"bucket": "vlw-test"
            // .. More options to come (probably)
		},
        // Default settings for each archive item
		"archive": {
            // The password used to encrypt all archives
			"password": "mypassword",
            // The compression level to use when "compress" is true for an item
			"compression_level": 10,
            // Default archive location when "path_temp" is null for an item
			"default_path_temp": "/tmp/output"
		}
	},
    // Array of archive items, see next section
	"archive": []
}

Each archive item uses the following structure:

{
    // Enables or disables compression for this directory. STORE will be used if disabled.
    "compress": true,
    // Store the encrypted archive in this directory temporarily while its being uploaded to S3.
    "path_temp": "/tmp/",
    // The relative path from the bucket root directory to store the uploaded object
    "path_target_to": "/myarchive.7z",
    // An absolute path (very important) to the target folder to upload
    "path_target_from": "/my/archive"
}

Common parent directories

One of the key features of this script is that it can perform different archiving procedures for a subdirectories and their parent directories.

If you have the directory /my/archive with the following config:

{
    "compress": true,
    "path_temp": null,
    "path_target_to": "/myarchive.7z",
    "path_target_from": "/my/archive"
}

And a subdirectory /my/archive/subdirectory with the following config:

{
    "compress": true,
    "path_temp": null,
    "path_target_to": "/my-subdirectory.7z",
    "path_target_from": "/my/archive/subdirectory"
}

The /my/archive/subdirectory will be excluded from the /my/archive archive since it has an overriding archive configuration.

CLI

Available command line argument with run.py:

arg	Name	Default	Description
-s	--sleep	2	Set a global sleep duration between commands
-a	--autorun	False	Archive each item in the .config.json archive array
-d	--dryrun	False	Perform a dry run. Archives will not be uploaded to S3.
-l	--log-level	StdoutLevel.STANDARD	Set a custom log level when printing to the console. See /src/Enums.py#StdoutLevel