3rd

3rd is a small Linux backup tool for the off-site copy in a 3-2-1 backup setup. It creates encrypted 7z archives from configured files or directories, uploads them to AWS S3, and can restore them back into their original directory structure.

The main advantage is that related paths can be backed up as separate archives while still restoring cleanly as one tree. This is useful when a large directory contains a subdirectory of binaries, media, or other files that should use different compression, encryption, storage, or upload settings.

The backup script also includes a "last change detection", so unchanged targets are skipped if the newest version already exists in S3. This check is performed locally by keeping track of when each target was modified and uploaded.

Features

• Per-file or per-directory backup configuration.
• Independent compression level, password, temp directory, and S3 destination per target.
• Optional 7-Zip volume splitting for very large archives.
• Uploads use the S3 Glacier Deep Archive storage class.
• Split archives for configured child targets.
• .placeholder files in parent archives that point to the S3 archive containing the excluded child target.
• Restore-in-place behavior: all archives preserve enough path structure to extract into the same target tree.
• Last change detection, so unchanged targets are skipped if the newest version already exists in S3.

3rd is a wrapper around the AWS CLI aws and the 7zip CLI 7z. It is intended for Linux; other operating systems are untested.

Requirements

• Python 3 (v3.11.2)
• 7zip CLI (v26.02)
• AWS CLI (v2.32.23)
• Write access to an AWS S3 bucket
• Read access to the same bucket when restoring (optional)

Installation

Clone the repository:

git clone https://codeberg.org/vlw/3rd
cd 3rd

Create your config file:

cp -p .example.config.json .config.json

Edit .config.json and set the absolute paths and S3 destinations for your backups.

Configuration

The config file is a JSON array. Each object describes one archive target:

[
	{
		"password": "mypassword",
		"compression": 10,
		"volume_size": false,
		"abspath_temp": "/tmp",
		"last_uploaded": null,
		"abspath_target": "/home/me/archive",
		"abspath_destination": "s3://my-bucket/archive.7z"
	}
]

Fields:

Key	Description
password	7z AES-256 password. Set to false to disable encryption.
compression	Compression level from 0 to 10. Use 0, false, or null to store without compression.
volume_size	7-Zip volume size for fragmented archives, such as "100M" or "10G". Set to false or null to create one archive file.
abspath_temp	Directory for temporary archives before upload or after download. Use null for the system temp directory.
last_uploaded	Modified datetime recorded after successful upload. Use null for a first run.
abspath_target	Absolute path to the file or directory to archive or restore.
abspath_destination	S3 URL for the archive. Uploaded archives use the S3 Glacier Deep Archive storage class.

Note

last_uploaded is managed by backup.py. When the current target modified datetime matches last_uploaded, the target is skipped without creating an archive.

Volume Splitting

Set volume_size when a single archive may become too large to handle in one chunk. This adds 7-Zip's -v argument and uploads each generated part using the configured destination as the base name:

s3://my-bucket/archive.7z.001
s3://my-bucket/archive.7z.002
s3://my-bucket/archive.7z.003

The last part will be less that volume_size if the bytes don't exactly match.

{
	"password": "mypassword",
	"compression": 10,
	"volume_size": "10G",
	"abspath_temp": "/tmp",
	"last_uploaded": null,
	"abspath_target": "/home/me/archive",
	"abspath_destination": "s3://my-bucket/archive.7z"
}

Set volume_size to false to upload one file to the exact abspath_destination.

Split Archives

You can configure both a parent directory and one or more paths inside it:

[
	{
		"password": "mypassword",
		"compression": 10,
		"abspath_temp": null,
		"last_uploaded": null,
		"abspath_target": "/home/me/archive",
		"abspath_destination": "s3://my-bucket/archive.7z",
	},
	{
		"password": false,
		"compression": 0,
		"abspath_temp": null,
		"last_uploaded": null,
		"abspath_target": "/home/me/archive/videos",
		"abspath_destination": "s3://my-bucket/archive-videos.7z",
	}
]

In this example, /home/me/archive/videos is not included in archive.7z. Instead, the parent archive contains:

archive/videos.placeholder

The placeholder file contains the S3 destination for the split archive:

s3://my-bucket/archive-videos.7z

This allows parent archives to be heavily compressed while skipping files like compressed video from recompressing. "Ultra" compression will waste a lot of CPU cycles on already-compressed files.

Backup

Run a backup:

python3 backup.py -i .config.json

Dry run:

python3 backup.py -i .config.json --dryrun

When a target has not changed since last_uploaded, the command prints a skipped message and does not run 7z or AWS for that target.

Restore

Restore all configured archives:

python3 restore.py -i .config.json

Archives are downloaded into abspath_temp, extracted into the correct path structure (abspath_target), and then removed from the temp directory.

When volume_size is set, restore downloads all matching abspath_destination.* volume files and extracts from the first part.

After restore, generated .placeholder files are removed automatically. To only remove placeholders without downloading from AWS:

python3 restore.py -i .config.json --placeholders-only

CLI

backup.py:

Argument	Description
-s, --sleep	Global sleep duration between logged commands.
-i, --input	Config file path. Defaults to .config.json.
-l, --log-level	Console log level. See src/Enums.py.
-d, --dryrun	Create archives but do not upload; preserved archives remain in temp storage.

restore.py:

Argument	Description
-s, --sleep	Global sleep duration between logged commands.
-i, --input	Config file path. Defaults to .config.json.
-l, --log-level	Console log level. See src/Enums.py.
--placeholders-only	Remove generated .placeholder files without downloading or extracting archives.

Cron

Example weekly backup:

30 2 * * 3 cd /opt/3rd && /usr/bin/python3 backup.py -i .config.json

This runs every Wednesday at 02:30.