mirror of
https://codeberg.org/vlw/3rd.git
synced 2026-01-12 06:46:00 +01:00
120 lines
No EOL
3.9 KiB
Markdown
120 lines
No EOL
3.9 KiB
Markdown
# 3rd
|
|
A script to automate the "off-site copy" in the 3-2-1 Backup strategy with encryption, uploading to AWS S3, and independent definition of compression method and [temporary] archive storage locations for uploading large archives to S3, with support for independent configurations for subdirectories as well.
|
|
|
|
This script is a wrapper for the AWS CLI `aws` and the 7zip CLI `7z` and is meant to be run on Linux. Other operating systems are untested.
|
|
|
|
## Key features
|
|
- Archive encryption before uploading to AWS S3.
|
|
- Independent compression level, archive location, S3 storage location, for directories and subdirectories.
|
|
|
|
# Installation
|
|
Make sure you have the following prerequisites before starting:
|
|
- Python 3 installed.
|
|
- The [7zip CLI](https://www.7-zip.org/download.html) installed.
|
|
- The [AWS CLI](https://aws.amazon.com/cli/) installed and configured with write access to your target bucket.
|
|
|
|
|
|
1. **Clone this repository**
|
|
|
|
```
|
|
git clone https://codeberg.org/vlw/3rd
|
|
```
|
|
|
|
2. **Copy `.example.config.json` to `.config.json`**
|
|
|
|
```
|
|
cp -p .example.config.json .config.json
|
|
```
|
|
|
|
3. **Set environment variables in `.config.json`**
|
|
|
|
[See the config file documentation for more information](#config).
|
|
|
|
4. **Run `run.py` in autorun mode**
|
|
|
|
```
|
|
python3 run.py -a
|
|
```
|
|
[See the CLI section for a list of all available arguments](#cli).
|
|
|
|
|
|
## Optional cron
|
|
Schedule this backup script to run with a crontab entry, for example:
|
|
```
|
|
30 2 * * 3 cd /opt/3rd && /usr/bin/python3 run.py -a
|
|
```
|
|
Which will run at 2:30 each Wednesday.
|
|
|
|
# Config
|
|
The config file `.config.json` is used to define parameters and which directories to archive (in autorun mode).
|
|
|
|
```json
|
|
{
|
|
"config": {
|
|
"cloud": {
|
|
// Name of the target AWS S3 bucket
|
|
"bucket": "vlw-test"
|
|
// .. More options to come (probably)
|
|
},
|
|
// Default settings for each archive item
|
|
"archive": {
|
|
// The password used to encrypt all archives
|
|
"password": "mypassword",
|
|
// The compression level to use when "compress" is true for an item
|
|
"compression_level": 10,
|
|
// Default archive location when "path_temp" is null for an item
|
|
"default_path_temp": "/tmp/output"
|
|
}
|
|
},
|
|
// Array of archive items, see next section
|
|
"archive": []
|
|
}
|
|
|
|
```
|
|
|
|
Each archive item uses the following structure:
|
|
```json
|
|
{
|
|
// Enables or disables compression for this directory. STORE will be used if disabled.
|
|
"compress": true,
|
|
// Store the encrypted archive in this directory temporarily while its being uploaded to S3.
|
|
"path_temp": "/tmp/",
|
|
// The relative path from the bucket root directory to store the uploaded object
|
|
"path_target_to": "/myarchive.7z",
|
|
// An absolute path (very important) to the target folder to upload
|
|
"path_target_from": "/my/archive"
|
|
}
|
|
```
|
|
|
|
## Common parent directories
|
|
One of the key features of this script is that it can perform different archiving procedures for a subdirectories and their parent directories.
|
|
|
|
If you have the directory `/my/archive` with the following config:
|
|
```json
|
|
{
|
|
"compress": true,
|
|
"path_temp": null,
|
|
"path_target_to": "/myarchive.7z",
|
|
"path_target_from": "/my/archive"
|
|
}
|
|
```
|
|
And a subdirectory `/my/archive/subdirectory` with the following config:
|
|
```json
|
|
{
|
|
"compress": true,
|
|
"path_temp": null,
|
|
"path_target_to": "/my-subdirectory.7z",
|
|
"path_target_from": "/my/archive/subdirectory"
|
|
}
|
|
```
|
|
The `/my/archive/subdirectory` will be **excluded** from the `/my/archive` archive since it has an overriding archive configuration.
|
|
|
|
# CLI
|
|
Available command line argument with `run.py`:
|
|
|
|
arg|Name|Default|Description
|
|
--|--|--|--
|
|
`-s`|`--sleep`|2|Set a global sleep duration between commands
|
|
`-a`|`--autorun`|False|Archive each item in the .config.json archive array
|
|
`-d`|`--dryrun`|False|Perform a dry run. Archives will not be uploaded to S3.
|
|
`-l`|`--log-level`|`StdoutLevel.STANDARD`|Set a custom log level when printing to the console. See `/src/Enums.py#StdoutLevel` |