Duplicity & S3 Backup and Restore

Duplicity & S3 Backup and Restore

We recently found ourselves needing a new incremental file backup system…  something we could use to remotely store, something fast, something secure, and ultimately something reliable.

In steps Duplicity.

Duplicity allows us to sync incremental file backups to our cloud storage flawlessly; it has helped reduce backup file size, allows us to encrypt the backups, and reduce the amount of bandwidth needed for transferring both backups and restores.   Overall giving our developers a much needed break from their manual backups.

The script in our repo contains an installer that will setup everything you will need on your servers for this, though please do note that the restore is incomplete.    While we have verified manual account & app restores, we have not been able to perfect account based restores.   What this means is if your account on your server has more than 1 application in it, you will not be able to restore all of the applications in one sitting and will have to manually restore each app.

Requirements

Installation

wget https://gitlab.com/westernmass-hosting/other/wmh-backup/raw/master/release/wmh_br && sudo chmod +x wmh_br && sudo mv wmh_br /usr/bin/ && wmh_br -U

This will download the compiled binary, make it executable, move it to your usr/bin for system execution, and run it’s built-in dependency installer/updater

Updates

Just run the following command: sudo wmh_br -U

This will update the compiled binary and run it’s built-in dependency installer/updater

Setup Instructions

  • Download and install the app as shown above
  • Run the following command: wmh_br -C to configure your app, primary backup parameters, and your keys

Configuration

The following options are necessary for running this application. You will be prompted for all, when you run wmh_br -C. The configuration is written to a hidden config file that will be read when the app is run.

  • Days Between Full: How many days should the backup system run a full backup? Default 30
  • Retention: How many days should the backup system retain older backups? Default 30
  • Security Hash: A password or hash to encrypt the backup. Default 64 character alpha-numeric string. This will be shown to you for your records in case you need to restore to any other machine. Please make a note of it.
  • Backup Name: A name to identify your backup, this defaults to the machines hostname if nothing is entered.
  • AWS Key: Your Amazon AWS API Key ID
  • AWS Secret: Your Amazon AWS API Secret
  • AWS S3 Bucket: Your Amazon S3 Bucket name
  • AWS S3 Region: The region your S3 bucket resides in – For more info: https://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region
  • Start Path: The starting path were the backup should begin looking
  • Primary Application Name: The application folders name. For RunCloud this is “webapps”, for ServerPilot this is “apps”, for CPanel this is “public_html”
  • MySQL Admin Username: The administrators username. This account must have all privileges
  • MySQL Admin Password: The administrators password.

USAGE:

usage: wmh_br [-h] [-V] [-U] [-C] [-b {all,acct,app,other}]
              [-r {app,db,other}] [-i INCLUDE] [-A ACCT] [-a APP]
              [-d DATABASE]

arguments:
  -h, --help            show this help message and exit
  -V, --version         Show the program version
  -U, --update          Install or update the backup system
  -C, --configure       Configure the backup system
  -b {all,acct,app,other}, --backup {all,acct,app,other}
                        Run the backup
  -r {app,db,other}, --restore {app,db,other}
                        Run the restore. For other, use the app switch to
                        specify the path(s) to restore. Ex: --app=/home/me
                        Single path only
  -i INCLUDE, --include INCLUDE
                        If you would like to include additional paths to
                        backup, pass them as a comma-delimited list here. Ex:
                        --include=/home/me,/home/you,/home/theother
  -A ACCT, --acct ACCT  What account are we going to backup or restore?
                        Required if account backup or restore is selected, it
                        is also required if application backup or restore is
                        selected.
  -a APP, --app APP     What application are we going to backup or restore?
                        Required if backup or restore is selected. If
                        selected, you must also include the account.
  -d DATABASE, --database DATABASE
                        The database name we should backup/restore, or ALL.
  --hash HASH           The hash from the machine you are attempting to
                        restore the backup for. This should only be used if
                        you are trying to access the backed up files from
                        another machine.
  --name NAME           The backup name set in the configuration. This should
                        only be used if you are trying to access the backed up
                        files from another machine.

USAGE EXPLAINED

  • Command: wmh_br -h|--help

    • Description: displays the usage for the app
  • Command: wmh_br -V|--version

    • Description: displays the version number of the app
  • Command: wmh_br -U|--update

    • Description: installs or updates the dependencies for the app
  • Command: wmh_br -C|--configure

    • Description: prompts for the configuration values you will need for this to work
  • Command: wmh_br -b|--backup {all,acct,app,other}

    • Description: runs the backup methods. Only “all” works by itself. acct requires the acccount flag to be passed as well. app requires both the account flag, and app flag to be passed. You can also “include” other paths to backup by passing them to the “include” flag
    • Examples:
      • wmh_br -b=all – backs up all accounts and all applications in those accounts
      • wmh_br -b=acct -A=mine – backs up all applications in the “mine” account
      • wmh_br -b=acct -A=mine -a=myapp – backs up the application “myapp” in the “mine” account
      • wmh_br -b=acct -A=mine -a=myapp -i=/home/otheruser/otherpath,/home/user3/ – backs up the application “myapp” in the “mine” account, also includes the 2 paths provided with the “include” flag
      • wmh_br -b=other -i=/home/otheruser/otherpath,/home/user3/ – backs up only the paths in the “include” flag. You must specify the included paths if “other” is used
  • Command: wmh_br -r|--restore {app,db,other}

    • Description: runs the restore methods. All restores will prompt for a date to restore from, and confirmation to overwrite the original location. If you choose not to overwrite, the backup will be left in the /tmp/restore folder. NOTE If you do not overwrite, you will need to manually delete the contents of /tmp/restore before trying to restore another backup
    • Examples:
      • wmh_br -r=app -A=mine -a=myapp – restores the application “myapp” in the account “mine”
      • wmh_br -r=db -d=test_db – restores the database “test_db”
      • wmh_br -r=other -a=/home/user3 – restores the “/home/user3” directory only
      • wmh_br -r=app -A=mine -a=myapp --hash=THE_HASH_FROMANOTHER_MACHINE --name=THE_OTHERMACHINES_BACKUPNAME – restores the application “myapp” in the account “mine” from the machine specified with the hash and name flags. If one is specified and the other is not, both will be ignored. These will also never overwrite anything, and will be left inside /tmp/restore NOTE You will need to manually delete the contents of /tmp/restore before trying to restore another backup
  • Command: wmh_br -d|--database [ALL|databasename]

    • Description: backs up the database specified, or all of them
    • Examples:
      • wmh_br -d=ALL – backs up all databases on the server
      • wmh_br -d=test_db – backs up the database “test_db”
Kevin
22 Years of PC and server maintenance & over 17 years of web development/design experience; you can rest assured that I take every measure possible to ensure your computers are running to their peak potentials. I treat them as if they were mine, and I am quite a stickler about keeping my machines up to date and optimized to run as well as they can.