2020-06-12
md
Domoticz Backup Scripts
DNS and Static IP Address->

There have been a lot of changes in my personal "chaos manor" (perhaps the reader will remember the column by that name by the late Jerry Pournelle in Byte). Moving my office so that it is now adjacent to the workshop took a lot of work. Indeed, I have yet to complete rearranging things. However one of the first thing I wanted to set up was a reasonable backup strategy... once again. In this post, I discuss an important part of that endeavour, backing up the home automation server.

Table of Contents

  1. Local Backup Network
  2. Pushing a Domoticz Database Backup
  3. Pulling a Domoticz Database Backup
  4. Bells and Whistle
  5. Conclusion

Local Backup Network toc

The following image shows a part of the hybrid home network which will be called the local backup network.

local backup network

A Raspberry Pi, running the last version of Raspbian before Raspberry Pi OS was introduced, is the host system for Domoticz which is the home automation system overseeing numerous IoT devices about the house. The home automation system is mostly wireless, running on the 2.4 MHz Wi-Fi band. As can be expected from a home automation controller, this Raspberry Pi is always running.

The backup server and my desktop are connected to the local area network with a 1 Gb/s Ethernet bus. The backup server is not a very powerful system, it is a 3.2 GHz dual-core Pentium 4 surplus computer bought from my employer just before I retired. It has 3 GB of memory and 1 TB hard drive. It is run as a headless server with a fresh installation of Debian 4.19.118-2 (2020-04-29) x86_64. I was a bit surprised to find that distribution was even more sparse than the "lite" versions of Raspbian and Armbian meant for small single-board computers like the Raspberry Pi and La Frite and so on. The server is connected to mains power through a Wi-Fi switch which means that it can be remotely turned on or off as desired. Most of the time it is off.

The desktop is an older consumer grade I7 based machine from a major player. I have just replaced two 1 TB hard drives on the machine with two 0.5 TB solid-state drives. There is an additional hard drive with three 1 TB partitions. One is dedicated to Timeshift system snapshots, the second is used for storing photographs and the third is a backup of important directories copied from the original two hard drives and backup server that I wanted to save and have yet to install back on the SSDs. Like the backup server, that machine can be off at any particular moment.

The hardware modifications of the desktop was the impetus for rethinking my backup strategy. Here is what is planned at the moment.

The Domoticz database used to be backed up to the other systems with Syncthing. However, I need to rethink that approach as the database is being updated very frequently. This would result in frequent transfers of the complete database. While I never noticed a bandwidth problem, I would prefer something less obtrusive. Hence this post about sending compressed backup copies of the database at less frequent intervals from the Raspeberry Pi to the other machines on the network.

Pushing a Domoticz Database Backup toc

It was not at all difficult to find a script to take care of what I wanted. The Domoticz wiki has a page about this very topic, Script to backup to FTP-server (only Domoticz database). Here is a slightly modified version. Beside using the SFTP protocol (File transfer over SSH) instead of the FTP protocol, I have included some error reporting to the system log.

#!/bin/bash # Bash script on Domoticz server that will back up its database to a # local machine (mDns host name: backserver.local) as a compressed # file: /home/becky/DomoticzBackups/domoticz_20200528-1342.db.gz # Local SFTP server parameters, must be set correctly # SERVER="backserver.local" # could be IP address USERNAME="becky" PASSWORD="xxxxxxxx" DESTDIR="~/DomoticzBackups/" # trailing "/" necessary # Domoticz parameters # HOST="goldserver.local" # or explicit IP address ok; "127.0.0.1" and "localhost" may not work DOMOTICZ_PORT="8080" # default HTTP port, could be different # Create timestamped backup files # TIMESTAMP=`/bin/date +%Y-%m-%d_%H-%M` BACKUPFILE="domoticz_$TIMESTAMP.db" # backups will be named "domoticz_YYYY-MM-DD_HH-MM.db.gz" BACKUPFILEGZ="$BACKUPFILE".gz # Ask Domoticz to backup the database # curl -s http://$HOST:$DOMOTICZ_PORT/backupdatabase.php > /tmp/$BACKUPFILE curlerr=$? # Stop if this did not work, logging error # if [ $curlerr -ne 0 ]; then logger -p err "Domoticz backup database not created (error: $curlerr)" exit 1 fi # Compress the database # gzip -9 /tmp/$BACKUPFILE # and upload it # curl -k -u "$USERNAME:$PASSWORD" -T "/tmp/$BACKUPFILEGZ" "sftp://$SERVER/$DESTDIR" curlerr=$? # delete the backup up data base # /bin/rm /tmp/$BACKUPFILEGZ # log error if one has occured # if [ $curlerr -ne 0 ]; then logger -p err "Domoticz database not backed up (error: $curlerr)" exit 2 fi
Download upload-domoticzdb.sh to ~/.local/bin/upload

The wiki script suspended the Domoticz service while at the same time asking it to create the backup database.
service domoticz.sh stop /usr/bin/curl -s http://$DOMO_IP:$DOMO_PORT/backupdatabase.php > /tmp/$BACKUPFILE service domoticz.sh start
Somehow, that seems to me to be an impossible feat to realize. It would make sense to suspend Domoticz if the database were to be directly copied, but here the automation server is instructed to back up its own database just as when the function is launched manually from the Web interface.

The script, imaginatively named upload, is made executable. Since it is in the ~/.local/bin directory which is included in the PATH, it can be launched very simply.

woopi@goldserver:~ $ chmod +x .local/bin/upload woopi@goldserver:~ $ printenv PATH /home/woopi/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/games:/usr/games woopi@goldserver:~ $ ls -l domoticz/domoticz.db -rw-r--r-- 1 woopi woopi 1048576 Jun 1 17:45 domoticz/domoticz.db woopi@goldserver:~ $ upload % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 243k 0 0 100 243k 0 153k 0:00:01 0:00:01 --:--:-- 153k 100 243k 0 0 100 243k 0 153k 0:00:01 0:00:01 --:--:-- 153k

While the database was slightly more than 1M bytes, only 243K bytes needed to be transferred to the backup server after compression.

When the backup server is not on line, the following error occurs.

woopi@goldserver:~ $ upload % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- 0:00:04 --:--:-- 0curl: (6) Could not resolve host: backserver.local

The error code is logged.

woopi@goldserver:~ $ journalctl -t 'woopi' -- Logs begin at Sun 2020-05-31 20:53:38 ADT, end at Mon 2020-06-01 17:29:46 ADT. -- Jun 01 17:24:34 goldserver woopi[3771]: Domoticz database not backed up (error: 6)

The exit codes are listed at the end on the curl man page.

woopi@goldserver:~ $ man curl ... EXIT CODES There are a bunch of different error codes and their corresponding error messages that may appear during bad con‐ ditions. At the time of this writing, the exit codes are: 1 Unsupported protocol. This build of curl has no support for this protocol. 2 Failed to initialize. ...

Look carefully at the transffered size. Initially, I was getting ridiculously small sizes, such as 111 bytes as seen below.

woopi@goldserver:~ $ bash upload % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 111 0 0 100 111 0 240 --:--:-- --:--:-- --:--:-- 240 100 111 0 0 100 111 0 240 --:--:-- --:--:-- --:--:-- 240

When this happened, I ran the actual curl command without redirecting the output to a file to see what was happening.

woopi@goldserver:~ $ curl -s http://localhost:9071/backupdatabase.php <html><head><title>Unauthorized</title></head><body><h1>401 Unauthorized</h1></body></html>

I could have looked at the downloaded file on backserver to find the same information. The point is that the Domoticz web server is reporting an authorization error. As it happens, password protection is enabled, and 127.0.0.1 is not in the list of local (no authorization required) networks. Consequently, Domoticz would not accept any request from 127.0.0.1 without the password. Similarly localhost, which gets resolved to 127.0.0.1, would not work.

Pulling a Domoticz Database Backup toc

There is a problem with the previous script. It will obviously fail if the backup server is not on line, which is often true in my home system. The solution is to have the backup server "pull" the Domoticz database. This is actually quite easy to do.

#!/bin/bash # Bash script on backup server that will obtain the Domoticz database from the host server # and save to file: /home/becky/DomoticzBackups/domoticz-20200528-1342.db.gz # Domoticz parameters # HOST="goldserver.local" # or explicit IP address ok; "127.0.0.1" and "localhost" may not work PORT="8080" # default HTTP port, could be different # Local directory to contain downloaded database DESTDIR=/home/becky/DomoticzBackups/ # Create timestamped backup files # TIMESTAMP=`/bin/date +%Y%m%d-%H%M` BACKUPFILE="domoticz_$TIMESTAMP.db" # backups will be named something like "domoticz-20200622-1042.db" # Ask Domoticz to copy the database to the local destination directory # curl -s "http://$HOST:$PORT/backupdatabase.php" > $DESTDIR$BACKUPFILE curlerr=$? # Stop if this did not work, logging error # if [ $curlerr -ne 0 ]; then logger -p err "Domoticz backup database not created (error: $curlerr)" exit 1 fi # Compress the database # gzip -9 $DESTDIR$BACKUPFILE
Download: download-domoticzdb.sh

There are two "problems" associated with this approach. The first is rather obvious: no backup will be done if the backup server is not turned on. I have some ideas about that but let's kick that down the road. The other problem with this approach is that the rather big uncompressed database is being sent over the LAN do the local backup server and it's the latter that compresses the file. That's easy enough to fix. Let the download script merely the upload script on the Domoticiz host machine.

The versatile SSH protocol makes it easy to execute a command on a remote machine. Here is an example where the uname command will be run on the Raspberry Pi from the backup server.

becky@backserver:~$ ssh woopi@goldserver.local uname -a woopi@goldserver.local's password: xxxxxxxx not echoed to screen Linux goldserver 4.19.75-v7+ #1270 SMP Tue Sep 24 18:45:11 BST 2019 armv7l GNU/Linux

The good news is that uname was executed as desired. The bad news is that it was necessary to enter a password. This is not acceptable for a utility that should eventually be run automatically at regular intervals. Well, this is Linux so someone has already encountered this situation and provide mere mortals like myself with a solution. Actually, I know of two solutions. The first I tried was installing sshpass.

becky@backserver:~$ sudo apt install sshpass ... becky@backserver:~$ sshpass -p 'xxxxxxxx' ssh woopi@goldserver.local uname -a Linux goldserver 4.19.75-v7+ #1270 SMP Tue Sep 24 18:45:11 BST 2019 armv7l GNU/Linux

Now that we know that this works, it's a just a small step to apply this technique to launching the upload script on the Raspberry Pi.

becky@backserver:~$ sshpass -p 'xxxxxxxx' ssh woopi@goldserver.local /home/woopi/.local/bin/upload % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 243k 0 0 100 243k 0 322k --:--:-- --:--:-- --:--:-- 322k 100 243k 0 0 100 243k 0 322k --:--:-- --:--:-- --:--:-- 322k

There is an even easier solution, although I have been avoiding it for quite some time: just set up an SSH key pair. In addition to the usual common username/password method of establishing the identity of a user trying to connect, SSH uses public-key (also called asymmetric) cryptography. Contrary to what I thought, this is a simple two-step procedure. The first step is to generate a public and a private key on the backup server.

becky@backserver:~ $ ssh-keygen Generating public/private rsa key pair. Enter file in which to save the key (/home/becky/.ssh/id_rsa): Created directory '/home/becky/.ssh'. Enter passphrase (empty for no passphrase): I left this empty Enter same passphrase again: again, leave empty Your identification has been saved in /home/becky/.ssh/id_rsa. Your public key has been saved in /home/becky/.ssh/id_rsa.pub. The key fingerprint is: SHA256:eeJ4nkIGdCARLviKw6XUpJWGl7E1Thbg6OCspRomTxr becky@backserver The key's randomart image is: +---[RSA 2048]----+ | @ | | + . | | . B . | | o * . | | X * S | | + O + . . | | . . S . o | | . . o | | . . | +----[SHA256]-----+

The next step is to send the generated public key over to the Raspberry Pi.

becky@backserver:~ $ ssh-copy-id woopi@goldserver.local /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/becky/.ssh/id_rsa.pub" The authenticity of host 'goldserver.local (192.168.1.22)' can't be established. ECDSA key fingerprint is SHA256:JsUTFyRmvXsFVhgxKV5QrJsUT4ECMgeeJ4nkIGhbg6O. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys woopi@goldserver.local's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'woopi@goldserver.local'" and check to make sure that only the key(s) you wanted were added.

As suggested, let's try to execute a remote command.

becky@backserver:~$ ssh woopi@goldserver.local uname -a Linux goldserver 4.19.75-v7+ #1270 SMP Tue Sep 24 18:45:11 BST 2019 armv7l GNU/Linux

Because that worked then, not surprisingly, it it possible to download the Domoticz database by remotely invoking the Raspberry Pi upload script without need to supply any password.

becky@backserver:~ $ ssh woopi@goldserver.local /home/woopi/.local/bin/upload % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 237k 0 0 100 237k 0 411k --:--:-- --:--:-- --:--:-- 410k 100 237k 0 0 100 237k 0 408k --:--:-- --:--:-- --:--:-- 408k

When using ssh to remotely execute the upload script on the Raspberry Pi, it is necessary to specify the complete path to the script. This is because /home/woopi/.local/bin is added to the PATH environment from the .profile configuration file when the user woopi logs in. But there is no login in this situation.

Bells and Whistle toc

If the backup server were always on, I would not bother with "pulling" the database to back it up as shown in the last section. Instead, I would simply set up a regularly scheduled cron job on the Raspberry Pi to push the database to the backup server. Here is an example.

woopi@goldserver:~ $ crontab -e

Add the following line, ajusting the time as desired.

# Twice a day, backup the Domoticz database 30 6,18 * * * /home/woopi/.local/bin/upload

As set up above, upload will be executed twice each day, at 6:30 and 18:30. This is a much lower frequency than the hourly backups done by Domoticz itself if automatic backups are enabled (SetupSettingsSystemAutomatic Backup). However, currently not many changes are being made to the home automation system so only device logs are likely to be lost in 12 hours, which in my case is not that important. In any case, changing the frequency of the backups is easily done.

As I have explained, the backup server is mostly off line and when it is turned on, to push changes to a local VCS repository, it will not be on line for a very long time. So I have adopted a different approach. I created a cron task on the backup server that is executed each time it is booted. In essence, the Domoticz database will be backed up each time I back up some source code.

Since this was the first time the crontab file was edited on the backup server, the editor to use had to be specified. There were only two choices offered on that lean Debian system but, thankfully, nano was one of them. I continue to stubbornly avoid learning vim.

becky@backserver:~$ crontab -e no crontab for becky - using an empty one Select an editor. To change later, run 'select-editor'. 1. /bin/nano <---- easiest 2. /usr/bin/vim.tiny Choose 1-2 [1]: 1

I added the following line to the file.

@reboot sleep 240 && /home/becky/.local/bin/saveddb

Each time the machine is booted, the saveddb (for "save domoticz database") script will be executed after a four-minute delay. The local script was added to provide extra functionality. I don't want database backups to accumulate, so any backup file more than a month old is deleted before downloading the current Domoticz database. Since I am a bit paranoid, older backups are only deleted if there will remain at least one backup database (necessarily younger than a month) in the directory.

#!/bin/bash ### User params ############### # Number of days to keep old Domoticz databases DAYS=31 # Directory where Domoticz databases are saved with trailing separator SAVE_DIR=/home/becky/DomoticzBackups/ # Rmote SSH account on computer hosting Domoticz REMOTE_USER=woopi@goldserver.local # Path to the upload script on the computer hosting Domoticz UPLOAD_PATH=/home/woopi/.local/bin/upload ############################### COUNT=$(find $SAVE_DIR -name "domoticz*db.gz" -mtime -$DAYS | wc -l) if [ $COUNT -gt 0 ] then echo "$COUNT files younger than $DAYS days, deleting older files (if any)" find $SAVE_DIR -name "domoticz*db.gz" -mtime +$DAYS -delete else echo "No files younger than $DAYS days, older files (if any) not deleted" fi ssh $REMOTE_USER $UPLOAD_PATH
Download: pull-domoticzdb.sh

Wait there's more to come... Being the forgetful kind, I decided to add an e-mail warning if more than 15 days have elapsed since the last backup of the Domoticiz database. There is not much to that. Basically, a time stamp is created each time the database is backed up with the upload script. Since the current time is being used in that script, only one line had to be added at the end of it.

echo $TIMESTAMP > ~/domoticz/backup.stamp

I could just as easily put touch ~/domoticz/backup.stamp as the content of the file is never used. Here is the Python script that checks how old the time stamp is and sends an e-mail if necessary.

#!/home/woopi/.syspy/bin/python # coding: utf-8 from urllib.request import urlopen # with python 3.x #from urllib2 import urlopen # with python 2.7 import os, time try: delta = time.time() - os.path.getmtime("/home/woopi/domoticz/backup.stamp") #print("Time since last modified: {} seconds".format(delta)) except: delta = -1 backup.stamp not found # Send e-mail if it's been more than 15 days (1296000 seconds) since # the Domoticz database has been saved. # if (delta > 1296000) or (delta < 0): cmd = "/home/woopi/.syspy/bin/python /home/woopi/.syspy/pymail -s 'Alerte' -m 'Plus de 15 jours depuis la sauvegarde de la base de données de Domoticz'" #print('Excuting ', cmd) os.system(cmd)
Download: checkbackup.py

The Python script, named checkbackup, was saved in the same ~/.syspy Python 3 virtual environment as the pymail script which it uses (see Raspberry Pi and Domoticz Watchdog for more details). The last bit to make this functionnal is to execute the script at regular intervals with cron. Don't forget to make the script executable beforehand.

woopi@goldserver:~ $ chmod +x .syspy/checkbackup woopi@goldserver:~ $ crontab -e

The following line is added to the crontab file.

# Once a day, send e-mail if Domoticz db not backed up 15 7 * * * /home/woopi/.syspy/checkbackup

Each morning, at 7:15, I will receive an email if a backup has not been done in the last 15 days.

Conclusion toc

This is not really a conclusion because doing proper backups is an ongoing problem which warrants new approaches every now and then. The next step in that never-ending quest is off-site backups. Perhaps if I come up with a novel solution, I will present it in a future post.

There are many ways to achieve my goal. I am sure that it would be possible to obtain pretty much the same thing as described above with Syncthing with the appropriate settings. I may go back to that approach latter. Indeed, I may try both simultaneously for a while. Another possibility is the venerable rsync.

DNS and Static IP Address->