md
The Domoticz Time Synchronization Problem
December 9, 2019
<-Home Automation Servers on Raspbian Buster Lite --

The solution that I was using for the problem causing Domoticz to shut down if there was time synchronization problem does not work in Ubuntu 18.04 (Bionic). The problem is discussed at length and a new solution using a systemd unit file to start the service is provided.

The first four sections of this post were previousl in the the guide to Home Automation Servers on Raspbian Buster Lite. To lighten that discussion, three of those sections have been removed from the guide and replaced with the last section presented here.

Table of Contents

  1. Installing Domoticz
  2. Time Syncrhonization Problem
  3. Auto Restart Domoticz Service
  4. Wait for systemd-timesync
  5. Use a Unit File to Start Domoticz

Installing Domoticz toc

Installing Domoticz is as simple as always when following The "easy" way instructions.

woopi@goldserver:~ $ curl -L install.domoticz.com | sudo bash % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 16257 100 16257 0 0 12007 0 0:00:01 0:00:01 --:--:-- 12015 ::: ::: You are root. ::: Verifying free disk space... ... after answering a couple of questions about ports and directories:
Ready... ┌─────────────────────┤ Installation Complete! ├─────────────────────┐ │ │ │ Point your browser to either: │ │ │ │ HTTP: 192.168.1.101:8080 │ │ HTPS: 192.168.1.101:443 │ │ │ │ Wiki: https://www.domoticz.com/wiki │ │ Forum: https://www.domoticz.com/forum │ │ │ │ The install log is in /etc/domoticz. │ │ │ │ │ │ <Ok> │ │ │ └────────────────────────────────────────────────────────────────────┘
... ::: Installation Complete! Configure your browser to use the Domoticz using: ::: 192.168.1.101:8080 ::: 192.168.1.101:443

The installation script will install the git and libudev-dev packages.

Start a browser and go to the address specified. The home automation system will display the following page.

This is a good place to express all my gratitude to the Domoticz development team as well as to the community.

Check that everything is functioning correctly. Something like the following should be displayed.

woopi@goldserver:~$ sudo systemctl status domoticz.service ● domoticz.service - domoticz_service Loaded: loaded (/etc/systemd/system/domoticz.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2019-10-17 10:37:43 AST; 6min ago Main PID: 2010 (domoticz) Tasks: 14 (limit: 1072) Memory: 5.3M CGroup: /system.slice/domoticz.service └─2010 /home/domoticz/domoticz/domoticz -www 8080 -sslwww 443 Oct 10 10:37:43 goldserver domoticz[2010]: 2019-12-08 18:37:43.891 Status: WebServer(HTTP) started on address: :: with port 8080 Oct 10 10:37:43 goldserver domoticz[2010]: 2019-12-08 18:37:43.897 Status: WebServer(SSL) started on address: :: with port 443 Oct 10 10:37:43 goldserver domoticz[2010]: 2019-12-08 18:37:43.902 Starting shared server on: :::6144 Oct 10 10:37:43 goldserver domoticz[2010]: 2019-12-08 18:37:43.902 Status: TCPServer: shared server started... Oct 10 10:37:43 goldserver domoticz[2010]: 2019-12-08 18:37:43.903 Status: RxQueue: queue worker started... Oct 10 10:37:45 goldserver domoticz[2010]: 2019-12-08 18:37:45.904 Status: EventSystem: reset all events... Oct 10 10:37:45 goldserver domoticz[2010]: 2019-12-08 18:37:45.906 Status: EventSystem: reset all device statuses... Oct 10 10:37:45 goldserver domoticz[2010]: 2019-12-08 18:37:45.908 Status: EventSystem: Started Oct 10 10:37:45 goldserver domoticz[2010]: 2019-12-08 18:37:45.908 Status: EventSystem: Queue thread started...

If an error about a missing Python library appears,

Oct 10 10:37:45 goldserver domoticz[2010]: 2019-12-08 18:37:45.907 Status: EventSystem - Python: Failed dynamic library load, install the latest libpython3.x library that is available for your platform.

then install the python3-dev package.

woopi@goldserver:~$ sudo apt install python3-dev -y

Time Problem with Domoticz toc

Everything looks to be in order but this may not the case. Power down the Raspberry Pi, wait six minutes or more and then turn the power on making sure that the device has access to the Internet. Try to open the Domoticz web page. It may be that the web server cannot be reached because the domoticz.service started and then mysteriously stopped.

woopi@goldserver:~ $ sudo systemctl status domoticz.service ● domoticz.service - LSB: Home Automation System Loaded: loaded (/etc/init.d/domoticz.sh; generated) Active: active (exited) since Thu 2019-10-17 11:11:32 ADT; 41min ago Docs: man:systemd-sysv-generator(8) Process: 474 ExecStart=/etc/init.d/domoticz.sh start (code=exited, status=0/SUCCESS) Oct 17 11:11:30 goldserver systemd[1]: Starting LSB: Home Automation System... Oct 17 11:11:32 goldserver domoticz.sh[474]: 2019-10-17 11:11:32.643 Status: Domoticz V4.10717 (c)2012-2019 GizMoCuz Oct 17 11:11:32 goldserver domoticz.sh[474]: 2019-10-17 11:11:32.662 Status: Build Hash: b38b49e5, Date: 2019-05-09 08:04:08 Oct 17 11:11:32 goldserver domoticz.sh[474]: 2019-10-17 11:11:32.663 Status: Startup Path: /home/woopi/domoticz/ Oct 17 11:11:32 goldserver domoticz.sh[474]: domoticz: Domoticz is starting up.... Oct 17 11:11:32 goldserver domoticz[484]: Domoticz is starting up.... Oct 17 11:11:32 goldserver domoticz[627]: Domoticz running... Oct 17 11:11:32 goldserver systemd[1]: Started LSB: Home Automation System. Oct 17 11:17:37 goldserver domoticz[627]: Domoticz stopped...

Even if you do not have this known problem, please read on because it could present itself later on.

Note the 6-minute gap between the last two entries in the log. As waaren explains in a Domoticz Forum entry Crashes on cold start on Raspberry and other systems without RTC, the server "interprets this time [gap] as a hang of some critical internal processes" and shuts down. The Domoticz server will work after a reboot or after restarting the service manually, which is not an acceptable solution.

As waaren says, I should not have encountered the problem because there was a real-time clock (RTC) (DS3231, see part 3 of this series of posts) on the Raspberry Pi. Fortunately, its battery was dead so it was not doing anything. I changed the RTC battery and the service no longer crashes on cold starts. Clearly, this could be a recurring problem when the battery expires again, so the Domoticz installation needs to be improved to handle the situation when the RTC is no longer functioning.

On the older Raspberry Pi there will not be a real-time clock. So I tried using the startupdelay option when starting Domoticz, by editing the start up script.

woopi@goldserver:~ $ sudo nano /etc/init.d/domoticz.sh

I added a 30 second delay to the DAEMON_ARGS parameter.

DAEMON_ARGS="$DAEMON_ARGS -startupdelay 30"

The result was disappointing.

woopi@goldserver:~ $ sudo systemctl status domoticz.service ● domoticz.service - LSB: Home Automation System Loaded: loaded (/etc/init.d/domoticz.sh; generated) Active: active (exited) since Fri 2019-10-18 12:32:28 AST; 3min 27s ago Docs: man:systemd-sysv-generator(8) Process: 472 ExecStart=/etc/init.d/domoticz.sh start (code=exited, status=0/SUCCESS) Oct 17 22:23:54 goldserver systemd[1]: Starting LSB: Home Automation System... Oct 17 22:23:55 goldserver domoticz.sh[472]: 2019-10-17 22:23:55.961 Status: Domoticz V4.10717 (c)2012-2019 GizMoCuz Oct 17 22:23:55 goldserver domoticz.sh[472]: 2019-10-17 22:23:55.970 Status: Build Hash: b38b49e5, Date: 2019-05-09 08:04:08 Oct 17 22:23:55 goldserver domoticz.sh[472]: 2019-10-17 22:23:55.971 Status: Startup Path: /home/pi/domoticz/ Oct 17 22:23:55 goldserver domoticz.sh[472]: 2019-10-17 22:23:55.971 Status: Startup delay... waiting 30 seconds... Oct 18 12:32:28 goldserver domoticz.sh[472]: domoticz: Domoticz is starting up.... Oct 18 12:32:28 goldserver domoticz[481]: Domoticz is starting up.... Oct 18 12:32:28 goldserver domoticz[640]: Domoticz running... Oct 18 12:32:28 goldserver systemd[1]: Started LSB: Home Automation System. Oct 18 12:32:36 goldserver domoticz[640]: Domoticz stopped...

Clearly the time had not been updated when the script was started but after the 30 second delay when domoticz was starting up the time was correct. Nevertheless, the service was stopped.

In his forum post, waaren suggests ways of working around the problem. The most elegant may very well be Alexandre Gambier's Domoticz restart service which is discussed next. I devised a different solution based on the built-in smtp client systemd-timesync. It works well in Raspbian Buster but it does not work in Ubuntu Bionic (18.04) or Armbian images based on Ubuntu. Starting Domoticz with a systemd unit file has proven to be a generally valid solution. If impatient, click on this link Use Unit File to Start Domoticz to go directly to what is now my preferred solution, jumping over the discussion of the other solutions.

Auto Restart Domoticz Service toc

Installing the service is a simple affair: create a directory and then add a short unit file in that directory.

woopi@goldserver:~ $ sudo mkdir /etc/systemd/system/domoticz.service.d woopi@goldserver:~ $ sudo nano /etc/systemd/system/domoticz.service.d/restart.conf

Copy the content of Alexandre Gambier's unit file:

[Service]
Type=forking
PIDFile=/run/domoticz.pid
RemainAfterExit=no
Restart=on-failure
RestartSec=5s

into the text editor and then exit nano (CtrlX), saving the file when prompted. Reload all unit files and check that the restart configuration service has been installed.

woopi@goldserver:~ $ sudo systemctl daemon-reload woopi@goldserver:~ $ sudo systemctl status domoticz ● domoticz.service - LSB: Home Automation System Loaded: loaded (/etc/init.d/domoticz.sh; generated) Drop-In: /etc/systemd/system/domoticz.service.d └─restart.conf Active: active (running) since Thu 2019-10-17 12:22:36 ADT; 13min ago ...

Check by powering down the Pi.

woopi@goldserver:~ $ sudo shutdown now Connection to goldserver.local closed by remote host. Connection to goldserver.local closed.

Wait at least six minutes and then power up and look at the logs.

woopi@goldserver:~ $ journalctl -u domoticz Oct 17 15:29:09 goldserver domoticz.sh[478]: 2019-10-17 15:29:09.629 Status: Domoticz V4.10717 (c)2012-2019 GizMoCuz Oct 17 15:29:09 goldserver domoticz.sh[478]: 2019-10-17 15:29:09.638 Status: Build Hash: b38b49e5, Date: 2019-05-09 08:04:08 Oct 17 15:29:09 goldserver domoticz.sh[478]: 2019-10-17 15:29:09.638 Status: Startup Path: /home/woopi/domoticz/ Oct 17 15:29:09 goldserver domoticz.sh[478]: domoticz: Domoticz is starting up.... Oct 17 15:29:09 goldserver domoticz[490]: Domoticz is starting up.... Oct 17 15:29:09 goldserver domoticz[545]: Domoticz running... Oct 17 15:55:56 goldserver domoticz[545]: Domoticz stopped... Oct 17 15:55:59 goldserver systemd[1]: domoticz.service: Main process exited, code=killed, status=10/USR1 Oct 17 15:55:59 goldserver systemd[1]: domoticz.service: Failed with result 'signal'. Oct 17 15:56:04 goldserver systemd[1]: domoticz.service: Service RestartSec=5s expired, scheduling restart. Oct 17 15:56:04 goldserver systemd[1]: domoticz.service: Scheduled restart job, restart counter is at 1. Oct 17 15:56:04 goldserver domoticz.sh[658]: 2019-10-17 15:56:04.897 Status: Domoticz V4.10717 (c)2012-2019 GizMoCuz Oct 17 15:56:04 goldserver domoticz.sh[658]: 2019-10-17 15:56:04.897 Status: Build Hash: b38b49e5, Date: 2019-05-09 08:04:08 Oct 17 15:56:04 goldserver domoticz.sh[658]: 2019-10-17 15:56:04.897 Status: Startup Path: /home/woopi/domoticz/ Oct 17 15:56:04 goldserver domoticz.sh[658]: domoticz: Domoticz is starting up.... Oct 17 15:56:04 goldserver domoticz[663]: Domoticz is starting up.... Oct 17 15:56:04 goldserver domoticz[664]: Domoticz running.

That works very well, the service was stopped as before but then the restart service kicked in and Domoticz was restarted and remained running.

woopi@goldserver:~ $ sudo reboot Connection to goldserver.local closed by remote host. Connection to goldserver.local closed. after waiting a minute or two michel@hp:~$ ssh woopi@goldserver.local or michel@hp:~$ ssh pi@192.168.1.101 woopi@goldserver:~ $ journalctl -u domoticz Oct 17 16:06:24 goldserver domoticz.sh[429]: 2019-10-17 16:06:24.121 Status: Domoticz V4.10717 (c)2012-2019 GizMoCuz Oct 17 16:06:24 goldserver domoticz.sh[429]: 2019-10-17 16:06:24.129 Status: Build Hash: b38b49e5, Date: 2019-05-09 08:04:08 Oct 17 16:06:24 goldserver domoticz.sh[429]: 2019-10-17 16:06:24.129 Status: Startup Path: /home/woopi/domoticz/ Oct 17 16:06:24 goldserver domoticz.sh[429]: domoticz: Domoticz is starting up.... Oct 17 16:06:24 goldserver domoticz[439]: Domoticz is starting up.... Oct 17 16:06:24 goldserver domoticz[449]: Domoticz running...

As promised, the drop-in restart service did not restart Domoticz this time because it was not stopped. The only niggling problem is a warning shown in the service status report or when stop or restarting the service.

woopi@goldserver:~ $ sudo systemctl status domoticz Warning: The unit file, source configuration file or drop-ins of domoticz.service changed on disk. Run 'systemctl daemon-reload' to reload units. ● domoticz.service - LSB: Home Automation System Loaded: loaded (/etc/init.d/domoticz.sh; generated) Drop-In: /etc/systemd/system/domoticz.service.d └─restart.conf

To get rid of the restart drop-in service, just delete the domoticz.service.d directory in /etc/systemd/system and reload the service unit files.

woopi@goldserver:~ $ sudo rm -r /etc/systemd/system/domoticz.service.d woopi@goldserver:~ $ sudo systemctl daemon-reload

Wait for systemd-timesync toc

I decided to use another approach based on the one proposed by waaren. It's interest for me lies in the fact that it works with the Raspberry Pi 3 with a working or non-working RTC and with the Raspberry Pi 1 which does not have an RTC and where ntp will not be installed. All that is required is a few extra lines in the Domoticz start script.

woopi@goldserver:~ sudo nano /etc/init.d/domoticz.sh

# # Function that starts the daemon/service # do_start() { # Wait at most count seconds for time synchronization # See http://sigmdel.ca/guide_buster02_en.html#syncdomoticz count=30 while [ ! -f "/run/systemd/timesync/synchronized" ] do count=$((count-1)) if [ $((count)) -lt 1 ] then echo "Could not sync" return 2 fi echo -n "." sleep 1 done echo "Sync'd" # Return # 0 if daemon has been started # 1 if daemon was already running # 2 if daemon could not be started ... }

Insert a while loop at the beginning of the do_start() function. It will wait until systemd-timesync has created a file named synchronized to indicate that it has obtained a valid date and time. If systemd-timesync, an sntp client, has not synchronized with a server within n seconds, the do_start() will be exited returning a value of 2 meaning the daemon could not be started. I take no credit for this, I just adapted the script by jcyr found on his reply in the Raspberry Forum. There are other approaches that could be of interest in that discussion. The following shows the the change in action when an ssh session was opened just after the Raspberry Pi without a hardware real time clock was rebooted.

michel@hp:~$ ssh woopi@goldserver.local woopi@goldserver.local's password: xxxxxx not echoed to the screen ... woopi@goldserver:~ $ sudo systemctl status domoticz ● domoticz.service - LSB: Home Automation System Loaded: loaded (/etc/init.d/domoticz.sh; generated) Active: activating (start) since Mon 2019-10-21 20:00:04 ADT; 16s ago Docs: man:systemd-sysv-generator(8) Cntrl PID: 520 (domoticz.sh) Tasks: 2 (limit: 2319) Memory: 884.0K CGroup: /system.slice/domoticz.service ├─520 /bin/sh /etc/init.d/domoticz.sh start └─735 sleep 1 Oct 21 20:00:04 goldserver systemd[1]: Starting LSB: Home Automation System... woopi@goldserver:~ $ sudo systemctl status domoticz ● domoticz.service - LSB: Home Automation System Loaded: loaded (/etc/init.d/domoticz.sh; generated) Active: active (running) since Mon 2019-10-21 20:00:38 ADT; 7s ago Docs: man:systemd-sysv-generator(8) Process: 520 ExecStart=/etc/init.d/domoticz.sh start (code=exited, status=0/SUCCESS) Tasks: 22 (limit: 2319) Memory: 23.2M CGroup: /system.slice/domoticz.service └─756 /home/woopi/domoticz/domoticz -daemon -www 8080 -sslwww 447 Oct 21 20:00:04 goldserver systemd[1]: Starting LSB: Home Automation System... Oct 21 20:00:37 goldserver domoticz.sh[520]: ........................Sync'd Oct 21 20:00:38 goldserver domoticz.sh[520]: 2019-10-21 20:00:38.115 Status: Domoticz V4.10717 (c)2012-2019 GizMoCuz Oct 21 20:00:38 goldserver domoticz.sh[520]: 2019-10-21 20:00:38.122 Status: Build Hash: b38b49e5, Date: 2019-05-09 08:04:08 Oct 21 20:00:38 goldserver domoticz.sh[520]: 2019-10-21 20:00:38.123 Status: Startup Path: /home/woopi/domoticz/ Oct 21 20:00:38 goldserver domoticz.sh[520]: domoticz: Domoticz is starting up.... Oct 21 20:00:38 goldserver domoticz[755]: Domoticz is starting up.... Oct 21 20:00:38 goldserver domoticz[756]: Domoticz running... Oct 21 20:00:38 goldserver systemd[1]: Started LSB: Home Automation System.

The status of the domoticz.service was checked just after booting and as can be seen, the install script was still waiting for systemd-timesync. The second time around, the sntp client had finally got the time from an Internet time server after about 20 seconds. If the Domoticz service were to be restarted, it would be "Sync'd" right away.

Here is the unformatted while loop to add to the start script.

Use Unit File to Start Domoticz toc

The modifications to the domoticz.sh script proposed above will not work in Ubuntu 18.04 (Bionic) which is the distribution on my desktop. I have encountered the same problem with a version of Armbian based on Ubuntu 18.04 (Bionic). Even though Ubuntu 18.04 is based on Debian Buster, it uses an older version of systemd with a systemd-timesync that does not create the /run/systemd/timesync/synchronized file.

It is easy enough to ascertain the version of systemd.

woopi@goldserver:~ $ systemd --version systemd 241 (241) ...
michel@hp:~$ systemd --version systemd 237

So at some time between version 237 found on Ubuntu 18.04 and version 241 found on Rasbian Buster, systemd-timesync started creating the syncrhonized file.

It is possible to avoid the crash problem by starting Domoticz with an systemd unit file with appropriate parameters instead of the init script. Indeed the Linux page on the Domoticz Wiki recommends using the appropriated method for starting a service on Linux distributions with systemd as the init program.

The first step is to create the unit file with nano or an editor of choice.

woopi@goldserver:~$ sudo nano /etc/systemd/system/domoticz.service

Here is an example unit file. The condition After=time-sync.target will ensure that the service is installed only once the system time has been synchronized. The parameter AmbientCapabilities=CAP_NET_BIND_SERVICE is needed if a TCP port less than 1024 is specified which is the case below where port 443 will be used for secured HTTP connections. See the Wiki page for more details, especially if an older version of Raspbian is being used.

[Unit] Description=domoticz_service After=time-sync.target [Service] User=woopi Group=users ExecStart=/home/woopi/domoticz/domoticz -www 8080 -sslwww 443 WorkingDirectory=/home/woopi/domoticz AmbientCapabilities=CAP_NET_BIND_SERVICE Restart=on-failure RestartSec=1m [Install] WantedBy=multi-user.target

Of course, the user name woopi needs to be adjusted and note that it appears 3 times in the unit file.

Now stop Domoticz if it is running and remove the script (saving it in case something goes wrong). Reload daemons, start the Domoticz service and check its status to ensure everything is correct.

woopi@goldserver:~$ sudo /etc/init.d/domoticz.sh stop [ ok ] Stopping domoticz.sh (via systemctl): domoticz.service. woopi@goldserver:~$ sudo mv /etc/init.d/domoticz.sh domoticz-sh woopi@goldserver:~$ sudo systemctl daemon-reload woopi@goldserver:~$ sudo systemctl start domoticz.service woopi@goldserver:~$ sudo systemctl status domoticz.service ● domoticz.service - domoticz_service Loaded: loaded (/etc/systemd/system/domoticz.service; disabled; vendor preset Active: active (running) since Mon 2019-12-09 15:38:10 AST; 17s ago Main PID: 2235 (domoticz) Tasks: 16 (limit: 1072) Memory: 11.8M CGroup: /system.slice/domoticz.service └─2235 /home/domoticz/domoticz/domoticz -www 8080 -sslwww 443 ...

The following command will instruct systemd to automatically start the service at boot time.

woopi@goldserver:~$ sudo systemctl enable domoticz.service Created symlink /etc/systemd/system/multi-user.target.wants/domoticz.service → /etc/systemd/system/domoticz.service.

This command, which creates a symbolic link to the newly created Domoticz unit file, needs to be launched only once.

When booting up after power has been off for more than 10 minutes or so, do not worry if initially the Domoticz service does not appear to be functioning:

woopi@goldserver:~$ sudo systemctl status domoticz.service ● domoticz.service - domoticz_service Loaded: loaded (/etc/systemd/system/domoticz.service; enabled; vendor preset: enabled) Active: activating (auto-restart) (Result: signal) since Sun 2019-12-08 19:20:00 AST; 53s ago Process: 881 ExecStart=/home/domoticz/domoticz/domoticz -www 8080 -sslwww 443 (code=killed, signal=USR1) Main PID: 881 (code=killed, signal=USR1)

It will be restarted in about one minute by which time the time-sync.target will have been reached and the service will be running.

woopi@goldserver:~$ sudo systemctl status domoticz.service ● domoticz.service - domoticz_service Loaded: loaded (/etc/systemd/system/domoticz.service; enabled; vendor preset: enabled) Active: active (running) since Sun 2019-12-08 22:28:29 AST; 2h 35min ago Main PID: 503 (domoticz) Memory: 4.1M CGroup: /system.slice/domoticz.service └─503 /home/sancho/domoticz/domoticz -www 8080 -sslwww 443 Dec 08 22:28:29 goldserver domoticz[503]: 2019-12-08 22:28:29.954 Status: WebServer(SSL) started on address: :: with port 443 Dec 08 22:28:29 goldserver domoticz[503]: 2019-12-08 22:28:29.969 Starting shared server on: ::::6144 Dec 08 22:28:29 goldserver domoticz[503]: 2019-12-08 22:28:29.975 Status: TCPServer: shared server started... Dec 08 22:28:29 goldserver domoticz[503]: 2019-12-08 22:28:29.978 Status: RxQueue: queue worker started... Dec 08 22:28:31 goldserver domoticz[503]: 2019-12-08 22:28:31.980 Status: EventSystem: reset all events... Dec 08 22:28:31 goldserver domoticz[503]: 2019-12-08 22:28:31.987 Status: EventSystem: reset all device statuses... Dec 08 22:28:32 goldserver domoticz[503]: 2019-12-08 22:28:32.023 Status: PluginSystem: Entering work loop. Dec 08 22:28:32 goldserver domoticz[503]: 2019-12-08 22:28:32.311 Status: Python EventSystem: Initalizing event module. Dec 08 22:28:32 goldserver domoticz[503]: 2019-12-08 22:28:32.317 Status: EventSystem: Queue thread started... Dec 08 22:28:32 goldserver domoticz[503]: 2019-12-08 22:28:32.320 Status: EventSystem: Started

Hopefully that is the end of this saga.

<-Home Automation Servers on Raspbian Buster Lite --