md
Updating Raspbian to Stretch
Update: January 20, 2019
January 10, 2019
-- Installing an Older Version of Lighttpd->

With the New Year, I decided it was time to tackle once again the task of updating the Raspbian operating system on my Raspberry Pi (Model B Plus Rev 1.2) to the latest version of Stretch updating the home automation system Domoticz at the same time. I had been holding back because of problems using the newest version of Lighttpd found in Raspbian Stretch to stream the output from Mpjg-Streamer over a secure connection. I have not solved that problem yet, but I have been experimenting with running a VPN server on the Raspberry Pi and that would be a secure work around.

I expected that updating to Raspbian Stretch from Jessie would be without problems. Especially since Stretch has been out for a while now and the Raspberry Pi Foundation has issued a number of updates. And I have tested it, with Bluetooth notably and managed to do better with the newer version. But I did run into little problems that nevertheless took some time to figure out. Also, I have gotten better at setting up the Raspberry Pi so I decided that this recounting of my experience might be of use to others.

To be clear then, this post is just another one of my series on setting up a Raspberry Pi. I suspect that it would be of interest mostly to new fans of small single-board computers, in this case the Raspberry Pi. Assuming that many of these would be new to Linux, I have added sidebars entitled Information for neophytes in which I ramble about what that I think could be of help to beginners.

Table of Contents

  1. Getting Stretch on an SD card
  2. Headless Setup
  3. Initial Boot
  4. Change the Default User
  5. Updating the Operating System
  6. Installing Domoticz
  7. Installing mosquitto
  8. Installing MJPG-Streamer for Use With a Webcam
  9. Using MJPG-Streamer with a Raspberry Pi Camera Module
  10. Installing Mochad
  11. Real-Time Clock
  12. Temperature and Humidity Sensor
  13. Logs
  14. Conclusion

Getting Stretch on an SD card toc

The home automation system cannot be offline while I am experimenting with updates. Instead everything is being done on a Raspberry Pi Model 3 B. Once that new system is running, I hope to do the same thing on the old Model B which I have found more than adequate as a home automation server.

As usual, I downloaded the latest Lite version of Raspbian:

  Raspbian Stretch Lite
  Minimal image based on Debian Stretch
  Version: November 2018
  Release date: 2018-11-13
  Kernel version: 4.14

from https://www.raspberrypi.org/downloads/raspbian/. I followed the instructions to copy the image onto a micro-SD card in Installing operating system images but there was a minor glitch.

An older version of Etcher was causing an error similar to that reported in Ubuntu 18.04 cannot launch etcher 1.4.6. Getting a newer version of the program from balena at https://www.balena.io/etcher/ resolved the issue. I removed the older version, by deleting the application etcher-electron-1.4.6-x86_64.AppImage, which I had installed in a directory named ~/.local/bin (short form for /home/michel/.local/bin). I also removed the configuration directory /home/michel/.config/etcher-electron and the application desktop file: /home/michel/.local/share/applications/appimagekit-Etcher.desktop.

Installing the current version of Etcher in a Linux system is quite simple. I extracted the application, balena-etcher-electron-1.4.9-x86_64.AppImage from the downloaded archive and copied it to ~/.local/bin (it could be just about anywhere else). I then started the application and let it install a desktop file. From then on I could start Etcher in the usual way. And, as before, it was a breeze to copy the operating system image onto the SD card with the application. It is not even necessary to decompress the downloaded archive containing the image file.

All these file management chores, copying, deleting, extracting from archives and so on, can all be done with "point and click" ease using Double Commander. It is one of the first utilities I install in all the Linux distributions I use. It even works in Windows, macOS and FreeBSD. Highly recommended.

In the initial post, I installed the new version of Raspbian Stretch on a Raspberrry Pi 3 model B. Then 10 days later I installed the operating system on my usual home automation server a Raspberry Pi 2 model B. The latter does not have wireless capabilities, so I use a Wi-Fi-USB dongle. Installation on the older Raspberry Pi was exactly the same as with the newer model except for a minor difference. Overclocking is possible on the older model but not on the newer model. This can be done with raspi-config.
January 20, 2019

Headless Setup toc

Do something often enough, you should get better at it.

The Raspberry Pi is used as a headless server with no monitor and no keyboard. It is a hassle to dig out a monitor and keyboard just for the initial configuration of such a system. Since connectivity with the local network will be by Wi-Fi, it would be great if that could also be set up before the first boot. Well, it can; even more can be achieved if the desktop computer used to create the SD image is running Linux.

I followed the instructions from Raspian France: Mettre en place une Raspberry Pi sans écran ni clavier, but similar information is available in many places. For example, Brian Lough has a YouTube video, A Truly Headless Setup for your Pi.

The setup is in two or three parts. I am assuming that the (micro) SD card containing the Raspbian image is still connected to the desktop computer and its boot partition is showing up as drive connected to the system. If not, remove the card from the reader, wait awhile and insert it in the reader again. The partition should show up as its file system is the ubiquitous FAT32 which most operating system can read. If the desktop is running Linux the second, much bigger partition, rootfs will also show up. Windows will not read a Ext4 partition. I don't know about macOS.

  1. Enable ssh. All that is needed for this is to create a file of any size, any content, named ssh in the boot partition. It can even be an empty file.
    michel@hp:~$ cd /media/michel/boot michel@hp:/media/michel/boot$ touch ssh michel@hp:/media/michel/boot$ ls -l ssh just checking -rw-r--r-- 1 michel michel 0 jan 3 11:04 ssh

    Information for neophytes

    These commands, shown on a greenish background, are typed out in a virtual console or a terminal which is a wrapper program that runs a shell or command processor on my desktop Ubuntu system. The terminal is easily started with the simultaneous three key press: AltCtrlT.

    The desktop computer name is hp and my user name is michel, hence the shell starts each command line with michel@hp:. After the colon, comes the present working directory (pwd in the jargon). In this case it is initially ~ which is short for the current user home directory which here is /home/michel. Finally, the shell writes the prompt which here is $. It can be different dependant on the shell being used. That's Linux for you, there is more than one shell.

    The first command, cd (for change directory), changes the present working directory. The second command, touch, is used to change the time stamp of a file. But here we are exploiting a side effect of the command; if the named file does not exist, it is created. The third command, ls, lists the content of the present working directory if no parameters follow it. In this instance, the -l parameter asks for a detailed listing. The added filename ssh will limit the listing to matching files. We can then see that the file ssh has indeed been created and that its size is 0.

    Instead of using touch, a file with minimal content named ssh could have been created with a text editor such as nano (see below). It would not matter what the file contained.

    Do not forget to press the Enter key after typing each command. That is how the shell knows that you are done and that it is time for it to do its job. A long time ago, back when most computer programs were punched on cards made of light cardboard, I caused the reboot of a laboratory system when I first used an interactive terminal because I hadn't realized the need for the Enter. I will not go into the details, suffice it to say I am still red-faced about ruining someone's experiment. We all have to learn.

    Because Double Commander allows me to launch a terminal in a specific directory, I did not even have to do the three-finger salute mentioned, nor issue the change directory command cd shown above.
  2. Create the wpa_supplicant configuration file with Wi-Fi credentials. I used nano from the command line, but any text editor such as Geany could probably be used. The file, wpa_supplicant.conf, goes in the boot partition alongside the ssh file created in the previous state. You have to adjust the network credentials of course.
    michel@hp:/media/michel/boot$ nano wpa_supplicant.conf
    country=ca <<-- 2 letter country code update_config=1 ctrl_interface=/var/run/wpa_supplicant network={ scan_ssid=1 ssid="wifi_network_name" psk="wifi_password" }

    Information for neophytes

    GNU nano is a small text editor found on most Linux distributions including Ubuntu and, as we shall see later, Raspbian. It is a relatively simple to use; all keystrokes, with the exception of control and meta sequences, enter text into the file being edited at the current cursor position. The most important control sequences are displayed at the bottom. There is an on-line manual.

  3. Add static IP address for Ethernet and wireless network interfaces. This requires access to the rootfs partition. As far as I know it cannot be done in the Windows. The fixed IP address is added to the dhcpcd configuration file. I prefer to back up that file found in the rootfs/etc directory before modifying it.
    michel@hp:/media/michel/rootfs/etc$ sudo cp dhcpcd.conf dhcpcd.bak michel@hp:/media/michel/rootfs/etc$ sudo nano dhcpcd.conf

    Alternatively, nano can be made to backup the original when saving a changed version of the file.

    michel@hp:/media/michel/rootfs/etc$ sudo nano -B dhcpcd.conf

    The following lines, with appropriate changes, need to be added to the configuration file.

    ... interface eth0 static ip_address=192.168.1.100/24 static routers=192.168.1.1 static domain_name_servers=192.168.1.1 interface wlan0 static ip_address=192.168.1.101/24 static routers=192.168.1.1 static domain_name_servers=192.168.1.1

    Information for neophytes

    All the IP addresses shown above need to be adjusted to take into account the parameters of the local area network. The router IP address (often called gateway elsewhere) and the DNS server IP address are usually the same and often, but not always, similar to the one shown above. But some local area networks (LAN) use the so-called 24-bit address block starting at 10.0.0.0 and others use the 20-bit block starting at 172.16.0.0. All the LANs I have used were in the 16-bit block starting at 192.168.0.0, with the only exceptions being a virtual private network and the virtual machines in Oracle VM VirtualBox. If you want to learn more about these private networks look at the Wikipedia article.

    I think it is best to assign different IP addresses to the two interfaces, unless it is certain that the Ethernet and wireless interfaces will never be simultaneously connected to the same local network.

Of course, if Wi-Fi is not needed then there is no need to create the wpa_supplicant.conf file and there is no need to set up the wlan0 network interface in dhcpcd.conf. Similarly, if the Ethernet interface will not be used, if using a Raspberry Pi Zero for example, then there is no need to define a static address for it in the configuration file. If physically possible, I would still recommend setting up the Ethernet interface, just in case the Wi-Fi connection initially fails?

Initial Boot toc

It is now time for the initial boot. All that needs to be done is to safely eject the SD card from the desktop reader, then place it in the SD slot of the Raspberry Pi and, lastly turn the power on. You may have to wait until the LED activity stops and the red LED is steadily on to open an ssh session.

Since I was able to set up a static IP, it was very easy to open a remote session with ssh.

michel@hp:/media/michel/rootfs/etc$ cd $home michel@hp:~$ ssh pi@192.168.1.101 The authenticity of host '192.168.1.101 (192.168.1.101)' can't be established. ECDSA key fingerprint is SHA256:h8AxJbL5t4WltEu415klkv7LS9px2CFfdorz74Dc0JM. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '192.168.1.101' (ECDSA) to the list of known hosts. pi@192.168.1.101's password: raspberry not echoed on screen! Linux raspberrypi 4.14.79-v7+ #1159 SMP Sun Nov 4 17:50:20 GMT 2018 armv7l ... SSH is enabled and the default password for the 'pi' user has not been changed. This is a security risk - please login as the 'pi' user and type 'passwd' to set a new password. pi@raspberrypi:~ $

Information for neophytes

Opening a remote session on the Raspberry Pi means that shell commands (things like cd, ls and cat) can be executed on the Rapsberry Pi from another computer on the same network as the Raspberry Pi. It comes down to using the desktop keyboard and monitor as if they were connected to the Raspberry Pi.

Of course you have to use a program on the desktop to do this. On Linux systems, it would most likely be the command line utility calledssh, part of the OpenSSH suite of tools which implements the SSH (Secure Shell) protocol. Most Windows users would probably use PuTTY instead of ssh to open a remote session on the Raspberry Pi. From what I can make out, Mac OS X users can use the built-in ssh client in a terminal window as shown above but there are alternatives.

It is necessary to log into an existing user account and provide the correct password when starting a remote session or a local session with a keyboard and monitor connected to the Raspberry Pi. Initially there is only one user account: pi and the password is raspberry. Note how ssh doesn't recognize the secure key returned by the Raspberry Pi (ECSDA is the cryptography algorithm used to generate the key) so it asks if it is alright to connect. Once you have answered yes, in full, it will store the key and will not ask again.

ssh is a secure replacement for telnet. Anyone else remember using NCSA telnet over a Crynwr packet driver on PC running MS-DOS? I would open sessions on the IT department minicomputer, a PDP-11 as I remember. I forget exactly why? Probably to send e-mails, which was a complicated thing to do in the mid '80s. I do remember accessing gopher sites that way in the early '90s and even building a demonstration gopher site for our department trying to get the institution to get on with it. It never did, but then HTTP took over.

If it had not been possible to set a static IP address, then try to connect to raspberrypi.local as shown here.

michel@hp:~$ ssh pi@raspberrypi.local

This will work in a Linux system where the Avahi utility is installed which is the case for most major distributions. It should also work in MAC OS (where Bonjour takes care of the zeroconf protocols). It probably will not work in a fresh install of Windows, but it might if Bonjour was added to connect to a networked printer, for example.

If another Raspberry Pi with the default raspberrypi hostname was already connected to the network, then the newer Pi's hostname would be mangled to avoid conflicts. Its qualified host name address could be something like raspberrypi-1.local. You could use a zeroconf service browser to try to find it.

It would be very awkward to run a home automation server such as Domoticz on a computer with a dynamically assigned IP address. Accordingly, if it had not been possible to set a fixed IP address previously, it should be done at this point.

pi@raspberrypi:~ $ nano /etc/dhcpcd.conf

You just need to add the static addresses as shown in part 3 of the previous section.

As suggested by the log-on screen, you really should change the password and I strongly suggest that you change the hostname to avoid collisions with other Raspberry Pi you may connect to the network. For illustrative purposes, I set the host name to goldserver. There are a couple of other changes that I always do. I set the correct time zone, but I do not bother changing the locale or keyboard map. Minimizing the amount of memory allocated to the GPU is a good idea. Use the configuration utility raspi-config to change these settings.

pi@raspberrypi:~ $ sudo raspi-config

Reboot as suggested, and then after a while reconnect with ssh.

michel@hp:~$ ssh pi@goldserver.local or michel@hp:~$ ssh pi@192.168.1.1

Change the Default User toc

For added security, it is not a bad idea to change the default user. In the past, I would simply add a new user and eventually removed the default user pi. But that was a hit-or-miss approach which usually meant hunting down a missing group membership for the new user when something new did not work. Luckily, Jim McDonnell at Unix etc. has a very good explanation of how to proceed systematically to avoid such problems. I just followed the instructions in How to Rename the Default Raspberry Pi User, executing the scripts found in that article. There were a couple of error messages.

sed: can't read lightdm/lightdm.conf: No such file or directory sed: can't read polkit-1/localauthority.conf.d/60-desktop-policy.con: No such file or directory

Because the listed configuration files are not present in the Lite version of Raspbian there were no real errors, as mentioned in the article.

I did run into another little problem: sudo would complain that it could not resolve host goldserver, which meant it could not obtain its IP address. It turned out that I had to edit the hosts file.
woopi@goldserver:~ $ sudo nano /etc/hosts sudo: unable to resolve host goldserver nevertheless nano started and hosts could be edited
127.0.0.1 localhost ::1 localhost ip6-localhost ip6-loopback ff02::1 ip6-allnodes ff02::2 ip6-allrouters 127.0.1.1 goldserver <<--- change to new hostname
This was because of an error on my part; I had changed the host name by directly editing the hostname file instead of using the raspi-config utility.

Updating the Operating System toc

Before installing any major new piece of software, it is always important to update the operating system. This is something that is done more or less automatically in full blown desktop systems such as Ubuntu. On the Lite version of Raspbian, it has to be done manually.

woopi:goldserver~ $ sudo apt-get update && sudo apt-get upgrade ... The following packages will be upgraded: base-files libperl5.24 libpolkit-agent-1-0 libpolkit-backend-1-0 libpolkit-gobject-1-0 libssl1.0.2 libssl1.1 libwbclient0 openssl perl perl-base perl-modules-5.24 policykit-1 python-rpi.gpio raspberrypi-sys-mods raspi-config samba-common 17 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. Need to get 10.3 MB of archives. After this operation, 120 kB of additional disk space will be used. Do you want to continue? [Y/n] y ...

This operation can take some time, especially with the single core Raspberry Pi 2 model B and even longer for the old Raspberry Pi 1. Nevertheless, the Raspberry Pi 2 manages quite well as a home automation server.

You may want to automate the procedure somewhat by adding the -y switch.

woopi:goldserver~ $ sudo apt-get update && sudo apt-get -y upgrade

Now you may be time to get a coffee, tea or other beverage.

Information for neophytes

Newcomers to Debian Linux might wonder what these commands do. The first, apt-get update, scans the default Debian repository and any other repositories that you may have added (see below for an example) to update the list of currently available software packages in the repositories. The second, apt-get upgrade will replace currently installed packages with any newer version if available. It is important to realize that apt-get upgrade uses to local list of available packages to decide if a newer version is available, it does not scan the repositories itself. That why it is important to execute apt-get update first.

If the -y option were to be added to the sudo apt-get upgrade command, it would not be necessary to confirm, with a y, that the operation is to be completed. I prefer not to do in order to see clearly how much disk space will be occupied by the upgrade.

Other Linux distributions have different package managers: Red Hat and CentOS uses yum, Arch Linux uses pacman and so on.

Even within Debian the story is rather more complex. As far as I know, APT (Advanced Packaging Tool) is actually a "front end" for DPKG the Debian Package Management System. There are other front ends, such as Synaptic, which I occasionally use on my Ubuntu desktop, and Aptitude.

With its inclusion in Ubuntu 16.04 in 2016, a new front end called simply apt started to become popular. Basically, it aggregates the commands spread across the apt-get and apt-cache utilities. You will find in my posts that I sometimes use apt update and apt upgrade instead of the apt-get versions shown above. They are basically interchangeable. The same is true for apt show and apt-get show, but I prefer to use apt-cache policy to get pithier information about available versions of a package as well as the installed version when a package is already installed. For more information see Difference Between apt and apt-get Explained.

Here are three commands that could prove useful to new users of the Linux shell. First, to end a session, use the exit command. If you were running a remote session with ssh then that will bring you back into the desktop terminal program.

woopi@goldserver:~ $ exit logout Connection to goldserver.local closed. michel@hp:~$

The Raspberry Pi will continue to chug along when you logout. Processes including servers such as a Web server and a home automation server that were running will continue to function.

To reboot the Raspberry Pi, issue the reboot command which is not too surprising. But this has to be done as a superuser (with adminitrator priviliges in Windows jargon), which requires using the sudo prefix as done on a number of occasions above.

woopi@goldserver:~ $ sudo reboot Connection to goldserver.local closed by remote host. Connection to goldserver.local closed. michel@hp:~$

You may not see the connection closed messages. The other command is used to shut down the Raspberry Pi in an orderly fashion. It is always best to issue that command and waiting for a little while before removing power to a Rasbperry Pi.

woopi@goldserver:~ $ sudo shutdown now Connection to goldserver.local closed by remote host. Connection to goldserver.local closed. michel@hp:~$

Be careful with these commands. I have lost a lot of work when I carelessly typed the command without looking at the prompt. Instead of shutting down a Raspberry Pi, I entered the command in a Ubuntu shell with the obvious consequence that my desktop computer became very quiet very quickly. You have been warned.


Installing Domoticz toc

Installing Domoticz has always been a very simple matter in the past when following The "easy" way instructions. But for some reason, I was getting an error.

woopi@goldserver:~ $ curl -L install.domoticz.com | sudo bash % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0bash: line 1: syntax error near unexpected token `newline' bash: line 1: `<!DOCTYPE html>' 100 11643 100 11643 0 0 11699 0 --:--:-- --:--:-- --:--:-- 11701

That means that the shell, bash, was choking on the first line of the installation script which was <!DOCTYPE html>. That was strange but after some thought, and running the curl command without the pipe to bash, I realized that the default index page was being returned from domoticz.com and not the script which was presumably at install.domoticz.com. Then it dawned on me that this was probably a consequence of a broken or missing redirection of secure HTTP requests. Something which I have struggled with on my site. So using the full URL solved the problem.

woopi@goldserver:~ $ curl -L https://install.domoticz.com | sudo bash % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 16257 100 16257 0 0 12007 0 0:00:01 0:00:01 --:--:-- 12015 ::: ::: You are root. ::: Verifying free disk space... ... after answering a couple of questions about ports and directories: Ready... ┌─────────────────────┤ Installation Complete! ├─────────────────────┐ │ │ │ Point your browser to either: │ │ │ │ HTTP: 192.168.1.101:8080 │ │ HTPS: 192.168.1.101:443 │ │ │ │ Wiki: https://www.domoticz.com/wiki │ │ Forum: https://www.domoticz.com/forum │ │ │ │ The install log is in /etc/domoticz. │ │ │ │ │ │ <Ok> │ │ │ └────────────────────────────────────────────────────────────────────┘

The installation script will install the git and libudev-dev packages.

In the past, I would copy the database, various bash and python scripts and floor plan images from a running Domoticz installation to a new one and it worked flawlessly. It was a bit more complicated this time because of changes between the newest version of the software and the older version I was using. It was a minor problem, floor plans had to be deleted and rebuilt in the new version.

I get browser cache refresh failures when opening the freshly installed Domoticz web page. I caught on that this problem shows up in very specific circumstances. With Firefox the failure occurs when using HTTPS, but not when using private browsing. Presumably in the latter case, there is no cache to refresh. With Chromium, the cache failure occurred whenever HTTPS is used no matter if incognito browsing is used or not. It seems that the problem is independent of Domoticz and stems from the use of a self-signed TLS certificate.

When using my Android tablets to access the server, I got cache problems even with insecure access. Some tabs, notably Scenes and Temperature, were empty at first and then after some time would become populated with garbled entries. I deleted home screen icons to start Domoticz and the complete browser history and that fixed the problems.

This is a good place to express all my gratitude to the Domoticz development team as well as to the community.

Installing mosquitto toc

An MQTT broker is a necessary part of my home automation system. The mosquitto broker is available in the Rasbian Stretch repository, so I proceeded to install it and the optional utilities to get mosquitto_sub and mosquitto_pub.

woopi@goldserver: $ sudo apt-get install mosquitto mosquitto-clients

After rebooting, I checked and found that the broker was running automatically.

woopi@goldserver:~ $ sudo systemctl status mosquitto.service ● mosquitto.service - LSB: mosquitto MQTT v3.1 message broker Loaded: loaded (/etc/init.d/mosquitto; generated; vendor preset: enabled) Active: active (running) since Tue 2019-01-08 01:46:18 AST; 11h ago Docs: man:systemd-sysv-generator(8) Process: 316 ExecStart=/etc/init.d/mosquitto start (code=exited, status=0/SUCC CGroup: /system.slice/mosquitto.service └─382 /usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf Jan 08 01:46:18 goldserver systemd[1]: Starting LSB: mosquitto MQTT v3.1 message Jan 08 01:46:18 goldserver mosquitto[316]: Starting network daemon:: mosquitto. Jan 08 01:46:18 goldserver systemd[1]: Started LSB: mosquitto MQTT v3.1 message

Most Recent Version of Mosquitto

Version 1.4.10 of the broker was installed, while version 1.5.5 had been working on the old home automation system. The reason is that I had manually installed a newer version of Mosquitto in Jessie. This can be done in Stretch contrary to what I wrote in the first version of this post. Here is how I did on another Raspberry Pi 3 with a fresh installation of Raspbian. Its host name was testbed.

pi@testbed:~ $ wget -O - http://repo.mosquitto.org/debian/mosquitto-repo.gpg.key | sudo apt-key add - ... 2019-01-21 12:25:54 (50.6 MB/s) - written to stdout [3120/3120] OK pi@testbed:~ $ sudo wget -O /etc/apt/sources.list.d/mosquitto-stretch.list http://repo.mosquitto.org/debian/mosquitto-stretch.list ... 2019-01-19 12:27:26 (2.03 MB/s) - ‘/etc/apt/sources.list.d/mosquitto-stretch.list’ saved [51/51] pi@testbed:~ $ sudo apt-get update | sudo apt-get -y install mosquitto ... Setting up mosquitto (1.5.5-0mosquitto1) ... Processing triggers for libc-bin (2.24-11+deb9u3) ... Processing triggers for systemd (232-25+deb9u6) ...

This perhaps cryptic sequence of commands is equivalent to the somewhat dated instructions found in a document perhaps inaptly named Mosquitto Debian repository. I also consulted the repository read me txt.

Verification that it all worked was not too difficult.

pi@testbed:~ $ sudo systemctl status mosquitto ● mosquitto.service - LSB: mosquitto MQTT v3.1 message broker Loaded: loaded (/etc/init.d/mosquitto; generated; vendor preset: enabled) Active: active (running) since Mon 2019-01-21 12:32:19 AST; 1min 20s ago Docs: man:systemd-sysv-generator(8) CGroup: /system.slice/mosquitto.service └─14774 /usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf Jan 21 12:32:19 testbed systemd[1]: Starting LSB: mosquitto MQTT v3.1 message br Jan 21 12:32:19 testbed mosquitto[14768]: Starting network daemon:: mosquitto. Jan 21 12:32:19 testbed systemd[1]: Started LSB: mosquitto MQTT v3.1 message bro

To be doubly sure, I opened two session on the home automation server. In the first I subscribed to all topics with the newly created mqtt broker.

woopi@goldserver:~ $ mosquitto_sub -h testbed.local -v -t "#"

A message was sent to the broker in the second session.

woopi@goldserver:~ $ mosquitto_pub -h testbed.local -t "test/topic" -m "hello there!"

This message was then displayed in the first session.

woopi@goldserver:~ $ mosquitto_sub -h testbed.local -v -t "#" test/topic hello there!

This confirmed that version 1.5.5 of the Mosquitto mqtt broker was functioning. However, I have not encountered problems with the older version of the broker on the home automation system and decided to keep it.


Installing MJPG-Streamer for Use With a Webcam toc

I have already documented how to use MJPG-Streamer in conjunction with Domoticz to stream video from a USB webcam. Everything in part 2. Software works without change in Rasbian Stretch. There is no point going into detail again, here is the sequence of commands needed to install the video streaming software with its prerequisites.

woopi@goldserver:~ $ sudo apt-get install -y cmake libjpeg8-dev ... woopi@goldserver:~ $ wget https://github.com/jacksonliam/mjpg-streamer/archive/master.zip ... woopi@goldserver:~ $ unzip master.zip ... woopi@goldserver:~ $ cd mjpg-streamer-master/mjpg-streamer-experimental woopi@goldserver:~/mjpg-streamer-master/mjpg-streamer-experimental $ make ... woopi@goldserver:~/mjpg-streamer-master/mjpg-streamer-experimental $ sudo make install ...

The executable binary file is placed in the /usr/local/bin directory, the input and output libraries in /usr/local/lib/mjpg-streamer/ and the served web pages in /usr/local/share/mjpg-streamer/www. Here is the command to stream the video from a webcam (Logitech HD Webcam C270) running at 10 frames per second at a 1280x720 pixel resolution. Of course the webcam should be connected to the Raspberry Pi before this command is given.

woopi@goldserver:~ $ mjpg_streamer -i "/usr/local/lib/mjpg-streamer/input_uvc.so -n -f 10 -r 1280x720" \ > -o "/usr/local/lib/mjpg-streamer/output_http.so -p 8085 -w /usr/local/share/mjpg-streamer/www" MJPG Streamer Version.: 2.0 i: Using V4L2 device.: /dev/video0 i: Desired Resolution: 1280 x 720 i: Frames Per Second.: 10 i: Format............: JPEG i: TV-Norm...........: DEFAULT o: www-folder-path......: /usr/local/share/mjpg-streamer/www/ o: HTTP TCP port........: 8085 o: HTTP Listen Address..: (null) o: username:password....: disabled o: commands.............: enabled

Given the specified port, the URL of the MJPG-Streamer Web page is http://goldserver.local:8085. Use the IP address of the Raspberry Pi if mDNS host names cannot be resolved.

Satisfied with the result I removed the installation directory but kept the installation archive moving it to a download directory and renaming it.

woopi@goldserver:~ $ mkdir downloads woopi@goldserver:~ $ mv master.zip downloads/mjpg-streamer.zip woopi@goldserver:~ $ ls downloads mjpg-streamer.zip

Using MJPG-Streamer with a Raspberry Pi Camera Module toc

I also have a V.1 Raspberry Pi camera module (Rev 1.3). To test it with MJPG-Streamer, I powered down the Raspberry Pi, removed the USB webcam, and connected the ribbon cable as shown in How to install / use the Raspberry Pi Camera. Use the connector between the 3.5mm RCA jack and the HDMI connector. Pull up the black tab on the connector about 2 mm, slide the ribbon into the slot with the metallic connectors facing the HDMI port. Press down the black tab to lock the ribbon cable.

After powering up, use the configuration utility raspi-config to enable the camera.

woopi@goldserver:~ $ sudo raspi-config

It is the first option in the Interfacing Options screen. It will be necessary to reboot for the change to take effect and then to open a new session on the Raspberry Pi. Using the default values of the input library for the camera module provided with MJPG-Streamer is very simple.

woopi@goldserver:~ $ mjpg_streamer -i "/usr/local/lib/mjpg-streamer/input_raspicam.so" \ > -o "/usr/local/lib/mjpg-streamer/output_http.so -p 8085 -w /usr/local/share/mjpg-streamer/www" MJPG Streamer Version.: 2.0 i: fps.............: 5 i: resolution........: 640 x 480 i: camera parameters..............: Sharpness 0, Contrast 0, Brightness 50 Saturation 0, ISO 0, Video Stabilisation No, Exposure compensation 0 Exposure Mode 'auto', AWB Mode 'auto', Image Effect 'none' Metering Mode 'average', Colour Effect Enabled No with U = 128, V = 128 Rotation 0, hflip No, vflip No ROI x 0.000000, y 0.000000, w 1.000000 h 1.000000 o: www-folder-path......: /usr/local/share/mjpg-streamer/www/ o: HTTP TCP port........: 8085 o: HTTP Listen Address..: (null) o: username:password....: disabled o: commands.............: enabled i: Starting Camera Encoder Buffer Size 81920

I wanted to use the same parameters as used with the Webcam. It is possible but options are defined differently with this input library.

woopi@goldserver:~ $ mjpg_streamer -i "/usr/local/lib/mjpg-streamer/input_raspicam.so -x 1280 -y 720 -fps 10" \ > -o "/usr/local/lib/mjpg-streamer/output_http.so -p 8085 -w /usr/local/share/mjpg-streamer/www" MJPG Streamer Version.: 2.0 i: fps.............: 10 i: resolution........: 1280 x 720 i: camera parameters..............: ...

More information about the input module can be found on the mjpg-streamer github.

I found the results acceptable, but the camera module seemed more sluggish than the webcam. Something to investigate later.

Installing Mochad toc

For the sake of completeness installation of the Mochad daemon will be presented here. This was already covered in a previous post: Domespic [3] Gateways - Install Mochad. The daemon (or service) is used in conjunction with a CM19A USB PC Transceiver to receive RF transmissions from a battery powered remote control, the HR12A PalmPad, and pass them on to Domoticz to control IoT devices. I had hoped to remove all X10 based technology from my home automation system and just a few days ago the last X10 module was retired from service. However, the RF remote is simple, quite versatile and it is often used. Given its high WAF, it will stay.

Sourceforge still serves mochad version 0.1.16 as the latest version of the service even though version 0.1.17 is available and needed in newer versions of Raspbian. The following shows how to install the later version. However you should check for a newer version here: https://sourceforge.net/projects/mochad/files/ and adjust the wget commands accordingly.

woopi@goldserver:~ $ sudo apt-get install libusb-1.0-0-dev Reading package lists... ... After this operation, 1,711 kB of additional disk space will be used. Do you want to continue? [Y/n] y ... Setting up libusb-1.0-0-dev:armhf (2:1.0.21-1) ... woopi@goldserver:~ $ wget -O mochad.tgz https://sourceforge.net/projects/mochad/files/mochad-0.1.17.tar.gz/download ... woopi@goldserver:~ $ tar xf mochad.tgz ... woopi@goldserver:~ $ cd mochad-0.1.17/ woopi@goldserver:~/mochad-0.1.17 $ ./configure ... woopi@goldserver:~/mochad-0.1.17 $ make ... woopi@goldserver:~/mochad-0.1.17 $ sudo make install

I then plugged in the CM19A into a free USB port and waited a little while for the system to find it. After that I tested that it received RF codes from the remote.

woopi@goldserver:~ $ lsusb Bus 001 Device 005: ID 0bda:8179 Realtek Semiconductor Corp. RTL8188EUS 802.11n Wireless Network Adapter Bus 001 Device 004: ID 0bc7:0002 X10 Wireless Technology, Inc. Firecracker Interface (ACPI-compliant) Bus 001 Device 003: ID 0424:ec00 Standard Microsystems Corp. SMSC9512/9514 Fast Ethernet Adapter Bus 001 Device 002: ID 0424:9514 Standard Microsystems Corp. SMC9514 Hub Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub woopi@goldserver:~ $ nc localhost 1099 01/19 17:23:07 Rx RF HouseUnit: C1 Func: On 01/19 17:23:09 Rx RF HouseUnit: C1 Func: Off 01/19 17:23:12 Rx RF HouseUnit: C2 Func: On

Use a CtrlC combination to exit the netcatprogram.

Using the remote with Domoticz is quite simple since it is directly supported hardware. For details see Domespic [4] Domoticz - Add X10/Mochad to Domoticz.

As before, I cleaned up the home directory, saving the downloaded archive.

woopi@goldserver:~ $ rm -r mochad-0.1.17/ woopi@goldserver:~ $ mv mochad.tgz downloads/

Real-Time Clock toc

A real time clock, based on the DS3231 chip is connected to the Raspberry Pi. The instructions, showing how to install modules for that little daughter board, written back in March 2017 (with an update in May of the same year), Real Time Clock, DS3231, for Domoticz on the Raspberry Pi, work in the newer version of Raspbian. This is a recap of the information.

First check that the I2C interface has been enabled.

woopi@goldserver:~ $ lsmod | grep i2c_ i2c_bcm2835 6465 0 i2c_dev 6674 0

If nothing is displayed, use the configuration utility raspi-config to enable loading the I2C kernel module in the Interfacing Options page. Once I2C has been enabled, a couple of packages are installed.

woopi@goldserver:~ $ sudo apt-get install -y python-smbus i2c-tools ... woopi@goldserver:~ $ sudo i2cdetect -y 1 0 1 2 3 4 5 6 7 8 9 a b c d e f 00: -- -- -- -- -- -- -- -- -- -- -- -- -- 10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 50: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 60: -- -- -- -- -- -- -- -- 68 -- -- -- -- -- -- -- 70: -- -- -- -- -- -- -- --

The first command installed 81.8 kB of archives. The second verified that the clock module was found at address 0x68. If you are willing to skip that step, it is not necessary to install i2c-tools.

Add the real time clock module to the kernel which involves adding a line in the modules file.

woopi@goldserver:~ $ sudo nano /etc/modules
# /etc/modules: kernel modules to load at boot time. # # This file contains the names of kernel modules that should be loaded # at boot time, one per line. Lines beginning with "#" are ignored. i2c-dev rtc-ds1307

The real-time clock can be used once the system is rebooted. The utility used to do that, hwclock, requires superuser rights and should be called with the sudo prefix. There is really no use case for manually reading the clock of a headless server, instead the internal clock is updated from the hardware (real-time) clock whenever the system reboots. But for that to work, the address of the chip has to be set.

woopi@goldserver:~ $ sudo nano /etc/rc.local
#!/bin/sh -e # # rc.local # # This script is executed at the end of each multiuser runlevel. # Make sure that the script will "exit 0" on success or any other # value on error. # # In order to enable or disable this script just change the execution # bits. # # By default this script does nothing. # Print the IP address _IP=$(hostname -I) || true if [ "$_IP" ]; then printf "My IP address is %s\n" "$_IP" fi echo ds1307 0x68 > /sys/class/i2c-adapter/i2c-1/new_device hwclock -s exit 0

The changes take effect after a reboot.

woopi@goldserver:~ $ sudo reboot wait for some time, then open new session michel@hp:~$ ssh woopi@goldserver.local woopi@goldserver.local's password: my-password not echoed to screen Linux goldserver 4.14.79+ #1159 Sun Nov 4 17:28:08 GMT 2018 armv6l ... woopi@goldserver:~ $ sudo hwclock -r 2019-01-19 14:20:04.599529-0400

The real-time clock could drift over time. So a cron task is defined to update the system time from NTP (network time protocol) servers on the internet once an hour. After that, the real-time clock is updated from the system time.

# Edit this file to introduce tasks to be run by cron. # # Each task to run has to be defined through a single line # indicating with different fields when the task will be run # and what command to run for the task # # To define the time you can provide concrete values for # minute (m), hour (h), day of month (dom), month (mon), # and day of week (dow) or use '*' in these fields (for 'any').# # Notice that tasks will be started based on the cron's system # daemon's notion of time and timezones. # # Output of the crontab jobs (including errors) is sent through # email to the user the crontab file belongs to (unless redirected). # # For example, you can run a backup of all your user accounts # at 5 a.m every week with: # 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/ # # For more information see the manual pages of crontab(5) and cron(8) # # m h dom mon dow command 45 * * * * sudo ntpd -gq; sudo hwclock -w

That way, if power is interrupted, the hardware clock has a reasonably accurate value and will keep time more or less accurately because of its internal battery. When power is restored, the Raspberry Pi will reboot and set its time from the real-time clock even if it cannot reach an NTP server.

Temperature and Humidity Sensor toc

As for reading the temperature and humidity sensor, a DHT11, things appear to be simpler than what I wrote in Temperature Sensors on a Raspberry Pi hosting Domoticz back in June of 2017. It looks like it is not necessary to edit the /boot/config.text file. On the other hand, I had to install python pip which was not mentioned in that post.

woopi@goldserver:~ $ sudo apt-cache policy python-pip python-pip: Installed: (none) Candidate: 9.0.1-2+rpt2 Version table: 9.0.1-2+rpt2 500 500 http://archive.raspberrypi.org/debian stretch/main armhf Packages 9.0.1-2 500 500 http://raspbian.raspberrypi.org/raspbian stretch/main armhf Packages woopi@goldserver:~ $ sudo apt install python-pip Reading package lists... Done ... Do you want to continue? [Y/n] y ...

Installation and testing of the Adafruit python library for DHT11 and DHT22 (AM2302) sensors is relatively straight forward. Instead of cloning the Github repository, I just downloaded a zip archive, extracted its content and installed it.

woopi@goldserver:~ $ wget https://github.com/adafruit/DHT-sensor-library/archive/master.zip ... woopi@goldserver:~ $ unzip master.zip ... woopi@goldserver:~ $ cd Adafruit_Python_DHT-master/ woopi@goldserver:~/Adafruit_Python_DHT-master $ sudo python setup.py install running install ... woopi@goldserver:~/Adafruit_Python_DHT-master $ cd .. woopi@goldserver:~ $ python Python 2.7.13 (default, Sep 26 2018, 18:42:22) [GCC 6.3.0 20170516] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import Adafruit_DHT >>> print(Adafruit_DHT.read(11, 24)) (20.0, 19.0) >>> exit()

Since that was working I removed the installation directory but kept the installation archive moving it to a download directory and renaming it.

woopi@goldserver:~ $ rm -r DHT-sensor-library-master/ woopi@goldserver:~ $ mv master.zip downloads/Adafruit_Python_DHT.zip

My original post, Temperature Sensors on a Raspberry Pi hosting Domoticz, contains Python scripts that update a virtual temperature and humidity sensor in Domoticz with values read from the DHT11.

Logs toc

I changed the system configuration to save wear on the SD card by keeping the system journal in a file in RAM .

pi@goldserver:~ $ sudo nano /etc/systemd/journald.conf
# This file is part of systemd. # [Journal] Storage=volatile Compress=yes RuntimeMaxUse=48M

I also enabled the UDP module of the log server on the Raspberry Pi. The configuration file is new slightly different from what I wrote in my previous post: Syslog Server on Raspbian.

woopi@goldberg:~/pythons $ sudo nano /etc/rsyslog.conf
# /etc/rsyslog.conf Configuration file for rsyslog. # # For more information see # /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html ################# #### MODULES #### ################# module(load="imuxsock") # provides support for local system logging module(load="imklog") # provides kernel logging support #module(load="immark") # provides --MARK-- message capability # provides UDP syslog reception module(load="imudp") <-- enabled by removing leading '#' input(type="imudp" port="514") <-- enabled by removing leading '#' # provides TCP syslog reception #module(load="imtcp") #input(type="imtcp" port="514")

This change is to capture log messages from IoT devices running Tasmota by Theo Arends. Logging has to be enabled in those devices for this to function.

Conclusion toc

The last major piece of software to install is a web server. As explained before, I used Lighttpd but I am having some problems with it. If I cannot get the latest version to act as a secure reverse proxy for the MJPG-Streamer web server, I could try installing another package or I could try to revert to the older version that worked in Jessie.

To be honest, this is just being pigheaded. I just hate being stumped by something that worked previously, so I would really like to solve the problem. However, I am pretty sure I will not use the reverse proxy. I will install the latest version of Lighttpd, mostly to support OTA updates of IoT devices on the local network. But I will also install a VPN server on the Raspberry Pi to secure access to the system from outside the local network (i.e. over the Internet).

OpenVpn is presently operating rather well on the older Raspberry Pi running Jessie. It might be a good time to check out WireGuard. There is a project on Github by Arian Mihalko to facilitate its installation onto a Rapsberry Pi and Dr Zzs has a video on the subject.

So this is a good point to back up the SD card. As I experiment with all the above, I could very well want or have to come back to this point which will be easy with the backup. Here is how I do it on my Ubuntu desktop.

michel@hp:~/Documents/domotique$ df -h Sys. de fichiers Taille Utilisé Dispo Uti% Monté sur udev 5,9G 0 5,9G 0% /dev tmpfs 1,2G 2,1M 1,2G 1% /run ... tmpfs 1,2G 16K 1,2G 1% /run/user/121 tmpfs 1,2G 52K 1,2G 1% /run/user/1000 /dev/sde1 44M 23M 22M 51% /media/michel/boot /dev/sde2 7,4G 1,2G 5,9G 17% /media/michel/rootfs michel@hp:~/Documents/domotique$ sudo umount /dev/sde2 michel@hp:~/Documents/domotique$ sudo umount /dev/sde1 michel@hp:~/Documents/domotique$ sudo dd bs=4M if=/dev/sde of=backup_2019_01_08_strech.img 1920+0 enregistrements lus 1920+0 enregistrements écrits 8053063680 bytes (8,1 GB, 7,5 GiB) copied, 474,673 s, 17,0 MB/s
-- Installing an Older Version of Lighttpd->