2020-08-11
md
MQTT with Domoticz
Flashing a Sonoff in the Arduino IDE (in French)-> <-Workflow when Working with a Headless Raspberry Pi

In the first post of this series, I showed how to install Domoticz, a home automation server on a Raspberry Pi. In this step, I will install an MQTT broker on a Raspberry Pi 3. This is a communication protocol which will be used as glue between Domoticz and ESP8266 based hardware that I am using to control devices.

For reference, I am following my own previous posts Add an MQTT Broker to the Raspberry Pi and Add MQTT to Domoticz for the most part. Nevertheless, I hope to improve on the latter. While this installation is done on a Raspberry Pi 3, it should be possible to follow the instructions and install the broker on any model of the Raspberry Pi. Indeed, I have installed Domoticz and the Mosquitto MQTT broker on ARM based boards from Orange Pi running Armbian in similar fashion.

Table of Contents

  1. What is MQTT?
  2. Installing Mosquitto MQTT
  3. Testing Mosquitto MQTT
  4. MQTT Topics
  5. Adding a Virtual Switch in Domoticz
  6. Enabling MQTT in Domoticz
  7. Domoticz MQTT Topics
  8. Testing MQTT in Domoticz

What is MQTT? toc

The Message Queue Telemetry Transport, MQTT for short, is a messaging protocol which has gained momentum for establishing communication between devices in the fast-growing "Internet of Things" sector. Interestingly, the protocol was developed back in 1999, just as the IoT was attracting some interest.

MQTT uses a publish-subscribe paradigm. We all understand point-to-point connections between peers. It is the basis of telephone conversations between two people. The client-server connection is another well-known point-to-point link. To display a web page, a web browser, the client, requests the page from an HTML server that may send the page or may refuse to do so. In these types of links, both parties must intentionally address each other. With MQTT, devices can publish data or subscribe to receive data from a "broker". Some devices do both. When a device publishes data, it does not need to know which devices want to read it. A device that has subscribed, receives data when it becomes available without having to request it from a particular device. Instead the broker acts as a clearing house; it receives all published data organized by "topics", and sends out that data to all subscribers of specific "topics".

Using a broker means that publishing data from one to many is possible at the same time as reading data from many to one. Here is a typical arrangement:

Icons made by EpicCoders from www.flaticon.com is licensed by CC 3.0 BY

The air conditioner is registered with the MQTT broker to receive all data on the topics "temperature" and "humidity". The temperature and humidity sensors are not registered to receive data, they only publish data to the broker.

Every ten minutes, the temperature sensor publishes the room temperature. In practice this means it sends a single number, say 23, under the topic "temperature" to the broker indicating that it is currently 23° C. Similarly, every fifteen minutes the humidity sensor sends data, 62 for example, under the topic "humidity" to the broker signifying that the humidity factor is 62%. In turn, the broker passes on these messages to the air-conditioning unit. It decides on the basis of these two measurements if it needs to turn on or off. This is the Internet of Things at work.

A home automation program with built in MQTT support can be added to the system to provide "intelligence" and control. The architecture of the system could be something like this:

Icons made by EpicCoders from www.flaticon.com is licensed by CC 3.0 BY

The home automation (ha) program is registered with the MQTT broker to receive all data on the topics "temperature", "pressure" and "air conditioner status". The air conditioner is registered with the MQTT broker to receive all data on the topic "air conditioning".

As before, the temperature and humidity sensors publish data to the broker. But now the ha program is registered to receive that information. It decides if the air conditioner should be turned on or off by taking into consideration the sensor data, the time of day, the delay since the last time the unit turned on or off, the presence of people in the house and so on. It instructs the air conditioner to turn on or off by publishing to the topic "air conditioning". The air conditioner publishes changes to its on or off state under the "air conditioner status" topic. Because the ha program subscribes to that topic, it knows when the air conditioning has been turned on or off by the homeowner.

Things are not quite that simple, of course. MQTT is a "transport" protocol, it says nothing about the content of messages. Is that 23 from the temperature sensor 23° centigrade or 23° Fahrenheit? Unless the temperature and humidity sensors are manufactured by the same firm that makes the air conditioner, and unless one of these devices hosts an MQTT broker, there is very little chance that it would work "out of the box". For true IoT, where one can buy sensors from one producer, appliances from another, and a bridge with home automation software from a third firm, there is a need for a universal IoT message syntax that MQTT could transport.

While some effort has been expanded on this subject (see Homie, an MQTT convention for the IoT by Marvin Roger), something widely used has yet to emerge as far as I know. Nevertheless, there is a good case for using MQTT. It is supported in Domoticz. Using a standard protocol is always a better idea than creating an ad hoc solution. But as will become evident, a lot of "glue" has to be applied to cobble something together.

Installing Mosquitto MQTT toc

The only real requirement about the host of the MQTT broker is that the computer be always on and connected to the local area network. The Raspberry Pi hosting the Domoticz server is probably the best choice for most. Luckily, one of the best known open source implementation of MQTT, Mosquitto, runs on the little board.

The original instructions written for Raspbian Jessie are outdated. I have found that the Mosquitto package contained in the Raspbian repositories since Stretch may not be the most up to date but they are recent enough. This makes for a simple installation.

pi@rpi2b:~ $ sudo apt update ... pi@rpi2b:~ $ sudo apt install mosquitto -y

It may be useful to do sudo apt upgrade -y after the update and to reboot before installing the mosquitto package. See Installing mosquitto in Home Automation Servers on Raspbian Buster Lite for more details.

Installation instructions for Raspian Jessie (outdated)

I was able to install the broker without problems following these instructions. To use the Mosquitto Debian repository at Mosquitto.org, you must import the repository package signing key and then make the repository available to apt:

pi@rpi2b:~ $ wget http://repo.mosquitto.org/debian/mosquitto-repo.gpg.key pi@rpi2b:~ $ sudo apt-key add mosquitto-repo.gpg.key pi@rpi2b:~ $ cd /etc/apt/sources.list.d/ pi@rpi2b:/etc/apt/sources.list.d $ sudo wget http://repo.mosquitto.org/debian/mosquitto-jessie.list

The package can now be installed and, to clean up, the Mosquitto repository key can be erased.

pi@rpi2b:/etc/apt/sources.list.d $ cd $home pi@rpi2b:~ $ sudo apt-get udpate pi@rpi2b:~ $ sudo apt-get install mosquitto pi@rpi2b:~ $ rm mosquitto-repo.gpg.key

The broker is now installed. Make sure that it is there:

pi@rpi2b:~ $ systemctl status mosquitto ● mosquitto.service - Mosquitto MQTT Broker Loaded: loaded (/lib/systemd/system/mosquitto.service; disabled) Active: active (running) since Tue 2017-05-30 12:06:43 ADT; 9min ago Docs: man:mosquitto(8) https://mosquitto.org/ Main PID: 1547 (mosquitto) CGroup: /system.slice/mosquitto.service └─1547 /usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf

Reboot and make sure the Mosquitto was started by asking systemctl for the status of mosquitto as above. If the mosquitto.service did not start, you could try enabling it:

pi@rpi2b:~ $ sudo systemctl enable mosquitto.service Synchronizing state for mosquitto.service with sysvinit using update-rc.d... Executing /usr/sbin/update-rc.d mosquitto defaults Executing /usr/sbin/update-rc.d mosquitto enable

Hopefully, Mosquitto will now start automatically when booting Raspbian.

While not essential, you may want to install the Mosquitto clients on the Raspberry Pi:

pi@rpi2b:~ $ sudo apt-get install mosquitto-clients

Example use of the clients will be found in the next section.

Testing Mosquitto MQTT toc

Now that an MQTT broker is installed, I recommend that you read Elliot Williams' excellent instructions at Hackaday as an introduction to using the protocol. Or you can follow along here right away.

I installed Mosquitto clients on my Ubuntu desktop using the Software Center. It can be done with Synaptic or even with apt-get in a terminal window. The keyboard combination that opens a terminal in Ubuntu is Alt Ctrl T.

michel@hp:~ $ sudo apt-get install mosquitto-clients

Instead you could open ssh sessions with the Raspberry Pi and use its Mosquitto clients if they are installed.

The clients are two programs: mosquitto_sub to subscribe to the broker in order to receive messages and mosquitto_pub to publish messages to the broker. We will open two terminals, one that subscribes to messages, and one to send messages. To avoid confusion, I changed the title to Subscribe in one terminal (look in the Terminal menu) and to Publish in the other.

In the Subscribe terminal subscribe to all topics that begin with "home" with the MQTT broker on the Raspberry Pi at IP address 192.168.1.22:

michel@hp:~$ mosquitto_sub -h 192.168.1.22 -v -t "home/#"

As can be seen in the image, nothing much happens. The program is running but nothing will appear until the MQTT broker gets a message under the "home" topic.

In the Publish terminal, publish a message to the MQTT broker.

michel@hp:~$ mosquitto_pub -h 192.168.1.22 -t "home/temperature" -m "21"

The broker will send this message on to the mosquitto_sub process still running in the Subscribe terminal. That process will display the received message, topic first, message after:

Play with this. Open a third terminal and subscribe to the same topic. Then publish a new message and you will see both subscribing clients will display the message. Publish to other subtopics such as "home/humidity". Try publishing a message with "test/temperature" as a topic.

To exit the subscribe clients, press the Ctrl C key combination.

MQTT Topics toc

From the above, it should be obvious that MQTT topics are hierarchical and represented in a fashion quite similar to full file paths (using the Unix '/' separator). And, somewhat like filenames, there are wildcards:

# which represents all lower level topics. Hence subscribing to home/# means that the broker will send all messages with topics such as home, home/humidity, home/temperature, home/temperature/centigrade, home/temperature/fahrenheit and so on.
+ which represents a single level in the hierarchy of topics. Hence subscribing to home/+/switches means that messages with topics such as home/bedroom/switches and home/kitchen/switches will be received, but not home/switches nor home nor switches.

For obvious reasons, the characters '/', '#' or '+' cannot be part of a topic. It is not a good idea to include these characters in a user name or client id (see Security advisory CVE-2017-7650 at mosquitto.org. You can use UTF-8 encoded text in topics such as /saison/été. At least this works in Linux. How it works in Windows, which uses 16 bit Unicode instead of UTF-8, I have yet to check.

When debugging MQTT messaging, it may be useful to subscribe to all topics:

michel@hp:~$ mosquitto_sub -h 192.168.1.22 -v -t "#"

Adding a Virtual Switch in Domoticz toc

So far no devices have been added to the Domoticz database on the Raspberry Pi 3. It will be difficult to test MQTT in conjunction with the home automation program without at least one virtual switch. The procedure to add a virtual switch, which eventually will be linked to a physical switch is almost exactly the same as the one followed in X10 On/Off Switches with a CM11A Interface in Domoticz.

The first step is to create a "dummy" hardware; Domoticz does not know how to handle the WiFi switches that I will be installing later on.

  1. Click on the Setup tab.
  2. Select Hardware in the drop-down menu.
  3. Enter a name for the hardware; I chose Virtual.
  4. For the Type select Dummy (Does nothing, use for virtual switches only) in the drop-down list.

    Add virtual hardware

  5. Click on the Add button. The added hardware will be added in a table at the top of the page:
  6. This step is done only once.

To add a virtual switch

  1. Click on the Create Virtual Sensors in the newly created Virtual hardware.
  2. Fill in the Name field, select Switch for Sensor Type in the Create Virtual Sensor window:

A message will pop up confirming the creation of the sensor and stating that it can be found in the devices' list. To see the list, click on the Setup tab and then select Devices.

The newly created switch can also be found on the Switches tab.

Clicking on the virtual switch bulb icon, found on the Switches tab or in the list of devices, will toggle its image to the on or off state. Try it and you will see a message pop up for a couple of seconds saying that the device has been turned on or off as appropriate. Of course, nothing actually happens because this virtual switch is not connected to anything yet. How this works is shown in the next section.

Enabling MQTT in Domoticz toc

The first step in adding MQTT capabilities to Domoticz is to add the 'hardware'. The procedure is similar to adding the dummy hardware in the previous secion

  1. Click on the Setup tab.
  2. Select Hardware in the drop-down menu.
  3. Enter a name for the hardware. Again showing how imaginative I am, I chose MQTT.
  4. For the Type select MQTT Client Gateway with LAN interface in the drop-down list.
  5. Set the Remote Address and Port of the MQTT broker. In my case, the broker is on the same host as the Domoticz server so I used localhost but I could have used the full IP address 192.168.1.22 or perhaps 127.0.0.1. I set the Port to 1883 which is the Mosquitto MQTT default port which I not changed.
    Add virtual hardware
    Make sure that Publish Topic is set to out; I think that is the default.
  6. Click on the Add button. The added hardware will be included along with the Virtual hardware in the table at the top of the page.

Domoticz MQTT Topics toc

It might be a good idea to reread the section on MQTT Topics in general before reading this summary of the MQTT topics used by the home automation system. Once the MQTT "hardware" is enabled in Domoticz it will publish MQTT messages on the domoticz/out topic whenever the state of a device changes. At the same time, it subscribes to the domoticz/in topic and will take action whenever it receives a message with the correct syntax. It's that simple, Domoticz uses only two topics.

Topic: domoticz/out
Publisher: Domoticz
Subscribers: home automation devices
Purpose: Domoticz sends messages under that topic to the MQTT broker when it wants one or more devices to perform some action such as turn the lamp on/off, set the light dim level, etc.
Topic: domoticz/in
Publisher: home automation device
Subscriber: Domoticz
Purpose: a home automation device sends messages under that topic to the MQTT broker when it wants to inform the home automation server that its state has changed usually because it has been manually turned on, off, dimmed, etc. Domoticz will use these messages to update its device database and web interface to show the new status of the device..

When MQTT "hardware" is enabled in Domoticz, the latter will publish a message using the domiticz/out whenever the state of a device is changed no matter if the device is controlled by MQTT messages or not. For example, if weather "hardware" has been enabled, then each time the temperature is reported by the weather service, Domoticz updates the database, updates the displayed temperature on the Web interface and sends out a message domiticz/out with the data. My "intelligent" garage door closer uses that information to decide how long to wait before automatically closing the garage door.

This description of message topics is valid only if the Publish Topic field in the MQTT Client Gateway with LAN interface hardware is set to out (or out + /, I suppose).

Since the MQTT protocol is a publish/subscribe service, it is easy to send messages and listen to messages. This is done in the next section to verify that everything works correctly.

Testing MQTT in Domoticz toc

It would be possible to subscribe to the domoticz/out topic with the mosquitto_sub utiliy in a terminal.

michel@hp:~$ mosquitto_sub -h 192.168.1.22 -v -t "domoticz/out"

Then in another terminal it would be possible to subscribe to the domoticz/in topic to see all the MQTT messages related to Domoticz.

michel@hp:~$ mosquitto_sub -h 192.168.1.22 -v -t "domoticz/in"

Of course, the correct IP address should be used. However I suggest subscribing to topic "#" as shown below in order to see all messages sent to the MQTT broker in sequence. That means that messages sent to Domoticz and messages sent by Domoticz will be seen. Then generate outbound Domoticz MQTT messages by toggling any switch on and off in the Switches tab sheet of the Domoticz web interface.

michel@hp:~$ mosquitto_sub -h 192.168.1.22 -v -t "#" domoticz/out { "Battery" : 255, "RSSI" : 12, "dtype" : "Light/Switch", "id" : "00014051", "idx" : 1, "name" : "Test Switch", "nvalue" : 1, "stype" : "Switch", "svalue1" : "0", "switchType" : "On/Off", "unit" : 1 } domoticz/out { "Battery" : 255, "RSSI" : 12, "dtype" : "Light/Switch", "id" : "00014051", "idx" : 1, "name" : "Test Switch", "nvalue" : 0, "stype" : "Switch", "svalue1" : "0", "switchType" : "On/Off", "unit" : 1 }

Leave the terminal open to see more messages later on.

These are typical JSON formatted Domoticz messages. For details about their structure see Domoticz API/JSON URL's. The real Wi-Fi switch will have to subscribe with the MQTT broker to receive all domotic/out messages then handle these to turn itself on or off. To do that, it will have to parse the message, look at the idx field to see if the message is meant for itself and then interpret the nvalue field to see if it should turn itself on or off. This looks much more daunting than it is thanks to the work of many persons that have developed open source libraries for the ESP8266. If something like the Tasmota firmware (see below) is used, then it will all be done transparently.

Conversely, the Wi-Fi switch will have to send MQTT messages to the broker to report that it has been manually turned on or off. The topic for those types of messages is domoticz/in because Domoticz subscribes to that topic. The home automation system will then update the status of the virtual switch in its database and its displayed status on the web interface. Here is a typical exchange of messages that occurs as displayed in the subscribed terminal.

domoticz/in {"idx":86,"nvalue":1,"svalue":"","Battery":81,"RSSI":6} ... domoticz/out { "Battery" : 81, "RSSI" : 6, "description" : "", "dtype" : "Light/Switch", "id" : "000140A6", "idx" : 86, "name" : "zoozee3", "nvalue" : 1, "stype" : "Switch", "switchType" : "On/Off", "unit" : 1 }

Note how the message sent by the switch is much shorter than those published by Domoticz. Note also how the later responds almost immediately with its own outgoing message confirming that the switch is on. If you don't already have a Wi-Fi switch that sends out messages, it can be simulated by publishing a message to the MQTT broker. I am assuming that the virtual switch that was installed in the previous section has an index (idx) of 1. The index is shown in the devices list. To see the list, click on the Setup tab and then select Devices . If it is not 1, then adjust the value when publising the following MQTT message in a second terminal.

michel@hp:~$ mosquitto_pub -h 192.168.1.22 -t "domoticz/in" -m '{ "idx" : 1, "nvalue" : 1}'

Since the mosquitto_sub client is subscribed to all message, the message published under topic domoticz/in will show up in the subscribing terminal as the almost instantaneous reply from Domoticz.

... domoticz/in { "idx" : 1, "nvalue" : 1} ... domoticz/out { "Battery" : 81, "RSSI" : 6, "description" : "", "dtype" : "Light/Switch", "id" : "00014051", "idx" : 86, "name" : "Test Switch", "nvalue" : 1, "stype" : "Switch", "switchType" : "On/Off", "unit" : 1 }

The Domoticz log also shows that it got the incoming message.

2017-05-31 13:14:38.446 MQTT: Topic: domoticz/in, Message: { "idx" : 1, "nvalue" : 1}

To see the log click on the Setup tab and then select Log. Then return to the Switches tab, and verify that the bulb icon of the virtual switch is set to the on state. This takes a little while so be patient.

To turn the virtual switch state to off, a similar MQTT message can be sent:

michel@hp:~$ mosquitto_pub -h 192.168.1.22 -t "domoticz/in" -m '{ "idx" : 0, "nvalue" : 0}'

Again adjust the idx value to reflect the correct index of the virtual switch. Again, the terminal in which mosquitto_sub is running will show the exchange of messages through the broker.

... domoticz/in { "idx" : 1, "nvalue" : 0} ... domoticz/out { "Battery" : 81, "RSSI" : 6, "description" : "", "dtype" : "Light/Switch", "id" : "00014051", "idx" : 1, "name" : "Test Switch", "nvalue" : 0, "stype" : "Switch", "switchType" : "On/Off", "unit" : 1 }

The real test will be the addition of actual hardware device using the MQTT protocol. Shelly devices are well known for their built-in MQTT Support. The firmware of other ESP8266 based devices, such as the Sonoff Basic, can be replaced to add MQTT control. While there are other possibilies, I have been using Tasmota by Theo Arends for a number of years with good results. I have a French language post (see Flashing a Sonoff with the Arduino IDE) showing how to flash the Sonoff Basic while Flash Tasmota on a Sonoff in DIY mode in Linux available in English and French shows how to replace the firmware of a Sonoff Mini or Basic R3. Note that there are many other tutorials on the Web which cover simpler approaches. Be prepared to see many more MQTT messages when performing the above tests with a Tasmota device because, by default, it publishes often to other topics as can be seen below.

15:49:31 MQT: tele/kitchen/LWT = Online (retained) 15:49:31 MQT: cmnd/kitchen/POWER = 15:49:31 MQT: tele/kitchen/INFO1 = {"Module":"Generic","Version":"6.4.1(tweak)","FallbackTopic":"cmnd/DVES_399992_fb/","GroupTopic":"sonoffs"} 15:49:31 MQT: tele/kitchen/INFO2 = {"WebServerMode":"Admin","Hostname":"kitchen","IPAddress":"192.168.0.129"} 15:49:31 MQT: tele/kitchen/INFO3 = {"RestartReason":"Software/System restart"} 15:49:31 MQT: stat/kitchen/RESULT = {"POWER":"OFF"} 15:49:31 MQT: stat/kitchen/POWER = OFF
Flashing a Sonoff in the Arduino IDE (in French)-> <-Workflow when Working with a Headless Raspberry Pi