December 19, 2016

MQTT is a machine-to-machine (M2M)/"Internet of Things" connectivity protocol. It was designed as an extremely lightweight publish/subscribe messaging transport.

Source: MQTT.ORG

Not a bad collection of trending terms: 'M2M', 'IoT' and 'pub/sub'. But what of 'MQTT' itself, what does it represent? The FAQ says "MQTT stands for MQ Telemetry Transport" and MQ comes from IBM's WebSphere MQ family of products. While this seems to be the flavour of the month, it turns out that the messaging protocol was invented back in 1999.

At one point, I read somewhere that there was no software problem the could not be solved with an additional layer of abstraction. That is what MQTT is, another layer on top of TCP/IP. So much for 'lightweight', implementing IP-TCP-MQTT to get a couple of devices talking is no easy task. I remember writing an IP-UPD stack for MS-DOS in Turbo Pascal many years ago and that was quite an effort. I never got to add TCP, technology moved faster than me, MS-DOS was replaced by Windows and a TCP/UDP/IP stack became a standard part of the OS. And again technology is moving so fast, that we now have 3$ chips, the ESP8266 for one, with a TCP/IP stack.

Publish, Subscribe Messaging

We all understand point-to-point connections between peers. It is the basis of telephone conversations between two people. The client-server connection is another well known point-to-point link. To display a web page, a web browser, the client, requests the page from an HTML server that may send the page or may refuse to do so.

In the MQTT world of devices, each can publish data or subscribe to receive data. Some devices do both. When a device publishes data, it does not need to know which devices want to read it. A device that has subscribed, receives data when it becomes available without having to request it from a particular device. Instead a "broker" acts as a clearing house; it receives all published data organized by "topics", and sends out that data to all subscribers of specific "topics".

Using a broker means that publishing data from one to many is possible at the same time as reading data from many to one. Here is a typical representation of the architecture:

Icons made by EpicCoders from www.flaticon.com is licensed by CC 3.0 BY

The air conditioner is registered with the MQTT broker to receive all data for the topics "temperature" and "humidity".

Every ten minutes, the temperature sensor publishes the current temperature. In practice this means it sends the temperature, say 23, under the topic "temperature" to the broker meaning that it is now 23° C. Similarly, every fifteen minutes the humidity sensor sends the humidity, say 62, under the topic "humidity" to the broker signifying that the humidity factor is 62%. In turn, the broker passes on these messages to the air conditioning unit. It decides on the basis of these two measurements if need to turn on or off. This is the Internet of Things at work.

This scenario is a pretty good case for using MQTT which is supported in Domoticz. Using a standard protocol that is supported by the home control application is probably better than creating an ad hoc solution. Of course this means that I have to familiarize myself with the MQTT protocol. The first step is to install an MQTT broker.

Installing Mosquitto MQTT

Since the Raspberry Pi hosting the home automation software will always be on, it makes sense that it should host the MQTT broker. One of the best known Open source MQTT broker is Mosquitto which is what I installed following the instructions found on that site.

To use the Mosquitto Debian repository at Mosquitto.org, you must import the repository package signing key and then make the repository available to apt:

pi@rpi2b:~ $ wget http://repo.mosquitto.org/debian/mosquitto-repo.gpg.key pi@rpi2b:~ $ sudo apt-key add mosquitto-repo.gpg.key pi@rpi2b:~ $ cd /etc/apt/sources.list.d/ pi@rpi2b:/etc/apt/sources.list.d $ sudo wget http://repo.mosquitto.org/debian/mosquitto-jessie.list

The package can now be installed and, to clean up, the Mosquitto repository key can be erased.

pi@rpi2b:/etc/apt/sources.list.d $ cd $home pi@rpi2b:~ $ sudo apt-get update pi@rpi2b:~ $ sudo apt-get install mosquitto pi@rpi2b:~ $ rm mosquitto-repo.gpg.key

The broker is now installed.

There is no requirement that the broker be on the same computer as the Domoticz server. The broker could be installed on any computer on the network including a router. If it happens to be running OpenWrt then it should be very easy to integrate the broker as Mosquitto provides a binary.

Playing with MQTT

There is not much that can be done with just the MQTT broker alone. We will add some mosquito clients. You can follow Elliot Williams excellent instructions which have been my guide to MQTT. Instead of installing Mosquitto clients on the Rasberry Pi as suggested by Elliot, I installed them on my Ubuntu desktop using the Software Center. It can be done with Synaptic or even with apt-get in a terminal window. The keyboard combination that opens a terminal in Ubuntu is Alt + Ctrl + T.

michel@hp:~ $ sudo apt-get install mosquitto-clients

The clients are two programs mosquitto_sub to subscribe to the broker in order to receive messages and mosquitto_pub to publish messages to the broker We will open two terminals, one that subscribes to messages, and one to send messages. To avoid confusion, I changed the terminal title to Subscribe in one terminal (look in the Terminal menu) and to Publish the other.

In the Subscribe terminal subscribe to all topics that begin with "home" with the MQTT broker on the Raspberry Pi at IP address 192.168.0.22 :

michel@hp:~$ mosquitto_sub -h 192.168.0.22 -v -t "home/#"

In the Publish terminal, publish a message to the MQTT broker

michel@hp:~$ mosquitto_pub -h 192.168.0.22 -t "home/temperature" -m "21"

The broker will send this message on to the mosquitto_sub process still running in the Subscribe terminal. That process will display the received message, topic first, message after:

Play with this. Open a third terminal and subscribe to the same topic. Then publish a new message and you will see both subscribing clients will display the message. Publish to other subtopics such as "home/humidity". Try publishing a message with "test/temperature" as a topic.

I installed MyMQTT on an Android tablet. That application can both subscribe and publish from a brooker. I had fun seeing everything updated as I published on one or the application and thinking that a small Raspberry Pi was acting as broker while running my home automation application at the same time. Of course the next step is to get the Domoticz server and the Mosquitto broker talking to each other. But before, there is a bit more to learn about the MQTT protocol.

MQTT Topics

From the above, it should be obvious that mqtt topics are hierarchical and represented in a fashion quite similar to full file paths (using the Unix '/' separator). And, somewhat like filenames, there are wildcards:

# which represents all lower level topics. Hence subscribing to home/# means that the broker will send all messages with topics such as home, home/humidity, home/temperature, home/temperature/centigrade, home/temperature/fahrenheit and so on.
+ which represents a single level in the hierarchy of topics. Hence subscribing to home/+/switches means that messages with topics such as home/bedroom/switches and home/kitchen/switches will be received, but not home/switches nor home nor

For obvious reasons, you cannot have an '/', '#' or '+' as part of a topic. You can also use UTF-8 encoded text in topics such as /saison/été. At least this works in Linux. How it works in Windows, which uses 16 bit Unicode instead of UTF-8, I have yet to check.

Quality of Service

There are three levels of Quality of serivce (QoS) in the protocol. They define the lengths to which the broker will try to ensure that a message is received.

Source: Valerie Lampkin, What is MQTT ....

As I see it, QoS level 0 is fast, simple, and may not work. Its like the X10 protocol except that it is fast and probably much more dependable. I assume this level would be adequate for turning on lights. Remember, this is running over TCP which does its own handshaking to ensure delivery.

Using QoS levels 1 or 2 is just as simple from a user's point of view. They do mean that the broker saves all published messages until it is certain that all subscribers have received them. Of course, miracles are out of the question. The broker cannot know in advance that a device will subscribe in five minutes for a topic which has just been published. So these levels of service apply to already established subscriptions. In that case, the subscriber can log off and then log back on and, at that time, it will receive all message published since it was last on line.

Retained Messages

There is one scenario,, where it would be useful to obtain a retained message as explained by Elliot Williams. Assume a temperature sensor publishes data every fifeen minutes. What happens if an application first subscribes for that data? In the worse case scenario it could be almost fifteen minutes before it has any data. If the sensor published with the ... to be completed

Reference: Mosquitto man Page mqtt-7

Second Thoughts about MQTT

The rosy scenario describing MQTT as the glue for the IoT is pretty far off the mark. For one thing, MQTT is a "transport" protocol, it says nothing about the content of messages. Unless the temperature and humidity sensors are manufactured by the same firm that makes the air conditionner, and unless one of these devices hosts an MQTT broker, there is very little chance that it would work. For true IoT, where one can buy sensors from one producer, appliances from another, and a bridge with home automation software from a third firm, there is a need for a universal IoT message syntax that MQTT could transport.

Many have thought about this before. For example, Marvin Roger has devised Homie which he calls an MQTT convention for the IoT. He has implemented this convention for the ESP8266 as an Aduino library. This project is worth investigating.

In the meantime, I will look at using Domoticz to control devices that hopefully will be able to use MQTT to communicate with the program. Looking at our first diagram, we will now have something like this

Icons made by EpicCoders from www.flaticon.com is licensed by CC 3.0 BY

The home automation (ha) program is registered with the MQTT broker to receive all data for the topics "temperature", "pressure" and "air conditioning". The air conditioner is registered with the mqtt broker to receive all data for the topic "air conditioning".

Every ten minutes, the temperature sensor publishes the current temperature. In practice this means it sends the temperature, say 23, under the topic "temperature" to the broker meaning that it is now 23° C. Similarly, every fifteen minutes the humidity sensor sends the humidity, say 62, under the topic "humidity" to the broker signifying that the humidity factor is 62%. In turn, the broker passes on these messages to the ha program. It decides on the basis of these two measurements if there is a need to turn on or off the air conditioning. If there is a change, it publishes a message to that effect to the broker under the topic "air conditioning". The broker then sends on the message to the air conditioner that will turn on or off as desired.

If I decide to turn on the air conditioner, the machine publishes a message to the broker about its new status under the topic "air conditioning". The broker passes on that message to the ha program that can update the air conditioner status.

Clearly, there is no absolute need for the broker, the sensors could send their message directly to the ha program. The ha program and the air conditioner could be speaking to each other using a peer-to-peer protocol. MQTT is just another layer of abstraction. Furthermore, it is not sufficient. Unless we are very lucky, there needs to be yet another layer, a translation layer which converts the contents of messages from the sensor and air conditioner to a format that the ha program can handle. Translates in the other direction is needed for the air conditioner also.

The translator is yet another layer of abstraction, but if it worked it would mean that in principle, the home automation program could be replaced without too much work.