2021-07-19
md
DIY Tasmota Backups
<-DIY Tasmota Backups (version 0.3)

There is a well-regarded backup solution for Tasmota devices: TasmoBackupV1 by danmed. I wanted something more nimble that did not require PHP and Sqlite (or equivalent) to centralize all the saved configurations in a database. A simple directory on a drive with clearly dated backup files would suffice. My do-it-yourself solution can be found on Github: tasmotasbacker. This is a relatively lightweight (< 4MB) utility with a graphic user interface written in Free Pascal / Lazarus which hopefully others will find useful.

Version 0.9.1 of tasmotasbacker is a combination of

Both those older versions remain available should the combined functionality not be needed, although as far as I know, there is no significant cost to using the newest version. The old documentation about these versions, which is similar to this post, can be found here: DIY Tasmota Backups (version 0.3.

Table of Content

  1. Getting the Tasmota Configuration
  2. Finding Tasmota Devices
    1. MQTT Discovery
    2. HTTP Discovery
  3. Using the Backup Utility
    1. The Initial Screen of the Wizard
    2. Finding Devices with MQTT Discovery
    3. Finding Devices with HTTP Discovery
    4. The List of Found Devices
    5. Setting the Backup Parameters
    6. Backup Results
    7. Saving or Resetting Options
  4. Programmer Notes
    1. HTTP Timing
    2. Timeout Units
    3. Mosquitto Header File
    4. Discovery by Other Means
  5. Final Remarks

Getting the Tasmota Configuration toc

It is a simple matter to download the configuration of a Tasmota device if its IP address is known. By default Tasmota runs a web server which can be used to control and manage the device. Just start a Web browser and enter the IP address in the address bar. The image on the left below is the main web page of a simple on/off device, a Sonoff module.

As can be seen it is mostly a menu. Click on Configuration to bring up the next menu which is the image shown on the right above. Click on the Backup Configuration button. This will bring up a download window asking if the file is to be downloaded or not.

Click on Save File to complete the download into the directory of your choice. The Restore Configuration just below the Backup Configuration button can then be used to copy the saved configuration file back onto the device if that should be necessary. This can be quite useful when playing with the many configuration options available in Tasmota. I also find it practical to restore the configuration of a temperamental Tuya dimmer which for some reason loses its configuration when there is some glitch or interruption in the mains power. Strangely, two other identical dimmers never seem to suffer the same fate.

While downloading a single configuration is a simple enough operation, it becomes tedious really quickly when one wants to back up the configuration of multiple devices. I have a modest home automation system but it nevertheless has some 20 Tasmota devices and I have never had the patience to go through all of them at once to make backups even if I know that would be a good idea.

Luckily Theo Arendts, the original creator of Tasmota, and his collaborators have provided a mechanism that can be used to automate the process. A simple HTTP request

http://192.168.1.105/dl

will bring up the last dialog shown to save the configuration file without manually clicking on buttons in a GUI. Note that many sources show a questions mark at the end of the path (http://<ip_address>/dl?). However a question mark is used at the start of an optional query, and since there is no query I did not include it. Nevertheless /dl and /dl? both work.

As expected a utility such as wget or curl can get the device configuration.

michel@hp:~$ wget 192.168.1.105/dl -O lampe-table.bin --2021-06-21 22:04:42-- http://192.168.1.105/dl Connecting to 192.168.1.105:80... connected. HTTP request sent, awaiting response... 200 OK Length: 4096 (4.0K) [application/octet-stream] Saving to: ‘lampe-table.bin’ lampe-table.bin 100%[===================>] 4.00K --.-KB/s in 0.02s 2021-06-21 22:04:42 (206 KB/s) - ‘lampe-table.bin’ saved [4096/4096] michel@hp:~$ curl 192.168.1.105/dl -o lampe-table.bin % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 4096 100 4096 0 0 27863 0 --:--:-- --:--:-- --:--:-- 2786

Or to get a "quieter" output

michel@hp:~$ wget 192.168.1.105/dl -O lampe-table.bin -q michel@hp:~$ curl 192.168.1.105/dl -o lampe-table.bin --no-progress-meter

Note how the output file name is preceded by an upper case O in the wget command and a lower case o in the curl and these options are case sensitive.

Given a list of valid IP addresses, I suppose it would be fairly simple to cobble a Bash script that would download all desired device configurations.

Finding Tasmota Devices toc

The problem is getting that list of valid IP addresses of the Tasmota devices on the local network if the devices obtain dynamic IP addresses from a DHCP server. One way would be to poke, as it were, IP addresses and keeping track of those addresses which produced a valid response. My preferred approach is to send query Tasmota devices through the MQTT broker to which they are connected and obtain their IP address directly from the device. As usual, each method has advantages and drawbacks.

There are other ways of proceeding and some of them are discussed below, but only the first two methods are implemented.

MQTT Discovery toc

The technique is pretty straight forward: subscribe to the stat/+/STATUS5 topic and then publish an MQTT message with payload 5 to the cmnd/tasmotas/status topic. All Tasmota devices connected to the broker will respond with the requested status 5 (StatusNET) message which includes the device hostname and IP address. The box below shows the reply from one device.

topic: stat/lampe-table/STATUS5 payload: { "StatusNET": { "Hostname":"lampe-table", "IPAddress":"192.168.1.105", "Gateway":"192.168.1.1", "Subnetmask":"255.255.255.0", "DNSServer":"192.168.1.1", "Mac":"84:83:82:E0:E1:E2", "Webserver":2, "WifiConfig":4, "WifiPower":17.0 } }

The topic to which the devices publish will contain the device MQTT Topic which by default would be something like tasmota_495A64 but which on my devices is the same as the hostname. As can be seen, the hostname itself is the first field in the payload.

Obviously, this approach will work only if

  1. an MQTT broker is available,
  2. the Tasmota devices are connected to the MQTT broker,
  3. the MQTT Group topic of all devices has not been changed to something now forgotten.

On that last point, I believe that initially there was a single group topic and its default value was sonoffs. That changed to tasmotas in version 7.1.0. To address the possibility that there may be older versions of the firmware on a device or that the group topic may have been changed, the tasmotasbacker utility will accept multiple group topics. Using both tasmotas and sonoffs should mean that the published message will be received by most Tasmota devices connected to the broker.

Since version 8.2.0.2 of Tasmota, up to four group topics can be defined in the firmware itself (see Add multiple GroupTopic (x=1-4) (default disabled) #8014). That means that newer versions of the firmware will subscribe to MQTT messages under the topics cmnd/xxxxx/status where xxxxx are the specified group topics and will reply to a message with payload 5 to any onf of those topics. So as long as at least one topic specified in the utility is among one of the group topics in Tasmota the device should reply and be added to the download list.

I find this approach fast and reliable, but it has a drawback. It will not reach Tasmota that are not connected to the MQTT broker. I can imagine that some would not use MQTT. A home automation system built on Tasmota devices, ha-bridge and Amazon Echo/Dot controllers would not require an MQTT broker for example.

HTTP Discovery toc

The Python scripts tas-backup and Tasmota-Config-Backup by dragonflyuk and Yuval adopt a more direct approach. For each IP address in a subnet or in a smaller range, they send an HTTP request for the status of the device expecting it to be a Tasmota device. If the answer is reasonable, they obtain the hostname from the reply and go on to download the configuration and save its content to the a file.

The utility does something similar. For each IP address, say 192.168.1.5, it makes two requests.

http://192.168.1.5/cm?cmnd=hostname http://192.168.1.5/cm?cmnd=topic

With a Tasmota device it should get back JSON formatted replies such as {"Hostname":"lampe"} and {"Topic":"tasmota_495A64"}. Of course the value will be different, but the keys "Hostname" and "Topic" must be as shown. It is very unlikely that a non Tasmota peripheral would give these exact replies, so in the advent that the expected replies are obtained, the IP address is added to the list of Tasmota devices.

Once the timing problems discussed later on are solved, this is a reliable method which should find all the devices whether configured correctly to connect to an MQTT broker or not. The downside of this approach is that it can take a long time, most of which is probably wasted querying IP addresses that are not even being used. Perhaps to alleviate this problem, the script by dragonflyuk starts by sending a ping to the peripheral and only on getting a valid response does it go on to formulate an HTTP status request. Unfortunately, the basic Free Pascal units from fcl-net and fcl-web do not have this functionality.

If the DHCP allocates dynamic IP addresses in a narrow range, it does help to search only those IP addresses in that range.

Using the Backup Utility toc

The utility adheres more or less to the wizard paradigm. Saving a set of Tasmota configuration files is done by progressing through a series of screens or windows, modifying the parameters as needed. In the last screen of the utility, there is the possibility to save all modified parameters. Hopefully, future downloads will be mostly a matter of clicking on a few buttons.

The Initial Screen of the Wizard toc

Not unexpectedly, the purpose of the first screen is to choose how the IP address of Tasmota devices will be discovered.

As explained before, there are two choices.

To continue, click on the Next button on the bottom right of the window. The screen next displayed will depend on the chosen discovery method. All screens after the next one will be the same no matter which discovery method is chosen.

There is also the possibility to exit the program by pressing the Quit button on the bottom left of all windows in the wizard.

Finding Devices with MQTT Discovery toc

The parameters of the MQTT broker needed to obtain the IP addresses of all Tasmota devices connected to the broker must be defined in the second screen show here.

Of course, the IP address of the broker as well as the TCP port it uses have to be set correctly. The default MQTT port is 1883 as shown. The user and password fields are empty by default, but if the broker requires a user name and password then they must be specified. Clicking on the eye icon to the right of the MQTT broker password can display or hide it as desired. The MQTT topics used to send status queries must be specified in the Topics field. If there is more than one topic, they must be separated by commas; spaces are ignored so topics with embedded spaces cannot be used. By default, two topics are defined: tasmotas which is the newer default MQTT topic defined in Tasmota and sonoffs which was the default topic in older version of the firmware.

Once all the fields are correct, click on the Next button to start the process of getting the list of IP addresses. Before an attempt is made to connect to the broker, an attempt to make a TCP connection to the host:port end point is made. If this connection is established, then the utility goes on to connect to the broker, subscribe to topics, publish status query messages and so on. If the connection cannot be established then an error message is displayed and the process of finding devices is not started. This extra step has been added in the latest version of the utility because it was found that the program would take a very long time to recover if an incorrect host IP or TCP port was specified. The Timeout field is used to specify the maximum number of seconds the utility will wait for a reply when establishing that first TCP connection verifying the presence of the broker.

Instead of moving on the performing the discovery, it is possible to return to the initial screen by clicking on the Back button or to quit the program altogether.

Finding Devices with HTTP Discovery toc

The range of IP addresses to be scan has to be defined in the second screen of the tamsotasbacker utility. There is a choice between

  1. Scanning the complete subnet (except for its first and last IP addresses) could take a long time. The subnet is defined in Classless Inter-Domain Routing (CIDR) notation where the number of bits is the number of leading 1 bits in the network mask. So 192.168.1.0/24 defines the 24 bit subnet mask 255.255.255.0 which means the range of IP addresses will be 192.168.1.0 to 192.168.1.255. Do not worry too much about the initial IP as long as it is that range. In other words 192.168.1.0/24 , 192.168.1.1/24 and 192.168.1.100/24 all define the same range of IP addresses.
  2. Scanning a range of IP addresses, which presumably would be the pool of dynamic IP addresses used by the DHCP server. If the range of dynamic range contains an appreciably smaller number of IP than the full subnet, it will reduce the discovery time markedly to select this option. If there happens to be a few Tasmota devices with static IP addresses, then these can be listed in the Include box. Do not forget to check the Include checkbox in that case.

A hint while hovering the mouse over each of the two radio buttons will dislay the maximum time the scan could take.

These times depend on the values entered in two other fields which are the maximum number of attempts made to connect to a presumed Tasmota device at an IP address and the maximum number of seconds to wait for a reply from the peripheral at that address if any.

To further reduce the time the scan takes, use the appropriately named Exclude box to list IP addresses known to be assigned to non Tasmota devices. The Exclude check box must be checked otherwise the list of IP addresses in the box is ignored and they will be scanned if they fall in the interval.

To edit the list of IP addresses in the the Exclude or Include boxes, double click on the box.

The remaining two options can be adjusted to determine the maximum time the utility could spend trying to determine if a Tasmota device is found at an IP address.

There is an obvious tradeoff: increase the time-out period and the number of connection attempts to ensure that all Tasmota devices are found, but accept that the scan will take longer. Some experimentation will have to be done to determine appropriate values. For those not using a binary release but instead compiling the utility, there is some help available in this regard as explained in section 4.1 HTTP Timing. Since the program will display the list of all found devices in the next screen, and since it will be possible to back out that screen to return to this screen if it is obvious that devices have not been found, it should not be too difficult to arrive at acceptable values.

Instead of clicking on the Next button that starts the process of getting the list of valid IP addresses, it is possible to press the Back button to return to the first screen to choose a different discovery method.

Warning:
While the HTTP scan is ongoing, the hourglass or system busy cursor will be shown. The user interface will not be very responsive during this process. It may look like it is possible to press the Quit button to interrupt the scan, but that will not work. The scan must complete before control is returned to the user.

The List of IP addresses of Found Devices toc

As Tasmota devices are found they are added to a list on the third screen of the utility, whichever version is used. Besides the device IP address, the device hostname and its MQTT topic, which by default would be something like tasmota_495A64, are displayed. On my Tasmota devices, the MQTT topic and the hostname are the same as shown next. Both of these are displayed because it will be possible to choose one or the other to form the name of the file in which the device configuration will be saved.

By default, all found devices are selected for backups. Clicking on the check box to the left of a device toggles its selection. Unchecking the All unselects all the devices, checking it once more selects all devices.

If no devices are listed or if there seems to be missing devices, it may be that some parameter was incorrectly specified in the previous screen. Activating the Back button will return to the previous screen where corrections can be made to try again.

Once the desired devices have been selected for backup, press on the Next button to define the backup options.

Setting the Backup Parameters toc

There are numerous options, but once they are defined there should be little reason to change them when making subsequent backups.

The first option is to specify in which directory the backups will be saved. The second option is the file extension appended to each file. By default it is .dmp which is the extension used by the Tasmota firmware, but as can be seen it has been modified to .config in this instance. The date shown will be the current date and should probably be left at that. Whatever date is chosen, it will added to the file name. To the right, the format of the date can be specified. There are 3 choices:

This last is recommended because alphabetic sorting of the saved files will give a rational order. There is a choice about how the file name of each backup is composed. The first choice starts with the Tasmota device name followed by the date while the order is reversed in the second choice. My preference is to start with the date. That makes it easy to identify and delete old backups but it does mean that it is easier to delete an older backup of a device absent from the more recent backups.

It is possible to choose the hostname or the MQTT device topic as the device name. Currently, I am using the hostname because mDNS is enabled in Tasmota and it is easy to reach the device web page using the hostname.local URL in Linux systems. However I think that mDNS support has been dropped since version 9.1.0 of Tasmota, so it may make more sense to use the device topic with more recent versions of the firmware if the device is connected to an MQTT broker.

The last two options are

The default values (2 tries and a timeout of 4 seconds) work well in my case. The Wi-Fi network is relatively fast and the backups are performed on a Linux Mint 20.1 system with a 4th generation i7 CPU. It's almost always the case that the download is accomplished in 200 to 300 ms on the first try. However, there are about 2 or 3 failures when the backup is done from a Windows 10 system with a 4th generation i5 CPU on the very same network. Increasing the timeout to 5 seconds does not entirely eliminate the problem. So on that Windows system I have set the values to 3 tries with a timeout out of 6 seconds. Obviously, multiple attempts to download the configuration with long delays before giving up will not slow down the download itself. By the time the utility gets around to requesting configuration downloads from Tasmota devices, it already knows that the IP addresses are valid and it has already managed to communicate directly or indirectly with every Tasmota device on the list. So to avoid a failed download, it seems reasonable to set a fairly high value for the timeout just in case the device is busy when the request is made or there is a lot of traffic on the network at the time. Having said that, each time the application repeats an HTTP request, it doubles the timeout delay, so it is not too clear which is preferable, a low timeout of 1 or 2 seconds with a higher attempt count or a long timeout with a reduce number of attempts. Again, some judicious experimentation would be a good strategy when first using the utility.

Once the options have been set, click on the Backup button to start the backup process.

Backup Results toc

As a copy of each Tasmota configuration is saved to a file, its name is added to the list in the backup result screen.

The saved file name will be shown when the download succeeded. Unselected devices will be listed as well and an error message for those devices for which it was not possible to get the configuration will be displayed. If one or more downloads failed, it may be useful to save the content of the result grid. Bring up the context menu, by clicking on the grid with the right button of the mouse. A pop-up menu will appear. Its first choice will copy the content of the grid to the clipboard as a delimited file. The delimiter between fields can be set to the comma or the tab character with the other two options in the pop-up menu. Clicking twice in rapid succession on the grid with the left button will copy the content of the grid as a tab delimited file without going through a menu.

In most cases where the download failed, retrying with a longer timeout will probably solve the problem. Once the process is finished the Options button will be enabled. Click on it to save any changes made to the options in the previous screens.

Saving or Resetting Options toc

The last optional step is to save or reset the options. All the utility options are shown along with their values and those values that are different from those already saved in the utility configuration file are identified by a check mark. The check marks can be set or reset at will. The values shown for the options cannot be changed. The choices are pressing on the

No matter which button is pressed, the program will be halted, so only one choice can be made.

The options are saved in a classic .ini text file where the options are saved as key=value pairs. The configuration file is named options.ini. Where the file is saved depends on the opertating system.


Programmer Notes toc

This section will be of interest only to those willing to compile the source code or that may be curious about some design decisions.

HTTP Timing toc

Because the HTTP requests sent by the utility are blocking, timeouts have to be specified otherwise the program could hang. As seen in the previous section, there are parameters that can be tweaked to set the number of HTTP connection attempts made and their timeout. To get a better understanding of the whole process, the utility can be compiled with the DEBUG_HTTP_REQUEST directive defined in the personalized project options. The utility will then log the times at which HTTP requests are made and replies are obtained. To see the time log in Linux, start the application from a terminal. In Windows, the log is saved to a file named tasmotasbacker.log in the same directory as the executable when the latter is exited. Here is an example of the time log during downloads.

time diff message 21:54:00.121083758 0 HttpRequest(url: http://192.168.1.116/dl, maxtries: 2, timeout: 1000) 21:54:00.121083864 106 request returns after 1 tries with code = 200, data = ??\M?^`abcelmg?Yc?lmooq????OYX???U?????????????????????????????? 21:54:00.121083867 0 HttpRequest(url: http://192.168.1.105/dl, maxtries: 2, timeout: 1000) 21:54:01.121084099 232 request returns after 1 tries with code = 200, data = ??\M?_`abceljg??c?lmooq????OYX???U?????????????????????????????? 21:54:01.121084104 0 HttpRequest(url: http://192.168.1.110/dl, maxtries: 2, timeout: 1000) 21:54:01.121084156 52 request returns after 1 tries with code = 200, data = ??\M?_`abcelkg0"c?lmooq????OYX???U?????????????????????????????? 21:54:01.121084160 0 HttpRequest(url: http://192.168.1.104/dl, maxtries: 2, timeout: 1000) 21:54:01.121084202 42 request returns after 1 tries with code = 200, data = ??\M~^`abcelng??c?lmooq????OYX???U?????????????????????????????? 21:54:01.121084206 0 HttpRequest(url: http://192.168.1.103/dl, maxtries: 2, timeout: 1000) 21:54:01.121084324 118 request returns after 1 tries with code = 200, data = ??\M?_`abcel|g;?c?lmooq????OYX???U??????????????????????????????

The fractional part of the seconds entry in the time stamp is the system tick count. The diff column shows the number of tick counts between two entries in the log which is nominally the number of elapsed milliseconds between them. Since this tick counter is reset to 0 for each new HTTP request, it is easy to see the time taken by the device to reply. As can be seen when that particular set of downloads was performed, all HTTP requests were answered in less than 0.3 seconds so the 1 second timeout had no effect, but such good result may not obtain in all cases.

Here is the time log for some of the HTTP requests made while scanning the local network for Tasmota devices.

time diff message 11:56:34.132856187 0 HttpRequest(url: http://192.168.1.100/cm?cmnd=hostname, maxtries: 2, timeout: 4000) 11:56:34.132856218 31 request returns after 1 tries with code = 200, data = <!DOCTYPE html>?<html>?<head>? <meta http-equiv="Content-Type" c 11:56:34.132856234 0 HttpRequest(url: http://192.168.1.101/cm?cmnd=hostname, maxtries: 2, timeout: 4000) 11:56:34.132856500 266 Exception class EHTTPClient, message Unexpected response status code: 404, code 404, try 1 11:56:34.132856500 0 request returns after 1 tries with code = 404, data = 11:56:34.132856515 0 HttpRequest(url: http://192.168.1.102/cm?cmnd=hostname, maxtries: 2, timeout: 4000) 11:56:35.132856765 250 request returns after 1 tries with code = 200, data = {"Hostname":"entree"} 11:56:35.132856765 0 HttpRequest(url: http://192.168.1.102/cm?cmnd=topic, maxtries: 2, timeout: 4000) 11:56:35.132856890 125 request returns after 1 tries with code = 200, data = {"Topic":"entree"} ... 11:56:35.132857531 0 HttpRequest(url: http://192.168.1.155/cm?cmnd=hostname, maxtries: 2, timeout: 4000) 11:56:39.132861546 4015 Exception class ESocketError, message Connection to 192.168.1.155:80 timed out., code 5, try 1 11:56:39.132861546 0 http request attemp 1 failed with code: 5 11:56:47.132869578 8032 Exception class ESocketError, message Connection to 192.168.1.155:80 timed out., code 5, try 2 11:56:47.132869578 0 http request attemp 2 failed with code: 5 ...

As can be seen when a request for the hostname is made using the syntax expected by Tasmota, there can be four types of responses (actually there can be more but that is not important here). The first case, a reply came from a Web server running on 192.168.1.100 which is my desktop machine. Not surprisingly, the reply was an HTML document saying that the requested page was not available. But since that was an HTML page, the result code was 200 which is, unusually, the no error value. Since the expected JSON formatted reply was not found, the IP address was deemed non-valid by the utility and it moved on to the next IP address. Connection to another web server at 192.168.1.101 was established. It was not running Tasmota and it replied with a straightforward 404 "Page not found" error. The utility then immediately moved on to the next IP address where it found a Tasmota device and it obtained both the host name. Consequently it queried the device again to obtain its MQTT topic with success. So the logic of the IP scan done by the utility is rather obvious. Any HTTP result code other than 200 is interpreted to mean that the IP address is not valid. If the result code is 200 then the data returned by the device is examined to find a "Hostname" key. If that is not found then the IP is not valid. If the key is found, then another HTTP request is made but this time the key is "Topic". As can be seen, these two queries usually do not take much time: 375 ms in the example above although given the timeout value of 4 seconds and the attempts value of 2 the time could be 2*(4 + 8) = 24 seconds (don't forget that on the second try the timeout is doubled). This worst-case scenario is unlikely, of course. Unfortunately, something akin to the worst-case scenario will occur when scanning an IP address that is not used by a peripheral, which is the case for 192.168.1.155 as shown above. Of course both tries timed out (code 5) as the utility waited for a reply that would never come. Thankfully, no attempt to make a second request for the Topic but there is nevertheless a 12 seconds delay.

So there is an obvious tradeoff: increase the time-out period and the number of tries to ensure that all Tasmota devices are found, but accept that the scan will take longer. There is an obvious gain to be had by restricting the scan to a subset of all possible IP addresses on the local network. The utility provides a little help. Hover the mouse over the radio buttons selecting either a scan of all IP addresses or a scan of a sub range of IP addresses. A hint will show the worst-case scenario. This result is not entirely accurate because as can be seen there is some overhead in addition to the timeout period and because only the number of addresses in the Exclude and Include boxes is used in the calculation without regard to whether they are effective or not.

Timeout Units toc

I learned more or less the hard way that seconds should be used to set the timeout values for HTTP requests. Initially timeouts were measured in milliseconds but the results were difficult to interpret. That was the reason for adding the DEBUG_HTTP_REQUEST directive discussed above. Once the times were logged, it became obvious that increasing a timeout from 1000 ms to 1500 ms had no discernable impact. Furthermore any value less than 1000 ms meant that the request would always time out. Here is the explanation for what is happening.

The timeout property of an HTTP client accepts a value measured in milliseconds. Let's use a 1500 ms (1.5 seconds) period:

TFPHTTPClient.ConnectTimeOut := 1500;

The value is then passed on to a socket stream class:

TSocketsStream.SetIOTimeout(1500);

Looking at the function, the situation becomes clearer.

procedure TSocketStream.SetIOTimeout(AValue: Integer); Var E : Boolean; {$ifdef windows} opt: DWord; {$endif windows} {$ifdef unix} time: ttimeval; {$endif unix} begin if FIOTimeout=AValue then Exit; FIOTimeout:=AValue; ... {$ifdef unix} time.tv_sec:=avalue div 1000; time.tv_usec:=(avalue mod 1000) * 1000; E:=fpsetsockopt(Handle, SOL_SOCKET, SO_RCVTIMEO, @time, sizeof(time))<>0; ...

We see that in unix the time record will be set as follows:

time.tv_sec = 1 time.tv_usec = 500000

A one second and half a million microsecond time value does correspond to 1.5 seconds. Apparently, the microseconds part of the time record is completely ignored in Linux. In other words, the effective timeout is AValue div 1000 seconds which means the millisecond value is rounded down to the biggest multiple of 1000 equal to or less than the AValue:

nominal timeouteffective timeout
millescondssecondsmillisecondsseconds
25002.520002
20002.020002
15001.510001
10001.010001
5000.500
0000

The last column explains why seconds were chosen as the units for timeouts in the utilities.

It must be pointed out that this is what happens in Linux and it may be that Windows does use millisecond timing. It did not really matter if the program is to perform in the same fashion no matter on which platform it is run.

Mosquitto Header File toc

Discovery of Tasmota devices by means of an HTTP scan was added to this version of the utility for two reasons.

In order to achieve these goals, I had to modify the Pascal header mosquitto.pas to make it possible to load the library at run-time. As written by Károly Balogh (chainq), the library is statically linked at compile time which explains why the original 0.3.6 version of tasmotasbacker would simply refuse to run on systems where the mosquitto library is not installed. The approach I used may not be the best one or may not even be correct, but it does seem to work. In any case, the modified header file will have to be reworked. At the beginning of what was a long process, I reused procedural types when functions had the same set of formal parameters even if the parameter names were different. Having a unique procedural type for each external function would be consistent and at the same time it will make it easier to adjust if ever there is a change in the library.

Discovery by Other Means toc

Sharp-eyed readers picked up from the first image that my Tasmota devices are using version 9.1.0 which is not the latest available. One of the reasons is that, as far as I know, support for mDNS is not enabled by default in newer versions of the firmware. In principle,

Multicast DNS (mDNS) provides the ability to perform DNS-like operations on the local link in the absence of any conventional Unicast DNS server. ...
In particular, the ability to look up DNS resource record data types (including, but not limited to, host names) in the absence of a conventional managed DNS server is useful.
Source: Multicast, DNS RFC 6762

Since mDNS advertising is enabled on all the Tasmota devices at home, it should be possible to find them.

michel@hp:~$ avahi-browse _http._tcp --parsable --resolve --terminate | grep '^=.*tasmota' =;enp4s0;IPv4;lampes-tele;Web Site;local;lampes-tele.local;192.168.1.102;80;"devicetype=tasmota" =;enp4s0;IPv4;torchere;Web Site;local;torchere.local;192.168.1.122;80;"devicetype=tasmota"

This may not work with older versions of the firmware is used, I am not sure when the "device=tasmota" string was added. It may be wise to change the grep matching pattern to '^=' but other peripherals will be listed. Unfortunately, conventional tools for resolving Zeroconf host names will not work.

michel@hp:~$ michel@hp:~$ nslookup torchere.local Server: 127.0.0.53 Address: 127.0.0.53#53 ** server can't find torchere.local: NXDOMAIN michel@hp:~$ adnshost torchere.local adns: /etc/resolv.conf:18: unknown option `trust-ad' torchere.local does not exist

The same is true of the THostResolver class in unit resolve which is part of fcl-net. I understand that the Indy package may be able to do this(see mDNS name resolution with Lazarus (hostname.local). Even if that were true, I have not found mDNS reliable enough, only a small portion of the IoT devices show up at any one time in avahi-browse for example, even when the --terminate flag is not on the command line.

One obvious solution to the whole problem of finding the IP address of the Tasmota devices it to assign them static IP addresses. Then it would be a simple matter of keeping the IP addresses in a text file and reading it when the program starts. Adding entries in the /etc/hosts file as shown here

127.0.0.1 localhost 127.0.1.1 hp 192.168.1.122 torchere 192.168.1.102 lampes-tele # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters

would add a degree of flexibility. The THostResolver class can obtain the IP address from the file. So now it would be possible to use a list of hosts to identify the Tasmota devices on the local network. While this certainly works in Linux and in all probability in Windows also, it's not a good solution. For one thing, it will be time consuming to set static IP addresses in each Tasmota device. For another, the hosts file will have to be adjusted on every machine connected to the local network on which tasmotasbacker might be run. The management of static addresses by juggling hosts files is by no means a simple thing on anything but the very simplest of networks. Indeed, there is a reason for DHCP!

I think the solution in this vein would be to run a DNS server such as dnsmasq on a local machine. I will not go into details, but it is something that I want to do once I finally reconfigure my home network. That has been on the to-do list for a long time.

Finally, I'll mention another possibility. Each Tasmota device has a unique MAC address and there is a lot of information on the Web that says that it is easy to obtain the IP address of a peripheral given its MAC address. The usual recommandation is "ping the broadcast address of the network from any Windows machine and then check out the ARP table afterwards." As such that did not work for me in Windows 10 nor in Linux. But here is what did work.

In Windows 10

  1. Create dopings.bat, a batch file to ping all IP addresses between 192.168.1.100 to 192.168.1.200 (adjust as required),
    @echo off FOR /L %%A in (100,1,200) DO ping -n 1 -w 250 192.168.1.%%A > NUL

    or, to get some feedback while the 101 pings are sent out,

    @echo off FOR /L %%A in (100,1,200) DO ping -n 1 -w 250 192.168.1.%%A > NUL & echo 192.168.1.%%A
  2. Clear the ARP table and then display it to confirm that it contains basically no entries.
    C:\Users\michel>arp -d * C:\Users\michel>arp -a Interface : 192.168.1.16 --- 0xc Adresse Internet Adresse physique Type 192.168.1.135 00-01-02-03-04-03f dynamique ...
  3. Run the batch file and display the ARP table
    C:\Users\michel>dopings C:\Users\michel>arp -a Interface : 192.168.1.16 --- 0xc Adresse Internet Adresse physique Type 192.168.1.1 00-01-02-03-04-05 dynamique ... 192.168.1.100 00-01-02-03-04-06 dynamique 192.168.1.101 00-01-02-03-04-04 dynamique 192.168.1.135 00-01-02-03-04-03 dynamique ... 192.168.1.255 ff-ff-ff-ff-ff-ff statique /div>

In Linux

  1. Create dopings.sh, a bash script file to ping all IP addresses between 192.168.1.100 to 192.168.1.200 (adjust as required),
    #!/bin/bash for i in {100..200} do ping -c 1 -W 1 192.168.1.$i >/dev/null 2>&1 done
  2. Clear the ARP table then display it.
    michel@hp:~$ sudo ip -s -s neigh flush all ... *** Round 1, deleting 9 entries *** *** Flush is complete after 1 round *** michel@hp:~$ arp -en Adresse TypeMap AdresseMat Indicateurs Iface 192.168.1.1 ether 00-01-02-03-04-04 C enp4s0 192.168.1.16 ether 00-01-02-03-04-0A C enp4s0
  3. Run the scrip and display the ARP table.
    michel@hp:~$ bash doping.sh michel@hp:~$ arp -en | grep ' C ' 192.168.1.16 ether 00-01-02-03-04-04 C enp4s0 192.168.1.1 ether 00-01-02-03-04-04 C enp4s0 192.168.1.118 ether 00-01-02-03-04-a1 C enp4s0 192.168.1.114 ether 00-01-02-03-04-b1 C enp4s0 ... 192.168.1.122 ether 00-01-02-03-04-c1 C enp4s0
  4. There's another possibility in Linux which might be simpler and faster if nmap is installed.
    michel@hp:~$ sudo nmap -sn 192.168.1.100-200 --open Starting Nmap 7.80 ( https://nmap.org ) at 2021-07-15 16:03 ADT Nmap scan report for 192.168.1.100 Host is up (0.15s latency). MAC Address: 00-01-02-03-04-06 (Espressif) Nmap scan report for 192.168.1.101 Host is up (0.14s latency). MAC Address: 00-01-02-03-04-04 (Espressif) ... Nmap scan report for 192.168.1.134 Host is up (0.23s latency). MAC Address: 00-01-02-03-04-b4 (Espressif) Nmap scan report for hp (192.168.1.135) Host is up. Nmap done: 101 IP addresses (31 hosts up) scanned in 3.80 seconds

All but one of these additional approaches require running one or two processes, capturing the output of one of them and then "scrapping" the output to compare MAC addresses with a user-provided list of MAC addresses of Tasmota devices. There is no doubt it can be done. However I think they represent brittle solutions. A local DNS server would represent a better solution, where the static mapping of MAC and IP addresses of Tasmota devices would be centralized in one place and any changes made would not have repercussions elsewhere in the system. Some may be able to do something almost as good by using the router static IP address table. Unfortunately, my ISP provided router has a static IP address table limited to about five entries so this approach has not been pursued.


Final Remarks toc

Right from the start, I wanted to use the MQTT broker to establish the list of Tasmota devices on the network because my MQTT client had already proven the feasibility of the approach. It really did not take much of an effort to add the ability to download the Tasmota configurations.

It seemed that it would be very simple to add an IP scan à la dragonflyuk and Yuval. Could not have been more wrong. tasmotasbackup0 took me much longer than the tasmotasbackup even though "only" one function of the source code and the first sheet of the wizard were being replaced. At first I could not find an IPv4 unit that filled my needs, so I had to write one using NLDIPTypes.pas by the Dutch Delphi Community as a starting point. That turned out to be a lot of fun, using advanced records and operator overloads for the first time. Then I had to add two more forms to edit the lists of included and excluded IP addresses. It was while working out the details of the IP scan that the timing problems discussed above became obvious, which was a good thing. All in all, a good learning experience even if somewhat tedious.

In retrospect, I should have investigated the management of the Tasmota configuration files more thoroughly. It would have made more sense to create a console application that performed the download procedure given a configuration file. That way a GUI application could have handled the options much like tasmotasbackup does but passed on the actual download task to the console application. At the same time, the console application could be launched by a scheduled task to automate the process which is really the best way to handle backups. Another possibility is to add a command line switch to the utilities so that it performs a backup with the current options instead of starting the interactive user interface. This is something to look into once I tire of manually backing up the Tasmota configurations. To be fair, that is something I do very rarely, as I do not update the device firmware with every new version of Tasmota, so it may be some time if ever before I get around to this idea.

<-DIY Tasmota Backups (version 0.3)