There is a well-regarded backup solution for Tasmota devices: TasmoBackupV1 by danmed. I wanted something more nimble that did not require PHP and Sqlite (or equivalent) to centralize all the saved configurations in a database. A simple directory on a drive with clearly dated backup files would suffice. My do-it-yourself solution can be found on Github: tasmotasbacker and tasmotasbacker0. These are relatively light utilities with a graphic user interface written in Free Pascal / Lazarus which hopefully others will also find useful.
tasmotasbackerwhich is a combination of version 0.3.6 of the utility and
tasmotasbacker0is available. The older versions described here remain available should the combined functionality not be needed, although as far as I know, there is no significant cost to using the newest version. The post entitled DIY Tasmota Backups now describes the newest version.
Table of Content
- Getting the Tasmota Configuration
- Finding Tasmota Devices
- The Backup Utility
- HTTP Timeouts
- Final Remarks
It is a simple matter to download the Tasmota configuration of a device if its IP address is known. By default Tasmota runs a web server which can be used to control and manage the device. Just start a Web browser and enter the IP address in the address bar. The image on the left below is the main web page of a simple on/off device, a Sonoff module.
As can be seen it is mostly a menu. Click onto bring up the next menu which is the image shown on the right above. Click on the button. This will bring up a download window asking if the file is to be downloaded or not.
Click on Save File to complete the download into the directory of your choice. The just below the button can then be used to copy the saved configuration file back onto the device if that should be necessary. This can be quite useful when playing with the many configuration options available in Tasmota. I also find it practical to restore the configuration of a temperamental Tuya dimmer which for some reason loses its configuration when there is some glitch or interruption in the mains power. Strangely, two other identical dimmers never seem to suffer the same fate.
While downloading a single configuration is a simple enough operation, it becomes tedious really quickly when one wants to back up the configuration of multiple devices. I have a modest home automation system but it nevertheless has some 20 Tasmota devices and I have never had the patience to go through all of them at once to make backups even if I know that would be a good idea.
Luckily Theo Arendts, the original creator of Tasmota, and his collaborators have provided a mechanism that can be used to automate the process. A simple HTTP query
will bring up the last dialog shown to save the configuration file without manually clicking on buttons in a GUI. Note that many sources show a questions mark at the end of the path (
http://<ip_address>/dl?). However a question mark is used at the start of an optional query, and since there is no query I did not include it. Nevertheless
/dl? both work.
As expected a utility such as
curl can get the device configuration.
Or to get a "quieter" output
Note how the output file name is preceded by an upper case O in the
wget command and a lower case o in the
curl and these options are case sensitive.
Given a list of valid IP addresses, I suppose it would be fairly simple to cobble a Bash script that would download all desired device configurations.
The problem is getting that list of valid IP addresses of the Tasmota devices on the local network if the devices obtain dynamic IP addresses from a DHCP server. One way would be to poke, as it were, all "likely" IP addresses and if a valid response is obtained to save the configuration. That is, more or less, what dragonflyuk and Yuval do in their Python scripts: tas-backup and Tasmota-Config-Backup. I use a different approach, but that is not to say their approach is not valid. Indeed, the
tasmotasbacker0 version of the utility locates Tasmota devices by scanning all devices in a subnet or a limited range within that subnet.
My preferred approach is to connect to a MQTT broker, subscribe to the
stat/+/STATUS5 topic and to then publish an MQTT message to the topic
cmnd/tasmotas/status with payload
5. All Tasmota devices connected to the broker will respond with the requested status 5 (
StatuNET) message which includes the device hostname and IP address. The box below shows the reply from one device.
The topic to which the devices publish will contain the device (MQTT) Topic which by default would be something like
tasmota_495A64 but which on my devices is the same as the hostname. The hostname itself is the first field in the payload.
Obviously, this approach will work only if
- an MQTT broker is available,
- the Tasmota devices are connected to the MQTT broker,
- the MQTT Group topic of all devices has not been changed to something now forgotten.
On that last point, I believe that initially there was a single group topic and its default value was
sonoffs. That changed to
tasmotas in version 7.1.0. To address the possibility that there may be older versions of the firmware on a device or that the group topic has been changed, the
tasmotasbacker utility will accept multiple group topics. Using both
sonoffs should mean that the published message will be heard by most Tasmota connected to the broker.
Since version 188.8.131.52 of Tasmota, up to four group topics can be defined in the firmware itself (see Add multiple GroupTopic
xxxxx are the specified group topics. and will reply to published messages with payload
5. So as long as at least one topic specified in the utility is among one of the group topics in Tasmota the device should reply and be added to the download list.
Currently, there are two versions of the utility:
tasmotasbackupwhich finds Tasmota devices through an MQTT broker. This version uses the
mosquittolibrary which may be difficult to install on some systems. The advantage of this version is the speed at which Tasmota devices are found. On the other hand, devices that are not configured to use an MQTT broker or that have a broken MQTT configuration will not be found.
tasmotasbackup0which finds Tasmota devices by scanning all IP addresses. This version which only uses HTTP requests to find Tasmota devices does not require any library or third party server. The scan to find devices can be considerable longer than using an MQTT broker. However it may find Tasmota devices that are not configured to use an MQTT broker or that have a broken MQTT configuration.
Both versions adhere more or less to the wizard paradigm. Saving a set of Tasmota configuration files is done by progressing through a series of screens or windows, modifying the parameters as needed. In the last screen of the utility, there is the possibility to save all modified parameters. Hopefully, future downloads will be mostly a matter of clicking on a few buttons.
Only the first and last screen differ between the versions, so the following describes how to use both utilities at once.
The parameters of the MQTT broker needed to obtain the IP addresses of all Tasmota devices connected to the broker must be defined in the first screen show here.
Of course, the IP address of the broker as well as the TCP port it uses have to be set correctly. The default MQTT port is 1883 as shown. The user and password fields are empty by default, but if the broker requires a user name and password then they must be specified. Clicking on the eye icon to the right of the MQTT broker password can display or hide it as desired. The MQTT topics used to send status queries must be specified in the
Topicsfield. If there is more than one topic, they must be separated by commas; spaces are ignored so topics with embedded spaces cannot be used. By default, two topics are defined:
tasmotas which is the newer default MQTT topic defined in Tasmota and
sonoffs which was the default topic in older version of the firmware.
Once all the fields are correct, click on the Next button to start the process of getting the list of IP addresses. Before an attempt is made to connect to the broker, a TCP connection to the
host:port end point is opened. If this connection is established, then the utility goes on to connect to the broker, subscribe to topics, publish status query messages and so on. If the connection cannot be established then an error message is displayed and the process of finding devices is not started. This extra step has been added in the latest version of the utility because it was found that the program would take a very long time to recover if an incorrect host Ip or TCP port was specified. The
Timeout field is used to specify the maximum number of seconds the utility will wait for a reply when establishing a TCP connection verifying the presence of the broker.
The range of IP addresses to be scan has to be defined in the first screen of the
tamsotasbacker0 utility. If the range of dynamic addresses used by the DHCP server is limited, then it is best to limit the scan to that range. Otherwise, a scan of the complete subnet (except for its first and last IP addresses) could take a long time.
It is possible to exclude some IP addresses known not to be assigned to Tasmota devices in the appropriately named
Exclude box. The
Exclude check box must be checked otherwise the list of IP addresses in the box is ignored and they will be scanned if they fall in the interval. If a subrange of the subnet will be scanned it is possible to scan IP addresses outside the range by adding them into the
Include box. To edit the content of either of these boxes, double click on them.
There are two other fields to fill out. These are the maximum number of attempts made to connect to a presumed Tasmota device at an IP address and the maximum number of seconds to wait for a reply from the device.
Once all the fields are correct, click on the Next button to start the process of getting the list of valid IP addresses.
As Tasmota devices are found they are added to a list on the second screen of the utility, whichever version is used. Besides the device IP address, the device hostname and its MQTT topic, which by default would be something like
tasmota_495A64, are displayed. On my Tasmota devices, the MQTT topic and the hostname are the same as shown next. Both of these are displayed because it will be possible to choose one or the other to form the name of the file in which the device configuration will be saved.
By default, all found devices are selected for backups. Clicking on the check box to the left of a device toggles its selection. Unchecking the All unselects all the devices, checking it once more selects all devices.
If no devices are listed or if there seems to be missing devices, it may be that some parameter was incorrectly specified in the previous screen. Activating the Back button will return to the first screen where corrections can be made to try again.
Once the desired devices have been selected for backup, press on the Next button to define the backup options.
There are numerous options, but once they are defined there should be little reason to change them when making subsequent backups.
The first option is to specify in which directory the backups will be saved. The second option is the file extension appended to each file. By default it is
.dmp which is the extension used by the Tasmota firmware, but as can be seen it has been modified to
.config in this instance. The date which is added to the file can be set to a value different from the current date. This is probably something that should not be done. To the right, the format of the date can be specified. There are 3 choices:
- MDY month, day, year: 10-30-2021
- DMY day, month, year: 30-10-2021
- YMD year, month, day: 2020-10-30
This last is recommended because alphabetic sorting of the saved files will give a rational order. There is a choice about how the file name of each backup is composed. The first choice starts with the Tasmota device name followed by the date while the order is reversed in the second choice. My preference is to start with the date. That makes it easy to identify and delete old backups but it does mean that it is easier to delete a backup should there be missing devices in recent backups.
It is possible to choose the hostname or the MQTT device topic as the device name. Currently, I am using the hostname because mDNS is enabled in Tasmota and it is easy to reach the device web page using the
hostname.local URL in Linux systems. However I think that mDNS support has been dropped since version 9.1.0 of Tasmota, so it may make more sense to use the device topic with more recent versions of the firmware.to
The last two options are
Download attempts: the number of times the utility will try to download the configuration from a device
Download timeout: the time (in milliseconds) the utility waits for a reply from the device when it sends an HTTP request to download the configuration.
The default values (2 tries and a timeout of 4 seconds) work well in my case. The Wi-Fi network is relatively fast and the backups are performed on a Linux Mint 20.1 system with a 4th generation i7 CPU. It's almost always the case that the download is accomplished in 200 to 300 ms on the first try. However, there are about 2 or 3 failures when the backup is done from a Windows 10 system with a 4th generation i5 CPU on the very same network. Increasing the timeout to 5 seconds does not entirely eliminate the problem. So on that Windows system I have set the values to 3 tries with a timeout out of 6 seconds.
Once the options have been set, click on the Backup button to start the backup process.
As a copy of each Tasmota configuration is saved to a file, its name is added to the list in the backup result screen.
The saved file name will be shown when the download succeeded. Unselected devices will be listed as well and an error message for those devices for which it was not possible to get the configuration will be displayed. If one or more downloads failed, it may be useful to save the content of the result grid. Bring up the context menu, by clicking on the grid with the right button of the mouse. A pop-up menu will appear. Its first choice will copy the content of the grid to the clipboard as a delimited file. The delimiter between fields can be set to the comma or the tab character with the other two options in the pop-up menu. Clicking twice in rapid succession on the grid with the left button will copy the content of the grid as a tab delimited file without going through a menu.
In most cases where the download failed, retrying with a longer timeout will probably solve the problem. Once the process is finished the Options button will be enabled. Click on it to save any changes made to the options in the previous screens.
The last optional step is to save or reset the options. All the utility options are shown along with their values. Of course, the options are different in the two versions of the utility
Those values that are different from those already saved in the utility configuration file are identified by a check mark. The check marks can be set or unset at will. The values shown for the options cannot be changed. The choices are pressing on the
- Save selected button: this will save the value of all checked options and leaving the others unchanged,
- Reset selected button: this will reset all checked options to the program default values,
- Quit button: no change will be made to the utility configuration file.
No matter which button is pressed, the program will be halted, so only one choice can be made.
The options are saved in a classic
.ini text file where the options are saved as key=value pairs. The configuration file is named
options.ini. In Linux, it is save in the
~/.config/sigmdel/tasmotasbacker/ directory (or
~/.config/sigmdel/tasmotasbacker0/ directory for the
tasmotasbacker0 version). Of course
~ is the user home directory. So the fully expanded file name is
/home/<user>/.config/sigmdel/tasmotasbacker/options.ini. In Windows 10, the file is saved in the local
Because the HTTP requests sent by the utility are blocking, timeouts have to be specified otherwise the program could hang. There are two options that can be tweaked in the
Backup Parameters screen.
Download attempts- the maximum number of times an HTTP request for the Tasmota configuration is made.
Download timeout- the maximum time to wait for the reply to the first request. This period is doubled at each new attempt. Specified in seconds, the minimum is 1 second.
The default values (2 attempts at 4 seconds timeout) usually work well on my Linux Mint 20.1 system with a 4th generation i7 CPU. As can be seen when looking at the time log generated by the application when
DEBUG_HTTP_REQUEST directive is defined in
main0.pas in the case of
tasmotizer.0) and the
Download timeout had been reduced as much as possible to 1 second (1000 ms).
The fractional part of the seconds entry in the time stamp is the system tick count. The
diff column shows the number of tick counts between two entries in the log which is nominally the number of elapsed milliseconds between them. Since this tick counter is reset to 0 for each new HTTP request it is easy to see the time taken by the device to reply. As can be seen when that particular set of downloads was performed, all HTTP requests were answered in less than 0.3 seconds so the 1 second timeout had no effect. In all probability some experimentation will be necessary to set reasonable values and I found that the timeout had to be increased to 5 or even 6 seconds on a system with a 4th generation i5 running Windows 10 connected to the same local area network.
By the time the utility gets around to requesting configuration downloads from Tasmota devices, it already knows that the IP address is valid. So it would seem reasonable to set a fairly high value for the timeout just in case the device is busy when the request is made or there is a lot of traffic on the network at the time. Each time the application repeats an HTTP request, it doubles the timeout delay. So I am not too sure which is preferable, a low timeout of 1 or 2 seconds with a higher attempt count or a long timeout with a reduce number of attempts. Again, recompiling the application to enable the time log might be of use to answer that question.
I have found that the HTTP scan for valid IP addresses done by
tasmotasbacker0 present a more complicated problem. Here is the time log for a part of that set of requests obtained by defining the
DEBUG_HTTP_REQUEST directive as before.
Do not forget that the number of attempts and the timeout values when performing the scan are set with two different parameters
Connect attempts- the maximum number of times an HTTP request for the Tasmota hostname and MQTT topic is made.
Connect timeout- the maximum time to wait for the reply to the first request. This period is then doubled at each new attempt. Specified in seconds, the minimum is 1 second.
As can be seen when a request for the hostname is made using the syntax expected by Tasmota, there can be four types of responses (actually there can be more but that is not important here). The first case, a reply came from a Web server running on
192.168.1.100 which is my desktop machine. Not surprisingly, the reply was an HTML document saying that the requested page was not available. But since that was an HTML page, the result code was 200 which is, unusually, the no error value. Since the expected JSON formatted reply was not found, the IP address was deemed non-valid by the utility and it moved on to the next IP address. Connection to another web server at
192.168.0.101 was established. It was not running Tasmota and it replied with a straightforward 404 "Page not found" error. The utility then immediately moved on to the next IP address where it found a Tasmota device and it obtained both the host name. Consequently it queried the device again to obtain its MQTT topic with success. So the logic of the IP scan done in
tasmotasbacker0 is rather obvious. Any HTTP result code other than 200 is interpreted to mean that the IP address is not valid. If the result code is 200 then the data returned by the device is examined to find a
"Hostname" key. If that is not found then the IP is not valid. If the key is found, then another HTTP request is made but this time the key is
"Topic". As can be seen, these two queries usually do not take much time: 375 ms in the example above although given the timeout value of 4 seconds and the attempts value of 2 the time could be 2*(4 + 8) = 24 seconds (don't forget that on the second try the timeout is doubled). This worst-case scenario is unlikely, of course. Unfortunately, something akin to the worst-case scenario will occur when scanning an IP address that is not used by an peripheral, which is the case for
192.168.1.155 as shown above. Of course both tries timed out (code 5) as the utility waited for a reply that would never come. Thankfully, no attempt to make a second request for the
Topic but there is nevertheless a 12 seconds delay.
So there is an obvious tradeoff: increase the time-out period and the number of tries to ensure that all Tasmota devices are found, but accept that the scan will take longer. There is an obvious gain to be had by restricting the scan to a subset of all possible IP addresses on the local network. The utility provides a little help. Hover the mouse over the radio buttons selecting either a scan of all IP addresses or a scan of a sub range of IP addresses. A hint will show the worst-case scenario. This result is not entirely accurate because as can be seen there is some overhead in addition to the time-out period and because only the number of addresses in the Exclude and Include boxes is used in the calculation without regard to whether they are effective or not.
This subsection will be of interest only to those that may want to modify the code. I learned more or less the hard way that seconds should be used to set the time out values for HTTP requests. Initially the utilities used milliseconds to specify the timeout but the results were difficult to interpret. That was the reason for adding the
DEBUG_HTTP_REQUEST directive discussed above. Once the times were logged, it became obvious that increasing a timeout from 1000 ms to 1500 ms had no discernable impact. Furthermore any value less than 1000 ms meant that the request would always timeout. Here is the explanation for what is happening.
The time out property of an HTTP client accepts a value measured in milliseconds. Let's use a 1500 ms (1.5 seconds) period:
The value is then passed on to a socket stream class:
Looking at the function, the situation becomes clearer.
We see that in
time record will be set as follows:
A one second and half a million microsecond
time value does correspond to 1.5 seconds. Apparently, the microseconds part of the
time record is completely ignored in Linux. In other words, the effective timeout is
AValue div 1000 seconds which means the millisecond value is rounded down to the biggest multiple of 1000 equal to or less than the
|nominal timeout||effective timeout|
The last column explains why seconds were chosen as the units for timeouts in the utilities.
It must be pointed out that this is what happens in Linux and it may be that Windows does use millisecond timing. It did not really matter if the program is to perform in the same fashion no matter on which platform it is run.
Right from the start, I wanted to use the MQTT broker to establish the list of Tasmota devices on the network because my MQTT client had already proven the feasibility of the approach. It really did not take much of an effort to add the ability to download the Tasmota configurations.
In retrospect, I should have investigated the management of the Tasmota configuration files more thoroughly. It would have made more sense to create a console application that performed the download procedure given a configuration file. That way a GUI application could have handled the options much like
tasmotasbackup does but passed on the actual download task to the console application. At the same time, the console application could be launched by a scheduled task to automate the process which is really the best way to handle backups. Another possibility is to add a command line switch to the utilities so that it performs a backup with the current options instead of starting the interactive user interface. This is something to look into once I tire of manually backing up the Tasmota configurations. To be fair, that is something I do very rarely, as I do not update the device firmware with every new version of Tasmota, so it may take quite some time before I get around to this idea.
When writing the first version of this post, I thought that it would be very simple to add an IP scan à la dragonflyuk and Yuval. Could not have been more wrong.
tamotasbackup0 took me much longer than the
tasmotasback even though "only" one function of the source code and the first sheet of the wizard were being replaced. At first I could not find an IPv4 unit that filled my needs, so I had to write one using NLDIPTypes.pas by the Dutch Delphi Community as a starting point. That turned out to be a lot of fun, using advanced records and operator overloads for the first time. Then I had to add two more forms to edit the lists of included and excluded IP addresses. It was while working out the details of the IP scan that the timing problems discussed above became obvious, which was a good thing. All in all, a good learning experience even if somewhat tedious.
The obvious next step is to combine the two utilities into one. There are two reasons one would want to do this.
- When the faster MQTT based search for Tasmota devices seems to miss devices then a longer scan of all IP addresses can be undertaken.
- On systems where the mosquitto library is not installed, then the slower HTTP scan of IP addresses can be used.
There's already a prototype of this super utility working on my Linux machine but it will not do. The problem with this approach is the static linking of the
mosquitto library. Because of that the super utility will not run if the
mosquitto library is not installed, which is a failure with regard to the second argument. There is a solution for this problem, which is dynamic linking the
mosquitto library. Unfortunately, this will require quite a bit of boring rewritting of all the function declarations in
mosquitto.pas. A way to avoid this would be to catch the exception raised by the absence of the static library. Unfortunately, I don't even know if such an exception exists and, secondly, the program is aborted when the library cannot be found very early, even before the initialization code of the
mosquitto.pas unit is executed. Does anyone know of a way to catch the absence of a static library during that start of a Free Pascal program? Any help would be greatly appreciated.