It's happened before. For some known or unknown reason, the Thunderbird calendar on my computers and the Google Agenda calendar on my Android devices can no longer read the calendar hosted by the CalDAV server running of a remote web hosting server. This time there was a warning, my account was being moved to a new physical server. After a couple of days, the email accounts worked themselves out and they have been working fine. But more than a week after the move, it remains impossible to connect the local calendars to the remote server. To be honest, this has always been a bit iffy, with incompatible instructions about how to connect depending on where I obtained them such as the web email account or the C-Panel email admin page, and so on. Nevertheless it has always been possible to get two-way synchronization working, but given the time it's taking to resolve the situation, I decided to revisit CalDAV self hosting using Radicale.
Table of Content
- Executive Summary
- Install Radicale in a Python Virtual Environment
- Adding Basic User Authentication
- Setting up Radicale as a Service
- Uploading Calendars and Tasks
- Accessing Calendars and Tasks from Thunderbird
- Accessing Calendars and Tasks in Android
- Backups
- Does this Work?
Executive Summary
A few years ago, I had tried installing the radicale apt package on our (not so) hard-working home automation server, but got nowhere. I tried again and, as before, could not get this to work. The instructions for the Debian package are rather old (2021) and there are a lot of moving parts in that package. It seemed to need more knowledge than I can muster. Consequently, I tried the Simple 5-minute setup from the Radicale Getting started guide and that worked. That was encouraging and looking at the rest of the document I figured that something useful could be working relatively quickly. Here are the steps that I followed
- Create a Python 3 virtual environment - this is not covered in the Guide but it's my preferred way of installing Python scripts.
- Install Radicale in the virtual environment - just a
pip installcommand. - Create a user password file - using
md5encryption which means thatapache-utilswill not be needed. - Enable user authentication - this is just a configuration step.
- Create a systemd service file and enable it - just like in the guide except we are not running a Pyton module.
- Upload the existing calendar - it was easy.
- Connect clients to the server - again very easy... "once you know how"™.
Install Radicale in a Python Virtual Environment
The host is a converted Android TV box based on the Amlogic S912 8-core ARM processor with 2 Gbytes of memory. It is running a variant of Debian/Ubuntu: Armbian 23.11.0-trunk Lunar with Linux 6.1.60-ophub. While that may seem to be underpowered to run the numerous servers already functioning (Domoticz, Ha-Bridge, Mosquitto, Zigbee2MQTT, WireGuard, SSH, Syslog, Nginx and so on), most of the time the system load is less than 1 or 2%. There is no doubt that it can handle yet one more service.
I prefer to install any major Python script into its own virtual Python environment. This avoids any conflicts between library versions. It may be necessary to install python3-venv if it's not already available.
There's no harm running that command even when venv is present on the system.
mkvenv script here are instructions on how to install it. Look at the following section in that reference post for the definitions of the ve and ev bash aliases to activate and deactivate an environment. If you don't want to install the script and aliases, here is what mkvenv radicale and ve radicale do.
So now we can proceed with the installation of the Radicale server into the radicale virtual environment
We can now check the version number.
Let's proceed with the creation of a data directory in which Radicale will save its collections of calendars, tasks and adress books.
It will not harm anything, but it may not have been necessary to do that. As far as I know, the server will create the directory if it does exist.
To test the installation we will create a minimal configuration file specifying where calendars and addressbooks will be saved and so that it is possible to log into the Radicale server from any machine on the local network. That's needed in my case, because the version converted TV box is a headless Linux server without a graphical interface so that it is not possible to open a browser on that machine. I have to do that from the desktop computer.
Copy the following into the config file, substituting the correct user name instead of alex.
Starting the server could not be easier.
The INFO messages state that the radical.web.internal server is ready and that there is no authentication, meaning any user with any password can log in.
As the screen capture shows, the server is running. However this is not quite acceptable. Even if Radicale will be running behind a firewall, user authentication must be enabled before anyone can sign in to the server and click that Next button to create agendas and address books and so on.
Adding Basic User Authentication
Let's add a relatively simple authentication mechanism: only known users with the correct password will be allowed to log into the server. We will do this by creating the following user account.
user: jean
password: micrad5232 The password has to be encrypted, which can easily be done with the on-line utility Htpasswd Generator. It encrypted the password with the md5 cypher and displayed the entry jean:$apr1$0adiuv5z$DxZe4XqirMEP/Y48hiId./ to include in the user password file. Thanks to the ArchWiki Radicale page for this bit of information as I have no intention of installing the Apache HTTP server as Nginx is already running on the server.
A file named users can be created in the ~/.config/radicale/ directory containing that single username:password line.
It does not matter if alex or root owns the file. The configuration file, ~/.config/radicale/config, has to be adusted to specify the location of the user file and the encryption used.
If the radicale daemon is still running, stop it by pressing the CTRL+C key combination.
Start radicale again, so that the new configuration is used. Note that it is not mandatory to activate the Python virtual environment and the following command works just as well.
This time we see that htpasswd md5 encryption is used. There's a good chance that the browser will ask for authentication.
Enter the user name and password used to create the user file. Do the same again when asked for a user and password by radicale itself. This is the only user name and password combination that will work.
At this point we could create a new calendar or upload one.
Before looking at how to use radicale, let's complete the installation.
Setting up Radicale as a Service
Obviously, radicale should start automatically whenever the host computer boots. Since it is running on a variant of Debian, which uses systemd to manage daemons/services, it will be necessary to create a service unit file and to enable it. Thankfully, the Getting started guide has all the information we need. Be careful and do not follow the instructions in Linux with systemd as a user because those instructions create a service that only runs when user alex had a session open. I got bitten by that in an early version of this post. Once I realized what the problem was, I proceeded as quickly as possible to fix things as will be shown next.
The first steps is creating a new service file.
Here is the content of the service file.
By adding the User=alex line, Radicale will find the configuration file /home/alex/.config/radicale/config. Alternatively, -C /home/alex/.config/radicale/config could be appended to the ExecStart command.
For reasons I do not know, the guide also suggests moving the collections directory. Here I followed the instructions for a user configuration instead of a system-wide installation, but it does not matter much at this point. Consequently, I removed the current collections directory which did not contain anything of value and set a new path filesystem_folder in the configuration file. There's no need to create that new directory, radicale will take care of that.
So this should be the final version of the configurartion file.
So now it's just a matter of enabling the service and starting it.
We can check that the service is running as expected.
At this point, our CalDAV and CardDAV server is running and it will automatically start up whenever its host computer reboots. This is a bit of a mess where the instructions from the Radicale team were not followed very closely, but it does work. Nevertheless, it will have to be revisited later.
From now on, it’s a matter of using it. From my perspective there are two aspects to consider. First, I want to use the calendar that was in effect previously. Secondly, the clients must be connected to this server.
Uploading Calendars and Tasks
Once a Radicale user account is created, an empty collection (calendar, calendar and tasks and so on) can be created by clicking on the green (+) button.
More than likely, the user wants to upload a list of contacts (address book), an agenda (calendar) and a todo list (tasks) that already exist. In my case, there were three files with the information to transfer to the new server.
The addressbook01.vcf and tasks-2024-06-15.ics had been exported from Thunderbird. The calendar, which was no longer available in Thunderbird because of my futile attempts to reconnect to the Web hosted accounts, was exported from the Web interface of the e-mail account on the web hosting site.
I wanted to have the calendar and tasks together in one file so they were combined as will be explained below. It's not obvious that this was a good choice. It worked with the Thunderbird client which was most important to me, but the tasks are not visible in the Android client.
Looking at the end of the tasks file, it became obvious that each task started with a BEGIN:VTODO and ended with a END:VTODO marker.
So I copied the content of the file from the first BEGIN:VTODO to the last END:VTODO into the much larger calendar file. The question was where to insert that information. Again, examining the end of the calendar file showed that it was mostly composed of events tagged with BEGIN:VEVENT and END:VEVENT.
Consequently the copied todo records were pasted right after the last END:VEVENT and before END:VCALENDAR in the calendar file. The result was saved as calendar-task-2024-06-15.ics. That was the file to upload.
To proceed with that operation I clicked on the upload icon and got the following web page.
I clicked on the browse button (Parcourir... in French) and found the calendar file with the system file manager and selected it.
The actual upload is started by clicking on the Upload button. Depending on the size of the .ics file, it may take a fair amount of time to complete the operation. During the process, the following is displayed.
Click on the Close button once the collection has been uploaded.
We can see its URL:
If that URL is used in a Web browser, the server will return an `Calender.ics` file for downloading. Clients can thus obtain the current version of the calendar via that URL. However, as it turned out, that explicit URL was not needed.
Accessing Calendars and Tasks from Thunderbird
The Radicale documentation about its use in Thunderbird is a bit terse. Here is a series of pretty pictures showing how it can be done. I should mention that my desktop computer and my portable computer are both running version 115.12.0 (64-bit) of Thunderbird on Linux Mint. To add a calendar in the mail client, click on the Calendar tab if its visible, or click on the Calendar icon on the left icon bar.
Then click on the New Calendar... button at the bottom of the Calendars area. This brings up a Create New Calendar window.
Select the On the Network option and then click on the Next button.
Enter the user name for the account created in Radicale and its URL which, of course, will be different from what is shown above. Then click on the button.
Enter the password when asked for it.
jean beforehand which was micrad5232. Once the button is pressed, Thunderbird will display the calendars it found.
Click on the button.
As can be seen, Thunderbird displays the calendar. Of course, what is seen will depend on the selected view (Day, Week, etc.) and on the events stored in the calendar. Not shown here, but the tasks can be viewed by clicking on the task icon in the left bar. It's the clipboard with a check mark just below the calendar icon that is currently highlighted. Some properties of the calendar can be modified. To view and edit them, place the mouse cursor over the calendar entry and click with the right mouse button to see the context menu.
Select Properties in the menu (note that the screen capture was taken after I had already edited some properties including the calendar name).
Almost everything can be edited, except for the Username and Location fields. If those are wrong, the only way to change them is to unsubscribe to the calendar (context menu again) and then repeat the steps to create a new calendar.
The real test was to install the calendar in exactly the same way in Thunderbird running on the portable computer. Then I added a new event called test in the Thunderbird calendar on the portable and waited until the even became visible in the first instance of Thunderbird on the desktop machine.
If waiting up to half an hour to see the event appear is unacceptable, the calendar contextual menu has a Synchronize option which forces the Thunderbird client to reread the content of the collection on the Radicale server. It is possible to modify the time Thunderbird waits before rereading the calendar from the server in the calendar properties editor.
Accessing Calendars and Tasks in Android
The Radicale documentation lists DAVx⁵ it as the first supported client and as far as I can tell it is the only Android client mentioned. For that reason and given that DAVx⁵ does have a good rating on Google Play and has been downloaded many times, I should test it out. However, synchronizing the calendar on my telephone and tablet, both running Android 8, has done with a small application called CalDAV Sync Adapter for a number of years since it was obtained from the Google Play store. So far it has worked with Radicale so, for the time being at least, I will continue to use it. That said, I am loath to recommend doing the same. It is no longer available in the Google Play store, and its author, Gérald Garcia (gggard), has not updated the code since version 1.8.1 was released in 2013 (see the End of life issue).
There is a fork of CalDAV Sync Adapter by ennswi who renamed the project AndroidCalcDavSyncAdapter or ACalDAV for short. He moved the repository to GitLab about 9 years ago and came out with a new version (0.1.1) in 2016. The .apk is available from f-droid.org. It may be a better choice than the older adapter, but I have not tried it.
Backups
We have an HA home automation system. Redundancy is assured by two similar (but not identical) single-board computers running the same software stack. The crossover when a system fails is not quite perfect yet. Someone, usually me, has to detect that the home automation system is down and then someone, usually me again, has to plug in the other system in its place. The crossover will work more or less seamless only if the replacement system has all the same settings and data that was used and created by the now broken system. Ideally, all that information would be backed up to a network storage on a regular basis (or even better asynchronously whenever it is changed) so that it could be easily downloaded to the replacement system. Until that is in place, there's an ugly bash script that takes care of backing up the important data and settings onto a USB stick that is connected to the single-board computer.
The Radicale settings and data need to be backed up before installing it on the replacement system. Rereading my notes above, it is clear that for settings, there are only three files that matter.
It was a simple matter to add these three files to the backup script mentioned above. There remains the question of backing up the calendars and address books and so on that are managed by Radicale. Here is a bash script cobbled together based on the information in the Radicale document on storage locking.
The exclusions means that the .Radicale.props will be saved with the data. That's needed by the Radicale web server to display the Address book and Calendar, journal and tasks collections. When this script is launched it will display something similar to the following messages.
Just to belabour the point, /home/alex/domoticz/backups/ is the mount point for the USB stick. Presumably, it would be more dependable than the SD card which is the main storage medium of one of the two systems. As long as I remember to launch this script often enough (cron task?) this will work. Another possibility is to add it as a hook in the Radicale config.
This should not be done for long, because the entire collection is being stored whenever a single event, task or contact is changed. It would make more sense to track all changes to the data with a version control system such as git as suggested by the Radicale guide. Care is required because if implemented as suggested in the guide, the git repository would be stored in the same file system as the original collection and would therefore not constitute a back up. Until such time as a version control backup is setup, the hook will remain in effect, but a cron task will delete older backups once a day.
This bash magic is a simplification of the better answers in a 16 year old stackoverflow discussion which takes advantage of the fact that the file names do not contain spaces and that there are no subdirectories with the name collectionsxxxxtar.gz.
Does this Work?
It's early days, but, so far, this is working well. As shown above, clients connected to the local network can synchronize in both directions with the Radicale server. Using a WireGuard tunnel, it has been possible to modify the calendar with the Android telephone and see that the copy on the Radicale server was updated. It also works in the other direction, changes made with Thunderbird on the desktop computer showed up on the Android phone connected to the Radicale server via the VPN.
Consequently, I am happy with this state of affairs. In a few days, if I still cannot connect to the CalDAV server at the web hosting site, I will raise a ticket, but there is no rush now. Eventually, when that is repaired, it will be possible to compare self hosting with a commercial service and I hope to be in a position to decide which is best. In any case, there will be a time when this site will be shutdown and at that time I may be very happy to have the possibility to self-host a CalDAV server.
As for Radicale itself, it lives up to its promise: it is a small but powerful CalDAV (calendars, to-do lists) and CardDAV (contacts) server [which] works out-of-the-box, no complicated setup or configuration required. The lesson for me was to stay away from the Debian package and to just follow the instructions given by the developers. Of course, you may have a different experience. Thank you very much to everyone at Kozea.