July 3, 2017

Hardware and software projects rarely work as planned in the first place. The virtual Domoticz temperature and humidity sensor log displayed recurrent yet irregular anomalous behaviour since installed a few days ago. I modified the Python script that reads the values from the DHT11 sensor and managed to improve the situation.

It has become apparent that monitoring the house temperature and humidity could evolve into a useful project with an impact on our comfort. Of course, nothing is ever as simple as first thought. Below I will talk about some difficulties I am encountering with trying to establish a measure of environmental home comfort. I have yet to come up with a solution, but that's ok, I am probably months away from a hardware project involving the dehumidifier and air exchanger.

Table of Contents

  1. Improved Python Script
  2. Logging Sensor Readings
  3. Relative Humidity and Dew Point
  4. Human Comfort
  5. Domoticz Sensor

  1. Improved Python Script
  2. The temperature and humidity graph displayed by Domoticz has numerous spikes corresponding to what I suppose are failed attempts at reading the sensor. To see the graph, open the Temperature tab and click on Log button of the sensor.

    I knew that the 0°C, 0% temperature and humidity values could not be right and probably corresponded to problems while reading the DHT11 sensor. It is a simple thing to add some data logging to the Python script that performed that task. Here is the first attempt.

    #!/usr/bin/python # coding: utf-8 # parameters DHT_type = 11 OneWire_pin = 24 room_temp_idx = 35 cpu_temp_idx = 36 url_json = "http://192.168.0.45:9071/json.htm?type=command¶m=udevice&idx=" verbose = 0 # 1 to print out information to the console, 0 for silence log = 1 # 1 to log readings to dht11_log.txt, 0 to suspend logging import sys import Adafruit_DHT import urllib if log > 0: from datetime import datetime # read and report dht11 temperature and humidity humidity, temperature = Adafruit_DHT.read_retry(DHT_type, OneWire_pin) if log > 0: with open('/home/pi/pythons/dht11_log.txt', 'a') as f: f.write(datetime.now().strftime('%Y-%m-%d %H:%M:%S') + ' t: '+str(temperature) + ', h: '+str(humidity) $ cmd = url_json + str(room_temp_idx) + "&nvalue=0&svalue=" + str(temperature) + ";" + str(humidity) + ";0" ... continue on as before

    Notice that the user defined parameters were moved to the top of the file. That way the datetime module is imported only if logging the data from the sensor. Logging really just means appending a line with the time, temperature and humidity readings to a text file. Here is the log after a few minutes.

    pi@domo:~ $ cat pythons/dht11_log.txt 2017-07-02 14:05:03 t: 24.0, h: 70.0 2017-07-02 14:10:10 t: 24.0, h: 71.0 2017-07-02 14:15:02 t: 24.0, h: 71.0

    After some time, the log reveals that the script is unable to get data from the sensor.

    2017-07-02 18:55:06 t: 25.0, h: 68.0 2017-07-02 19:00:40 t: None, h: None 2017-07-02 19:05:40 t: None, h: None 2017-07-02 19:10:13 t: 25.0, h: 68.0

    Just what could be the problem is not clear but at the very least, the python script can be modified to not pass on bogus temperature and humidity values to the home automation server. Indeed the Adafruit example script tests that the temperature and humidity values are valid before doing anything with them.

    When reading the data fails, the read_try function will have made fifteen attempts at getting the data from the sensor waiting 2 seconds between each attempt. In all, this will take 40 seconds. I experimented a bit, thinking that systematically waiting the minimum time allowed between reads might not be the best approach. I am not one for repeating a failed action without changing anything fifteen times and hopping for a different outcome. In the end I settled on three attempts at the default two seconds intervals, followed by three attempts at four second intervals and, if needed, a final three attempts at eight second intervals. It is early yet, but that seems to give good results. Failure to read valid values from the sensor does take 49 seconds which is somewhat longer than before.

    Here is the corrected Python script.

    #!/usr/bin/python # coding: utf-8 # parameters DHT_type = 11 OneWire_pin = 24 room_temp_idx = 35 cpu_temp_idx = 36 url_json = "http://192.168.0.45:9071/json.htm?type=command¶m=udevice&idx=" verbose = 0 # 1 to print out information to the console, 0 for silence log = 2 # 0 no logging, 1 log bad data only, 2 log failed attempts also, 3 log everything import sys import Adafruit_DHT import urllib if log > 0: from datetime import datetime # orignal code # read and report dht11 temperature and humidity # humidity, temperature = Adafruit_DHT.read_retry(DHT_type, OneWire_pin) # bad_data = (humidity is None) or (temperature is None) # first try 3 times at 2 second intervals humidity, temperature = Adafruit_DHT.read_retry(DHT_type, OneWire_pin, 3) bad_data = (humidity is None) or (temperature is None) # if read failed, try 3 times at 4 second intervals if bad_data: if log > 0: with open('/home/pi/pythons/dht11_log.txt', 'a') as f: f.write(datetime.now().strftime('%Y-%m-%d %H:%M:%S') + ' failed reading in 3 tries at 2 second intervals\n') humidity, temperature = Adafruit_DHT.read_retry(DHT_type, OneWire_pin, 3, 4) bad_data = (humidity is None) or (temperature is None) # if read failed, try 3 times at 8 second intervals if bad_data: if log > 0: with open('/home/pi/pythons/dht11_log.txt', 'a') as f: f.write(datetime.now().strftime('%Y-%m-%d %H:%M:%S') + ' failed reading in 3 tries at 4 second intervals\n') humidity, temperature = Adafruit_DHT.read_retry(DHT_type, OneWire_pin, 3, 8) bad_data = (humidity is None) or (temperature is None) if (log > 0) and bad_data: with open('/home/pi/pythons/dht11_log.txt', 'a') as f: f.write(datetime.now().strftime('%Y-%m-%d %H:%M:%S') + ' failed reading in 3 tries at 8 second intervals\n') if (log > 2): with open('/home/pi/pythons/dht11_log.txt', 'a') as f: f.write(datetime.now().strftime('%Y-%m-%d %H:%M:%S') + ' t: '+str(temperature) + ', h: '+str(humidity) + '\n') if bad_data: sys.exit('Execution of temps.py stopped with bad temperature or humidity data') if humidity > 60: HUM_STAT = '3' elif humidity > 25: HUM_STAT = '1' else: HUM_STAT = '2' cmd = url_json + str(room_temp_idx) + "&nvalue=0&svalue=" + str(temperature) + ";" + str(humidity) + ";" + HUM_STAT hf = urllib.urlopen(cmd) if verbose > 0: print 'Données lues: température ambiante {0:0.1f}°C, humidité {1:0.1f}%'.format(temperature, humidity) print 'URL JSON pour Domoticz: ' + cmd print 'Réponse: ' + hf.read() hf.close # read and report cpu temperature cpuTemp = int(open('/sys/class/thermal/thermal_zone0/temp').read()) / 1e3 cmd = url_json + str(cpu_temp_idx) + "&nvalue=0&svalue=" + str(cpuTemp) hf = urllib.urlopen(cmd) if verbose > 0: print 'Donnée lue: température du Raspberry Pi: {0:0.1f}°C'.format(cpuTemp) print 'URL JSON pour Domoticz: ' + cmd print 'Réponse: ' + hf.read() hf.close
    Download the script by clicking here.

    Every now and then, there is a failure to read the sensor. I will continue to monitor this in an effort to see if there is a pattern of sorts. Perhaps the failed readings correspond to times when the Raspberry Pi is busy, such as when an automatically backing up the Domoticiz database. Or perhaps when handling a notification that involves taking a snap shot from the Web Cam.

    Some may have noticed that this newer version of the script does set a value for HUM_STAT in the Json URL other than "0". For reasons explained below, this is not a bad solution after all.

  3. Logging Sensor Readings
  4. The logging facility was easily added to the Python script, but was it necessary? There is a Download CSV option in the Chart context menu of the temperature and relative humidity chart displayed when looking at the sensor log. The menu is displayed when clicking on the three horizontal bar button at the top right edge of the temperature charts. Unfortunately, that option did not work. Fortunately, a Forum discussion on the subject had the solution.

    The discussion mentions two newer versions of the export-csv.js script: 1.4.5 and 1.4.8. I tried both and found that 1.4.5 worked with my version 3.5877 of Domoticz.

    Here are the detailed instructions. First go to the Domoticz Javascript directory. Second rename the current (compressed) export-csv script, just in case. And then download the (uncompressed) script file from Github.

    pi@domo:~ $ cd domoticz/www/js pi@domo:~/domoticz/www/js $ mv export-csv.js.gz export-csv.js.gz-bak pi@domo:~/domoticz/www/js $ wget https://raw.githubusercontent.com/highcharts/export-csv/v1.4.5/export-csv.js

    To try version 1.4.8, use wget https://raw.githubusercontent.com/highcharts/export-csv/master/export-csv.js.

  5. Relative Humidity and Dew Point
  6. Relative humidity and dew point are two measure of the amount of water vapour in the air.

    Relative humidity is ratio of the actual amount of water vapour in the air compared to the maximum water vapour the air could hold at the current temperature. This ratio of the water vapour content of the air to its capacity is expressed as a percentage.

    The water vapour capacity of air is not a constant; it varies with the ambient temperature. Colder air has a lower capacity to hold water vapour compared to warmer air.

    While the above is often seen, it seems to be an incorrect view of things as air does not "hold" water vapour. To quote Steve Horstmeyer, Relative Humidity....Relative to What? The Dew Point Temperature...a better approach it's all about energy, or more specifically it's about the available thermal energy to do the work of evaporation. The temperature of a gas is a measure of the average kinetic energy of the molecules that make up that gas. "So instead of saying that warm air has a greater water vapour capacity, say, that warm air provides more thermal energy to evaporate more water". Of course, if the water is itself warmer, then its molecules have greater kinetic energy and that also increases the likelihood that some water molecules will break free from the liquid and escape into the air. I only mention this last bit because it would explain why it is more humid in the summer near big bodies of water.

    The dew point is the temperature to which the air would need to be cooled so as to obtain 100% relative humidity. Thus the dew point is a temperature measure on the Celsius (most of the world) or Fahrenheit (USA) scale. The dew point is a lower temperature than the ambient temperature unless relative humidity is already at 100%.

    Since the two concept are measures of the same physical entity, there is a non trivial mathematical relationship between them which actually involves another variable vapour pressure. Here is an often used approximation, the Magnus formula, which only involves relative humidity, dew point and a couple of constants.

    RH: relative humidity (%) DP: dew point (°C) T: temperature (°C) b: 17.625 c: 243.04 Dew point from relative humidity q = ln(RH/100) + (b*T)/(c+T) DP = c*q/(b-q) Relative humidity from dew point RH = 100*exp[ b*DP/(c+DP) - b*T/(c+T) ] or RH = 100*exp[ b*(DP/(c+DP) - T/(c+T)) ]
    Source: Brian McNoldy. The same conversion, presented in a slightly different form, is found in R.L. Snyder Humidity Conversion except that the constants are a: 17.27, b: 237.3.

    There is an even simpler approximation proposed by Mark G. Lawrence

    DP = T-(100-RH)/5 RH = 100 - 5(T - DP)
    valid when the relative humidity is over 50%. As will be seen below, this appealingly simple approximation may not be useful, especially for Canadian houses in the winter where relative humidity is much lower.

  7. Human Comfort
  8. Human comfort depends on temperature, humidity and other factors. Looking for an authoritative reference for the relationship between relative humidity and human comfort, I found the following

    For humans relative humidity below 25% feels uncomfortable dry. Relative humidity above 60% feels uncomfortable wet. Human comfort requires the relative humidity to be in the range 25 - 60% RH. (The Engineering ToolBox)
    and
    In most Canadian cities, ideal indoor relative humidity levels are 35% in the winter and 50% in the summer. ASHRAE specifies a range between 25 and 60%. (pg 22, Indoor Air Quality in Office Buildings: A Technical Guide, Health Canada)

    These values appear to be based on ASHRAE Standard 55-1992 Thermal Environmental Conditions for Human Occupancy (or a more recent version). There is a similar EU standard: CEN - EN 15251 Indoor Environmental Input Parameters for Design and Assessment of Energy Performance of Buildings Addressing Indoor Air Quality, Thermal Envirnoment, Lighting and Acoustics. I do not have these documents, but I assume this comfort scale is for a narrow band of normal house temperatures given the inverse relationship between relative humidity and temperature. Look at the Center for the Built Environment Thermal Comfort Tool.

    As mentioned in my previous post, the US National Weather Service stated that it is better to look at the dew point instead of relative humidity to judge how dry or humid it feels. It argues that it feels dryer on a cold day with 100% relative humidity than on a hot day with 50% relative humidity:

    TemperatureRelative humidityDew PointPerception
    80°F/27°C50%59.7°F/15.4°Chumid
    30°F/-1.1°C100%30°F/-1.1°Cdry
    As can be seen the lower the dew point the less humid it feels.

    The service gives the following guidelines for the summer months:

    Dew PointPerception
    > 64°F> 18°Coppressive
    56 – 64°F13 – 18°Csticky/muggy
    < 56°F< 13°Cdry and comfortable

    There are many other such tables some of which have a broader range of dew points. Here are two others.

    How does it feel? by Brian Newdorff

    Dew PointPerception
    > 75°F> 24°CMiserable
    71 – 75°F21 – 24°COppressive
    66 – 70°F18 – 21°CUncomfortable
    61 – 65°F16 – 18°CSticky
    56 – 60°F13 – 16°CComfortable
    < 56°F< 13°CPleasant

    From the Wikipedia Dew Point page which is basically the same as found in Steve Horstmeyer.

    Dew pointHuman perceptionRelative humidity at 32°C (90°F)
    > 80°F> 26°CSeverely high, even deadly for asthma related illnesses73% and higher
    75 – 80°F24 – 26°CExtremely uncomfortable, fairly oppressive62 – 72%
    70 – 74°F21 – 24°CVery humid, quite uncomfortable52 – 61%
    65 – 69°F18 – 21°CSomewhat uncomfortable for most people at upper edge44 – 51%
    60 – 64°F16 – 18°COK for most, but all perceive the humidity at upper edge37 – 43%
    55 – 59°F13 – 16°CComfortable31 – 36%
    50 – 54°F10 – 12°CVery comfortable26 – 30%
    < 50°F< 10°CA bit dry for some25% and lower

    It's almost as if there is too much information. Recall that there are only three possible comfort values for a Domoticz temperature and humidity sensor "wet", "comfortable" and "dry", plus an apparently out of band "normal".

    I decided to try the following mapping from dew point temperature to the Domoticz classification.

    Dew pointHuman perceptionRelative Humidity
    at 21°Cat 26°C
    > 68°F> 20°Cwet> 94%> 69.5%
    50 – 68°F10 – 20°Ccomfortable49,4 – 94%36.5  – 69.5%
    < 50°F< 10°Cdry< 49.4%< 36.5%

    I chose to calculate the relative humidity for two temperatures: 21°C/70°F, which is more or less the average indoor temperature in Canadian houses in the winter, and 26°C/79°F which is the average maximum daily temperature in July in Montréal, Québec where I remember how it often felt muggy downtown.

    That is an interesting situation. Apparently the Health Canada target 35% relative humidity for heated houses in the winter would be judged too dry by many. Yet it is within the comfort zone prescribed by ASHREA. To investigate this a bit further, I created a small spreadsheet that shows the dew point temperature as a function of relative humidity and temperature. It is colour coded in accordance with the above mapping: red are the dry, green the comfortable and blue the wet situations.

    The CBE Tool is much more versatile and does show that a relative humidity of 35% at a temperature of 21°C is just within the comfort zone when normal winter clothing is worn.

    Clearly, the dew point temperature will not be sufficient to accurately calculate the comfort zone.

  9. Domoticz Sensor
  10. The whole point of this long discussion was about setting the HUM_STAT value of the virtual temperature and humidity sensor. The simplest is to use the relative humidity value obtained from the physical sensor as already done in the Python script reading the DHT11.

    if humidity > 60: HUM_STAT = '3' elif humidity > 25: HUM_STAT = '1' else: HUM_STAT = '2'

    We could replace those six lines with a calculation of the dew point and use it to set HUM_STAT.

    import math b = 17.625 c = 243.04 q = math.log(humidity/100.0) + (b*temperature)/(c + temperature) dp = c*q/(b-q) if dp > 20: HUM_STAT = '3' elif dp > 10: HUM_STAT = '1' else: HUM_STAT = '2'

    However, if you do want to use the dew point, there is another way since Domoticz calculates it when the virtual sensor values for the temperature and humidity are set. Hence there is no need to calculate the dew point. Instead a Lua script can update the variable with little work.

    pi@domo:~ $ nano domoticz/scripts/lua/script_device_temp+humi.lua
    -- name and index of virtual temperature + humidity sensor DName = 'Temp' DIDX = 35 commandArray = {} if (devicechanged[DName]) then -- read the values of the temperature + humidity sensor -- tonumber(string.format("%.1f", exact)) is used to avoid displaying nearest -- floating point such as 21.200000877596 for 21.2 tp = tonumber(otherdevices_temperature[DName].format("%.1f", exact)) hd = otherdevices_humidity[DName] -- Domoticz truncates value to floor integer dp = otherdevices_dewpoint[DName] -- not used to display --hum_stat can be one of 0=Normal (not used), 1=Comfortable, 2=Dry, 3=Wet if dp > 20 then hum_stat = 3 -- wet elseif dp > 10 then hum_stat = 1 -- confortable else hum_stat = 2 -- dry end commandArray[DIDX] = { ['UpdateDevice'] = DIDX..'|0|'.. tp ..';'.. hd ..';'.. hum_stat } end return commandArray
    Download the script by clicking here.

    The script picks up the temperature, relative humidity and dew point temperature from Domoticz and assigns a value to hum_stat. The virtual device is then updated. Luckily, no outgoing MQTT message is generated so that an infinite loop is not a possibility. There would of course be no need to calculate a meaningful HUM_STAT value. Instead simply assign it a 0 value:

    cmd = url_json + str(room_temp_idx) + "&nvalue=0&svalue=" + str(temperature) + ";" + str(humidity) + ";0"

    I would prefer to calculate the dew point in the Python script instead of the Lua script if no further action is taken with the temperature and relative humidity values. And that is just because Domoticz goes through every Lua script with a name starting with script_device on every event. But if another device, say an air conditioner, an air exchanger, a dehumidifier or a humidifier, is to be controlled by Domoticz, then the last approach seems reasonable.