Installing domoticz on raspberry pi (1/3)

By Atomstar on Sunday 2 December 2018 16:46 - Comments (0)
Categories: Linux, Smarthome, Views: 2.179

Although there are many guides on this topic available, I'm recording my personal setup here for my reference and with the hope that some issues I ran into will help others.

Goal

As goal for my Rpi/Domoticz project I had the following:
  • Record (smart) meter data and graph it
  • Cheap/low-power solution (i.e. no Intel NUC)
  • Versatile wrt interfaces (e.g. bluetooth/wifi/GPIO)

Hardware

Initially I ran on a RPi0, which is fine for domoticz but lacked the power to run grafana and influxdb which I added later, for which I upgraded to an RPi3B plus. Remember: premature optimisation is the root of all evil.
RPi Zero vs 3B+ pro/con
Pros:
  • + save 30e (cost + electricity);
  • + save power;
  • + can still play some emulator games after overclocking (All the Game Boys, GG, Master System, Genesis, NES, SNES and Turbografx at 60fps.)
  • + ...still faster than Pi1;
Cons:
  • - not sufficient to run desktop (source)
  • - No ethernet jack;
  • - no 3.5mm jack out
Rpi0
  • rpi zero W (11e)
  • 16 gb raspbian (10e)
  • usb charger 2A (9e)
  • housing (7e)
  • P1 kabel (20e)
RPi3B
  • Rip 3B+ (36e)
  • 64 GB (14e)
  • Housing (7e)
  • usb charger 2A (9e)
  • P1 kabel (20e)

Setup Raspbian

  1. Get Raspbian image -- use light image where possible, only use full image if you need desktop to debug etc.
    1. Optionally: enable SSH and wifi to run Rpi headless out of the box. See here, here or here.
  2. Write image to SD
    1. Enable Wifi: connect, login and enable
    2. Enable SSH from desktop: instructions
  3. Optional: customize hostname
    1. sudo hostname <newname>
    2. sudo vi /etc/hostname
    3. sudo vi /etc/hosts
  4. Secure Pi: instructions
  5. Optional: remove large packages
    1. Find large packages dpkg-query -Wf '${Installed-Size}\t${Package}\n' | sort -n
    2. Find reverse dependencies to check what you can safely remove: [mono]apt-cache rdepends jetty[/mon] or dry-run purging it: aptitude -s purge libplrpc-perl.
    3. Remove graphical user interface: sudo aptitude remove x11-common. N.B. aptitude will try to solutions that satisfy all dependencies, the first solution for me was to keep the package, the second solution (enter . or n to see) proposed to delete many/all packages that depend on x11-common and was the one I wanted.
  6. Set timezone to Europe/Amsterdam: sudo timedatectl set-timezone Europe/Amsterdam

Install domoticz

  1. Optional: create dedicated, non-root domoticz user: useradd domoticz
  2. Install domoticz: curl -L install.domoticz.com | sudo bash
  3. Chown files correctly: chown domoticz:domoticz * (see this pull request)
  4. Optional: enable non-root user to run domoticz:
    1. Allow USB access as normal user (required to read out P1 smart meters with USB cable): sudo usermod -a -G dialout YOURUSERNAME
    2. Change PID location to /tmp and and update startup script: see this commit
  5. Fix libssl error for https access (./domoticz: error while loading shared libraries: libssl.so.1.0.0: cannot open shared object file: No such file or directory)
    1. Switch to beta: cd domoticz; ./updatebeta (source)
    2. Alternatively, download obsolete SSL lib
  6. Add weather forecast, using station PWS:IUTRECHT432. N.B. IDEBILT9 (has data gaps, temperature was wrong, 16 vs 10 deg).
  7. Enable camera (for machine vision reading meter data later on)
    1. Use pre-applied adhesive tape to glue camera to PCB
    2. Shutdown Pi
    3. Use flat cable supplied with RPi (Zero) to connect to RPi
    4. Boot Pi
    5. Enable camera in raspi-config
    6. Reboot Pi
    7. Enable X11 forwarding
    8. Use PiCameraApp to view live camera: sudo apt install python python-tk python-pil.imagetk (N.B. this never worked for me)
  8. Reduce filesystem usage to prevent SD card wear
    1. SD cards have a limited lifetime. While data on SD cards is hard to find, SSDs with much more robust memory can support 200-6000x write cycles for consumer laptops (e.g. Samsung SSDs or these other consumer SSDs or Intel's server SSD). Likely, an SD card has worse performance than an SSD, such that 200x write cycles is probably optimistic. This means for a 64GB card which you want to use 30 years, you can write 1 GB per day (200*64/365/30). This is reasonable, so these tricks might not be required, although it still helps to prevent your SD card filling up. Again, premature optimization is the root of all evil, so be careful
    2. Disable domoticz automatic backup of database (66 MB/day = 24.000MB/yr)
    3. Disable swap (load unknown): sudo apt-get remove dphys-swapfile or sudo systemctl disable dphys-swapfile (source)
    4. Use noatime (default in raspbian)
    5. Reduce rsyslog syncing needs (default in raspbian)
  9. Improve security
    1. Domoticz is a monolithic program that works great if you run it as root. It has a built-in custom webserver, which is very convenient, but security-wise risky. I recommend against opening domoticz to the outside world
    2. Add real SSL certs from letsencrypt (not tested)
    3. Add mainstream webserver (apache, nginx): guide here. This did not work: reverse proxy has extremely slow performance (on RPi0). Since domoticz is a lot of json requests, this puts a (too) heavy load on the reverse proxy.
  10. Use external bluetooth adapter (e.g. for improved range)
    1. Check new device is available and works: [mono]hcitool dev && hcitool -i hci0 scan[mono]
    2. Disable built-in adapter so SBFspot uses external one: rfkill list | grep DE:AD:BE:EF:AA:AA; sudo rfkill block X
  11. Disable WiFi to prefer ethernet connection: Add dtoverlay=pi3-disable-wifi to /boot/config.txt

Manual fixes in database

In case some measurement is wrong, here is a (rather laborious) method to fix that:
  1. Determine meter & correct readings (from domoticz)
  2. Stop domoticz: sudo service domoticz stop
  3. Backup domoticz.db: cp domoticz.db domoticz.db.backup
  4. Get SQLite editor (e.g. DB Browser for SQLite)
  5. Copy domoticz.db to machine with SQLite editor: scp pi@raspberrypi.home:domoticz/domoticz.db .
  6. Update meter
    1. Fix DeviceStatus table - updates sValue
    2. Fix Meter table - update / delete wrong data
    3. Fix Meter_Calender - update / delete wrong data
    4. Alternatively, data is in "MultiMeter" or other tables
  7. Copy domoticz.db back to domoticz machine and restart domoticz: scp domoticz.db pi@raspberrypi.home:domoticz/ and sudo service domoticz start

Smarthome data logging with RPi/domoticz/influxdb/grafana

By Atomstar on Sunday 2 December 2018 16:01 - Comments (1)
Categories: Linux, Smarthome, Views: 363

When we got a smart meter and solar panels I wanted to record this data. Along the way I added more sensors and I've extended the software

Goals

  1. Record (smart) meter data and graph it (now including: electricity smart meter via USB, PV via bluetooth, heat meter via machine vision, water meter via line sensor, temperature via EvoHome)
  2. Log data in generic, long-lasting format locally (i.e. no externally hosted solution)
  3. Robust
  4. Interactive graphs on smartphone (domoticz, plot.ly, Highcharts, dygraphs)
  5. Customisable graphs (like grafana)
  6. Cheap/low-power solution (i.e. no Intel NUC)
  7. Versatile wrt interfaces (e.g. bluetooth/wifi/GPIO/USB)
  8. Not used to control or actuate. For that I use COTS products that work and don't crash :p

Architecture

Data model
Units
I use only SI units for storing meter data so I can combine e.g. heat meter (native unit MJ) with smart meter (native unit Wh) with PV (native unit kWh).
  1. Power: W, J/s
  2. Energy: J
  3. Temperature: deg C
  4. Wind: m/s
  5. Wind direction: deg
  6. Voltage: V
  7. Water: liter
  8. Rain: mm
Database scheme
Influxdb can only combine / add / subtract data from the same measurement. Keep this in mind when designing your scheme.

Generic format: <unit>, type=<quantity>, device=<datasource>, subtype=<device qualifier> value1=X value2=Y value3=Z <timestamp>.

Fully worked out, this looks like:
Power,type=elec,device=<kaifa/sma/smartplug> energy=X power=Y
Power,type=heat,device=<landisgyr> energy=X
Power,type=radiation,device=solar energy=X
temperature,type=heating,device=livingroom actual=X setpoint=Y
temperature,type=device,device=sma actual=X
temperature,type=weather,device=XXweatherloc actual=X
wind,type=weather,device=XXweatherloc wind=X
winddirection,type=weather,device= XXweatherloc winddir=X
voltage,type=elec,device=sma, DC=X AC=Y
water,type=potable,device=sensus water=X
rain,type=weather,device=XXweatherloc rain=X

Optionally this could all be in one table (called 'measurement' in influxdb parlance)
Polling cadence
  • Temperature rooms: 1x/5 minutes, resolution: 0.5 deg C
  • Power: 1x/min
  • PV: 1x/min
  • Water: on-demand incremental
  • Heat: 1x/minute
Data usage at 1 measurement/minute is: 1440min/day * 2byte/measurement 3kb/day/metric = 1 MB/yr/metric.
Software architecture
  • Easy access visualization: domoticz
  • Database: influxdb
  • Full visualization only accessibly locally: grafana
  • Selected visualizations remotely accessible on smartphone: Dygraphs
  • Data collection: domoticz with pushers to influxdb, as well as custom scripts to push directly to influxdb.

Implementation

Implementation occurred more or less chronologically in this order. I've written the text including hindsight to fix initial mistakes:
  1. Installing domoticz on Rpi
  2. Extending domoticz with influxdb/grafana
  3. Connecting sensors to Rpi/domoticz

Alternatives

Before settling on the above solution I've considered (and rejected) the following alternative solutions for various reasons:
  • hass.io + influxdb: NOK hass.io has no influxdb export, and appears too heavy (never actually tried)
  • OpenHab: NOK extremely slow on RPi0, briefly tried on RPi3B but still slow.
  • prometheus: not as widely supported as influxdb. Might actually be better because of more powerful query language
  • rpi-monitor / pimatic: appears not mature enough
  • vis.js / Dygraphs / Highcharts / Rickshaw / Plotly: too complicated

Sources

https://blog.jokielowie.c...f-latwe-i-piekne-wykresy/
https://community.openhab.org/t/oh-webserver/36756/10
https://www.circuits.dk/install-grafana-influxdb-raspberry/
https://prometheus.io/doc.../#prometheus-vs.-influxdb
http://dirkgroenen.nl/pro...-endpoints-documentation/
http://blog.videgro.net/2017/09/smart-meter/
https://community.openhab...her-chart-meteogram/39324
https://www.thethingsnetw...d-push-your-iot-data/1788

Reading out TP-Link HS110 on Linux/Raspberry Pi

By Atomstar on Sunday 25 November 2018 13:47 - Comments (2)
Categories: Linux, Smarthome, Views: 788

I bought a TP-Link HS110 version 3.0 smart wi-fi plug for 30 EUR to read out power usage of some appliances. I have a Raspberry Pi 3B+ to connect this to running domoticz and influxdb/grafana to visualise the data.

The device identifies itself as:
{
  "system": {
    "get_sysinfo": {
      "sw_ver": "1.4.3 Build 171009 Rel.104144",
      "hw_ver": "2.0",
      "type": "IOT.SMARTPLUGSWITCH",
      "model": "HS110(EU)",
      "mac": "ab:cd:ef:gh:ij:kl",
      "dev_name": "Smart Wi-Fi Plug With Energy Monitoring",
      "alias": "TP-LINK_Smart Plug_IJKL",
      "relay_state": 1,
      "on_time": 336,
      "active_mode": "none",
      "feature": "TIM:ENE",
      "updating": 0,
      "icon_hash": "",
      "rssi": 33,
      "led_off": 0,
      "longitude_i": 0,
      "latitude_i": 0,
      "hwId": "044A516EE63C875F9458DA25C2CCC5A0",
      "fwId": "00000000000000000000000000000000",
      "deviceId": "8006FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF",
      "oemId": "1998A14DAA86E4E001FD7CAF42868B5E",
      "err_code": 0
    }
  }
}
where mac, the last 4 digits of alias, and the last 36 digits of deviceId have been masked.

Device setup

Here we connect the HS110 to our existing WiFi network without using the TP-link Kasa app and without it ever connecting to the TP-link cloud server.
  1. Plug into wall socket, the wifi symbol will blink green-amber.
  2. Get python clients https://github.com/softScheck/tplink-smartplug
  3. Connect to HS110 access point TP-LINK_Smart Plug_XXXX.
    1. Get initial information:
      ./tplink_smartplug.py -t 192.168.0.1 -c info
  4. Disable cloud access:
    1. Set devs.tplinkcloud.com to resolve to 127.0.0.1 (or similar)
    2. Set cloud server to new dummy value of trusted party: using
      ./tplink_smartplug.py -t 192.168.0.1 -j '{"cnCloud":{"set_server_url":{"server":"xs4all.nl"}}}'
      N.B. err_code:0 means no error
  5. Bind to existing WiFi network:
    ./tplink_smartplug.py -t 192.168.0.1 -j '{"netif":{"set_stainfo":{"ssid":"WiFi","password":"123","key_type":3}}}'
    N.B. This command is not encrypted so it will leak your SSID and network. There is no way around it as the official Kasa app uses the same protocol.
  6. Recommend to give it a fixed IP

Device read out

Now the device is connected to our WiFi network and we can use it as a measurement device
  1. Find out new IP
    1. On your router, look at connected devices, write down IP
    2. Ping broadcast address to find IP of all connected devices responding to ping: e.g. ping 192.168.0.255, the TP-link HS110 will respond
  2. Read out device:
    ./tplink_smartplug.py -t 192.168.0.134 -c energy

Implementing in Domoticz/Influxdb

Here I automate measurement data collection on my Rpi:
Influxdb
  1. Get updated script from https://github.com/tvwerkhoven/tplink-smartplug, store somewhere
    git clone https://github.com/tvwerkhoven/tplink-smartplug.git
  2. Install as crontab, e.g.
    */1 * * * * /home/pi/workers/hs110_reader/tplink_smartplug.py -t 172.16.0.204 -c energy --influxdb http://172.16.0.133:8086 smarthome --influxdb_energy "energy,type=elec,device=hs110" --influxdb_power "power,type=elec,device=hs110"
Domoticz
Todo, probably: create virtual sensor/meter, update script to push to domoticz, adapt cron, done.

Accuracy

Accuracy is comparable to Brennenstuhl PM 231E offline power meter, which has a reported accuracy of 1% or 0.2W.
15W incandescent bulb
  • Reference measurement: 15.00.2W (average-by-eye)
  • TP-Link measurement: 15.030.09W (average of 10 measurements)
1300W heating iron
  • Reference measurement: 13405W (average-by-eye)
  • TP-link measurement: 1342W5W (average of 14 measurements)
N.B. the heating iron's load drops 20W over the timeframe of a minute, such that these uncertainties are upper limits.

Power usage

The TP-link HS110 itself uses 4.30.2 Watt (as measured by Brennenstuhl PM 231E)

Speed

The plug starts up in 8 seconds, and then takes 3 seconds to join the WiFi network.

Alternatives

  1. Elgato Eve Energy works with Bluetooth (vs WiFi), compatible with Homekit. Could work on Linux using Homebridge, but this sounded like more trouble than reading out over Wi-Fi/HTTP
  2. Fibaro Wall Plug works with Z-wave (vs WiFi), requires additional Z-wave dongle, which I don't have. It's also more expensive than the TP-link. After that it could work on Linux, e.g. via this OpenHab post or this Youtube Tutorial

Sources


Tweakers vormt samen met Tweakers Elect, Hardware.Info, Autotrack, Nationale Vacaturebank en Intermediair de Persgroep Online Services B.V.
Alle rechten voorbehouden © 1998 - 2018 Hosting door True