Weekend project: energy monitoring with telegraf, modbus, InfluxDB, Flux and Grafana
In this blog post, I will share how to obtain a view on the net electricity use and generation of my and your house. We’ll use modbus RTU and TCP to get our data, store it in InfluxDB, query it with InfluxQL and Flux and visualize it with Grafana. The end result looks a bit like this:
Curious? Read on!
The premise
Our house is designed according to the passive house standard, which means it’s designed and oriented in such a way that it requires less than 15kWh/m2/year for heating and cooling. A small air/water heat pump, powered by electricity, supplies the little energy still required for a comfortable home environment.
Recently, I had some time for a small weekend project. My goal was to:
- Collect data to gain insights into the energy use during the day
- Collect data from the PV installation about energy production
- Calculate the level of self-consumption we have without a home battery. In the future, I will use this to calculate the ROI of a home battery.
So, let’s do this!
Collecting data for energy use
Since electricity is the only energy source for our house, I can resort to monitoring the energy use of all our devices with a simple electricity meter. I bought a 3-phase DIN-rail mounted energy meter with modbus RTU interface. There are plenty of cheap modules to be found. I bought a DDS024MR.
Next, I used a Kunbus RevPI Core 3+ with USB to RS485 converter. The RevPI is an industrial-grade Raspberry PI which Ekopak also uses to monitor its water treatment installations. It’s a rather expensive module for a home project but I intend to use this for more automation projects in the future so I think it’s worth it.
First I built a small cabinet to wire everything up. See the picture below. Hardware is not my forte :)
Since I will use Telegraf with the modbus input plugin, I therefore used the following config in the telegraf.conf TOML file to get data from the DDS024MR module.
[[inputs.modbus]]
name = "energymeter"
slave_id = 11
timeout = "1s"
busy_retries = 2
busy_retries_wait = "1s"
controller = "file:///dev/ttyUSB0"
baud_rate = 9600
data_bits = 8
parity = "E"
stop_bits = 1
transmission_mode = "RTU"input_registers = [
{ measurement = "voltage_phase_a", name = "value", byte_order = "ABCD", data_type = "FLOAT32-IEEE", scale=1.0, address = [0,1]}{ measurement = "voltage_phase_b", name = "value", byte_order = "ABCD", data_type = "FLOAT32-IEEE", scale=1.0, address = [2,3]},{ measurement = "voltage_phase_c", name = "value", byte_order = "ABCD", data_type = "FLOAT32-IEEE", scale=1.0, address = [4,5]},{ measurement = "current_phase_a", name = "value", byte_order = "ABCD", data_type = "FLOAT32-IEEE", scale=1.0, address = [8,9]},{ measurement = "current_phase_b", name = "value", byte_order = "ABCD", data_type = "FLOAT32-IEEE", scale=1.0, address = [10,11]},{ measurement = "current_phase_c", name = "value", byte_order = "ABCD", data_type = "FLOAT32-IEEE", scale=1.0, address = [12,13]},{ measurement = "power_active_total", name = "value", byte_order = "ABCD", data_type = "FLOAT32-IEEE", scale=1.0, address = [16,17]},{ measurement = "power_active_a", name = "value", byte_order = "ABCD", data_type = "FLOAT32-IEEE", scale=1.0, address = [18,19]},{ measurement = "power_active_b", name = "value", byte_order = "ABCD", data_type = "FLOAT32-IEEE", scale=1.0, address = [20,21]},{ measurement = "power_active_c", name = "value", byte_order = "ABCD", data_type = "FLOAT32-IEEE", scale=1.0, address = [22,23]},{ measurement = "power_reactive_total", name = "value", byte_order = "ABCD", data_type = "FLOAT32-IEEE", scale=1.0, address = [24,25]},{ measurement = "power_reactive_a", name = "value", byte_order = "ABCD", data_type = "FLOAT32-IEEE", scale=1.0, address = [26,27]},{ measurement = "power_reactive_b", name = "value", byte_order = "ABCD", data_type = "FLOAT32-IEEE", scale=1.0, address = [28,29]},{ measurement = "power_reactive_c", name = "value", byte_order = "ABCD", data_type = "FLOAT32-IEEE", scale=1.0, address = [30,31]},{ measurement = "power_factor_a", name = "value", byte_order = "ABCD", data_type = "FLOAT32-IEEE", scale=1.0, address = [42,43]},{ measurement = "power_factor_b", name = "value", byte_order = "ABCD", data_type = "FLOAT32-IEEE", scale=1.0, address = [44,45]},{ measurement = "power_factor_c", name = "value", byte_order = "ABCD", data_type = "FLOAT32-IEEE", scale=1.0, address = [46,47]},{ measurement = "frequency", name = "value", byte_order = "ABCD", data_type = "FLOAT32-IEEE", scale=1.0, address = [54,55]},{ measurement = "energy_active_total", name = "value", byte_order = "ABCD", data_type = "FLOAT32-IEEE", scale=1.0, address = [256,257]},{ measurement = "energy_reactive_total", name = "value", byte_order = "ABCD", data_type = "FLOAT32-IEEE", scale=1.0, address = [1024,1025]},
]
Collecting data from PV inverter
Our SMA PV inverter allows to retrieve data from it via a REST API, a cloud service, modbus RTU and modbus TCP. Since I hadn’t used modbus TCP before and the REST API needed a client implementation, I decided to opt for the modbus TCP route.
First step: add the PV inverter to the network and enable modbus TCP in the SMA control panel by opening the control panel as the installer and activating modbus TCP. See the SMA manual for this.
Second step: get a list of modbus addresses. This should have been easy but took me a while to find the proper document. Here it is: https://my.sma-service.com/s/article/SMA-Modbus-Interface-SMA-SunSpec-Modbus-Interface?language=en_US (scroll to SMA_Modbus-TI-en-15.pdf)
Now, bring everything together with the following telegraf.confsnippet:
[[inputs.modbus]]
name = "SMA"
slave_id = 3
timeout = "5s"
controller = "tcp://a.b.c.d:502"
holding_registers = [
{ measurement = "sma_status", name = "value", byte_order = "ABCD", data_type = "UINT32", scale=1.0address = [30201,30202]},
{ measurement = "sma_temperature_derating", name = "value", byte_order = "ABCD", data_type = "UINT32", scale=1.0 address = [30219,30220]},
{ measurement = "sma_total_ac_energy_fed", name = "value", byte_order = "ABCDEFGH", data_type = "UINT64", scale=1.0 address = [30513,30514,30515,30516]},
{ measurement = "sma_daily_ac_energy_fed", name = "value", byte_order = "ABCDEFGH", data_type = "UINT64", scale=1.0 address = [30517,30518,30519,30520]},
{ measurement = "sma_operating_time", name = "value", byte_order = "ABCDEFGH", data_type = "UINT64", scale=1.0 address = [30521,30522,30523,30524]},
{ measurement = "sma_feedin_time", name = "value", byte_order = "ABCDEFGH", data_type = "UINT64", scale=1.0 address = [30525,30526,30527,30528]},
{ measurement = "sma_active_power_total", name = "value", byte_order = "ABCD", data_type = "UINT32", scale=1.0 address = [30775,30776]},
]Now just start telegraf and the data is flowing in.
Well, not actually. It took me a while to find the correct modbus RTU and the modbus TCP configuration, even with the right documentation. For the interested reader, see the bottom of this article on how to sniff both protocols for debugging.
Visualizing the data in Grafana
Once the data is being collected, the visualization in Grafana is pretty straightforward. To keep things simple, I used InfluxQL to retrieve the raw data from both the PV panel production and the electricity consumption.
Next, using Grafana’s transformations, we multiple the energy consumption with -1 to make these values negative so they appear below 0 on the y-axis. Finally, the sum of both values gives a Net Energy use, a line that will indicate when I’m putting power on the net or drawing from it.
Finally, a few series overrides in Grafana hide the original (positive) power usage curve and hides it from the legend and tooltip. Then, the Netto Energy (one of the calculated fields) is drawn without a fill and a wider line thickness.
The end result looks like the image above, or like this:
Calculate how effectively I’m using the energy I’m generating
The Grafana visualization above gives a good first impression. But actually, we’re interested in calculating how “self-sufficient” we are.
What is our level of self-consumption?
Flux allows me to calculate this, with the following pseudo-code.
1. Get raw data in Watt units of measure at original resolution
2. Calculate the difference between production and consumption
3. Split the data sets on values above and below zero, that is, moments where we're injecting into or consuming from the electricity grid.
4. Integrate all points over time to get KWh units of measure.
5. Calculate the ratio of self-consumption.Let’s turn that into an actual script:
import "math"NET_ENERGY = from(bucket: "smarthome")
|> range(start: v.timeRangeStart, stop: v.timeRangeStop)
|> filter(fn: (r) => (r._measurement == "sma_active_power_total" and r._value < 7000.0) or r._measurement == "power_active_total")
|> keep(columns: ["_time", "_start", "_stop", "_value", "_field", "_measurement"])
|> pivot(rowKey:["_time"], columnKey: ["_measurement"], valueColumn: "_value")
|> fill(column: "sma_active_power_total", usePrevious: true) // fill some gaps
|> fill(column: "power_active_total", usePrevious: true) // fill some gaps
|> map(fn: (r) => ({ r with _value: float(v: r.sma_active_power_total) - r.power_active_total })) // convert int to float and calculate the difference
|> sort(columns: ["_time"], desc: false) // clean it up a little
|> window(every: 1d) // offset at UTC, should be local timezone but unsupported for now (https://github.com/influxdata/flux/issues/406)TOTAL_CONSUMPTION = NET_ENERGY
|> integral(unit: 1h, column: "power_active_total")TOTAL_PRODUCTION = NET_ENERGY
|> integral(unit: 1h, column: "sma_active_power_total")NET_PRODUCTION = NET_ENERGY
|> map(fn: (r) => ({ r with _value: if r._value > 0.0 then r._value else 0.0 }))
|> integral(unit: 1h, column: "_value")NET_CONSUMPTION = NET_ENERGY
|> map(fn: (r) => ({ r with _value: if r._value <= 0.0 then math.abs(x: r._value) else 0.0 }))
|> integral(unit: 1h, column: "_value")NET = join(tables: {C: NET_CONSUMPTION, P: NET_PRODUCTION}, on: ["_field", "_start"], method: "inner")
NETTP = join(tables: {N: NET, TP: TOTAL_PRODUCTION}, on: ["_field", "_start"], method: "inner")
join(tables: {NTP: NETTP, TC: TOTAL_CONSUMPTION}, on: ["_field", "_start"], method: "inner")
|> rename(columns: {_value_C: "net_consumption", _value_P: "net_production", power_active_total: "total_consumption", sma_active_power_total: "total_production"}) // make it more readable
|> group() // makes for a nice table in Grafana
|> map(fn: (r) => ({ r with
_error: (r.total_consumption - r.net_consumption) - (r.total_production - r.net_production ), // to check whether our calculations are correct
own_use_ratio: 1.0 - r. net_consumption / r.total_consumption, // to define the percentage of our daily electricity use that we generated ourselves
self_consumption_ratio: 1.0 - r.net_production / r.total_production // to define how much energy we used ourselves from the total amount we generated.
}))
|> drop(columns: ["_stop_C", "_stop_NTP", "_stop_P", "_stop_TC"]) // clean up our table
And add units to our table in Grafana:
The end result is a table that tells us quite a lot:
To complete this view and calculate the ROI of a home battery, I would need to compare the costs and revenue of having a utility contract with separate injection and usage counters, with the current costs of a ‘prosumer tariff’ in Belgium. For now, I still have an analog meter which basically functions as an infinitely large battery.
In conclusion, the open source tools I used here have shown their value in being able to build this in just a couple of hours time.
Debugging Modbus RTU traffic
To debug modbus RTU traffic with telegraf, I needed to monitor what data was flowing in and out of the ttyUSB0 serial device. I used socatto achieve this, shamelessly stolen from a blog somewhere.
$ socat /dev/ttyUSB0,raw,echo=0 SYSTEM:'tee in.txt | socat - "PTY,link=/tmp/ttyV0,raw,echo=0,waitslave" | tee out.txt'The file in.txt will contain the raw data that was sent to the modbus RTU slave. The out.txt file will contain the raw data replied by the modbus RTU slave.
Afterwards, you can open both files with a hex editor and see the bytes transferred.
Debugging Modbus TCP traffic
This is easier. I used a simple modbus debugging tool on a Windows VM running on my Mac and captured the network traffic.
sudo tcpdump -s 0 -W 10 -C 10 -w vm.pcap -i en0 tcp port 502Afterwards, the vm.pcap file can be opened with Wireshark to see which commands are sent and how the device replies.
