Merge pull request #1 from JasperE84/dev

Release 1.0.3

Author: JasperE84

Examples/docker-compose.yml

@@ -37,20 +37,20 @@ services:
     image: jsprnl/pyfusionsolardatarelay:latest
     #build: ./PyFusionSolarDataRelay
     restart: unless-stopped
-    network_mode: host
     depends_on:
       - influxdb
       - mosquitto
     environment:
       - pvdebug=True
       - pvsysname=inverter01
 
+      - pvfusionsolar=True
       - pvfusionsolarkkid=GET_THIS_STRING_FROM_YOUR_KIOSK_URL
-      - pvfusioninterval=120 #please note that the kiosk current power only seems to update each 30mins
+      - pvfusionminutecron=0,30 # Please note that the kiosk only seems to update each 30mins
 
       - pvinflux=True
       - pvinflux2=True
-      - pvifhost=localservername
+      - pvifhost=influxdb
       - pvif2protocol=http
       - pvif2org=acme
       - pvif2bucket=fusionsolar
@@ -61,9 +61,15 @@ services:
       - pvpvoutputapikey=GENERATE_THIS_AND_SYSTEMID_ON_PVOUTPUT.ORG
 
       - pvmqtt=True
-      - pvmqtthost=localservername
+      - pvmqtthost=mosquitto
       - pvmqttauth=False
 
+      - pvgridrelay=False
+      - pvgridrelaykenterean=000000000000000000
+      - pvgridrelaykentermeterid=0000000000
+      - [email protected]
+      - pvgridrelaykenterpasswd=secretpassword
+
     volumes:
       - /etc/localtime:/etc/localtime
 

Examples/pvoutput-measurement-result-example.png

@@ -1,6 +1,8 @@
 # Huawei FusionSolar Kiosk API to InfluxDB, MQTT and PVOutput relay
 This is a python project intended to fetch data from the **Huawei FusionSolar** public **kiosk** and relay it to **InfluxDB** and/or **PVOutput.org** and/or **MQTT**. 
 
+Additionally this project can also fetch and relay grid usage data from the Dutch meetdata.nl API provider by **Kenter**.
+
 Credits go to the [Grott project](https://github.com/johanmeijer/grott). Many bits of code, structure and ideas are borrowed from there.
 
 [![GitHub release](https://img.shields.io/github/release/JasperE84/PyFusionSolarDataRelay?include_prereleases=&sort=semver&color=2ea44f)](https://github.com/JasperE84/PyFusionSolarDataRelay/releases/)
@@ -14,31 +16,44 @@ A local settings file (such as .yml or .ini) has not been implemented yet, but p
 
 Check out [Examples/docker-compose.yml](https://github.com/JasperE84/PyFusionSolarDataRelay/blob/main/Examples/docker-compose.yml) for a docker configuration example.
 
+# Breaking changes in the release
+The fusionsolarinterval configuration paramters has been replaced by two cron settings defaulting to poll fusionsolar data each half our.
+
 # About Huawei FusionSolar Kiosk mode
 FusionSolar is Huawei's online monitoring platform for their PV inverters. FusionSolar features a kiosk mode. When enabled, a kiosk url is generated which is publically accessible. The kiosk web app fetches its data from a JSON backend. It is this backend where this project fetches the PV data. 
 Fetching data from the kiosk mode can be beneficial to those without direct access to the official API and/or the inverter Modbus-TCP. For instance when the inverter is logging to fusionsolar over a direct cellular connection configured and fitted by an installer unable to provide API access rights to third parties.
 
+# About Kenter's API and matching PVOutput intervals
+Fusion solar data fetching is planned by cron in order to exactly specify at what times the data should reload. This way, it is possible to synchronise the intervals of fusionsolar and gridkenter datapoints, which end up showing on PVOutput. That's relevant because if the gridkenter data class is fetched, meetdata.nl does not provide live measurements. Instead it provides historic measurements with a certain interval (15 minutes interval with the most recent data point 3 days old in my case). If this interval doesn't match the fusionsolar interval, then PVOutput will show distorted graphs because it won't have a datapoint for both PV production and grid usage for each interval. (Fusionsolar kiosk API only updates each half hour). See [this url](https://crontab.guru/) for help with finding the right cron config.
+
 # About PVOutput.org
 [PVOutput.org](https://pvoutput.org/) is a free service for sharing and comparing PV output data.
+![PVOutput dashboard screenshot](./Examples/pvoutput-measurement-result-example.png)
 
 # About InfluxDB
 [InfluxDB](https://www.influxdata.com/) is an open source time series database on which dashboards can easily be built. For instance using [Grafana](https://grafana.com/)
 
 # About MQTT
 MQTT is an OASIS standard messaging protocol for the Internet of Things (IoT). It is designed as an extremely lightweight publish/subscribe messaging transport that is ideal for connecting remote devices. MQTT can be used to relay the PV data to various home automation software such as [Home Assistant](https://www.home-assistant.io/)
 
+# About Kenter's meetdata.nl
+Kenter provides measurement services for **commercially rented** grid transformers. This project can fetch energy usage data from this API and post it to InfluxDB and PVOutput. MQTT is not supported for posting Kenter data, as Kenter's latest measurement data is usually 3 days old.
+
 # Configuration parameter documentation
 | Parameter | Environment variable | Description | Default |
 | --- | --- | --- | --- |
 | debug | pvdebug | Enables verbose logging | True |
 | pvsysname | pvsysname | Definition of 'measurement' name for InfluxDB | inverter01 |
+| fusionsolar | pvfusionsolar | Can be `True` or `False`, determines if fusionsolar kiosk API is enabled | True |
 | fusionsolarurl | pvfusionsolarurl | Link to the fusionsolar kiosk data backend | [Click url](https://region01eu5.fusionsolar.huawei.com/rest/pvms/web/kiosk/v1/station-kiosk-file?kk=) |
 | fusionsolarkkid | pvfusionsolarkkid | Unique kiosk ID, can be found by looking the kiosk URL and then taking the code after `kk=` | GET_THIS_FROM_KIOSK_URL |
-| fusioninterval | pvfusioninterval | Seconds between fusionsolar data polling and relay | 120 |
+| fusionhourcron | pvfusionhourcron | Hour component for python cron job to fetch and process data from fusionsolar. | * |
+| fusionminutecron | pvfusionminutecron | Minute component for python cron job to fetch and process data from fusionsolar | 0,30 |
 | pvoutput | pvpvoutput | Can be `True` or `False`, determines if PVOutput.org API is enabled | False |
 | pvoutputapikey | pvpvoutputapikey | API Key for PVOutput.org | yourapikey |
 | pvoutputsystemid | pvoutputsystemid | System ID for PVOutput.org, should be numeric | 12345 |
-| pvoutputurl | pvpvoutputurl | API url for PVOutput.org | [Click url](https://pvoutput.org/service/r2/addstatus.jsp)
+| pvoutputurl | pvpvoutputurl | API url for PVOutput.org live output posting | [Click url](https://pvoutput.org/service/r2/addstatus.jsp)
+| pvoutputbatchurl | pvpvoutputbatchurl | API url for PVOutput.org historic data batch posting (used for grid data from meetdata.nl) | [Click url](https://pvoutput.org/service/r2/addbatchstatus.jsp)
 | influx | pvinflux | Can be `True` or `False`, determines if InfluxDB processing is enabled | False |
 | influx2 | pvinflux2 | If `True` the InfluxDBv2 methods are used. If `False` InfluxDBv1 methods are used | True |
 | ifhost | pvifhost | Hostname of the influxdb server | localhost |
@@ -57,6 +72,16 @@ MQTT is an OASIS standard messaging protocol for the Internet of Things (IoT). I
 | mqttuser | pvmqttuser | MQTT Username | fusionsolar |
 | mqttpasswd | pvmqttpasswd | MQTT Password | fusionsolar |
 | mqtttopic | pvmqtttopic | MQTT Topic for publishing | energy/pyfusionsolar |
+| gridrelay | pvgridrelay | Can be `True` or `False`, determines if data is fetched from Kenter's meetdata.nl API | False |
+| gridrelaysysname | pvgridrelaysysname | Grid transformer name for InfluxDB transformer data | transformer01 |
+| gridrelayinterval | pvgridrelayinterval | Interval in seconds to fetch data from meetdata.nl and post to PVOutput and InfluxDB | 43200 |
+| gridrelaykenterurl | pvgridrelaykenterurl | Kenter API url for fetching transformer grid measurements | [Click url](https://webapi.meetdata.nl) |
+| gridrelaykenterean | pvgridrelaykenterean | EAN code for transformer on Kenter's www.meetdata.nl | XXX |
+| gridrelaykentermeterid | pvgridrelaykentermeterid | MeterID as shown on Kenter's www.meetdata.nl | XXX |
+| gridrelaykenteruser | pvgridrelaykenteruser | Username for Kenter's API | user |
+| gridrelaykenterpasswd | pvgridrelaykenterpasswd | Password for Kenter's API | passwd |
+| gridrelaydaysback | pvgridrelaydaysback | Kenter's meetdata.nl does not provide live data. Data is only available up until an X amount of days back. May vary per transformer. | 3 |
+| gridrelaypvoutputspan | pvgridrelaypvoutputspan | In my case meetdata.nl has datapoints for each 15mins. Setting this to a value of 2, will calculate averages over 2 datapoints spanning half an hour before posting to PVOutput. This way the datapoint interval between the grid usage data and fusionsolar PV production data matches, resulting in nice diagrams on PVOutput.org | 2 |
 
 # Grafana dashboard example
 A grafana dashboard export is included in the Examples subfolder in the Git repository.
@@ -79,5 +104,14 @@ Take the following steps to achieve this:
 Result:
 ![Xibo layout screenshot](./Examples/grafana-embedded-in-xibo-layout.png)
 
+# Changelog
+| Version | Description |
+| --- | --- |
+| 1.0.3 | Grid transformer usage measurement polling from Kenter's meetdata.nl API has been implemented |
+| 1.0.3 | Changed docker-compose.yml template not to use host networking mode |
+| 1.0.3 | pv.py now uses separate threads for PvRelay and GridRelay classes |
+| 1.0.3 | Implemented apscheduler's cron implementation to be able to specify exact moments to fetch fusionsolar data |
+| 1.0.3 | Code and method name refactoring including PvConf type hints in classes where this class was injected as method parameter |
+=======
 
 Released under [MIT](/LICENSE) by [@JasperE84](https://github.com/JasperE84).

README.md

@@ -0,0 +1,100 @@
+import requests
+from requests.auth import HTTPBasicAuth
+from datetime import datetime, timedelta
+import json
+from pvconf import PvConf
+
+
+class GridKenter:
+    def __init__(self, conf: PvConf, logger):
+        self.conf = conf
+        self.logger = logger
+        self.logger.debug("GridKenter class instantiated")
+
+    def fetch_gridkenter_data(self, days_back):
+        self.logger.info(
+            "Requesting data for {} from GridKenter API...".format(
+                self.conf.gridrelaysysname
+            )
+        )
+        req_time = datetime.now() - timedelta(days=days_back)
+        req_year = req_time.strftime("%Y")
+        req_month = req_time.strftime("%m")
+        req_day = req_time.strftime("%d")
+
+        try:
+            url = f"{self.conf.gridrelaykenterurl}/api/1/measurements/{self.conf.gridrelaykenterean}/{self.conf.gridrelaykentermeterid}/{req_year}/{req_month}/{req_day}"
+            self.logger.debug(f"Fetching URL: {url}")
+
+            response = requests.get(
+                url,
+                auth=HTTPBasicAuth(
+                    self.conf.gridrelaykenteruser, self.conf.gridrelaykenterpasswd
+                ),
+                verify=False,
+            )
+        except Exception as e:
+            raise Exception(
+                "Error fetching data from GridKenter API: '{}'".format(str(e))
+            )
+
+        try:
+            response_json = response.json()
+        except Exception as e:
+            raise Exception(
+                "Error while parsing JSON response from GridKenter API: '{}'".format(
+                    str(e)
+                )
+            )
+
+        if not "16180" in response_json:
+            raise Exception(
+                f"GridKenter API response does not contain '16180' (levering tbv allocatie) key. Data possibly not ready yet. Response: {json.dumps(response_json)}"
+            )
+
+        # Checking required realKpi elements and transforming kW(h) to W(h)
+        if len(response_json["16180"]) == 0:
+            raise Exception(
+                "'16180' (levering tbv allocatie) key does not contain data. Data possibly not ready yet. Response: {json.dumps(response_json)}"
+            )
+
+        grid_data_obj = {
+            "sysname": self.conf.gridrelaysysname,
+            "ean": self.conf.gridrelaykenterean,
+            "meter_id": self.conf.gridrelaykentermeterid,
+            "grid_net_consumption": [],
+        }
+
+        prev_ts = None
+
+        for measure in response_json["16180"]:
+            # Meting en valide, of handmatig goedgekeurd
+            if (measure["origin"] == "m" and measure["status"] == "v") or measure[
+                "status"
+            ] == "m":
+                # Calculate powerload
+                ts = datetime.fromtimestamp(measure["timestamp"])
+                if prev_ts == None:
+                    seconds_from_prev_ts = (
+                        ts - ts.replace(hour=0, minute=0, second=0, microsecond=0)
+                    ).total_seconds()
+                else:
+                    seconds_from_prev_ts = (ts - prev_ts).total_seconds()
+                prev_ts = ts
+                calculated_power = round(
+                    measure["value"] * 3600 / seconds_from_prev_ts, 3
+                )
+
+                # self.logger.debug(
+                #     f"Measurement local ts: {datetime.fromtimestamp(measure['timestamp']).strftime('%Y-%m-%d %H:%M:%S')} kWh: {measure['value']} kW (calculated): {calculated_power}"
+                # )
+
+                grid_data_obj["grid_net_consumption"].append(
+                    {
+                        "timestamp": measure["timestamp"],
+                        "interval_energy": measure["value"] * 1000,
+                        "interval_power_avg": calculated_power * 1000,
+                    }
+                )
+
+        return grid_data_obj

gridkenter.py

@@ -0,0 +1,58 @@
+import time
+from pvinflux import PvInflux
+from pvoutputorg import PvOutputOrg
+from pvconf import PvConf
+from gridkenter import GridKenter
+from pvmqtt import PvMqtt
+
+
+class GridRelay:
+    def __init__(self, conf: PvConf, logger):
+        self.conf = conf
+        self.logger = logger
+        self.logger.debug("GridRelay class instantiated")
+
+        self.gridkenter = GridKenter(conf, logger)
+        self.pvoutput = PvOutputOrg(conf, logger)
+        self.pvmqtt = PvMqtt(conf, logger)
+        self.pvinflux = PvInflux(self.conf, self.logger)
+        self.pvinflux_initialized = False
+
+        self.logger.info("Starting GridRelay on separate thread")
+        self.start()
+
+    def start(self):
+        self.logger.debug("GridRelay waiting 5sec to initialize docker-compose containers")
+        time.sleep(5)
+
+        while 1:
+            try:
+                grid_measurement_data = self.gridkenter.fetch_gridkenter_data(self.conf.gridrelaydaysback)
+                self.write_gridkenter_to_influxdb(grid_measurement_data)
+                self.write_gridkenter_to_pvoutput(grid_measurement_data)
+            except:
+                self.logger.exception(
+                    "Uncaught exception in GridRelay data processing loop."
+                )
+
+            self.logger.debug("Waiting for next interval...")
+            time.sleep(self.conf.gridrelayinterval)
+
+    def write_gridkenter_to_pvoutput(self, grid_measurement_data):
+        if self.conf.pvoutput:
+            try:
+                self.pvoutput.write_griddata_to_pvoutput(grid_measurement_data)
+            except:
+                self.logger.exception("Error writing GridData to PVOutput.org")
+
+    def write_gridkenter_to_influxdb(self, grid_measurement_data):
+        if self.conf.influx:
+            if self.pvinflux_initialized == False:
+                self.pvinflux_initialized = self.pvinflux.initialize()
+
+            if self.pvinflux_initialized:
+                self.pvinflux.pvinflux_write_griddata(grid_measurement_data)
+        else:
+            self.logger.debug("Writing data to Influx skipped, not initialized yet.")
+
+

gridrelay.py

@@ -1,7 +1,10 @@
 import sys
 import logging
+import time
+from threading import Thread
 from pvconf import PvConf
 from pvrelay import PvRelay
+from gridrelay import GridRelay
 
 # Logger
 logger = logging.getLogger()
@@ -10,7 +13,7 @@
 formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
 streamHandler.setFormatter(formatter)
 logger.addHandler(streamHandler)
-logger.info("PyFusionSolarDataRelay 1.0.2 started")
+logger.info("PyFusionSolarDataRelay 1.0.3 started")
 
 # Config
 conf = PvConf(logger)
@@ -21,10 +24,20 @@
 else:
     logger.setLevel(logging.INFO)
 
-# Start relay
-relay = PvRelay(conf, logger)
+# Start PvRelay and KenterRelay
 try:
-    relay.main()
+    if __name__ == '__main__':
+        if conf.fusionsolar:
+            fs_thread = Thread(target = PvRelay, args=[conf, logger])
+            fs_thread.daemon = True
+            fs_thread.start()
+        if conf.gridrelay:
+            gr_thread = Thread(target = GridRelay, args=[conf, logger])
+            gr_thread.daemon = True
+            gr_thread.start()
+    while True:
+        time.sleep(1)
 except KeyboardInterrupt:
     logger.info("Ctrl C - Stopping relay")
     sys.exit(0)
+