Merge pull request #105 from HotNoob/v1.1.10

V1.1.10

Author: HotNoob

.github/workflows/python-3.10.yml

@@ -3,6 +3,9 @@
 
 name: Python 3.10
 
+permissions:
+  contents: read
+  
 on:
   push:
     branches: [ "main" ]

.github/workflows/python-3.11.yml

@@ -3,6 +3,9 @@
 
 name: Python 3.11
 
+permissions:
+  contents: read
+
 on:
   push:
     branches: [ "main" ]

.github/workflows/python-3.12.yml

@@ -3,6 +3,9 @@
 
 name: Python 3.12
 
+permissions:
+  contents: read
+
 on:
   push:
     branches: [ "main" ]

.github/workflows/python-3.13.yml

@@ -3,6 +3,9 @@
 
 name: Python 3.13
 
+permissions:
+  contents: read
+
 on:
   push:
     branches: [ "main" ]

.github/workflows/python-3.9.yml

@@ -3,6 +3,9 @@
 
 name: Python 3.9
 
+permissions:
+  contents: read
+
 on:
   push:
     branches: [ "main" ]

.gitignore

@@ -28,3 +28,8 @@ classes/transports/*custom*
 
 input_registry.json
 holding_registry.json
+
+#ignore pypi / pyproject.toml output 
+dist/*
+build/*
+python_protocol_gateway.egg-info/*

Dockerfile

@@ -1,4 +1,4 @@
-FROM python:3.9-alpine as base
+FROM python:3.13-alpine as base
 FROM base as builder
 RUN mkdir /install
 WORKDIR /install

README.md

@@ -5,10 +5,12 @@
 ![Python 3.12](https://github.com/HotNoob/PythonProtocolGateway/actions/workflows/python-3.12.yml/badge.svg)
 ![Python 3.13](https://github.com/HotNoob/PythonProtocolGateway/actions/workflows/python-3.13.yml/badge.svg)
 
+[![PyPI version](https://img.shields.io/pypi/v/python-protocol-gateway.svg)](https://pypi.org/project/python-protocol-gateway/)
 [![CodeQL](https://github.com/HotNoob/PythonProtocolGateway/actions/workflows/github-code-scanning/codeql/badge.svg)](https://github.com/HotNoob/PythonProtocolGateway/actions/workflows/github-code-scanning/codeql)
 
 For advanced configuration help, please checkout the documentation :)
-https://github.com/HotNoob/PythonProtocolGateway/tree/main/documentation
+
+[/documentation](/documentation)
 
 # Python Protocol Gateway
 
@@ -17,8 +19,9 @@ Configuration is handled via a small config files.
 In the long run, Python Protocol Gateway will become a general purpose protocol gateway to translate between more than just modbus and mqtt. 
 
 For specific device installation instructions please checkout the documentation:
-Growatt, EG4, Sigineer, SOK, PACE-BMS
-https://github.com/HotNoob/PythonProtocolGateway/tree/main/documentation
+Growatt, EG4, Sigineer, SOK, PACE-BMS, Sigineer, ect...
+
+[/documentation/devices](/documentation/devices)
 
 # General Installation
 Connect the USB port on the inverter into your computer / device. This port is essentially modbus usb adapter.
@@ -28,7 +31,7 @@ Alternatively, connect a usb adapter to your rs485 / can port with appropriate w
 
 ### install as homeassistant add-on
 checkout:
-https://github.com/felipecrs/python-protocol-gateway-hass-addon/tree/master
+[PPG HASS Addon](https://github.com/HotNoob/python-protocol-gateway-hass-addon/tree/master)
 
 ### install requirements
 ```
@@ -49,21 +52,30 @@ nano config.cfg
 manually select protocol in .cfg
 protocol_version = {{version}}
 ```
+eg4_v58 = eg4 inverters
+eg4_3000ehv_v1 = eg4 inverters
 v0.14 = growatt inverters 2020+
 sigineer_v0.11 = sigineer inverters
-growatt_2020_v1.24 = alt protocol for large growatt inverters - currently untested
-srne_v3.9 = SRNE inverters - confirmed working-ish
-victron_gx_3.3 = Victron GX Devices - Untested
-solark_v1.1 = SolarArk 8/12K Inverters - Untested
+srne_v3.9 = SRNE inverters
+
 hdhk_16ch_ac_module = some chinese current monitoring device :P
-srne_2021_v1.96 = SRNE inverters 2021+ (tested at ASF48100S200-H, ok-ish for HF2430U60-100 )
+```
 
-eg4_v58 = eg4 inverters ( EG4-6000XP ) - confirmed working
-eg4_3000ehv_v1 = eg4 inverters ( EG4_3000EHV )
+Untested Protocols
 ```
+growatt_2020_v1.24 = alt protocol for large growatt inverters
+victron_gx_3.3 = Victron GX Devices
+solark_v1.1 = SolarArk 8/12K Inverters
+```
+
+For a complete list of protocols, explore:
+[/Protocols](/protocols)
 
-more details on these protocols can be found in the documentation:
-https://github.com/HotNoob/PythonProtocolGateway/tree/main/documentation
+For a more complete list of tested devices & protocols:
+[Tested Devices & Protocols](documentation/usage/devices_and_protocols.csv)
+
+more advanced details can be found in the documentation:
+[/Documentation](/documentation)
 
 ### run as script
 ```
@@ -108,8 +120,11 @@ once installed; the device should show up on home assistant under mqtt
 
 ```Settings -> Devices & Services -> MQTT ```
 
-more docs on setting up mqtt here: https://www.home-assistant.io/integrations/mqtt
-i probably might have missed something. ha is new to me.
+more docs on setting up mqtt here: 
+https://www.home-assistant.io/integrations/mqtt
+
+#### connect mqtt on home assistant with external mqtt broker
+[HowTo Connect External MQTT Broker To HomeAssistant](https://www.youtube.com/watch?v=sP2gYLYQat8)
 
 ### general update procedure
 update files and restart script / service
@@ -118,8 +133,6 @@ git pull
 systemctl restart protocol_gateway.service
 ```
 
-**if you installed this when it was called growatt2mqtt-hotnoob or invertermodbustomqtt, you'll need to reinstall if you want to update. **
-
 ### Unknown Status MQTT Home Assistant 
 If all values appear as "Unknown"
 This is a bug with home assistant's discovery that some times happens when adding for the first time. just restart the service / script and it will fix itself. 
@@ -148,17 +161,18 @@ As i dive deeper into solar monitoring and general automation, i've come to the
 
 So... don't mind me as i may add other devices such as battery bms' and... i have a home energy monitor on the way! so i'll be adding that when it arrives.
 
-### Rebranding Again... last time.
-if you installed this when it was called growatt2mqtt-hotnoob or InverterModBusToMQTT, you'll need to reinstall if you want to update. 
-
-
 ### donate
 this took me a while to make; and i had to make it because there werent any working solutions. 
-donations would be appreciated.
-![BitCoin Donation](https://github.com/HotNoob/growatt2mqtt-hotnoob/blob/main/images/donate_to_hotnoob.png?raw=true)
-
-```(btc) bc1qh394vazcguedkw2rlklnuhapdq7qgpnnz9c3t0```
+donations / sponsoring this repo would be appreciated.
 
-### Use Docker - untested
+### Use Docker
 - ```docker build . -t protocol_gateway ```
 - ```docker run --device=/dev/ttyUSB0 protocol_gateway```
+
+### Use Docker Image
+- ``` docker pull hotn00b/pythonprotocolgateway ``` 
+- ```docker run -v $(pwd)/config.cfg:/app/config.cfg --device=/dev/ttyUSB0 hotn00b/pythonprotocolgateway```
+
+See [config.cfg.example](/config.cfg.example)
+
+[Docker Image Repo](https://hub.docker.com/r/hotn00b/pythonprotocolgateway)

RELEASE.md

@@ -0,0 +1,8 @@
+things todo to perform a release.
+
+can try to automate some of these later.
+
+GitHub - https://github.com/HotNoob/PythonProtocolGateway/releases
+PyPi Package - https://pypi.org/project/python-protocol-gateway/
+HomeAssistant repo - https://github.com/HotNoob/python-protocol-gateway-hass-addon
+Docker Image - https://hub.docker.com/r/hotn00b/pythonprotocolgateway

classes/protocol_settings.py

@@ -188,6 +188,7 @@ class registry_map_entry:
     register_bit : int
     register_byte : int
     ''' byte offset for canbus ect... '''
+
     variable_name : str
     documented_name : str
     unit : str
@@ -208,6 +209,9 @@ class registry_map_entry:
     data_type_size : int = -1
     ''' for non-fixed size types like ASCII'''
 
+    data_byteorder : str = ''
+    ''' entry specific byte order little | big | '' '''
+
     read_command : bytes = None
     ''' for transports/protocols that require sending a command ontop of "register" '''
 
@@ -330,6 +334,19 @@ def get_registry_entry(self, name : str, registry_type : Registry_Type) -> regis
                 return item
 
         return None
+    
+    def get_code_by_value(self, entry : registry_map_entry, value : str, fallback=None) -> str:
+        ''' case insensitive '''
+        
+        value = value.strip().lower()
+
+        if entry.variable_name+"_codes" in self.codes:
+            codes = self.codes[entry.variable_name+"_codes"]
+            for code, val in codes.items():
+                if value == val.lower():
+                    return code
+                
+        return fallback
 
     def load__json(self, file : str = "", settings_dir : str = ""):
         if not settings_dir:
@@ -512,16 +529,30 @@ def process_row(row):
 
             #region data type
             data_type = Data_Type.USHORT
-
             data_type_len : int = -1
+            data_byteorder : str = ''
             #optional row, only needed for non-default data types
             if "data type" in row and row["data type"]:
+                data_type_str : str = ''
+
                 matches = data_type_regex.search(row["data type"])
                 if matches:
                     data_type_len = int(matches.group("length"))
-                    data_type = Data_Type.fromString(matches.group("datatype"))
+                    data_type_str = matches.group("datatype")
                 else:
-                    data_type = Data_Type.fromString(row["data type"])
+                    data_type_str = row["data type"]
+
+                #check if datatype specifies byteorder
+                if data_type_str.upper().endswith("_LE"):
+                    data_byteorder = "little"
+                    data_type_str = data_type_str[:-3]
+                elif data_type_str.upper().endswith("_BE"):
+                    data_byteorder = "big"
+                    data_type_str = data_type_str[:-3]
+
+
+                data_type = Data_Type.fromString(data_type_str)
+
 
 
             if "values" not in row:
@@ -658,6 +689,7 @@ def process_row(row):
                                             unit_mod= unit_multiplier,
                                             data_type= data_type,
                                             data_type_size = data_type_len,
+                                            data_byteorder = data_byteorder,
                                             concatenate = concatenate,
                                             concatenate_registers = concatenate_registers,
                                             values=values,
@@ -857,6 +889,10 @@ def load_registry_map(self, registry_type : Registry_Type, file : str = "", sett
     def process_register_bytes(self, registry : dict[int,bytes], entry : registry_map_entry):
         ''' process bytes into data'''
 
+        byte_order : str = self.byteorder
+        if entry.data_byteorder: #allow map entry to override byteorder
+            byte_order = entry.data_byteorder
+
         if isinstance(registry[entry.register], tuple):
             register = registry[entry.register][0] #can bus uses tuple for timestamp
         else:
@@ -869,14 +905,15 @@ def process_register_bytes(self, registry : dict[int,bytes], entry : registry_ma
             register = register[:entry.data_type_size]
 
         if entry.data_type == Data_Type.UINT:
-            value = int.from_bytes(register[:4], byteorder=self.byteorder, signed=False)
+            value = int.from_bytes(register[:4], byteorder=byte_order, signed=False)
         elif entry.data_type == Data_Type.INT:
-            value = int.from_bytes(register[:4], byteorder=self.byteorder, signed=True)
+            value = int.from_bytes(register[:4], byteorder=byte_order, signed=True)
         elif entry.data_type == Data_Type.USHORT:
-            value = int.from_bytes(register[:2], byteorder=self.byteorder, signed=False)
+            value = int.from_bytes(register[:2], byteorder=byte_order, signed=False)
         elif entry.data_type == Data_Type.SHORT:
-            value = int.from_bytes(register[:2], byteorder=self.byteorder, signed=True)
+            value = int.from_bytes(register[:2], byteorder=byte_order, signed=True)
         elif entry.data_type == Data_Type._16BIT_FLAGS or entry.data_type == Data_Type._8BIT_FLAGS or entry.data_type == Data_Type._32BIT_FLAGS:
+            val = int.from_bytes(register, byteorder=byte_order, signed=False)
             #16 bit flags
             start_bit : int = 0
             end_bit : int = 16 #default 16 bit
@@ -952,11 +989,20 @@ def process_register_bytes(self, registry : dict[int,bytes], entry : registry_ma
                 # If positive, simply extract the value using the bit mask
                 value = (register >> bit_index) & bit_mask
 
-        elif entry.data_type.value > 200 or entry.data_type == Data_Type.BYTE: #bit types
+        elif entry.data_type == Data_Type.BYTE: #bit types
+            value = int.from_bytes(register[:1], byteorder=byte_order, signed=False)
+        elif entry.data_type.value > 200: #bit types
             bit_size = Data_Type.getSize(entry.data_type)
             bit_mask = (1 << bit_size) - 1  # Create a mask for extracting X bits
             bit_index = entry.register_bit
+
+
+            if isinstance(register, bytes):
+                register = int.from_bytes(register, byteorder=byte_order)
+
             value = (register >> bit_index) & bit_mask
+
+
         elif entry.data_type == Data_Type.HEX:
             value = register.hex() #convert bytes to hex
         elif entry.data_type == Data_Type.ASCII:
@@ -986,6 +1032,11 @@ def process_register_bytes(self, registry : dict[int,bytes], entry : registry_ma
 
     def process_register_ushort(self, registry : dict[int, int], entry : registry_map_entry ):
         ''' process ushort type registry into data'''
+
+        byte_order : str = self.byteorder
+        if entry.data_byteorder:
+            byte_order = entry.data_byteorder
+
         if entry.data_type == Data_Type.UINT: #read uint
             if entry.register + 1 not in registry:
                 return
@@ -1058,7 +1109,7 @@ def process_register_ushort(self, registry : dict[int, int], entry : registry_ma
             else:
                 flags : list[str] = []
                 if end_bit > 0:
-                    end : int = 16 if end_bit >= 16 else end_bit    
+                    end : int = 16 if end_bit >= 16 else end_bit
                     for i in range(start_bit, end):  # Iterate over each bit position (0 to 15)
                         # Check if the i-th bit is set
                         if (val >> i) & 1:
@@ -1074,10 +1125,10 @@ def process_register_ushort(self, registry : dict[int, int], entry : registry_ma
                 bit_index = entry.register_bit
                 value = (registry[entry.register] >> bit_index) & bit_mask
         elif entry.data_type == Data_Type.HEX:
-                value = registry[entry.register].to_bytes((16 + 7) // 8, byteorder=self.byteorder) #convert to ushort to bytes
+                value = registry[entry.register].to_bytes((16 + 7) // 8, byteorder=byte_order) #convert to ushort to bytes
                 value = value.hex() #convert bytes to hex
         elif entry.data_type == Data_Type.ASCII:
-            value = registry[entry.register].to_bytes((16 + 7) // 8, byteorder=self.byteorder) #convert to ushort to bytes
+            value = registry[entry.register].to_bytes((16 + 7) // 8, byteorder=byte_order) #convert to ushort to bytes
             try:
                 value = value.decode("utf-8") #convert bytes to ascii
             except UnicodeDecodeError as e: