New Controller: LORA Direct

[enesbcs]

I am currently testing a new node-to-node communication form, LORA direct. (no WAN)

@TD-er
I am suggesting the following data structures:

struct Lora_SysInfoStruct
{
  byte header = 255;
  byte ID = 1;
  byte sourcelUnit;
  byte mac[6];
  word build;
  char Name[25];
  byte nodeType;
};

struct Lora_SensorInfoStruct
{
  byte header = 255;
  byte ID = 3;
  byte sourcelUnit;
  byte destUnit;
  byte destTaskIndex;
  byte sensorType;
  char taskName[26];
  char ValueNames[VARS_PER_TASK][26];
};

struct Lora_SensorDataStruct
{
  byte header = 255;
  byte ID = 5;
  byte sourcelUnit;
  byte destUnit;
  byte destTaskIndex;
  float Values[VARS_PER_TASK];
};

struct Lora_CommandDataStruct
{
  byte header = 255;
  byte ID = 7;
  byte sourcelUnit;
  byte destUnit;
  char commandline[64];
};

I've written a test-sketch for my BsFrance LoRa32u4 module based on the old ArduinoEasy code which i will publish soon. (this device has 2k ram + 32k flash)

The RPI is able to receive packages without problem, Duty cycle is handled at the sender side, in this case by the LoRa32u4, the RPI is a bridge-like device in the setup.

Referenced to:
#75

[TD-er]

Have you seen my recent PR regarding TTN?
letscontrolit/ESPEasy#2546

Of this PR, especially have a look at the JavaScript decoder functions which will be executed on the TTN server.
In short, i have added very basic support for all plugins to send out the standard 1 .. 4 float values, but will add more support for plugins to output more values in a compact format specific for that plugin.
See the raw encoder functions for Sysinfo and GPS:

https://github.com/letscontrolit/ESPEasy/blob/1a1b7ce219e793f4087748037209531b4df3de87/src/_P026_Sysinfo.ino#L147-L169

https://github.com/letscontrolit/ESPEasy/blob/1a1b7ce219e793f4087748037209531b4df3de87/src/_P082_GPS.ino#L523-L544

I am not sure if you can run JS code on the Pi, but if so we can maintain the same decoder files for both platforms.
If for some reason a plugin will later support more variables, it can be filtered on packet length or we may add a version byte to its stream.
Currently the raw stream does start with the plugin_id, which is a very easy filter for decoding it.

For LoRa, as well as LoRaWAN, there are limits on how long your air time can be. So you want to make sure the data stream is as short as possible and also unambiguous when decoding it.
That last part may be a bit difficult for some plugins where the user may select what value to output.
So that's why I came up with this RAW packed format.

I was planning to also add support for these kind of raw packed streams into the ESPeasy p2p network, so we can use it also among nodes, regardless of their network or platform.

See also the (incomplete) documentation for this.

[enesbcs]

Sure Javascript can be executed from inside Python with the Js2Py package.
I see this TTN thing, but i prefer (local) direct communication forms instead third party server (internet) integrations. They are technically interesting but not so practical solutions.
The raw streams can be useful for LoRa, ESPNOW and either Bluetooth P2P communcation in the future.

[TD-er]

I see this TTN thing, but i prefer (local) direct communication forms instead third party server (internet) integrations. They are technically interesting but not so practical solutions.

I never said it should be done solely via TTN (by the way you can also connect your own server to process the data yourself. It is then still encrypted over the internet)
It is just that TTN requires the decoders to be handling data in a certain way and if we want to make our own, why not borrow their ideas to make our work simpler so we can use the same decoders for different platforms?

The raw streams can be useful for LoRa, ESPNOW and either Bluetooth P2P communcation in the future.

That I totally agree upon and that's also why I put so much effort to add support for these RAW packed encoder thingies.
They can be re-used for a lot of things. (pun intended :) )
Also for storage on SPIFFS in large binary dumps. (See the cache controller I'm also working on)

[enesbcs]

It is just that TTN requires the decoders to be handling data in a certain way and if we want to make our own, why not borrow their ideas to make our work simpler so we can use the same decoders for different platforms?

I see. But how can we use javascript encoders/decoders in Arduino on device that has 2KiB RAM and 32KiB flash for example?

[TD-er]

It is just that TTN requires the decoders to be handling data in a certain way and if we want to make our own, why not borrow their ideas to make our work simpler so we can use the same decoders for different platforms?

I see. But how can we use javascript encoders/decoders in Arduino on device that has 2KiB RAM and 32KiB flash for example?

We don't.
But for those we could try to add a decoder to the same plugin.
It is just that having a generic all-in-one decoder to process all plugins is very practical to have on some more powerful nodes like R'pi or TTN servers etc. And just for those purposes it is potentially saving a lot of work if those decoders could be shared among platforms.
For the small nodes like the LoRa32u4, you probably are not interested in decoding all possible plugins, but only a handful. So for those it will need some C++ decoding.
But still it would be nice if we could also share those encoder/decoder codes in some way. (not sure how)

[TD-er]

In general the small nodes will only supply data from a sensor, so they just encode the data.
Only if a node does need to receive sensor data and also needs to decode it (instead of just forwarding it via another medium) then we need to have a decoder written for that platform.

[TD-er]

If you want to do direct LoRa communication from node to node, you still need to add some headers indicating who's sending, who has to receive it and what the payload is.
So the raw packed stream can be a good part of the payload, which format can be shared among platforms.
But you still need to do some coding on the node itself to process it.

It is just my hope that we can come up with some kind of generic way of generating and interpreting the payload per plugin, so we can add new ones on various platforms without a lot of work.
Each platform still needs to have its own wrappers to process the payload, but that only has to be done once per platform/medium.
The Payload structure should be done only once for all platforms and the payload encoding/content should be done only once per plugin.

That's my Utopian idea, but I am not sure it is achievable like this.

[enesbcs]

I am using currently the following (working) sketch with the LoRa32u4 for simple sending.
https://github.com/enesbcs/EasyLoraSensor
Currently the whole thing uses 1KiB RAM and 20KiB flash space.

Sending performed by the LoraDirect.ino in the same style as the ESPEasy P2P UDP networking works. Names only sent once at startup for minimizing traffic later, then the Task values sent at the requested intervals, if the Duty Cycle lets it.. Currently all four task value is sent, but can be modified to respect "ValueCount". On the receiver side (RPI) the received data stored in Dummy tasks without much thinking, as there are no real device beneath these tasks, they simply receive and forward values to other Controllers.

I am not sure what is the benefit of this decoders?
https://github.com/letscontrolit/ESPEasy/blob/mega/misc/TTN/packed_converter.js

[enesbcs]

The Payload structure should be done only once for all platforms and the payload encoding/content should be done only once per plugin.

What is the concrete payload structure that you recommend?

[TD-er]

I don't think starting with the existing p2p implementation is a bad idea (in fact it is a good idea to get started)
But the p2p layer is rather inefficient and limits the number of values to 4.
So that's why I later will add a new packet definition for the p2p protocol which carries the raw packed payload..
That's just a new ID in the p2p header so it will remain compatible with the old versions, just adding a new option that will only be able to be handled by newer builds.
So this introduction of my LoRa/TTN efforts is just to make sure that you will not spend a lot of time thinking about the same stuff and come up with almost the same solution but incompatible :)

About the the "converters" part.
I used it in my development to "clean up" parts that were not yet decoded. So initially it was just 80+ times the same code cut-and-pasted and then stripped out values that were not available on some plugins.
Just see it as a left-over to-do list of mine.
In the end we probably do not need the "converter" part anymore, or at least not with entries for all plugins, since the bulk of the decoding part is done in the decoders.js file.

[TD-er]

The Payload structure should be done only once for all platforms and the payload encoding/content should be done only once per plugin.

What is the concrete payload structure that you recommend?

This is the description of the header:
https://github.com/letscontrolit/ESPEasy/blob/1a1b7ce219e793f4087748037209531b4df3de87/misc/TTN/packed_decoder.js#L272-L288

• Plugin_id : 1 byte
• IDX: 2 bytes (same IDX entry as mainly used for Domoticz, only now 2 bytes)
• samplesetcount : 1 byte (see documentation on "sample set initiator"
• valuecount : 1 byte

And then the binary packed data stream of the plugin.
This header should be enough to help decode that.

[TD-er]

If for some reason the single byte of plugin ID is no longer enough, we can start with a special byte to signal the 2-byte plugin ID. (e.g. set the first bit high)

But please feel free to suggest changes. Right now this code is probably running on a handful of nodes and even less TTN accounts :)

[enesbcs]

* Plugin_id : 1 byte
* IDX: 2 bytes (same IDX entry as mainly used for Domoticz, only now 2 bytes)
* samplesetcount : 1 byte (see [documentation on "sample set initiator"](https://espeasy.readthedocs.io/en/latest/Controller/_Controller.html?highlight=sample%20set#controller-parameters)
* valuecount : 1 byte

And then the binary packed data stream of the plugin.

All right then you are suggesting something similar:

struct Lora_SensorDataStruct
{
 byte plugin_id;
 word idx;
 byte samplesetcount;
 byte valuecount;
 float Values[VARS_PER_TASK]; // valuecount * float
};

But in Raw LoRa communication there are no addressing (it has to be included in the payload), this is why i am using the old ESPEasy P2P packets, as they almost perfect... This is why i think that we can not treat a high level TTN/LoraWAN protocol payload as a simple RAW BLE/LoRa/ESPNOW payload.
In LoRa we need at least: 1 byte for sender address, 1 byte for destination address (with some reserved addresses for example 0 is broadcast and 1 for default gateway)

The idx approach is an interesting one, so at the receiver side the user has to create manually the device with this IDX to be able to receive? The old approach sending the DestinationTaskIndex is a little simpler i think. (creating automatically the task at the specific taskindex if the slot is free)

Sending valuecount is a nice idea, using this field i can use one generic struct without decoding stuff, only changing the buffer length to include the necessary ones at the end. :)

And what about the 1st introduction message which may include Nodename, Node MAC for generating a list of nodes? It will be a nice feature to know what nodes was active and when...