Merge pull request #40 from OS2iot/feature/docs-for-mqtt-support

Feature/docs for mqtt support

Author: fcv-iteratorIt

.gitignore

@@ -8,3 +8,4 @@ docProps
 word
 customXml
 [Content_Types].customXml
+.idea

source/exit-strategy/exit-strategy.rst

@@ -1,16 +1,28 @@
 Exit strategy
 =============
 
-There are two primary ways of systematically exporting the data from OS2iot:
+There are three primary ways of systematically exporting the data from OS2iot:
 
-1. Using the REST API
+1. Using the Export 
+
+    a. Ideal if you just need to get all the information about devices in an application.
+
+2. Using the REST API
 
     a. Ideal if you want to import/change it into something else, and OS2iot is still running.
 
-2. Exporting the database
+3. Exporting the database
 
     a. If OS2iot is no longer running, the postgres database will contain all the data from the system.
 
+Export using CSV export 
+-----------------------
+
+It is possible, when on the details page for an application, to get a csv file containing all the devices in the application. 
+
+This csv file is formatted to be compatible with the bulk import feature, button located right next to it, to make it possible to reimport the devices into a different application.
+
+
 Export using the REST API
 -------------------------
 The REST API is automatically exported from the backend via swagger. 

source/external-interface-design/external-interface-design.rst

@@ -20,7 +20,7 @@ Integration patterns
 Internal systems
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-Internally, OS2iot consists of four components shown in this figure.
+Internally, OS2iot consists of five components shown in this figure.
 
 |image2|
 
@@ -30,7 +30,7 @@ Two different patterns are used between the internal components:
    | Used between the frontend, backend and database.
 
 -  | **Publish-subscribe**
-   | Used in communication between the backend and MQTT broker for Chirpstack.
+   | Used in communication between the backend and MQTT broker for Chirpstack and the internal MQTT broker.
 
 External systems
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -50,11 +50,13 @@ IoT devices
 
 Figure 3 - IoT device integration overview
 
-All IoT device integrations except LoRaWAN use callbacks to send data to
+All IoT device integrations except LoRaWAN and MQTT use callbacks to send data to
 OS2iot using HTTPS in a request-response manner. Device management for 
 Sigfox (i.e. adding and modifying IoT devices) is done by
 sending requests to Sigfox's Cloud backend.
 
+MQTT devices uses the MQTT protocol to communicate with the broker.
+
 Data from Chirpstack devices are sent to OS2iot through Chirpstack using
 MQTT, where Chirpstack is the publisher and OS2iot is the subscriber.
 Device management is done from OS2iot by sending gRPC requests to
@@ -308,6 +310,40 @@ over the network by a device.
 
 .. _update-existing-device-1:
 
+MQTT
+^^^^
+
+There are two kinds of MQTT devices available. MQTT-publisher and MQTT-subscriber. These two devices works in different matters which will be described below.
+
+MQTT-publisher
+~~~~~~~~~~~~~~
+The MQTT-publisher device will make it possible for a physical device to communicate with the internal OS2IoT mosquitto broker.
+The MQTT-publisher is created in the OS2IoT backend and is created with the credentials that the device needs for communicating with the internal broker.
+
+The MQTT-publisher device can either be created with username/password or credentials. If the publisher is created with username/password it will use port 8885, and if created with certificate it will use port 8884.
+
+When a physical MQTT device will publish some data, then OS2IoT will check for the specific topic that the device is publishing to in the database, and if the topic is set in the database, it will process the data.
+If a MQTT-publisher device with the specific topic isn't created then the broker won't be able to find it in the database and therefore it will reject the data.
+
+The specific topic for the created device will be :code:`device/organizationID/applicationID/deviceID`.
+
+
+
+
+MQTT-subscriber
+~~~~~~~~~~~~~~~
+
+The MQTT-subscriber uses the MQTT protocol to subscribe to a topic on an external MQTT broker. A client is created in the OS2IoT backend.
+This client will connect to the external MQTT broker using the provided URL, port and authentication, and then subscribe to data on the provided topic.
+
+OS2IoT doesn't have any knowlegde of the external broker so it's totally up to the user to provide the different inputs.
+If the input isn't valid and a connection can't be made to the external broker, a flag will be set in the database which tells OS2IoT that the connection can't be made and then OS2IoT will stop trying to connect to the external broker.
+
+If the inputs from the user IS valid, then a connection will be made and the device will listen to any updates from the broker.
+
+The MQTT-subscriber device has the possibility to use either certificate or username/password to a external broker if needed.
+
+
 Sigfox
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
@@ -514,7 +550,7 @@ See `the seperate page for KOMBIT adgangsstyring <../kombit-adgangsstyring/kombi
 
 .. |image1| image:: media/image5.png
 .. |image2| image:: media/image6.png
-.. |image3| image:: media/image7.png
+.. |image3| image:: media/image5.png
 .. |image4| image:: media/image8.png
 .. |image5| image:: media/image9.png
 .. |image6| image:: media/image10.png

source/external-interface-design/media/image5.png

@@ -245,7 +245,13 @@ OS2IoT-backend takes several environment variables as configuration, if these ar
 +-------------------------------+------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------+
 | METADATA_SAVED_COUNT          | Maximum number of message metadata to store from an IoT device                                       | :code:`20`                                                                              |
 +-------------------------------+------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------+
-| DEVICE_STATS_INTERVAL_IN_DAYS | How many days back to fetch IoT device statistics                                                    | :code:`29`                                                                              |
+| MQTT_BROKER_HOSTNAME          | The hostname of the MQTT broker.                                                                     | :code:`localhost`                                                                       |
++-------------------------------+------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------+
+| MQTT_SUPER_USER_PASSWORD      | The password for the internal MQTT listener.                                                         | :code:`SuperUser`                                                                       |
++-------------------------------+------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------+
+| ENCRYPTION_SYMMETRIC_KEY      | A symmetric key that is used for encrypting                                                          | :code:`SecretKey`                                                                       |
++-------------------------------+------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------+
+| CA_KEY_PASSWORD               | The password for the Certificate Authority key.                                                      | :code:`os2iot`                                                                          |
 +-------------------------------+------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------+
 
 Logs levels
@@ -268,3 +274,48 @@ Defaults are set in :code:`OS2IoT-frontend/src/environments/environment.ts`
 +-------------------------------+--------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------+
 | TABLE_PAGE_SIZE               | Default page size of tables                                                                                  | :code:`25`                                                                              |
 +-------------------------------+--------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------+
+
+OS2IoT-Mosquitto broker
+^^^^^^^^^^^^^^^^^^^^^^^
+
+To get the mosquitto broker working, you have to create some certificates and update some values. These following steps is done with Windows. If you use linux, then write :code:`sudo` before the commands.
+
+Prerequisites: openssl installed and accesible from path
+
+Generate files:
+
+   1. Open the command prompt in administrator mode.
+
+   2. Create a certificate authority(CA) key with this command: :code:`openssl genrsa -des3 -out ca.key 2048`. You will be prompted to enter a password. It's very important that you save this password, since it will be used later.
+
+   3. Create the CA certificate with this command: :code:`openssl req -new -x509 -days 1826 -key ca.key -out ca.crt`. You will be asked to enter the password from the step before. After this, you will be prompted to enter informations. These values are not important, except one: "Common name". Common name HAS to be the ip/hostname of your broker.
+
+   4. Create the server key (for the broker) with the command: :code:`openssl genrsa -out server.key 2048`
+
+   6. Create the server signing request with the command: :code:`openssl req -new -out server.csr -key server.key`. You will be prompted to enter some informations. These values are not important, except one: "Common name". Common name HAS to be the ip/hostname of your broker. The rest of the values should not be exact the same as in step 4.
+
+   7. Create the server certificate (that is signed by the CA) with this command: :code:`openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -days 360`. You will be prompted to enter the password from step 3.
+
+If you want to get docker container with mosquitto running, then follow these steps:
+
+   1. Place the generated files, ca.key, ca.crt, server.key and server.crt from the above steps in the folder "OS2IoT-docker/configuration/mosquitto-broker-os2iot". You don't need the server.csr.
+
+   2. Open the mosquitto-os2iot.conf file placed in OS2IoT-docker/configuration/mosquitto-broker-os2iot in a text editor and update the values to match your database.
+
+   3. Copy the files ca.crt and ca.key and place them in OS2IoT-backend/resources.
+
+   4. Update the :code:`MQTT_BROKER_HOSTNAME` with the ip/hostname that you used for step 4 and 6, and :code:`CA_KEY_PASSWORD` with the password that you entered in step 3 in the docker-compose.yml file placed in OS2IoT-docker.
+
+If you want to use kubernetes to host mosquitto then you need some futher steps.
+
+Prerequisites: kubectl installed and accesible from path
+
+   1. Open a command prompt in administrator mode.
+
+   2. Create a secret with the server.key and server.crt with the command: :code:`kubectl create secret generic server-keys --from-file=server.key=path/to/server.key --from-file=server.crt=path/to/server.crt`. Replace path/to/ with the path to your server.key and server.crt, created in the steps above.
+
+   3. Create a secret with the ca.crt and ca.key with the command: :code:`kubectl create secret generic ca-keys --from-file=ca.crt=path/to/ca.crt --from-file=ca.key=path/to/ca.key`. Replace path/to/ with the path to your server.key and server.crt, created in the steps above.
+
+   4. Update the empty values in OS2IoT-docker/helm/charts/mosquitto-os2iot/values.yaml
+
+   5. Update the :code:`MQTT_BROKER_HOSTNAME` with the ip/hostname that you used for step 4 and 6 in the steps above, and :code:`CA_KEY_PASSWORD` with the password that you entered in step 3 in the steps above, in the file "OS2IoT-docker/helm/charts/os2iot-backend/deployment.yaml".

source/external-interface-design/media/image6.png

@@ -10,13 +10,15 @@ The following sequence diagram illustrates how data is processed when it is rece
 Data ingestion
 --------------
 
-OS2iot supports three types of devices:
+OS2iot supports four types of devices:
 
     1. LoRaWAN devices (using Chirpstack)
 
     2. SigFox devices
 
     3. Generic HTTP devices
+    
+    4. MQTT devices
 
 OS2iot ingests data for each of these device types in a slightly different way:
 
@@ -57,6 +59,30 @@ The endpoint :code:`/api/v1/receive-data?apiKey=<guid>` is used for this, where
 If the API-key is valid and corresponds to a device in OS2iot, then the response will be 204 No Content, and the message is further processed. 
 If the API-key is unknown, then the resposne will be 403 Bad Request with the message: :code:`MESSAGE.DEVICE-DOES-NOT-EXIST`
 
+MQTT
+^^^^
+
+OS2IoT supports two kinds of MQTT devices. An MQTT-publisher and an MQTT-subscriber. 
+
+OS2IoT supports two kinds of authorization for MQTT devices: Username/password and certificate.
+
+MQTT-publisher
+~~~~~~~~~~~~~~
+
+When an MQTT-publisher is created, the credentials for connecting to the internal MQTT-broker are generated. 
+These consists of a URL, a port and a topic for the device to send data to. 
+If the authorization type is certificate then the required certificates will be generated. If username/password is used these are manually entered.
+
+When a known MQTT-publisher sends data to the topic assigned to it, the message will be further processed. 
+If a device attempts to send to a different topic the message will be discarded.
+
+MQTT-subscriber
+~~~~~~~~~~~~~~~
+
+When an MQTT-subscriber is created a connection to the external broker is made using the entered credentials. 
+When a message is received by the external broker on the topic configured, OS2IoT will also receive the message and further process it.
+
+
 Payload transformation and storage
 ----------------------------------
 

source/external-interface-design/media/image7.png

@@ -93,13 +93,18 @@ DataTargetType
 ~~~~~~~~~~~~~~~~~
 
 1. HttpPush
+2. Fiware
+3. MQTT
+4. OpenDataDK
 
 IoTDeviceType
 ~~~~~~~~~~~~~~~~~
 
 1. GenericHttp
 2. LoRaWAN
 3. SigFox
+4. MQTTBroker
+5. MQTTSubscriber
 
 ErrorCodes 
 ~~~~~~~~~~~~~~~~~
@@ -184,5 +189,10 @@ SigFoxPayloadType
 5. RadioPlanningFrame
 6. Sensitv2
 
+AuthenticationType
+~~~~~~~~~~~~~~~~~~
+1. PASSWORD
+2. CERTIFICATE
+
 .. |image1| image:: ./media/image4.png