Merge pull request #313 from mapswipe/dev

v2.1.10

Author: Hagellach37

docs/source/cli.md

@@ -1,10 +1,15 @@
 # Command Line Interface
 
+THIS DOCUMENT IS OUTDATED: Please use the `--help` flag of the CLI.
+
+---
+
 This document describes how to use the command line interface of MapSwipe Worker.
 
 In our current deployment setup the commands of the MapSwipe Workers CLI are hard-coded in the Docker-Compose File.
 
 You can run these commands also using docker-compose:
+
 ```
 docker-compose run mapswipe_workers mapswipe_workers --help
 ```

docs/source/configuration.md

@@ -1,146 +1,79 @@
-<!--
-
-Then set up a Service Account Key file:
-1. Open [Google Cloud Service Accounts](https://console.cloud.google.com/iam-admin/serviceaccounts)
-2. Create a new Service Account Key file:
-    * set name (e.g. *dev-mapswipe-workers*)
-    * add roles, (e.g. `Storage Admin` and `Firebase Admin`) or use pre-defined role instead (e.g. `Custom Firebase Developer`)
-3. Download Key as file:
-    * select `.json` and save
-
+# Configuration Reference
 
+Most of the configuration is stored in environment variables.
+At the root of the GitHub repository an example file (`example.env`) with all possible configuration variables exists. To get started copy this file to `.env` and fill in missing variables. Once done source this file to make variables accessible as environment variables: `source .env` to either be used by docker-compose during deployment setup or by MapSwipe Workers directly.
 
+In following chapters configuration values and keys are discussed for each part of the MapSwipe Back-end.
 
 
-## Firebase
+## MapSwipe Workers
 
-Firebase is a central part of MapSwipe. In our setup we use *Firebase Database*, *Firebase Database Rules* and *Firebase Functions*. In the documentation we will refer to two elements:
-1. `your_project_id`: This is the name of your Firebase project (e.g. *dev-mapswipe*)
-2. `your_database_name`: This is the name of your Firebase database. It is very likely that this will be the same as your Firebase project name as well.)
+All configuration values for MapSwipe Workers are stored in environment variables.
 
-The `mapswipe_workers` module uses the [Firebase Python SDK](https://firebase.google.com/docs/reference/admin/python) to access *Firebase Database* services as administrator, you must generate a Service Account Key file in JSON format. For this we use the previously generated Service Account Key. (Check the *Google APIs and Services Credentials* section again if you don't have it.) Copy the file to `mapswipe_workers/config/serviceAccountKey.json`.
+Required environment variables are:
+- FIREBASE_API_KEY
+- FIREBASE_DB
+- GOOGLE_APPLICATION_CREDENTIALS
+- POSTGRES_DB
+- POSTGRES_HOST
+- POSTGRES_PASSWORD
+- POSTGRES_PORT
+- POSTGRES_USER
 
-The `mapswipe_workers` module further uses the [Firebase Database REST API](https://firebase.google.com/docs/reference/rest/database) to access *Firebase Database* either as a normal user or project manager.
+Mandatory environment variables are:
+- SLACK_CHANNEL
+- SLACK_TOKEN
+- SENTRY_DSN
 
-For both things to work you need to add your `database_name` in the configuration file. For the the REST API add also the previously generated *mapswipe_workers* api key. (Check the *Google APIs & Services Credentials* section again if you don't have it.) The firebase section in `mapswipe_workers/config/configuration.json` should look like this now:
+For satellite imagery access to at least one provider is needed. Define the API key as environment variable:
+- IMAGE_BING_API_KEY
+- IMAGE_MAPBOX_API_KEY
+- IMAGE_MAXA_PREMUIM_API_KEY
+- IMAGE_MAXAR_STANDARD_API_KEY
+- IMAGE_ESRI_API_KEY
+- IMAGE_ESRI_BETA_API_KEY
 
-```json
-"firebase": {
-  "database_name": "your_database_name",
-  "api_key": "mapswipe_workers_api_key"
-}
-```
+In addition to get access to Firebase a Service Account Key is required.
+The path the Service Account Key is defined in:
+- GOOGLE_APPLICATION_CREDENTIALS
 
-The `manager_dashboard` module uses the [Firebase JavaScript client SDK](https://firebase.google.com/docs/database/web/start) to access *Firebase Database* service as authenticated as MapSwipe user with project manager credentials. Add the previously generated *manager-dashboard* api key. (Check the *Google APIs & Services Credentials* section again if you don't have it.) Project-id refers to the name of your Firebase project (e.g. dev-mapswipe). The firebaseConfig in `mapswipe_dashboard/js/app.js` should look like this now:
+> Notes: When deploying using `docker` or `docker-compose` `POSTGRES_HOST` should have the value `postgres` and the Service Account Key (`serviceAccountKey.json`) should be copied to `mapswipe_workers/serviceAccountKey.json` as described in detail in [Deployment](deployment.md).
 
-```javascript
-var firebaseConfig = {
-    apiKey: "manager_dashboard_api_key",
-    authDomain: "your_project_id.firebaseapp.com",
-    databaseURL: "https://your_project_id.firebaseio.com",
-    storageBucket: "your_project_id.appspot.com"
-  };
-```
 
-The `firebase` module uses the [Firebase Command Line Interface (CLI) Tools](https://github.com/firebase/firebase-tools) to access *Firebase Database Rules* and *Firebase Functions*. You need a firebase token. Here's how you generate it:
-1. On a PC with a browser install the Firebase Command Line Tools ([https://firebase.google.com/docs/cli/](https://firebase.google.com/docs/cli/#install_the_firebase_cli))
-2. Run `firebase login:ci` to generate a Firebase Token.
-3. Save the Firebase Token to `.env` at the root of the cloned MapSwipe Backend repository: `echo "FIREBASE_TOKEN=your_token" >> .env`
-4. You should have an entry for the firebase token in your `.env` now:
-
-```bash
-FIREBASE_TOKEN="your_token"
-```
-
--->
-
-# Configuration Reference
+## Postgres
 
-This document provides details on all required configuration files:
+Required environment variables are:
+- POSTGRES_DB
+- POSTGRES_HOST
+- POSTGRES_PASSWORD
+- POSTGRES_PORT
+- POSTGRES_USER
 
-- `.env file`
-    * postgres password
-    * firebase token
-    * wal-g google storage prefix
-- `mapswipe_workers/config/configuration.json`
-    * firebase: api-key, databaseName
-    * postgres: host, port, database, username, password
-    * imagery: urls, api keys
-    * slack: token, username, channel
-    * sentry: dsn value
-- `mapswipe_workers/config/serviceAccountKey.json`
-    * check if file exists
-- `manager_dashboard/manager_dashboard/js/app.js`
-    * firebase: authDomain, apiKey, databaseUrl, storageBucket
-- `nginx/nginx.conf`
-    * server name
-    * ssl certificates, ssl certificates key
 
-You can run the script `test_config.py` to check if you set all the needed variables and file. The script will test the following files:
+### Postgres Backup
 
+On details of how the back-up works please refer to [Postgres Backup](backup.md).
 
-## .env
+Required environment variables are:
+- WALG_GS_PREFIX
 
-The environment file (`.env`) contains all variables needed by services running in Docker container.
+To gain access to Google Cloud Storage another Service Account Key is needed. Again refer to [Postgres Backup](backup.md) on how to create this file.
+The Service Account Key (`serviceAccountKey.json`) should be saved to `postgres/serviceAccountKey.json`
 
-```.env
-POSTGRES_PASSWORD=password
 
-# Google Cloud Storage path for backups of Postgres
-WALG_GS_PREFIX=gs://x4m-test-bucket/walg-folder
+### Manager Dashboard
 
-# Token for deployment of Firebase Rules and Functions
-FIREBASE_TOKEN=firebase_token
+`manager_dashboard/manager_dashboard/js/app.js`
 
-GOOGLE_APPLICATION_CREDENTIALS=google_application_credentials
 ```
-
-
-## MapSwipe Workers - Configuration
-
-`mapswipe_workers/config/configuration.json`:
-
-```json
-{
-    "postgres": {
-        "host": "postgres",
-        "port": "5432",
-        "database": "mapswipe",
-        "username": "mapswipe_workers",
-        "password": "your_mapswipe_db_password"
-    },
-    "firebase": {
-        "database_name": "your_firebase_database_name",
-        "api_key": "your_firebase_api_key"
-    },
-    "imagery":{
-        "bing": {
-            "api_key": "your_bing_api_key",
-            "url": "http://t0.tiles.virtualearth.net/tiles/a{quad_key}.jpeg?g=854&mkt=en-US&token={key}"
-        },
-        "digital_globe": {
-            "api_key": "your_digital_globe_api_key",
-            "url": "https://api.mapbox.com/v4/digitalglobe.nal0g75k/{z}/{x}/{y}.png?access_token={key}"
-        },
-        "sinergise": {
-            "api_key": "your_sinergise_api_key",
-            "url": "https://services.sentinel-hub.com/ogc/wmts/{key}?request=getTile&tilematrixset=PopularWebMercator256&tilematrix={z}&tilecol={x}&tilerow={y}&layer={layer}"
-        }
-    },
-    "slack": {
-        "token": "your_slack_token",
-        "channel": "your_slack_channel",
-        "username": "your_slack_username"
-    },
-    "sentry": {
-        "dsn": "your_sentry_dsn_value"
-    }
-}
+TODO
 ```
 
 
 ## NGINX
 
+`nginx/nginx.conf`:
+
 ```
 server {
     listen 80;
@@ -181,19 +114,3 @@ server {
     }
 }
 ```
-
-## Postgres Backup
-We back up the Postgres database using [Wal-G](https://github.com/wal-g/wal-g) and [Google Cloud Storage](https://console.cloud.google.com/storage). You could also set it up using another cloud storage service.
-
-First, create a new cloud storage bucket:
-1. Google Cloud Platform > Storage > Create Bucket
-2. Choose a bucket name, e.g. `your_project_id_postgres_backup`
-3. Select storage location > `Multi-Region` > `eu`
-4. Select storage class > `Coldline`
-
-We need to access Google Cloud Storage. For this we use the previously generated Service Account Key. (Check the *Google APIs and Services Credentials* section again if you don't have it.) Copy the file to `postges/serviceAccountKey.json`.
-
-```bash
-GCS_Link_URL=https://console.cloud.google.com/storage/browser/your_project_id_postgres_backup
-GCS_Link_for_gsutil=gs://your_project_id_postgres_backup
-```

docs/source/deployment_overview.md

@@ -13,6 +13,6 @@ MapSwipe utilizes a bunch of Google Cloud services:
     - Create a database: `> Develop > Database > Create Database`
 - [Google Cloud Storage](https://cloud.google.com/storage/) for backup of Postgres
 
-[Configuration and Credentials]() describes all needed configuration and credentials.
+[Configuration](configuration.md) describes all needed configuration and credentials.
 
-[Installation]() describes step-by-step how to setup the backend for the first time.
+[Installation](installation.md) describes step-by-step how to setup the backend for the first time.

docs/source/dev_setup.md

@@ -1,6 +1,6 @@
 # Development Setup
 
-In this document some tips and workflows for development are loosely collected. Those are independend of the production setup using Docker. A working Firebase Project (including Firebase Functions and Database Rules) is presupposed.
+In this document some tips and workflows for development are loosely collected. Those are independent of the production setup using Docker. A working Firebase Project (Including Firebase Functions and Database Rules) is presupposed. Get in touch to get access to an existing Firebase Project for development purposes.
 
 
 ## Installation
@@ -21,22 +21,28 @@ git checkout dev
 
 ### Configuration
 
-MapSwipe Workers looks for configuration in `~/.config/mapswipe_workers`. (XDG Base Directory Specification is respected). It expects three files:
+All configurations values are stored in environment variables.
 
-- `configuration.json`
-- `serviceAccountKey.json`
-- `logging.cfg`
+Please refer to the documentation on [Configuration](configuration.md) for further details.
 
-Please refer to the [configuration](configuration.md) and [setup](setup.md) documentation for further details.
 
-In addition the data directory for MapSwipe Workers needs to be created:
-`mkdir --parents .local/share/mapswipe_workers`
+### Keys
+
+To gain access to Firebase MapSwipe workers needs a Service Account Key (`serviceAccountKey.json`). Create on in Firebase and store it where ever you want. Make sure to store the path to the key in the `GOOGLE_APPLICATION_CREDENTIALS` environment variable.
+
+
+### Directories
+
+MapSwipe Workers needs access to a directory for data and logs.
+To create the directory run `mkdir --parents ~/.local/share/mapswipe_workers`.
+
+> Note: XDG Base Directory Specification is respected
 
 
 ### Install MapSwipe Workers Python Package
 
 1. Create a Python virtual environment with `system-site-packages` option enabled to get access to GDAL/OGR Python packages
-2. Activate the vitrual environment.
+2. Activate the virtual environment.
 3. Install MapSwipe Workers using pip.
 4. Run it.
 
@@ -50,11 +56,11 @@ mapswipe_workers --help
 
 ## Postgres
 
-Setup a local Postgres instance for MapSwipe Workers using the provided Dockerfile.
+Setup a local Postgres instance for MapSwipe Workers using the for development purposes provided Dockerfile.
 
 ```bash
 cd postgres/`
-docker build -t mapswipe_postgres .
+docker build -t mapswipe_postgres Dockerfile-dev
 docker run -d -p 5432:5432 --name mapswipe_postgres -e POSTGRES_DB=mapswipe -e POSTGRES_USER=mapswipe_workers -e POSTGRES_PASSWORD=your_password mapswipe_postgres
 ```
 
@@ -98,9 +104,9 @@ Firebase functions are used by Mapswipe Workers to increment/decrement or calcul
 - user.timeSpentMapping
 - user.contibutions{.projectId.groupId.{timestamp, startTime, endTime}}
 
-The functions will be triggert by incoming results from the Mapswipe App.
+Those functions will be directly or indirectly triggered by incoming results from the MapSwipe App.
 
-By using firebase functions those attributes can be calculated in real-time and be accessed by users immediately. The use of those functions also reduces the data-transfer between the Firebase Realtime Database and Mapswipe Workers.
+By using Firebase functions those attributes can be calculated in real-time and be accessed by users immediately. The use of those functions also reduces the data-transfer between the Firebase Realtime Database and Mapswipe Workers.
 
 On how to setup development enviroment and how to deploy functions to the Firebase instance please refer to the official [Guide on Cloud Function for Firebase](https://firebase.google.com/docs/functions/get-started).
 For more information refer to the official [Reference on Cloud Function for Firebase](https://firebase.google.com/docs/reference/functions/). For example function take a look at this [GitHub repository](https://github.com/firebase/functions-samples).

docs/source/installation.md

@@ -1,15 +1,15 @@
 # Installation
 
-This document describes how to setup all the parts of the MapSwipe back-end for the first time.
+This document describes how to setup all the parts of the MapSwipe back-end production.
 
-Please take also a look at our [Configuration Reference](https://mapswipe-workers.readthedocs.io/en/dev/configuration.html) which is a summary of all configurations and keys needed for deployment.
+Please take also a look at our [Configuration Reference](configuration.md) which is a summary of all configurations and keys needed for deployment.
 
 1. Firebase
 2. Postgres
 3. MapSwipe Workers
 4. API
 5. Manager Dashboard
-6. Lets Encrypt and NEGIX as proxy
+6. Lets Encrypt and NGINX as proxy
 
 For all those setups our main repository is required:
 

docs/source/notes.md

@@ -0,0 +1,48 @@
+Then set up a Service Account Key file:
+1. Open [Google Cloud Service Accounts](https://console.cloud.google.com/iam-admin/serviceaccounts)
+2. Create a new Service Account Key file:
+    * set name (e.g. *dev-mapswipe-workers*)
+    * add roles, (e.g. `Storage Admin` and `Firebase Admin`) or use pre-defined role instead (e.g. `Custom Firebase Developer`)
+3. Download Key as file:
+    * select `.json` and save
+
+
+## Firebase
+
+Firebase is a central part of MapSwipe. In our setup we use *Firebase Database*, *Firebase Database Rules* and *Firebase Functions*. In the documentation we will refer to two elements:
+1. `your_project_id`: This is the name of your Firebase project (e.g. *dev-mapswipe*)
+2. `your_database_name`: This is the name of your Firebase database. It is very likely that this will be the same as your Firebase project name as well.)
+
+The `mapswipe_workers` module uses the [Firebase Python SDK](https://firebase.google.com/docs/reference/admin/python) to access *Firebase Database* services as administrator, you must generate a Service Account Key file in JSON format. For this we use the previously generated Service Account Key. (Check the *Google APIs and Services Credentials* section again if you don't have it.) Copy the file to `mapswipe_workers/config/serviceAccountKey.json`.
+
+The `mapswipe_workers` module further uses the [Firebase Database REST API](https://firebase.google.com/docs/reference/rest/database) to access *Firebase Database* either as a normal user or project manager.
+
+For both things to work you need to add your `database_name` in the configuration file. For the the REST API add also the previously generated *mapswipe_workers* api key. (Check the *Google APIs & Services Credentials* section again if you don't have it.) The firebase section in `mapswipe_workers/config/configuration.json` should look like this now:
+
+```json
+"firebase": {
+  "database_name": "your_database_name",
+  "api_key": "mapswipe_workers_api_key"
+}
+```
+
+The `manager_dashboard` module uses the [Firebase JavaScript client SDK](https://firebase.google.com/docs/database/web/start) to access *Firebase Database* service as authenticated as MapSwipe user with project manager credentials. Add the previously generated *manager-dashboard* api key. (Check the *Google APIs & Services Credentials* section again if you don't have it.) Project-id refers to the name of your Firebase project (e.g. dev-mapswipe). The firebaseConfig in `mapswipe_dashboard/js/app.js` should look like this now:
+
+```javascript
+var firebaseConfig = {
+    apiKey: "manager_dashboard_api_key",
+    authDomain: "your_project_id.firebaseapp.com",
+    databaseURL: "https://your_project_id.firebaseio.com",
+    storageBucket: "your_project_id.appspot.com"
+  };
+```
+
+The `firebase` module uses the [Firebase Command Line Interface (CLI) Tools](https://github.com/firebase/firebase-tools) to access *Firebase Database Rules* and *Firebase Functions*. You need a firebase token. Here's how you generate it:
+1. On a PC with a browser install the Firebase Command Line Tools ([https://firebase.google.com/docs/cli/](https://firebase.google.com/docs/cli/#install_the_firebase_cli))
+2. Run `firebase login:ci` to generate a Firebase Token.
+3. Save the Firebase Token to `.env` at the root of the cloned MapSwipe Backend repository: `echo "FIREBASE_TOKEN=your_token" >> .env`
+4. You should have an entry for the firebase token in your `.env` now:
+
+```bash
+FIREBASE_TOKEN="your_token"
+```

mapswipe_workers/mapswipe_workers/auth.py

@@ -60,7 +60,7 @@ def query(self, query, data=None):
         self._db_connection.commit()
         self._db_cur.close()
 
-    def copy_from(self, f, table, columns):
+    def copy_from(self, f, table, columns=None):
         self._db_cur = self._db_connection.cursor()
         self._db_cur.copy_from(f, table, columns=columns)
         self._db_connection.commit()