AlertHub Feedback Pipeline

A feedback processing system that automatically categorises and analyses feedback from KoboToolbox using a comprehensive Early Warning System (EWS) framework, stores results in PostgreSQL with built‑in deduplication, and creates cases in EspoCRM. Multilingual support with automatic detection and translation.

Features

• EWS framework categorisation: 4 pillars (Risk Knowledge, Forecasting, Warning, Response) plus cross‑cutting Competencies and Values
• Robust deduplication: PostgreSQL submission_id unique constraint with ON CONFLICT DO NOTHING
• EspoCRM sync: Per‑submission check by cSubmissionid; creates only missing cases
• Multilingual: Automatic language detection with confidence; translation uses source=auto when uncertain
• Smart analysis: Urgency, complexity, sentiment, keywords
• Exports: Clean CSV export and an interactive Plotly HTML analytics summary

Quick start

1) Install dependencies

pip install -r requirements.txt

2) Environment

Create .env with your settings:

KoboToolbox

KOBO_API_TOKEN=your_token KOBO_FORM_ID=your_form_id KOBO_API_URL=https://kf.kobotoolbox.org/api/v2/

EspoCRM

ESPOCRM_URL=https://your-espocrm.com/api/v1/ ESPOCRM_USER=your_username ESPOCRM_PASSWORD=your_password

PostgreSQL

PG_HOST=localhost PG_PORT=5432 PG_DB=your_database PG_USER=your_username PG_PASSWORD=your_password

3) Run the pipeline

python main.py

4) Export data to CSV

python export_csv.py

Exports to exports/feedback_data.csv (overwritten each run).

5) Build the analytics summary

python export_analytics_summary.py

6) Testing

Run the test suite with the built-in runner:

python run_tests.py

• Skip stress tests:

python run_tests.py --skip-stress

• Run a specific test file or pattern:

python run_tests.py -k 'test_performance.py'
python run_tests.py -k 'test_stress_processing.py'

Project structure

alerthub-feedback-pipeline/ ├── main.py # Entry point ├── feedback_processor/ │ └── processor.py # Kobo → PostgreSQL → EspoCRM ├── utils/ │ ├── categoriser.py # EWS categorisation and analysis │ └── multilingual_handler.py # Language detection/translation ├── export_csv.py # Single-file CSV export ├── export_analytics_summary.py # Plotly analytics summary generator ├── run_tests.py # Test runner (CLI flags) ├── tests/ # Test suite (unit, integration, performance, stress) ├── requirements.txt # Dependencies ├── exports/ # Output folder (auto-created) └── .env # Local configuration