Danube

A lightweight and scalable Cloud-Native Messaging Platform with Cloud Object Storage (S3/GCS/Azure)

Danube is an open-source distributed messaging broker platform inspired by Apache Pulsar, designed to be cloud-native and cost-effective. Built with a Write-Ahead Log (WAL) architecture and persistent object storage integration, Danube delivers sub-second dispatch with cloud economics.

🚀 Get Started with Danube

Quick Start with Docker Compose - Deploy a cluster in seconds:

Create a directory and download the required files:

mkdir danube-docker && cd danube-docker

Download the docker-compose and broker configuration file:

curl -O https://raw.githubusercontent.com/danube-messaging/danube/main/docker/docker-compose.yml

curl -O https://raw.githubusercontent.com/danube-messaging/danube/main/docker/danube_broker.yml

Start the danube cluster:

docker-compose up -d

This launches a complete Danube cluster with:

• 2 High-Availability Brokers for topics failover
• ETCD for distributed metadata management
• MinIO S3-Compatible Storage for cloud-ready persistence
• Automatic bucket creation and configuration

Test the setup:

Produce messages with reliable delivery

docker exec -it danube-cli danube-cli produce \
  --service-addr http://broker1:6650 \
  --topic "/default/persistent-topic" \
  --count 100 \
  --message "Persistent message" \
  --reliable

Consume messages from the topic

docker exec -it danube-cli danube-cli consume \
  --service-addr http://broker1:6650 \
  --topic "/default/persistent-topic" \
  --subscription "persistent-sub" \
  --sub-type exclusive

📦 Run with Docker (choose one):

• Minimal stack (brokers + dependencies): docker/README.md
• Stack with Admin UI (UI + gateway + Prometheus): docker/danube_with_ui/README.md

📖 Complete Docker Setup Guide →

Architecture

🏗️ Cluster & Broker Characteristics

• Stateless brokers: Metadata in ETCD and data in WAL/Object Storage
• Horizontal scaling: Add brokers in seconds; partitions rebalance automatically
• Leader election & HA: Automatic failover and coordination via ETCD
• Rolling upgrades: Restart or replace brokers with minimal disruption
• Multi-tenancy: Isolated namespaces with policy controls
• Security-ready: TLS/mTLS support in Admin and data paths

Cloud-Native by Design - Danube's architecture separates compute from storage, enabling:

🌩️ Write-Ahead Log + Cloud Persistence

• Sub-millisecond producer acknowledgments via local WAL
• Asynchronous background uploads to S3/GCS/Azure object storage
• Automatic failover with shared cloud state
• Infinite retention without local disk constraints

⚡ Performance & Scalability

• Hot path optimization: Messages served from in-memory WAL cache
• Stream per subscription: WAL + cloud storage from selected offset
• Multi-cloud support: AWS S3, Google Cloud Storage, Azure Blob, MinIO

┌─────────────┐    ┌──────────────┐    ┌─────────────────┐
│  Producers  │───▶│ Danube WAL   │───▶│ Object Storage  │
└─────────────┘    │ (Sub-ms ACK) │    │ (S3/GCS/Azure)  │
                   └──────────────┘    └─────────────────┘
                          │                      ▲
                          ▼                      │
┌─────────────┐    ┌──────────────┐    ┌─────────────────┐
│  Consumers  │◀───│ Stream Reader│◀───│ Background      │
└─────────────┘    │ (WAL + Cloud)│    │ Uploader        │
                   └──────────────┘    └─────────────────┘

Core Capabilities

📨 Message Delivery

• Topics: Partitioned and non-partitioned with automatic load balancing
• Reliable Dispatch: At-least-once delivery with configurable storage backends
• Non-Reliable Dispatch: High-throughput, low-latency for real-time scenarios
• Flexible Schemas: Bytes, String, Int64, JSON with automatic serialization

🔄 Subscription Models

• Exclusive: Single consumer per subscription
• Shared: Load-balanced message distribution across consumers
• Failover: Automatic consumer failover with ordered delivery

🛠️ Developer Experience

• Multi-language clients: Rust, Go
• CLI Tools: Message publishing and consumption
• Admin CLI: Cluster, namespace, and topic management

Community & Clients

Official Clients

• Rust Client - Full-featured async client with examples
• Go Client - Production-ready client with examples

Community Contributions

Contributions in Python, Java, JavaScript, and other languages are welcome! Join our growing ecosystem.

Development & Contribution

Get involved - Danube is actively developed with new features added regularly.

🐛 Report Issues | 💡 Request Features | 📖 Development Guide

Project Structure

• danube-broker - Core messaging platform
• danube-persistent-storage - WAL and cloud storage integration
• danube-client - Async Rust client library
• danube-cli - Command-line producer/consumer tools
• danube-admin-cli - Cluster management utilities
• danube-admin-gateway - HTTP/JSON BFF for the Admin UI

---