Client based Azure OpenAI Simple Load Balancing

A simple round-robin load balancer for Azure OpenAI services using Azure Storage Tables for state management.

Overview

This project implements a simple client based round-robin load balancing that distributes requests across multiple Azure OpenAI service instances. It uses Azure Storage Tables to maintain consistent state across the multiple instances.

Architecture

Note:

This diagram shows a simplified targeted version of the solution architecture (within the same region). The actual bicep implementation creates two separate Azure OpenAI Cognitive Services accounts in different regions (East US 2 and Sweden Central) rather than a single Azure AI Foundry hub due to our playground limited Azure OpenAI quotes.

Prerequisites

• Azure CLI installed and authenticated
• Python 3.7 or higher
• Azure subscription with OpenAI services deployed

Installation

1. Clone this repository

2. Create Azure resource group:

   az group create \
     --name rg-azure-openai-realtime-demo \
     --location eastus2

3. Deploy Azure resources:

   az deployment group create \
     --resource-group rg-azure-openai-realtime-demo \
     --template-file main.bicep \
     --name main

4. Install Python dependencies:

   pip install -r requirements.txt

Configuration

The load balancer automatically configures itself using Azure deployment outputs:

from config import setup_from_deployment_outputs

backend1, backend2, storage_connection_string = setup_from_deployment_outputs()

Example Implementation

See usage_example.py for a complete example of how to use the load balancer in your application.

Project Structure

├── config.py              # Configuration and Azure deployment 
├── simple_roundrobin.py   # Main load balancer implementation
├── usage_example.py       # Example usage and testing
├── requirements.txt       # Python dependencies
├── main.bicep             # Azure infrastructure template

Azure Infrastructure

The project includes a Bicep template (main.bicep) for deploying the required Azure resources:

• Multiple Azure OpenAI service instances
• Azure Storage Account for state management

This is just a quick demo ..

This project is designed for demonstration purpose as several security and production considerations have been simplified,for example:

Security Considerations:

• Secrets in Outputs - The bicep template exposes API keys and storage account keys in deployment outputs (listKeys() functions). This triggers a warning Outputs should not contain secrets
• For Production - Use Azure Key Vault to store secrets and implement managed identities for authentication
• Access Keys - In production, retrieve keys programmatically when needed rather than exposing them in outputs