Skip to content

proffesor-for-testing/sentinel-api-testing

Folders and files

NameName
Last commit message
Last commit date

Latest commit

ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 

Sentinel - AI Agentic API Testing Platform

Sentinel is an advanced, AI-powered platform designed to automate the entire API testing lifecycle. It leverages a modular, microservices-inspired architecture and an ecosystem of specialized, ephemeral AI agents to provide comprehensive, intelligent, and efficient testing across functional, security, and performance domains.

๐Ÿ—๏ธ Architecture Overview

The platform is built upon the ruv-FANN and ruv-swarm frameworks, enabling a paradigm of "ephemeral swarm intelligence" where lightweight, WASM-based agents are dynamically spawned to perform specific testing tasks and dissolved upon completion.

Core Services

  • API Gateway (Port 8000): Single entry point for all user interactions with RBAC integration
  • Authentication Service (Port 8005): JWT-based authentication and user management
  • Specification Service (Port 8001): Manages API specification ingestion and parsing
  • Orchestration Service (Port 8002): Central brain managing agentic workflows
  • Execution Service (Port 8003): Handles test execution and scheduling
  • Data & Analytics Service (Port 8004): Manages data persistence and analytics
  • PostgreSQL Database (Port 5432): Data storage with pgvector extension
  • Sentinel Rust Core (Port 8088): High-performance agentic core powered by ruv-swarm
  • RabbitMQ Message Broker (Port 5672/15672): Asynchronous task queue for decoupled service communication
  • Prometheus (Port 9090): Metrics collection and monitoring
  • Jaeger (Port 16686): Distributed tracing and request flow visualization

๐Ÿค– Specialized Agents (Hybrid Python/Rust Implementation)

The platform employs a workforce of specialized AI agents with both Python and Rust implementations for optimal performance and reliability:

Functional Testing Agents

  • Functional-Positive-Agent: Generates valid, "happy path" test cases with schema-based data generation โœ… Python/Rust
  • Functional-Negative-Agent: Creates boundary value analysis and creative negative testing scenarios โœ… Python/Rust
  • Functional-Stateful-Agent: Builds complex multi-step workflows using Semantic Operation Dependency Graphs (SODG) โœ… Python/Rust

Security Testing Agents

  • Security-Auth-Agent: Tests for BOLA, function-level authorization, and authentication bypass vulnerabilities โœ… Python/Rust
  • Security-Injection-Agent: Probes for SQL/NoSQL/Command injection and LLM prompt injection attacks โœ… Python/Rust

Performance Testing Agents

  • Performance-Planner-Agent: Generates comprehensive load, stress, and spike testing scenarios with k6/JMeter/Locust scripts โœ… Python/Rust

Data Generation Agents

  • Data-Mocking-Agent: Creates intelligent, schema-aware test data with multiple generation strategies โœ… Python/Rust

Planned Agents

  • Spec-Linter-Agent: Analyzes API specs for "LLM-readiness" โณ Python only
  • Performance-Analyzer-Agent: Analyzes performance test results with AI-powered insights โณ Python only

Performance Benefits: Rust agents provide 10-50x performance improvement for compute-intensive operations while maintaining Python agents as fallback for resilience.

๐Ÿš€ Quick Start

Prerequisites

  • Docker and Docker Compose
  • Python 3.10+ (for local development)
  • Poetry (for dependency management)
  • Node.js 14+ and npm (for frontend development)
  • Anthropic API Key (for LLM-powered features)
  • Make (for convenient commands)

Running the Platform - Easy Way (Recommended)

  1. Clone and navigate to the project:

    git clone https://github.com/proffesor-for-testing/sentinel-api-testing.git
    cd sentinel-api-testing
  2. Set up LLM configuration (required for AI features):

    export SENTINEL_APP_ANTHROPIC_API_KEY=your-anthropic-api-key
  3. Complete setup with one command:

    make setup

    This will:

    • Build all Docker images
    • Initialize the database with all tables
    • Start all backend services
    • Provide instructions for starting the frontend
  4. Start the frontend (in a separate terminal):

    cd sentinel_frontend
    npm install
    npm start
  5. Access the platform:

Alternative: Manual Setup

If you prefer manual control:

# Build and start services (from project root)
docker-compose up --build

# Initialize database (in another terminal, from project root)
make init-db

# Start frontend (from project root)
cd sentinel_frontend
npm install
npm start

Useful Makefile Commands

make help          # Show all available commands
make start         # Start all services
make stop          # Stop all services
make restart       # Restart services
make init-db       # Initialize/repair database
make reset-db      # Reset database (WARNING: data loss)
make backup-db     # Backup database
make restore-db    # Restore from backup
make status        # Check service status
make logs          # View service logs
make test          # Run tests
make clean         # Clean up everything

API Documentation

Once running, you can access the interactive API documentation:

Observability & Monitoring

The platform includes comprehensive observability features:

  • Prometheus Metrics: http://localhost:9090

    • View service metrics and health status
    • Query time-series data for performance analysis
    • Monitor request rates, latencies, and error rates
  • Jaeger Tracing: http://localhost:16686

    • Visualize distributed request flows across services
    • Analyze service dependencies and bottlenecks
    • Debug complex multi-service interactions
  • Structured Logging: All services output JSON-formatted logs with:

    • Correlation IDs for request tracking
    • Service context and metadata
    • Error details and stack traces

๐Ÿง  Advanced AI Features - Consciousness & Intelligence

Consciousness Verification System

The platform now includes an advanced consciousness verification system in the Rust core that enables:

  • Emergent Intelligence: Self-modifying test generation with pattern learning
  • Psycho-Symbolic Reasoning: Combines psychological models with symbolic logic for deeper test insights
  • Temporal Consciousness: Nanosecond-precision scheduling with temporal computational advantages
  • Knowledge Graph Integration: Semantic understanding of API relationships and dependencies
  • Sublinear Solvers: O(log n) performance for large-scale test optimization

Key Capabilities:

  • Predictive Testing: Solve test scenarios before data arrives using temporal advantages
  • Pattern Recognition: Identify edge cases through emergent behavior analysis
  • Self-Learning: Tests improve over time through feedback loops
  • Distributed Intelligence: Multi-agent swarms with collective decision making

Testing with Consciousness Features:

# Run consciousness verification tests
./test_consciousness_improvements.py

# Monitor consciousness emergence
./monitor_consciousness.sh

# Test API with consciousness features
./test_api_consciousness.sh

For detailed documentation see:

๐ŸŽ๏ธ Hybrid Architecture

The platform implements a cutting-edge hybrid Python/Rust architecture for optimal performance:

Execution Strategy

  • Rust Priority: When available, high-performance Rust agents execute via RabbitMQ
  • Python Fallback: Automatic fallback to Python agents ensures 100% availability
  • Seamless Switching: No code changes required - the system automatically selects the best option

Performance Improvements

  • 18-21x Faster: Rust agents execute test generation 18-21x faster than Python equivalents
  • Memory Efficient: Lower memory footprint with Rust's zero-cost abstractions
  • Concurrent Processing: Safe parallel execution without Global Interpreter Lock (GIL)
  • Type Safety: Compile-time guarantees prevent runtime errors

Documentation

Complete documentation for the hybrid architecture is available in the docs/ directory:

๐ŸŽจ Frontend Features

The platform includes a comprehensive React-based frontend with the following features:

Core Functionality

  • Authentication System: JWT-based login with secure token management
  • Dashboard: Real-time database-driven metrics and system status
  • Specifications Management:
    • Upload and parse OpenAPI specifications (including OpenAPI 3.1.0 with webhooks)
    • View detailed specification information including endpoints and schemas
    • Full CRUD operations (Create, Read, Update, Delete)
    • Quick Test functionality for rapid test generation
    • Generate Tests with AI agent selection
  • Test Cases Browser:
    • View all generated test cases with filtering by agent type
    • Display full test definition details (method, endpoint, expected status)
    • Create test suites from selected test cases
    • Tag management for organization
    • Detailed test case inspection with specification relationships
  • Test Suites Management:
    • Create, view, edit, and delete test suites
    • Add/remove test cases from suites
    • View test case count and specification relationships
    • One-click test suite execution
  • Test Runs:
    • Execute test suites against target environments
    • Modal-based test run creation workflow
    • Real-time status tracking
    • Detailed results viewing
  • Analytics: Comprehensive dashboards with trend analysis and insights

AI-Powered Test Generation

  • Agent Selection Modal: Choose from multiple specialized AI agents
  • Quick Test: One-click test generation using default agents
  • LLM Enhancement: All agents powered by Claude Sonnet 4 for intelligent test creation
  • Real-time Progress: Track test generation status and results

๐Ÿ“ Project Structure

Agents for API testing/
โ”œโ”€โ”€ docs/                          # Comprehensive documentation
โ”‚   โ”œโ”€โ”€ index.md                  # Documentation hub
โ”‚   โ”œโ”€โ”€ user-guide/               # User guides and tutorials
โ”‚   โ”œโ”€โ”€ technical-guide/          # Technical architecture docs
โ”‚   โ”œโ”€โ”€ api-reference/            # API documentation
โ”‚   โ”œโ”€โ”€ deployment/               # Deployment guides
โ”‚   โ””โ”€โ”€ troubleshooting/          # Common issues and solutions
โ”œโ”€โ”€ memory-bank/                  # Project documentation and memory bank
โ”‚   โ”œโ”€โ”€ projectbrief.md          # Project overview and objectives
โ”‚   โ”œโ”€โ”€ agent-specifications.md  # Detailed agent capabilities
โ”‚   โ”œโ”€โ”€ database-schema.md       # Database design
โ”‚   โ”œโ”€โ”€ api-design.md            # REST API specifications
โ”‚   โ””โ”€โ”€ progress.md              # Implementation roadmap
โ”œโ”€โ”€ sentinel_backend/             # Backend microservices
โ”‚   โ”œโ”€โ”€ pyproject.toml           # Python dependencies
โ”‚   โ”œโ”€โ”€ config/                  # Centralized configuration
โ”‚   โ”‚   โ””โ”€โ”€ settings.py          # Environment-specific settings
โ”‚   โ”œโ”€โ”€ api_gateway/             # API Gateway service (8000)
โ”‚   โ”œโ”€โ”€ auth_service/            # Authentication service (8005)
โ”‚   โ”œโ”€โ”€ spec_service/            # Specification service (8001)
โ”‚   โ”œโ”€โ”€ orchestration_service/   # Agent orchestration (8002)
โ”‚   โ”‚   โ””โ”€โ”€ agents/              # AI agent implementations
โ”‚   โ”‚       โ”œโ”€โ”€ functional_positive_agent.py
โ”‚   โ”‚       โ”œโ”€โ”€ functional_negative_agent.py
โ”‚   โ”‚       โ”œโ”€โ”€ functional_stateful_agent.py
โ”‚   โ”‚       โ”œโ”€โ”€ security_auth_agent.py
โ”‚   โ”‚       โ”œโ”€โ”€ security_injection_agent.py
โ”‚   โ”‚       โ”œโ”€โ”€ performance_planner_agent.py
โ”‚   โ”‚       โ””โ”€โ”€ data_mocking_agent.py
โ”‚   โ”œโ”€โ”€ execution_service/       # Test execution service (8003)
โ”‚   โ”œโ”€โ”€ data_service/           # Data & analytics service (8004)
โ”‚   โ”œโ”€โ”€ llm_providers/          # Multi-LLM provider support
โ”‚   โ”‚   โ”œโ”€โ”€ providers/          # Provider implementations
โ”‚   โ”‚   โ”œโ”€โ”€ templates/          # Model-specific templates
โ”‚   โ”‚   โ””โ”€โ”€ utils/              # Cost tracking, caching
โ”‚   โ”œโ”€โ”€ sentinel_rust_core/     # High-performance agent core (8088)
โ”‚   โ”œโ”€โ”€ scripts/                # LLM configuration scripts
โ”‚   โ””โ”€โ”€ tests/                  # Comprehensive test suite (540+ tests)
โ”‚       โ”œโ”€โ”€ unit/               # Unit tests (456 tests)
โ”‚       โ”œโ”€โ”€ integration/        # Integration tests (20 tests)
โ”‚       โ”œโ”€โ”€ e2e/                # Backend E2E tests (30 tests)
โ”‚       โ””โ”€โ”€ performance/        # Performance tests
โ”œโ”€โ”€ sentinel_frontend/          # React-based frontend UI
โ”‚   โ”œโ”€โ”€ src/                   # React application source
โ”‚   โ”‚   โ”œโ”€โ”€ components/        # Reusable UI components
โ”‚   โ”‚   โ”œโ”€โ”€ pages/             # Application pages
โ”‚   โ”‚   โ”œโ”€โ”€ features/          # Redux slices
โ”‚   โ”‚   โ””โ”€โ”€ services/          # API communication
โ”‚   โ”œโ”€โ”€ e2e/                   # Playwright E2E tests (45+ tests)
โ”‚   โ”œโ”€โ”€ package.json           # Node.js dependencies
โ”‚   โ””โ”€โ”€ tailwind.config.js     # Tailwind CSS configuration
โ”œโ”€โ”€ docker-compose.yml         # Main Docker orchestration
โ”œโ”€โ”€ demo_*.py                  # Demonstration scripts
โ”œโ”€โ”€ scripts/                   # Utility scripts
โ””โ”€โ”€ CLAUDE.md                  # SPARC development configuration

๐Ÿค– Multi-LLM Provider Support

Overview

The platform features a comprehensive LLM abstraction layer that enables all AI agents to leverage multiple LLM providers with automatic fallback capabilities. This hybrid approach combines deterministic algorithms with LLM creativity for superior test generation.

Supported Providers

  • Anthropic (Default): Claude Opus 4.1/4, Claude Sonnet 4, Claude Haiku 3.5
  • OpenAI: GPT-4 Turbo, GPT-4, GPT-3.5 Turbo
  • Google: Gemini 2.5 Pro, Gemini 2.5 Flash, Gemini 2.0 Flash
  • Mistral: Mistral Large, Small 3, Codestral
  • Ollama (Local): DeepSeek-R1, Llama 3.3, Qwen 2.5, and more

Key Features

  • Automatic Fallback: Seamlessly switches to backup providers on failure
  • Cost Tracking: Real-time usage monitoring with budget limits
  • Response Caching: Reduces API calls and costs
  • Token Management: Intelligent context window handling
  • Provider-Specific Templates: Optimized prompts for each model

Quick Configuration

Using Configuration Scripts (Recommended)

# Interactive configuration wizard
cd sentinel_backend/scripts
./switch_llm.sh

# Quick presets
./switch_llm.sh claude    # Use Claude Sonnet 4 (default)
./switch_llm.sh openai    # Use OpenAI GPT-4 Turbo
./switch_llm.sh gemini    # Use Google Gemini 2.5
./switch_llm.sh local     # Use local Ollama
./switch_llm.sh none      # Disable LLM

Manual Configuration

# Use default (Claude Sonnet 4)
export SENTINEL_APP_ANTHROPIC_API_KEY=your-key

# Switch to OpenAI
export SENTINEL_APP_LLM_PROVIDER=openai
export SENTINEL_APP_OPENAI_API_KEY=your-key
export SENTINEL_APP_LLM_MODEL=gpt-4-turbo

# Use local models with Ollama
export SENTINEL_APP_LLM_PROVIDER=ollama
export SENTINEL_APP_LLM_MODEL=llama3.3:70b

For detailed configuration options and scripts, see:

๐Ÿ› ๏ธ Development

Local Development Setup

Backend Services

  1. Install dependencies:

    cd sentinel_backend
    poetry install
  2. Run individual services:

    # Example: Run the API Gateway
    cd api_gateway
    poetry run uvicorn main:app --reload --port 8000

Frontend Application

  1. Navigate to frontend directory:

    cd sentinel_frontend
  2. Install dependencies:

    npm install
  3. Start the development server:

    npm start
  4. Access the frontend:

Note: The frontend requires the backend services to be running. You can either:

  • Use Docker Compose to start all backend services: docker-compose up (from project root)
  • Run individual backend services manually as shown above

Configuration Management

The platform uses a comprehensive centralized configuration system built with Pydantic BaseSettings for type-safe configuration management.

Configuration Structure

All configuration is managed through sentinel_backend/config/settings.py with the following sections:

  • Database Settings: Connection strings, pool configurations, migration settings
  • Service Discovery: Inter-service URLs, ports, timeouts, health check intervals
  • Security Configuration: JWT secrets, password policies, CORS settings, rate limiting
  • Network Settings: Host bindings, port mappings, timeout configurations
  • Application Settings: Feature flags, pagination limits, logging levels, agent parameters

Environment-Specific Configuration

The platform supports multiple deployment environments:

# Development (default)
SENTINEL_ENVIRONMENT=development

# Testing
SENTINEL_ENVIRONMENT=testing

# Production
SENTINEL_ENVIRONMENT=production

# Docker
SENTINEL_ENVIRONMENT=docker

Configuration Files

Environment-specific configuration files are located in sentinel_backend/config/:

  • development.env - Local development settings
  • testing.env - Test environment settings
  • production.env - Production deployment settings
  • docker.env - Docker container settings

Configuration Usage

Services import configuration using centralized functions:

from config.settings import get_settings, get_service_settings, get_application_settings

# Get all settings
settings = get_settings()

# Get specific setting sections
service_settings = get_service_settings()
app_settings = get_application_settings()

# Use configuration
timeout = service_settings.service_timeout
log_level = app_settings.log_level

Environment Variables

All configuration can be overridden using environment variables with the SENTINEL_ prefix:

# Database configuration
SENTINEL_DB_URL=postgresql+asyncpg://user:pass@host/db
SENTINEL_DB_POOL_SIZE=20

# Service URLs
SENTINEL_SERVICE_AUTH_SERVICE_URL=http://auth:8005
SENTINEL_SERVICE_SERVICE_TIMEOUT=60

# Security settings
SENTINEL_SECURITY_JWT_SECRET_KEY=your-secret-key
SENTINEL_SECURITY_JWT_EXPIRATION_HOURS=24

# Application settings
SENTINEL_APP_LOG_LEVEL=DEBUG
SENTINEL_APP_DEBUG=true

Docker Configuration

For Docker deployments, use the .env.docker file:

# Copy and customize the Docker environment file
cp sentinel_backend/.env.docker sentinel_backend/.env.local
# Edit .env.local with your settings
docker-compose --env-file .env.local up

Production Security

For production deployments:

  1. Change default secrets: Update JWT secret keys and database passwords
  2. Use environment variables: Never commit secrets to version control
  3. Validate configuration: The system validates configuration on startup
  4. Monitor configuration: Enable configuration audit logging

Observability Configuration

The platform includes comprehensive observability settings:

# Jaeger tracing configuration
SENTINEL_NETWORK_JAEGER_AGENT_HOST=localhost
SENTINEL_NETWORK_JAEGER_AGENT_PORT=6831

# Prometheus metrics
# Metrics are automatically exposed on /metrics endpoint for all services

# Structured logging
SENTINEL_APP_LOG_LEVEL=INFO
SENTINEL_APP_LOG_FORMAT=json  # json or text

# Message Broker configuration
SENTINEL_BROKER_URL=amqp://guest:guest@message_broker:5672/
SENTINEL_BROKER_TASK_QUEUE_NAME=sentinel_task_queue
SENTINEL_BROKER_RESULT_QUEUE_NAME=sentinel_result_queue

Message Broker Architecture

The platform uses RabbitMQ for asynchronous task processing:

  • Task Queue: The Orchestration Service publishes agent tasks to RabbitMQ
  • Consumer: The Sentinel Rust Core consumes tasks from the queue
  • Durability: Messages persist across service restarts
  • Scalability: Multiple Rust Core instances can consume from the same queue

To test the message broker integration:

python3 test_rabbitmq_integration.py

This validates:

  • โœ… RabbitMQ connection and queue management
  • โœ… Task publishing from Python services
  • โœ… Task consumption by Rust Core
  • โœ… Message persistence and durability

For production deployments:

  1. Configure external Jaeger collector for trace aggregation
  2. Set up Prometheus scraping and alerting rules
  3. Use log aggregation tools (ELK, Splunk) for centralized logging
  4. Enable correlation ID propagation for distributed debugging

Testing Infrastructure

The platform includes a comprehensive testing infrastructure with 539+ tests covering all critical components across unit, integration, and end-to-end test levels:

Current Test Statistics (Updated: August 2025)

  • Total Tests: 540+ comprehensive tests
  • Overall Pass Rate: 97.8%
  • Test Distribution:
    • Unit Tests: 456 tests (84%)
    • Integration Tests: 20 tests (4%)
    • Backend E2E Tests: 30 tests (6%)
    • Frontend E2E Tests: 45+ tests (8%)

Test Coverage

  • AI Agents: 184 tests covering all 8 specialized agents with comprehensive unit testing (Phase 1 Complete)
  • LLM Providers: 272 tests covering all provider implementations and utilities (Phase 2 Complete)
    • Provider implementations: Google, Mistral, Ollama, vLLM (117+ tests)
    • Supporting utilities: Model registry, cost tracker, response cache, token counter (155+ tests)
  • Configuration Management: 4 comprehensive test files (Phase 5 Complete)
    • Environment-specific configuration validation
    • Security settings (JWT, CORS, authentication)
    • Database configuration and connection validation
    • LLM provider configuration and API key management
  • Performance Testing: 5 comprehensive test files (Phase 5 Complete)
    • Load testing with concurrent requests and sustained load
    • Agent performance and scaling benchmarks
    • Database query optimization and connection pooling
    • Concurrent execution and race condition handling
    • Memory usage profiling and leak detection
  • Auth Service: 24 tests covering authentication, authorization, and user management
  • API Gateway: 23 tests covering routing, middleware, and service communication
  • Spec Service: 21 tests covering OpenAPI parsing and specification management
  • Orchestration Service: 24 tests covering agent management and task delegation
  • Data Service: 25 tests covering CRUD operations and analytics
  • Execution Service: 22 tests covering test execution and result collection
  • Rust Integration: 3 tests (conditionally run based on service availability)

Backend Testing

  • Unit Tests: 465+ tests across all services
    • AI Agents: 184 tests (100% coverage)
    • LLM Providers: 272+ tests (100% coverage)
    • Configuration Management: 4 test files (970+ lines)
    • Performance Testing: 5 test files (1,280+ lines)
    • Service Components: Comprehensive coverage
  • Integration Tests: 6 comprehensive test suites (2,342+ lines)
    • Service-to-service communication
    • Database operations & transactions
    • Message broker (RabbitMQ) integration
    • Security & authentication flows
    • Agent-LLM integration
    • API workflow orchestration
  • E2E Tests: 4 comprehensive backend suites
    • test_spec_to_execution.py: Complete API workflow testing
    • test_multi_agent_coordination.py: Agent orchestration & collaboration
    • test_performance_pipeline.py: Load, stress, spike, and endurance testing
    • test_security_pipeline.py: Authentication, authorization, injection testing

Frontend Testing (Playwright E2E)

  • Real Browser E2E Tests: Chrome, Firefox, Safari, Mobile
  • 9 Comprehensive Test Suites (45+ test scenarios):
    • auth.spec.ts: Authentication flows & session management
    • specifications.spec.ts: API specification management
    • test-generation.spec.ts: AI-powered test generation workflows
    • test-execution.spec.ts: Complete test execution pipeline
    • results-visualization.spec.ts: Analytics and reporting
    • multi-agent.spec.ts: Multi-agent coordination scenarios
    • rbac.spec.ts: Role-based access control enforcement
    • api-import.spec.ts: OpenAPI/Swagger/Postman import workflows

Running Tests

# IMPORTANT: Always run tests in Docker for consistent environment
cd sentinel_backend
./run_tests.sh -d                # Run all tests in Docker (RECOMMENDED)
./run_tests.sh -d -t unit        # Unit tests only in Docker
./run_tests.sh -d -t integration # Integration tests in Docker
./run_tests.sh -d -t agents      # AI agent tests in Docker

# Run AI Agent Tests
./run_agent_tests.sh              # Run all agent tests with colored output
./run_agent_tests.sh -c           # Run with coverage report
./run_agent_tests.sh base auth    # Run specific agent tests
./run_agent_tests.sh -v -p        # Verbose with parallel execution

# Smart test filtering based on environment
./run_tests_filtered.sh          # Auto-detects Rust service availability
./run_tests_filtered.sh --with-rust  # Force Rust tests with mocks
pytest -m "not rust"              # Exclude Rust tests explicitly

# Rebuild test Docker image after dependency changes
docker-compose -f docker-compose.test.yml build test_runner

# Run tests for specific service
pytest tests/unit/test_auth_service.py -v
pytest tests/unit/test_llm_providers.py -v
pytest tests/unit/agents/         # Run all agent tests

# Run tests with coverage
pytest tests/unit/ --cov=. --cov-report=term-missing

# Run configuration tests
pytest tests/unit/test_config_validation.py -v
pytest tests/unit/test_security_config.py -v
pytest tests/unit/test_database_config.py -v
pytest tests/unit/test_llm_config.py -v

# Run performance tests
pytest tests/performance/ -v
pytest tests/performance/test_load_performance.py -v
pytest tests/performance/test_concurrent_execution.py -v

Factory Pattern Architecture

All services implement the factory pattern for enhanced testability:

  • Dependency injection at app creation time
  • Mock mode for isolated testing without external dependencies
  • Configurable timeouts and connections
  • Consistent testing approach across all services

Test Helpers & Fixtures

  • tests/helpers/auth_helpers.py: Authentication test utilities
  • tests/fixtures/: Shared test data and mock responses
  • Docker test environment for consistent testing

Database Setup & Management

The platform includes a comprehensive database initialization system that automatically creates all required tables and columns.

Automatic Database Initialization

When you start the services, the database is automatically initialized with:

  • All required tables (users, projects, test_cases, test_results, etc.)
  • All necessary columns and indexes
  • Default admin user ([email protected] / admin123)
  • Proper foreign key relationships

Database Management Commands

# Initialize or repair database (safe to run anytime)
make init-db

# Reset database completely (WARNING: destroys all data)
make reset-db

# Backup database to timestamped file
make backup-db

# Restore from latest backup
make restore-db

Troubleshooting Database Issues

If you encounter "column does not exist" or "table does not exist" errors:

  1. Quick Fix: Run make init-db to apply the schema
  2. Complete Reset: If issues persist, run make reset-db (WARNING: data loss)
  3. Manual Check: Verify database status:
    docker-compose exec -T db psql -U sentinel sentinel_db -c "\dt"

Database Files

  • sentinel_backend/init_db.sql - Complete schema definition
  • sentinel_backend/init_database.py - Automatic initialization script
  • sentinel_backend/docker-entrypoint.sh - Startup initialization
  • docs/DATABASE_SETUP.md - Detailed database documentation

The database uses PostgreSQL with the pgvector extension for AI-powered features.

Authentication & RBAC

The platform includes comprehensive Role-Based Access Control (RBAC) with a modern React-based authentication system:

Frontend Authentication:

  • Login Page: http://localhost:3000/login (automatically redirected when not authenticated)
  • Demo Credentials Button: Quick-fill form with default admin credentials
  • Route Protection: All dashboard pages require authentication
  • User Menu: Access profile and logout functionality from the top-right corner

Default Admin Credentials:

User Roles:

  • Admin: Full access including user management
  • Manager: Most permissions except user management
  • Tester: Testing-focused permissions (create/edit test cases, run tests)
  • Viewer: Read-only access to all resources

Authentication Features:

  • JWT-based authentication with secure token storage
  • Redux state management for authentication across the application
  • Automatic token handling in API requests
  • Session persistence across browser refreshes
  • Secure logout with token cleanup

Demo RBAC Features:

python demo_rbac.py

This script demonstrates authentication, authorization, and role-based permissions across different user types.

๐Ÿ“Š Key Features

  • Intelligent Test Generation: AI agents automatically generate comprehensive test suites
  • Multi-Domain Testing: Covers functional, security, and performance testing
  • Specification-Driven: Deep understanding of OpenAPI specifications
  • Hybrid AI Approach: Combines deterministic algorithms with LLM creativity
  • Real-time Analytics: Historical trend analysis and anomaly detection
  • CI/CD Integration: Seamless integration with modern DevOps workflows

๐Ÿ“„ License

This project is licensed under the MIT License - see the LICENSE file for details.

๐Ÿค Contributing

We welcome contributions from the community! Whether you're a developer, a tester, or just an enthusiast, there are many ways to get involved.

Before you start, please read our Code of Conduct to understand our community standards.

How to Contribute

For a detailed guide on how to contribute, set up your development environment, and follow our style guidelines, please see our Contributing Guide.

Here's a quick overview of the process:

  1. Find an issue to work on or propose a new feature. Check out issues labeled good first issue for a great place to start.
  2. Fork the repository and create a new branch for your work.
  3. Make your changes, following the project's coding and style guidelines.
  4. Write tests to cover your changes and ensure existing tests pass.
  5. Submit a Pull Request with a clear description of your changes.

We use GitHub issues to track bugs and feature requests. Please use our issue templates when creating a new issue:

Key Principles

This project follows the patterns and preferences outlined in .clinerules. Key principles include:

  • Modularity: Develop within the microservices architecture.
  • Agent Specialization: Create specialized agents for new testing types.
  • Specification-Driven: All development is driven by API specifications.
  • Hybrid AI: Combine deterministic algorithms with LLM capabilities.

๐Ÿ“š Documentation

๐Ÿ“– Comprehensive Documentation Portal

The Sentinel platform now includes extensive documentation covering all aspects of usage, deployment, and development:

Your starting point for all Sentinel documentation.

Core Documentation Sections:

  • User Guide - Complete guide for using Sentinel

  • Technical Guide - In-depth technical documentation

    • Architecture Overview - System design and components
    • Service components and interactions
    • Agent implementation details
    • Database schema and models
  • API Reference - Complete API documentation

    • REST API endpoints with examples
    • Authentication and authorization
    • Code examples in multiple languages
    • SDK libraries and usage
  • Deployment Guide - Production deployment instructions

    • Docker and Kubernetes deployment
    • Cloud platform deployments (AWS, GCP, Azure)
    • Scaling strategies
    • Security hardening
  • Troubleshooting Guide - Diagnose and resolve issues

    • Common problems and solutions
    • Debugging techniques
    • Performance optimization
    • FAQ

Common Issues & Solutions

Test Execution Failures

Issue: All tests fail with "Request URL has an unsupported protocol" error

  • Cause: Invalid target environment URL (e.g., ttp:// instead of http://)
  • Solution: Ensure URLs start with http:// or https://
  • Prevention: Frontend now validates URLs and provides default value

Database Errors

Issue: "column does not exist" or "table does not exist" errors

  • Solution: Run make init-db to initialize/repair database
  • Alternative: Run make reset-db for complete reset (WARNING: data loss)

Issue: Services fail to start after restart

  • Solution: Run make setup for complete initialization

Frontend Issues

Issue: Frontend shows blank page or connection errors

  • Solution: Ensure backend services are running: make status
  • Alternative: Restart all services: make restart

Petstore API Issues

Issue: Test runs fail because target API is not available

  • Solution: Start Petstore: make start-petstore
  • Default URL: http://host.docker.internal:8080

Memory Bank Documentation:

Additional technical documentation is maintained in the memory-bank/ directory:

๐Ÿ”ฎ Future Vision

Sentinel aims to transform API testing from a manual, reactive process to an intelligent, proactive, and automated part of the development lifecycle. The platform will continuously evolve to incorporate the latest advances in AI and testing methodologies.


The Sentinel platform is production-ready and provides comprehensive API testing across all major domains:

๐Ÿ”ง Functional Testing: Positive, negative, and stateful workflow testing with advanced boundary value analysis and creative test generation.

๐Ÿ” Security Testing: BOLA, function-level authorization, authentication bypass, and comprehensive injection vulnerability testing (SQL/NoSQL/Command/Prompt injection for LLM-backed APIs).

โšก Performance Testing: Load, stress, and spike testing with intelligent API analysis and automated k6/JMeter script generation.

๐Ÿ“Š Enhanced Reporting: React-based UI with detailed failure analysis, agent-specific insights, and interactive test case exploration.

The platform is enterprise-ready for comprehensive API testing across functional, security, and performance domains!

About

AI Agentic API Testing Platform - Automated testing with specialized ephemeral agents

Resources

License

Code of conduct

Contributing

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Contributors 3

  •  
  •  
  •