Skip to content

Commit 10ae2b9

Browse files
chrishayukchrishayuk
committed
improved observavility docs
1 parent 779c890 commit 10ae2b9

File tree

8 files changed

+1544
-2
lines changed

8 files changed

+1544
-2
lines changed

‎README.md‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -188,7 +188,7 @@ asyncio.run(main())
188188
|----------|----------------|
189189
| 📘 [CONFIGURATION.md](docs/CONFIGURATION.md) | **All config knobs & defaults**: ToolProcessor options, timeouts, retry policy, rate limits, circuit breakers, caching, environment variables |
190190
| 🚨 [ERRORS.md](docs/ERRORS.md) | **Error taxonomy**: All error codes, exception classes, error details structure, handling patterns, retryability guide |
191-
| 📊 [OBSERVABILITY.md](OBSERVABILITY.md) | **Metrics & tracing**: OpenTelemetry setup, Prometheus metrics, spans reference, PromQL queries |
191+
| 📊 [OBSERVABILITY.md](docs/OBSERVABILITY.md) | **Metrics & tracing**: OpenTelemetry setup, Prometheus metrics, spans reference, PromQL queries |
192192
| 🔌 [examples/hello_tool.py](examples/hello_tool.py) | **60-second starter**: Single-file, copy-paste-and-run example |
193193
| 🎯 [examples/](examples/) | **20+ working examples**: MCP integration, OAuth flows, streaming, production patterns |
194194

@@ -200,7 +200,7 @@ asyncio.run(main())
200200
| 🔌 **Connect to external tools** | MCP integration (HTTP/STDIO/SSE) | [MCP Integration](#5-mcp-integration-external-tools) |
201201
| 🛡️ **Production deployment** | Timeouts, retries, rate limits, caching | [CONFIGURATION.md](docs/CONFIGURATION.md) |
202202
| 🔒 **Run untrusted code safely** | Subprocess isolation strategy | [Subprocess Strategy](#using-subprocess-strategy) |
203-
| 📊 **Monitor and observe** | OpenTelemetry + Prometheus | [OBSERVABILITY.md](OBSERVABILITY.md) |
203+
| 📊 **Monitor and observe** | OpenTelemetry + Prometheus | [OBSERVABILITY.md](docs/OBSERVABILITY.md) |
204204
| 🌊 **Stream incremental results** | StreamingTool pattern | [StreamingTool](#streamingtool-real-time-results) |
205205
| 🚨 **Handle errors reliably** | Error codes & taxonomy | [ERRORS.md](docs/ERRORS.md) |
206206

@@ -1173,6 +1173,8 @@ pip install opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp prom
11731173
uv pip install chuk-tool-processor --group observability
11741174
```
11751175

1176+
> **⚠️ SRE Note**: Observability packages are **optional**. If not installed, all observability calls are no-ops—your tools run normally without tracing/metrics. Zero crashes, zero warnings. Safe to deploy without observability dependencies.
1177+
11761178
**Quick Start: See Your Tools in Action**
11771179

11781180
```python
@@ -1634,10 +1636,13 @@ async def create_secure_processor():
16341636
Check out the [`examples/`](examples/) directory for complete working examples:
16351637

16361638
### Getting Started
1639+
- **60-second hello**: `examples/hello_tool.py` - Absolute minimal example (copy-paste-run)
16371640
- **Quick start**: `examples/quickstart_demo.py` - Basic tool registration and execution
16381641
- **Execution strategies**: `examples/execution_strategies_demo.py` - InProcess vs Subprocess
16391642
- **Production wrappers**: `examples/wrappers_demo.py` - Caching, retries, rate limiting
16401643
- **Streaming tools**: `examples/streaming_demo.py` - Real-time incremental results
1644+
- **Streaming tool calls**: `examples/streaming_tool_calls_demo.py` - Handle partial tool calls from streaming LLMs
1645+
- **Schema helper**: `examples/schema_helper_demo.py` - Auto-generate schemas from typed tools (Pydantic → OpenAI/Anthropic/MCP)
16411646
- **Observability**: `examples/observability_demo.py` - OpenTelemetry + Prometheus integration
16421647

16431648
### MCP Integration (Real-World)

‎docs/GRAFANA-DASHBOARD.md‎

Lines changed: 341 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,341 @@
1+
# Grafana Dashboard Guide
2+
3+
This guide shows what the CHUK Tool Processor Grafana dashboard displays and how to use it.
4+
5+
## Dashboard Overview
6+
7+
The dashboard (`docs/grafana-dashboard.json`) provides complete observability for CHUK Tool Processor in a single screen with 10 panels.
8+
9+
## Dashboard Panels
10+
11+
### 1. Total Calls/Sec (Gauge)
12+
**Shows**: Current rate of tool calls per second across all tools
13+
14+
**PromQL Query**:
15+
```promql
16+
sum(rate(tool_executions_total[1m]))
17+
```
18+
19+
**What it tells you**: How busy your tool processor is right now
20+
- **High values**: Heavy usage, ensure adequate resources
21+
- **Low/zero values**: System idle or potential issues
22+
- **Sudden drops**: May indicate problems upstream
23+
24+
---
25+
26+
### 2. Error Rate % (Gauge)
27+
**Shows**: Percentage of tool calls failing
28+
29+
**PromQL Query**:
30+
```promql
31+
(sum(rate(tool_executions_total{status="error"}[5m])) /
32+
sum(rate(tool_executions_total[5m]))) * 100
33+
```
34+
35+
**What it tells you**: Overall system health
36+
- **< 1%**: Healthy system
37+
- **1-5%**: Investigate error patterns
38+
- **> 5%**: Critical - check errors immediately
39+
40+
---
41+
42+
### 3. Cache Hit Rate % (Gauge)
43+
**Shows**: Percentage of tool calls served from cache
44+
45+
**PromQL Query**:
46+
```promql
47+
(sum(rate(tool_cache_operations_total{result="hit"}[5m])) /
48+
sum(rate(tool_cache_operations_total{operation="lookup"}[5m]))) * 100
49+
```
50+
51+
**What it tells you**: Cache effectiveness
52+
- **High (>60%)**: Cache working well, saving resources
53+
- **Low (<20%)**: Consider increasing cache TTL or checking cache invalidation
54+
- **Zero**: Cache disabled or no repeated calls
55+
56+
---
57+
58+
### 4. Circuit Breaker Status (Gauge)
59+
**Shows**: Current state of circuit breakers (0=CLOSED, 1=OPEN, 2=HALF_OPEN)
60+
61+
**PromQL Query**:
62+
```promql
63+
max(tool_circuit_breaker_state)
64+
```
65+
66+
**What it tells you**: System resilience status
67+
- **0 (CLOSED)**: All tools healthy
68+
- **1 (OPEN)**: One or more tools are circuit-broken (too many failures)
69+
- **2 (HALF_OPEN)**: Testing recovery
70+
71+
**Action**: If showing OPEN, check which tool is failing and investigate
72+
73+
---
74+
75+
### 5. Tool Call Rate (Time Series Graph)
76+
**Shows**: Tool execution rate over time, broken down by tool name
77+
78+
**PromQL Query**:
79+
```promql
80+
sum by(tool) (rate(tool_executions_total[1m]))
81+
```
82+
83+
**What it tells you**: Tool usage patterns
84+
- Identify which tools are most used
85+
- Spot usage spikes (potential scaling issues)
86+
- Correlate with error rates to find problematic tools
87+
88+
---
89+
90+
### 6. Latency Percentiles (Stat Panels)
91+
**Shows**: P50, P95, and P99 latency for tool executions
92+
93+
**PromQL Queries**:
94+
```promql
95+
# P50 (Median)
96+
histogram_quantile(0.5,
97+
sum by(le) (rate(tool_execution_duration_seconds_bucket[5m])))
98+
99+
# P95
100+
histogram_quantile(0.95,
101+
sum by(le) (rate(tool_execution_duration_seconds_bucket[5m])))
102+
103+
# P99
104+
histogram_quantile(0.99,
105+
sum by(le) (rate(tool_execution_duration_seconds_bucket[5m])))
106+
```
107+
108+
**What it tells you**: Performance characteristics
109+
- **P50**: Typical execution time
110+
- **P95**: 95% of calls complete within this time
111+
- **P99**: Worst-case latency (outliers)
112+
113+
**Target values** (depends on your tools):
114+
- P50: < 100ms for local tools, < 500ms for API tools
115+
- P95: < 500ms for local tools, < 2s for API tools
116+
- P99: < 1s for local tools, < 5s for API tools
117+
118+
---
119+
120+
### 7. Success vs Error Rate (Time Series Graph)
121+
**Shows**: Success and error rates over time
122+
123+
**PromQL Queries**:
124+
```promql
125+
# Success rate
126+
sum(rate(tool_executions_total{status="success"}[1m]))
127+
128+
# Error rate
129+
sum(rate(tool_executions_total{status="error"}[1m]))
130+
```
131+
132+
**What it tells you**: System health trends
133+
- Spot when errors started
134+
- Correlate with deployments or external events
135+
- See if retries are helping (errors should be lower than without retries)
136+
137+
---
138+
139+
### 8. Cache Hit Rate by Tool (Time Series Graph)
140+
**Shows**: Cache hit rate for each tool over time
141+
142+
**PromQL Query**:
143+
```promql
144+
sum by(tool) (rate(tool_cache_operations_total{result="hit"}[5m])) /
145+
sum by(tool) (rate(tool_cache_operations_total{operation="lookup"}[5m]))
146+
```
147+
148+
**What it tells you**: Which tools benefit most from caching
149+
- **High hit rates**: Tool has stable, cacheable results
150+
- **Low hit rates**: Results change frequently or cache TTL too short
151+
- **Zero hit rate**: Tool not using cache or unique arguments every time
152+
153+
**Optimization**: Increase cache TTL for tools with high hit rates
154+
155+
---
156+
157+
### 9. Retry Rate % (Gauge)
158+
**Shows**: Percentage of tool calls that required retries
159+
160+
**PromQL Query**:
161+
```promql
162+
(sum(rate(tool_retry_attempts_total{attempt!="0"}[5m])) /
163+
sum(rate(tool_executions_total[5m]))) * 100
164+
```
165+
166+
**What it tells you**: Reliability of tools and external dependencies
167+
- **< 5%**: Normal transient errors
168+
- **5-15%**: Flaky tools or unreliable dependencies
169+
- **> 15%**: Critical reliability issues
170+
171+
**Action**: High retry rates mean:
172+
- External APIs are flaky (consider circuit breaker)
173+
- Network issues
174+
- Tools need better error handling
175+
176+
---
177+
178+
### 10. Top 10 Tools (Table)
179+
**Shows**: Most active tools with their metrics
180+
181+
**PromQL Queries**:
182+
```promql
183+
# Total calls
184+
topk(10, sum by(tool) (rate(tool_executions_total[5m])))
185+
186+
# Error count
187+
topk(10, sum by(tool) (rate(tool_executions_total{status="error"}[5m])))
188+
189+
# Average duration
190+
topk(10,
191+
sum by(tool) (rate(tool_execution_duration_seconds_sum[5m])) /
192+
sum by(tool) (rate(tool_execution_duration_seconds_count[5m])))
193+
```
194+
195+
**What it tells you**: Tool hotspots
196+
- Which tools are most frequently used
197+
- Which tools have the most errors
198+
- Which tools are slowest
199+
200+
**Use cases**:
201+
- Prioritize optimization efforts
202+
- Identify candidates for caching
203+
- Find tools that need better error handling
204+
205+
---
206+
207+
## Example Dashboard State
208+
209+
Based on live metrics from a running system:
210+
211+
```
212+
Total Calls/Sec: 15.2 calls/sec
213+
Error Rate: 2.1%
214+
Cache Hit Rate: 45.6%
215+
Circuit Breaker: 0 (CLOSED - Healthy)
216+
217+
Latency Percentiles:
218+
P50: 68ms
219+
P95: 234ms
220+
P99: 511ms
221+
222+
Top Tools:
223+
1. calculator - 9.2 calls/sec, 0% errors, 42ms avg
224+
2. weather - 3.1 calls/sec, 0% errors, 95ms avg
225+
3. database - 1.8 calls/sec, 5% errors, 178ms avg
226+
4. api_call - 0.9 calls/sec, 0% errors, 112ms avg
227+
5. slow_tool - 0.2 calls/sec, 0% errors, 687ms avg
228+
229+
Cache Performance:
230+
- weather: 87% hit rate (highly cacheable)
231+
- database: 72% hit rate (good caching)
232+
- calculator: 0% hit rate (unique calculations)
233+
234+
Retry Rate: 3.2% (normal transient failures)
235+
```
236+
237+
## Common Patterns
238+
239+
### Pattern 1: Healthy System
240+
- Error rate < 1%
241+
- Cache hit rate > 40%
242+
- Circuit breaker: CLOSED
243+
- Retry rate < 5%
244+
- P99 latency within acceptable range
245+
246+
### Pattern 2: Tool Overload
247+
- High call rate (>100 calls/sec)
248+
- Increasing P99 latency
249+
- Rising error rate
250+
- **Action**: Scale up concurrency or add rate limiting
251+
252+
### Pattern 3: External Dependency Failure
253+
- Specific tool error rate >10%
254+
- Circuit breaker: OPEN
255+
- High retry rate for that tool
256+
- **Action**: Check external service health, circuit breaker prevents cascading failure
257+
258+
### Pattern 4: Poor Cache Configuration
259+
- Low cache hit rate (<20%)
260+
- High P50 latency
261+
- **Action**: Increase cache TTL or review cache key generation
262+
263+
### Pattern 5: System Recovery
264+
- Circuit breaker: HALF_OPEN
265+
- Error rate decreasing
266+
- Retry rate elevated but declining
267+
- **Action**: Monitor - system is auto-recovering
268+
269+
## Import Instructions
270+
271+
1. **Configure Prometheus scraping:**
272+
```yaml
273+
# prometheus.yml
274+
scrape_configs:
275+
- job_name: 'chuk-tool-processor'
276+
scrape_interval: 15s
277+
static_configs:
278+
- targets: ['your-app:9090'] # Your metrics port
279+
```
280+
281+
2. **Import dashboard:**
282+
- Grafana → Dashboards → Import
283+
- Upload `docs/grafana-dashboard.json`
284+
- Select your Prometheus data source
285+
286+
3. **Adjust time range:**
287+
- Default: Last 1 hour
288+
- Recommendation: Last 15 minutes for active monitoring
289+
- Auto-refresh: Every 5 seconds
290+
291+
## Alerting Recommendations
292+
293+
Based on this dashboard, configure alerts for:
294+
295+
1. **Error Rate Alert**
296+
```promql
297+
(sum(rate(tool_executions_total{status="error"}[5m])) /
298+
sum(rate(tool_executions_total[5m]))) > 0.05
299+
```
300+
Fire when error rate > 5%
301+
302+
2. **Circuit Breaker Alert**
303+
```promql
304+
max(tool_circuit_breaker_state) == 1
305+
```
306+
Fire when any circuit breaker opens
307+
308+
3. **Latency Alert**
309+
```promql
310+
histogram_quantile(0.99,
311+
sum by(le) (rate(tool_execution_duration_seconds_bucket[5m]))) > 5.0
312+
```
313+
Fire when P99 latency > 5 seconds
314+
315+
4. **Retry Rate Alert**
316+
```promql
317+
(sum(rate(tool_retry_attempts_total{attempt!="0"}[5m])) /
318+
sum(rate(tool_executions_total[5m]))) > 0.15
319+
```
320+
Fire when retry rate > 15%
321+
322+
## Screenshot
323+
324+
![Grafana Dashboard](img/grafana-dashboard.png)
325+
326+
_Screenshot shows all 10 panels with live data from a running system_
327+
328+
## Next Steps
329+
330+
1. **After importing**: Let it run for 5-10 minutes to see patterns
331+
2. **Set baselines**: Note typical values for your workload
332+
3. **Configure alerts**: Based on your SLOs
333+
4. **Team review**: Share dashboard link with ops team
334+
5. **Incident response**: Bookmark for troubleshooting
335+
336+
---
337+
338+
For more information:
339+
- [OBSERVABILITY.md](../OBSERVABILITY.md) - Complete observability guide
340+
- [CONFIGURATION.md](CONFIGURATION.md) - Tuning parameters
341+
- [ERRORS.md](ERRORS.md) - Error handling guide

0 commit comments

Comments
 (0)