Commit 2223012
committed
feat: Add KV-Cache Manager for LLM inference (#223)
## Implementation
User-space KV-cache management system for LLM inference optimization.
### Features
- POSIX shared memory pools for cache storage
- Multiple eviction policies (LRU, LFU, TTL, Size-based)
- Persistent cache with save/restore
- Thread-safe operations
- CLI interface for cache management
- Comprehensive test suite
### Files
- kv_cache_manager.py: Core implementation
- test_kv_cache_manager.py: Test suite
### Usage
```bash
cortex cache create llama-cache --size 16G --tier cpu
cortex cache status llama-cache
cortex cache persist llama-cache
cortex cache restore llama-cache
cortex cache evict llama-cache --percent 25
```
Closes #2231 parent da3e635 commit 2223012
2 files changed
+1329
-0
lines changed
0 commit comments