Skip to content

Commit 2223012

Browse files
committed
feat: Add KV-Cache Manager for LLM inference (#223)
## Implementation User-space KV-cache management system for LLM inference optimization. ### Features - POSIX shared memory pools for cache storage - Multiple eviction policies (LRU, LFU, TTL, Size-based) - Persistent cache with save/restore - Thread-safe operations - CLI interface for cache management - Comprehensive test suite ### Files - kv_cache_manager.py: Core implementation - test_kv_cache_manager.py: Test suite ### Usage ```bash cortex cache create llama-cache --size 16G --tier cpu cortex cache status llama-cache cortex cache persist llama-cache cortex cache restore llama-cache cortex cache evict llama-cache --percent 25 ``` Closes #223
1 parent da3e635 commit 2223012

File tree

2 files changed

+1329
-0
lines changed

2 files changed

+1329
-0
lines changed

0 commit comments

Comments
 (0)