LMCache is an open-source KV cache management layer for large-model inference. It turns the temporary key-value state produced during prefill into reusable infrastructure that can persist across requests, sessions and serving-engine instances. The current project supports tiered offloading to CPU memory, local storage and remote backends, cross-engine cache reuse, cache observability, pluggable storage and transport, and non-prefix reuse. The maintainers describe LMCache as engine-independent and vendor-neutral rather than as the vLLM engine itself; integrations span serving engines, hardware vendors and storage systems. Vendor-reported performance gains depend on the model, hardware and amount of shared context, so buyers should benchmark their own long-context, multi-turn and RAG workloads. LMCache is best suited to platform teams operating repeated-prompt or shared-context inference at enough scale to justify a separate cache layer.
LMCache
Reusable KV cache infrastructure for scalable LLM inference
Open-source KV cache management layer that persists, offloads and reuses model key-value caches across requests and serving engines to reduce repeated prefill work and improve inference throughput.
Pricing
Free and open source under Apache-2.0. There is no software license fee; users pay for their own compute, storage and operations.
Platforms
Installable with pip and deployable as an engine-independent cache daemon or integrated backend. Supports tiered CPU, local and remote storage, multiple serving engines and production observability.
Categories
Tags
Alternatives
Related Tools
AIBrix
Cloud-native control plane for scalable GenAI inference
Open-source Kubernetes-native building blocks for deploying, routing and scaling GenAI inference, including an LLM gateway, autoscaling, LoRA management and KV-cache offloading.
LLM Compressor
Quantize and compress LLMs for optimized vLLM deployment
Open-source, Hugging Face-compatible library for quantizing and compressing LLMs (W8A8, W4A16, NVFP4/MXFP4, KV-cache and attention quantization, expert pruning) into vLLM-ready `compressed-tensors` checkpoints.
LMDeploy
Open-source toolkit for quantizing, deploying, and serving LLMs and vision-language models
LMDeploy is an Apache-2.0 toolkit for self-hosting LLM and vision-language model inference with TurboMind and PyTorch engines. It combines continuous batching, blocked KV cache, tensor parallelism, AWQ and KV-cache quantization with OpenAI-compatible APIs, multi-GPU distribution, offline pipelines, and production metrics.
OpenMeter
Real-time usage metering and billing for AI, API, and DevOps products
OpenMeter is an Apache-2.0 open-source metering platform that ingests usage events, aggregates them into meters, and supports limits, entitlements, and usage-based billing. It fits AI, API, and DevOps products that need token-level reporting, customer usage visibility, or finance-ready chargeback data. Teams can self-host the core platform or evaluate OpenMeter's managed offering.
Flexprice
Usage metering and billing infrastructure for AI, API, and SaaS products
Flexprice is an AGPL-3.0 open-source platform for real-time usage metering, usage-based pricing, credits, entitlements, and billing workflows. It helps engineering and finance teams turn token, API, and feature events into billable usage across managed-cloud or self-hosted deployments. Use it when a product needs finance-grade chargeback and customer usage controls, not only LLM traces.
MCP Context Forge
IBM-backed ContextForge gateway for federating MCP, A2A, REST, and gRPC APIs
MCP Context Forge is IBM’s Apache-2.0 ContextForge project for operating a gateway, registry, and proxy across MCP servers, A2A agents, REST APIs, and gRPC services. It centralizes discovery, authentication, policy controls, federation, and observability, with deployment paths through PyPI, Docker, and Kubernetes.