Mooncake is an open-source infrastructure layer for large-scale LLM inference and training. Its KV-cache-centric disaggregated architecture separates prefill and decode work while pooling otherwise underused CPU memory, DRAM and SSD/NVMe resources across a GPU cluster. The project combines Transfer Engine for topology-aware, multi-NIC movement across heterogeneous networks and accelerators with Mooncake Store for distributed KV-cache and model-weight storage, placement, replication and eviction. It also includes components for elastic expert-parallel and process-group execution, and provides integrations for engines and systems including vLLM and SGLang. Mooncake is not a standalone model-serving engine: teams attach it beneath an engine when they need cross-node state transfer or a shared cache pool. It is also complementary to LMCache rather than a duplicate; Mooncake's official integration uses Mooncake as the transfer and storage backend while LMCache supplies cache-management and reuse behavior. KTransformers, despite sharing the kvcache-ai organization, is a separate heterogeneous CPU-GPU inference and fine-tuning framework. Mooncake is best suited to platform teams operating multi-node inference, long-context, multi-turn or agentic workloads; deployment complexity and vendor-reported performance gains should be validated on the team's own hardware, topology and traffic.
Mooncake
Disaggregated KV cache storage and transfer for LLM serving
Open-source infrastructure for disaggregated LLM serving that pools KV caches across prefill and decode workers, with high-performance transfer, distributed storage and integrations for vLLM and SGLang.
Pricing
Free and open source under Apache-2.0. The official project sources reviewed do not document a paid Mooncake software tier; teams pay for their own compute, memory, storage, networking and operations.
Platforms
C++ infrastructure with Python packages plus Docker and Kubernetes deployment guides. Supports distributed DRAM/SSD/NVMe storage, TCP/RDMA/EFA-class transports, and integrations with vLLM, SGLang, LMCache and other serving stacks.
Categories
Tags
Use Cases
Alternatives
LMCache
Reusable KV cache infrastructure for scalable LLM inference
Open-source KV cache management layer that persists, offloads and reuses model key-value caches across requests and serving engines to reduce repeated prefill work and improve inference throughput.
llm-d
Kubernetes-native distributed LLM inference stack
llm-d is an open-source Kubernetes-native stack for distributed LLM inference with cache-aware routing and disaggregated serving. It separates prefill and decode stages across different GPU pools for optimal resource utilization, routes requests to nodes with warm KV caches, and integrates with vLLM as the serving engine. Apache-2.0 licensed with 2,900+ GitHub stars.
SGLang
Fast serving framework for LLMs and vision models
SGLang is an open-source serving framework for large language and vision-language models, designed for low latency and high throughput. It features RadixAttention for automatic KV cache reuse, compressed finite state machines for fast structured output generation, continuous batching, and tensor parallelism. With over 25,000 GitHub stars, it supports models like LLaMA, Mistral, Qwen, and Gemma on NVIDIA and AMD GPUs.
vLLM
High-throughput LLM serving engine
vLLM is an Apache-2.0 LLM inference and serving engine focused on high-throughput self-hosted model APIs. It combines PagedAttention, continuous batching, prefix caching, quantization options, OpenAI-compatible serving, structured outputs, metrics, Docker/Kubernetes deployment guidance and integrations with agent and LLM frameworks.
Related Tools
GPUStack
Open-source GPU control plane for scalable AI model serving
Open-source GPU cluster manager that configures vLLM, SGLang, TensorRT-LLM or custom engines, serves models through compatible APIs, and provisions SSH-accessible GPU instances across on-premises, Kubernetes and cloud environments.
LitServe
Build custom AI inference servers in pure Python
Open-source, FastAPI-based serving engine from Lightning AI for building custom inference APIs — models, agents, RAG, and pipelines — with built-in batching, streaming, and multi-GPU autoscaling.
LLM Compressor
Quantize and compress LLMs for optimized vLLM deployment
Open-source, Hugging Face-compatible library for quantizing and compressing LLMs (W8A8, W4A16, NVFP4/MXFP4, KV-cache and attention quantization, expert pruning) into vLLM-ready `compressed-tensors` checkpoints.
Open Notebook
Private, self-hosted research notebooks with flexible AI models, source chat, and podcasts
Open Notebook is an MIT-licensed, self-hosted alternative to NotebookLM for collecting sources, chatting over research, generating reusable transformations, and producing multi-speaker podcasts. Its Docker stack keeps notebook data under the user's control while supporting 18-plus model providers, including local Ollama and LM Studio workflows.
Text Embeddings Inference
Hugging Face's open-source inference server for embeddings, rerankers, and classifiers
Text Embeddings Inference is Hugging Face's Apache-2.0 server for high-throughput embedding, reranking, and sequence-classification models. TEI packages token-based dynamic batching, optimized Transformers kernels, Safetensors loading, OpenAI-compatible embedding endpoints, Prometheus metrics, and configurable OpenTelemetry tracing in deployable CPU and GPU images.
LMDeploy
Open-source toolkit for quantizing, deploying, and serving LLMs and vision-language models
LMDeploy is an Apache-2.0 toolkit for self-hosting LLM and vision-language model inference with TurboMind and PyTorch engines. It combines continuous batching, blocked KV cache, tensor parallelism, AWQ and KV-cache quantization with OpenAI-compatible APIs, multi-GPU distribution, offline pipelines, and production metrics.