LMDeploy is an open-source toolkit for teams that need to compress, deploy, and serve large language or vision-language models on infrastructure they control. The project provides two inference paths: the CUDA-focused TurboMind engine for optimized production serving and a Python-native PyTorch engine that lowers the barrier for experimentation and new model support. Official documentation covers offline pipelines, OpenAI-compatible and Anthropic-compatible endpoints, tool calling, structured output, LoRA serving, multi-model request distribution, tensor parallelism, context parallelism, speculative decoding, and production metrics. That breadth makes LMDeploy a serving layer rather than a hosted model provider or a desktop chat application.
Performance claims need careful attribution. LMDeploy's own documentation reports up to 1.8 times higher request throughput than vLLM for a cited configuration and reports 4-bit inference up to 2.4 times faster than FP16, but those are project benchmarks rather than independent guarantees. The documented optimization set is concrete: persistent or continuous batching, blocked KV cache, dynamic split-and-fuse scheduling, high-performance CUDA kernels, weight-only quantization, INT4 or INT8 KV-cache quantization, AWQ, GPTQ, automatic prefix caching, and multi-GPU distribution. The current install guide supports Python 3.10 through 3.13 and documents NVIDIA CUDA, AMD ROCm, Ascend, Cambricon and MACA paths, with hardware support varying by backend.
Choose LMDeploy when a team wants an Apache-licensed inference server with both model compression and serving in one stack, especially for InternLM, Qwen, Llama, DeepSeek, InternVL, or other models listed in its compatibility tables. It is not a managed endpoint: operators still own GPU sizing, drivers, model licenses, observability, scaling, security and cloud spend. Teams should validate their exact model, quantization mode, context length and accelerator against the supported-model tables before standardizing. vLLM, SGLang and Hugging Face Text Generation Inference remain useful alternatives when their model coverage, ecosystem integrations or operational conventions fit better.
