LitServe

LitServe is an open-source Python framework for writing your own inference server when off-the-shelf single-model servers are too rigid. You define how requests are handled, how models load, and how batching, routing, and streaming work; LitServe handles the concurrency, scaling, and deployment plumbing. It suits multi-model pipelines, agents, RAG systems, chatbots, and MCP servers, works with any PyTorch model, and can bring your own engine (including vLLM). Vendor framing (labeled): Lightning AI advertises LitServe as "2× faster than FastAPI" — treat that as a vendor benchmark and validate on your own model and payload sizes. Commercial-adjacent note: the OSS library is fully self-hostable anywhere, but the README heavily promotes the paid/managed Lightning Cloud (lightning deploy --cloud) for one-click autoscaling and monitoring; the managed platform is optional and separate from the Apache-2.0 core. Best fit for Python teams that want full control over inference logic without gluing together MLOps tooling, and who don't need a Kubernetes control plane. Distinct from vLLM: LitServe is a general serving layer, not a high-throughput single-model engine — you can run vLLM inside it, but they solve different problems.