Lemonade

Run LLMs locally with one command

Tool for running large language models locally on your machine with a simple CLI interface. Download and run Llama 3, Mistral, Gemma, Phi, Code Llama, and dozens of other open-source models with a single command. Features model management, GPU acceleration (NVIDIA/AMD/Apple Silicon), OpenAI-compatible API server, Modelfile for customization, and multi-model switching. Ideal for offline AI development, privacy-sensitive use cases, and local testing. 120K+ GitHub stars.

Run local LLMs with an intuitive desktop GUI and OpenAI-compatible API server.

Free desktop application by Element Labs for discovering, downloading, and running open-source LLMs locally. Features a curated Hugging Face model browser, side-by-side model comparison, parameter tuning, and an OpenAI-compatible API server on localhost:1234. Powered by llama.cpp with Metal acceleration for Apple Silicon.

Free, open-source local AI inference engine

LocalAI is an open-source local AI inference engine with 44K+ GitHub stars that runs LLMs, image generation, audio transcription, and embeddings entirely on consumer hardware without GPU requirements. Provides an OpenAI API-compatible REST endpoint as a drop-in replacement, supporting 1000+ models including LLaMA, Mistral, and Phi families. Features include text-to-speech, speech-to-text, function calling, constrained grammar output, and multi-modal capabilities all running locally.

Run LLMs as a single portable executable file

Llamafile by Mozilla packages a complete LLM — model weights, inference engine, and OpenAI-compatible API server — into a single executable file that runs on Mac, Windows, Linux, FreeBSD, and OpenBSD with no installation. Built on llama.cpp and Cosmopolitan Libc for cross-platform portability, it delivers GPU-accelerated inference when available and falls back to optimized CPU execution. Supports GGUF models with a built-in web chat UI and REST API for integration.

Heterogeneous CPU-GPU inference and SFT for large MoE models

Open-source framework for running and fine-tuning large Mixture-of-Experts models with heterogeneous CPU-GPU execution, optimized kernels, limited VRAM and SGLang or LLaMA-Factory integrations.

Official Kubernetes and Helm reference stack built on the vLLM inference engine

Official vLLM reference implementation for scaling the existing inference engine on Kubernetes with Helm, request routing, KV-cache offload, autoscaling and Prometheus/Grafana observability.

Distributed inference orchestration above vLLM, SGLang and TensorRT-LLM

Open-source, datacenter-scale orchestration layer that coordinates vLLM, SGLang and TensorRT-LLM across nodes with disaggregated serving, KV-aware routing, multi-tier cache management and automatic scaling.

Open-source GPU control plane for scalable AI model serving

Open-source GPU cluster manager that configures vLLM, SGLang, TensorRT-LLM or custom engines, serves models through compatible APIs, and provisions SSH-accessible GPU instances across on-premises, Kubernetes and cloud environments.

Disaggregated KV cache storage and transfer for LLM serving

Open-source infrastructure for disaggregated LLM serving that pools KV caches across prefill and decode workers, with high-performance transfer, distributed storage and integrations for vLLM and SGLang.

Reusable KV cache infrastructure for scalable LLM inference

Open-source KV cache management layer that persists, offloads and reuses model key-value caches across requests and serving engines to reduce repeated prefill work and improve inference throughput.