DeepInfra

Run LLMs as a single portable executable file

Llamafile by Mozilla packages a complete LLM — model weights, inference engine, and OpenAI-compatible API server — into a single executable file that runs on Mac, Windows, Linux, FreeBSD, and OpenBSD with no installation. Built on llama.cpp and Cosmopolitan Libc for cross-platform portability, it delivers GPU-accelerated inference when available and falls back to optimized CPU execution. Supports GGUF models with a built-in web chat UI and REST API for integration.

100% private document Q&A powered by local LLMs

PrivateGPT enables fully private document interaction using GPT-powered RAG without any data leaving your machine. Ingest documents (PDF, DOCX, TXT, and more) and chat with them using local LLMs via Ollama or remote providers. Built on LlamaIndex with Qdrant vector storage. 57,200+ GitHub stars, Apache 2.0 licensed. The go-to solution for air-gapped environments, regulated industries, and anyone who needs document Q&A without cloud data exposure.

Kubernetes-native distributed LLM inference stack

llm-d is an open-source Kubernetes-native stack for distributed LLM inference with cache-aware routing and disaggregated serving. It separates prefill and decode stages across different GPU pools for optimal resource utilization, routes requests to nodes with warm KV caches, and integrates with vLLM as the serving engine. Apache-2.0 licensed with 2,900+ GitHub stars.

Heterogeneous CPU-GPU inference and SFT for large MoE models

Open-source framework for running and fine-tuning large Mixture-of-Experts models with heterogeneous CPU-GPU execution, optimized kernels, limited VRAM and SGLang or LLaMA-Factory integrations.

Hugging Face's open-source inference server for embeddings, rerankers, and classifiers

Text Embeddings Inference is Hugging Face's Apache-2.0 server for high-throughput embedding, reranking, and sequence-classification models. TEI packages token-based dynamic batching, optimized Transformers kernels, Safetensors loading, OpenAI-compatible embedding endpoints, Prometheus metrics, and configurable OpenTelemetry tracing in deployable CPU and GPU images.

Open-source toolkit for quantizing, deploying, and serving LLMs and vision-language models

LMDeploy is an Apache-2.0 toolkit for self-hosting LLM and vision-language model inference with TurboMind and PyTorch engines. It combines continuous batching, blocked KV cache, tensor parallelism, AWQ and KV-cache quantization with OpenAI-compatible APIs, multi-GPU distribution, offline pipelines, and production metrics.

Multi-agent model API that orchestrates frontier models behind one OpenAI-compatible endpoint

Sakana Fugu is a hosted model-provider API that exposes a learned multi-agent system as one OpenAI-compatible model. It dynamically routes coding, code review, research, and reasoning tasks across a frontier-model pool, with Fugu for lower-latency work and Fugu Ultra for harder workloads where answer quality matters more than cost or speed.

Lifelike AI voice generation, cloning, and voice agents

ElevenLabs is an AI voice platform for text-to-speech, voice cloning, and conversational AI agents, built on models like Multilingual v2 and the low-latency Flash v2.5 and Turbo v2.5. Developers call its API to generate lifelike narration, clone voices from short audio samples, dub content across 30+ languages, add sound effects, and deploy real-time voice agents for customer service, IVR, and interactive apps, with SDKs for Python, JavaScript, and more.

Official Python SDK for the xAI API

The xAI Python SDK is the official Python client for the xAI API, giving developers a direct way to build Grok-powered apps without relying on community proxies or unofficial wrappers. It supports synchronous and asynchronous Python clients for chat completions, streaming responses, function/tool calling, and multimodal workflows, making it a clean fit for backend services, agents, notebooks, and developer tools that need programmatic xAI access.