llama.cpp

Cross-platform on-device AI inference SDK

RunAnywhere SDK is a production-ready toolkit for running AI models entirely on-device across iOS, macOS, Android, Web, React Native, and Flutter. It provides a unified C++ core with platform-specific bindings for LLM text generation via llama.cpp, vision-language models, Whisper speech-to-text, Piper text-to-speech, and on-device image generation. All processing stays local with zero cloud dependency, ensuring privacy and low latency for mobile and edge AI applications.

Cross-platform on-device AI model runtime

Nexa SDK enables running frontier LLMs and multimodal models locally across PC, mobile, IoT, and wearables with automatic hardware acceleration for GPU, NPU, and CPU. It supports Qwen, Gemma, Llama, DeepSeek models with Python/C++ desktop SDKs, Android/iOS mobile SDKs, and Docker for edge deployment. Includes an OpenAI-compatible API server with chat and function calling support.

Heterogeneous CPU-GPU inference and SFT for large MoE models

Open-source framework for running and fine-tuning large Mixture-of-Experts models with heterogeneous CPU-GPU execution, optimized kernels, limited VRAM and SGLang or LLaMA-Factory integrations.

Hugging Face's open-source inference server for embeddings, rerankers, and classifiers

Text Embeddings Inference is Hugging Face's Apache-2.0 server for high-throughput embedding, reranking, and sequence-classification models. TEI packages token-based dynamic batching, optimized Transformers kernels, Safetensors loading, OpenAI-compatible embedding endpoints, Prometheus metrics, and configurable OpenTelemetry tracing in deployable CPU and GPU images.

Open-source toolkit for quantizing, deploying, and serving LLMs and vision-language models

LMDeploy is an Apache-2.0 toolkit for self-hosting LLM and vision-language model inference with TurboMind and PyTorch engines. It combines continuous batching, blocked KV cache, tensor parallelism, AWQ and KV-cache quantization with OpenAI-compatible APIs, multi-GPU distribution, offline pipelines, and production metrics.

Multi-agent model API that orchestrates frontier models behind one OpenAI-compatible endpoint

Sakana Fugu is a hosted model-provider API that exposes a learned multi-agent system as one OpenAI-compatible model. It dynamically routes coding, code review, research, and reasoning tasks across a frontier-model pool, with Fugu for lower-latency work and Fugu Ultra for harder workloads where answer quality matters more than cost or speed.

Lifelike AI voice generation, cloning, and voice agents

ElevenLabs is an AI voice platform for text-to-speech, voice cloning, and conversational AI agents, built on models like Multilingual v2 and the low-latency Flash v2.5 and Turbo v2.5. Developers call its API to generate lifelike narration, clone voices from short audio samples, dub content across 30+ languages, add sound effects, and deploy real-time voice agents for customer service, IVR, and interactive apps, with SDKs for Python, JavaScript, and more.

Official Python SDK for the xAI API

The xAI Python SDK is the official Python client for the xAI API, giving developers a direct way to build Grok-powered apps without relying on community proxies or unofficial wrappers. It supports synchronous and asynchronous Python clients for chat completions, streaming responses, function/tool calling, and multimodal workflows, making it a clean fit for backend services, agents, notebooks, and developer tools that need programmatic xAI access.