# AI Inference

Local-first auditable AI agent workspace and append-only runtime

Apache Maka is an open-source, local-first AI agent workspace under the Apache Software Foundation that uses an append-only event log to record model messages, tool calls, and permissions for deterministic session replay and auditability.

Interactive architecture diagram and visual delta engine for AI agents

Archify is an open-source AI agent skill with 37,000+ GitHub stars that compiles typed JSON intermediate representations into interactive, animated system architecture diagrams with motion and visual delta diffs for Claude Code and Cursor.

ML inference platform for production AI models

Baseten is the inference platform for deploying AI models at scale with dedicated and pre-optimized model APIs and performance-optimized infrastructure. Specializes in image generation, transcription, text-to-speech, LLM serving, embeddings, and compound AI workloads. Delivers 75% latency reduction with 415ms cold starts and 3000+ concurrent scaling. Available as managed cloud or self-hosted, trusted by Cursor, Notion, Descript, and Sourcegraph for production inference.

Open-source GPU control plane for scalable AI model serving

Open-source GPU cluster manager that configures vLLM, SGLang, TensorRT-LLM or custom engines, serves models through compatible APIs, and provisions SSH-accessible GPU instances across on-premises, Kubernetes and cloud environments.

Kubernetes operator for serving AI inference workloads

KubeAI is an Apache-2.0 Kubernetes operator for deploying and scaling AI inference workloads, including LLMs, embeddings, reranking, and speech-to-text. It gives platform teams OpenAI-compatible endpoints, model proxy/controller primitives, model caching, scale-from-zero behavior, and cluster-native resource management for self-hosted inference on Kubernetes.

Data factory for AI teams and model training

Labelbox is a comprehensive data platform for AI teams handling reinforcement learning, evaluations, robotics, and human feedback workflows. Core capabilities include RL data generation with knowledge work rubrics, custom evaluations for private benchmarks and model comparisons, robotics data with full-stack video and trajectories, and an expert network of 1.5M+ knowledge workers including 50K+ PhDs. Trusted by 80% of leading AI labs for production data operations.

Disaggregated KV cache storage and transfer for LLM serving

Open-source infrastructure for disaggregated LLM serving that pools KV caches across prefill and decode workers, with high-performance transfer, distributed storage and integrations for vLLM and SGLang.

Distributed inference orchestration above vLLM, SGLang and TensorRT-LLM

Open-source, datacenter-scale orchestration layer that coordinates vLLM, SGLang and TensorRT-LLM across nodes with disaggregated serving, KV-aware routing, multi-tier cache management and automatic scaling.

Edge AI deployment SDK for heterogeneous SoCs

Roofline AI is a contact-sales edge AI deployment toolkit built around an MLIR- and IREE-based compiler. Its SDK compiles models ahead of time, a lightweight C runtime executes them across CPUs, GPUs, and NPUs, and a performance dashboard tracks latency, throughput, memory use, and model coverage across devices.

Cross-platform on-device AI inference SDK

RunAnywhere SDK is a production-ready toolkit for running AI models entirely on-device across iOS, macOS, Android, Web, React Native, and Flutter. It provides a unified C++ core with platform-specific bindings for LLM text generation via llama.cpp, vision-language models, Whisper speech-to-text, Piper text-to-speech, and on-device image generation. All processing stays local with zero cloud dependency, ensuring privacy and low latency for mobile and edge AI applications.

Hugging Face's open-source inference server for embeddings, rerankers, and classifiers

Text Embeddings Inference is Hugging Face's Apache-2.0 server for high-throughput embedding, reranking, and sequence-classification models. TEI packages token-based dynamic batching, optimized Transformers kernels, Safetensors loading, OpenAI-compatible embedding endpoints, Prometheus metrics, and configurable OpenTelemetry tracing in deployable CPU and GPU images.

NVIDIA's optimized AI model serving platform

Triton Inference Server is NVIDIA's open-source inference serving platform that deploys AI models from TensorRT, PyTorch, ONNX, TensorFlow, OpenVINO, Python, and more across cloud, data center, and edge environments. It supports dynamic batching, model ensembles, concurrent model execution on GPUs and CPUs, and real-time, streaming, and batch inference patterns. Includes Model Analyzer for profiling and Model Navigator for automated optimization.

Serverless AI inference for generative media at scale

fal.ai is a serverless AI inference platform providing ultra-low-latency APIs for generating images, videos, audio, and 3D models. With 600+ production-ready models and native Python and JavaScript SDKs, it eliminates GPU management while delivering 30-50% lower costs than alternatives. Automatic scaling with no cold starts and real-time streaming support make it ideal for interactive AI applications.

Official Kubernetes and Helm reference stack built on the vLLM inference engine

Official vLLM reference implementation for scaling the existing inference engine on Kubernetes with Helm, request routing, KV-cache offload, autoscaling and Prometheus/Grafana observability.