Devtron

LLM evaluation and prompt engineering platform

Braintrust is an AI observability and evaluation platform for tracing LLM applications, building datasets, running prompt/model experiments, scoring outputs and turning production feedback into regression tests. It fits teams that need repeatable quality gates for AI releases rather than one-off prompt demos.

Hybrid search and ML ranking engine at scale

Vespa is an open-source serving engine with 6K+ GitHub stars for hybrid search combining vector similarity, BM25 text ranking, and structured filtering in a single query. Built by Yahoo for web-scale, it handles billions of documents with millisecond latency. Features real-time indexing, ML model serving, tensor computation, and ACID-compliant writes. Supports custom ranking models, query federation, and geographic search. Used for recommendation systems, personalization, and RAG.

Deep document understanding RAG engine

RAGFlow is an open-source RAG engine with 76K+ GitHub stars that provides deep document understanding for building knowledge-based AI applications. Optimizes chunking for 20+ document types including PDFs, Word docs, presentations, and images using layout-aware parsing. Features template-based chunking strategies, citation with source references, multi-recall retrieval combining keyword and semantic search, and a visual knowledge base management interface with drag-and-drop document upload.

Heterogeneous CPU-GPU inference and SFT for large MoE models

Open-source framework for running and fine-tuning large Mixture-of-Experts models with heterogeneous CPU-GPU execution, optimized kernels, limited VRAM and SGLang or LLaMA-Factory integrations.

Official Kubernetes and Helm reference stack built on the vLLM inference engine

Official vLLM reference implementation for scaling the existing inference engine on Kubernetes with Helm, request routing, KV-cache offload, autoscaling and Prometheus/Grafana observability.

Distributed inference orchestration above vLLM, SGLang and TensorRT-LLM

Open-source, datacenter-scale orchestration layer that coordinates vLLM, SGLang and TensorRT-LLM across nodes with disaggregated serving, KV-aware routing, multi-tier cache management and automatic scaling.

Open-source GPU control plane for scalable AI model serving

Open-source GPU cluster manager that configures vLLM, SGLang, TensorRT-LLM or custom engines, serves models through compatible APIs, and provisions SSH-accessible GPU instances across on-premises, Kubernetes and cloud environments.

Disaggregated KV cache storage and transfer for LLM serving

Open-source infrastructure for disaggregated LLM serving that pools KV caches across prefill and decode workers, with high-performance transfer, distributed storage and integrations for vLLM and SGLang.

Reusable KV cache infrastructure for scalable LLM inference

Open-source KV cache management layer that persists, offloads and reuses model key-value caches across requests and serving engines to reduce repeated prefill work and improve inference throughput.