AI Monitoring & Observability

Open-source toolkit for building AI SRE incident response agents

OpenSRE is Tracer Cloud’s open-source public-alpha Python toolkit for building AI SRE agents that investigate and respond to production incidents. It ships 60+ tools across observability, databases, incident management, communications, deployment and protocol integrations, plus simulation/evaluation workflows for benchmarking agent accuracy before live pager use.

OpenTelemetry-based observability SDK for LLM applications

Traceloop is an LLM reliability platform built around OpenLLMetry, an Apache-2.0 OpenTelemetry instrumentation layer for GenAI applications. It traces calls across OpenAI, Anthropic, vector databases, LangChain, LlamaIndex, and other frameworks, then sends data to OTel-compatible backends or Traceloop Cloud. Current positioning adds monitoring, evaluation dashboards, CI/CD integration, prompt management, and enterprise/on-prem options.

AI red teaming and infrastructure security scanner by Tencent

AI-Infra-Guard is Tencent's open-source AI security platform providing one-click evaluation of AI infrastructure risks across five modules. It covers insecure config detection, multi-agent workflow evaluation, MCP server scanning across 14 risk categories, vulnerability scanning for 55+ AI frameworks with 1,000+ CVE mappings, and jailbreak evaluation for prompt robustness. Deployable via Docker with academic backing from Peking and Fudan Universities.

See where your AI coding tokens actually go

Open-source TUI dashboard and CLI that shows where your AI coding tokens actually go, broken down by task type, tool, model, MCP server, and project. CodeBurn reads local session data directly from Claude Code, Codex, Cursor, OpenCode, Pi, and GitHub Copilot — no wrapper, proxy, or API keys — and layers on one-shot success rates so you can see whether the AI nails work first try or burns budget on edit/test/fix retries. Ships with a macOS menu bar widget and CSV/JSON export.

Full-lifecycle AI agent optimization and monitoring

CozeLoop is an open-source AI agent optimization platform from ByteDance's Coze ecosystem providing full-lifecycle management from development to production monitoring. It enables developers to debug agent prompts, evaluate agent performance across test cases, optimize reasoning processes, and monitor deployed agents in real-time. Built on Go and React with SDKs for Go, Python, and Node.js, CozeLoop is designed for enterprise-grade AI agent development and operation.

AI Lakehouse with Feature Store for real-time ML

Hopsworks is a data-intensive AI platform combining a Python-centric Feature Store with MLOps capabilities for production ML systems. Provides sub-millisecond feature retrieval powered by RonDB, dual offline and online storage for batch and real-time inference, experiment tracking, model registry, and deployment pipelines. Available as managed cloud on AWS, Azure, and GCP, self-hosted on Kubernetes, or serverless platform.

Open-source AIOps alert management platform

Keep is an open-source AIOps platform that provides a single pane of glass for all alerts from monitoring tools like Datadog, PagerDuty, Grafana, and 50+ integrations. It uses AI to correlate, deduplicate, and enrich alerts, reducing noise and helping on-call teams focus on real incidents. Keep includes workflow automation, bidirectional sync with ticketing systems, and a modern web dashboard.

Smart LLM router that cuts inference costs up to 70%

Manifest is an open-source smart model router that intelligently routes LLM requests to the cheapest capable model, reducing inference costs by up to 70% without sacrificing output quality. It uses a 23-dimension scoring algorithm to evaluate 300+ models across providers including OpenAI, Anthropic, Google, and DeepSeek, with automatic fallbacks and budget controls. Manifest can be deployed as a cloud service, local plugin, or self-hosted Docker container with transparent routing logic.

Observability data accessible to AI agents via MCP

Netdata's MCP integration exposes infrastructure monitoring, discovery, and root-cause analysis capabilities to AI agents. Built into the 78K+ star Netdata monitoring platform, it lets agents query real-time metrics, explore system health, investigate incidents, and generate observability reports through the Model Context Protocol.

Unified LLM API gateway and proxy hub

New API is an open-source multi-tenant AI gateway that aggregates and distributes LLM API requests across providers like OpenAI, Claude, and Gemini through a unified proxy interface. It cross-converts requests into OpenAI-compatible, Claude-compatible, or Gemini-compatible formats, with built-in channel management, quota control, token-based authentication, and billing capabilities. Deploy via Docker with SQLite or MySQL for centralized model management.

Lightweight eval library for LLM applications

OpenEvals is a lightweight evaluation library from the LangChain team for testing LLM application quality using LLM-as-judge patterns. It provides pre-built prompt sets and evaluation functions that score model outputs against criteria like accuracy, relevance, coherence, and safety without requiring complex infrastructure. Available as both Python and JavaScript packages, OpenEvals complements OpenAI Evals with a simpler, framework-agnostic approach to quality measurement in agentic workflows.

AI testing and evaluation for agents and LLM apps

RagaAI Catalyst is a comprehensive Python SDK for observability, monitoring, and evaluation of LLM and agentic applications. Provides agent tracing with execution graph visualization, self-hosted dashboard with analytics, synthetic data generation, multi-metric evaluation framework, and guardrail management. Built for teams running production RAG systems and AI agents who need systematic testing, debugging, and performance optimization workflows.

AI-powered production incident resolution

Resolve AI automates production incident investigation, diagnosis, and remediation acting as an AI SRE that participates in every on-call rotation. Autonomously investigates incidents pursuing multiple hypotheses in parallel, validates against real evidence, creates code snippets and drafts PRs, generates post-mortems, and onboards new teammates with instant answers about code and infrastructure. Drives 5x faster MTTR and 87% faster incident investigations.

Intelligent model router that balances cost and quality across LLM providers

RouteLLM by LMSYS routes LLM requests to the most cost-effective model that can handle each query's complexity. It uses learned routing models to classify whether a query needs a powerful expensive model or can be handled by a cheaper alternative, reducing costs by up to 85% while maintaining quality. Supports OpenAI, Anthropic, and other providers through an OpenAI-compatible API.

AI-native observability for multi-agent systems

Sazabi is an AI-native observability platform designed for fast-moving engineering teams building with LLMs and multi-agent systems. Backed by leaders from Vercel and LangChain, it provides multi-agent tracing, tool-call visualization, and latency analysis for complex agentic workflows. Focuses on helping developers debug the complete path of requests through interconnected agents and tool calls.

AI production engineer that auto-triages and fixes alerts

Sonarly is a YC W26-backed AI production engineer that autonomously triages production alerts, deduplicates them by root cause, and sends ready-to-merge pull request fixes. It connects to monitoring tools like Sentry and Datadog, analyzes alert patterns to identify the underlying issue, and generates code fixes or optimization recommendations. Built on Claude APIs, Sonarly reduces mean time to resolution for production incidents while minimizing alert fatigue for engineering teams.

Open-source LLM gateway with built-in optimization and A/B testing

TensorZero is an open-source LLMOps platform in Rust that unifies an LLM gateway, observability, prompt optimization, and A/B experimentation in a single binary. It routes requests across providers with sub-millisecond P99 latency at 10K+ QPS while capturing structured data for continuous improvement. Supports dynamic in-context learning, fine-tuning workflows, and production feedback loops. Backed by $7.3M seed funding, 11K+ GitHub stars.

Open-source observability and self-healing layer for AI agents

TraceRoot is a YC S25-backed open-source observability platform purpose-built for AI agents and LLM apps. It combines OpenTelemetry-compatible tracing with an agentic debugging runtime that reads your source code, correlates failures with recent commits, and proposes fix PRs automatically. BYOK support spans seven LLM providers; the entire stack runs self-hosted via Docker Compose, with TraceRoot Cloud available for managed deployments.