# Observability

Usage metering and billing infrastructure for AI, API, and SaaS products

Flexprice is an AGPL-3.0 open-source platform for real-time usage metering, usage-based pricing, credits, entitlements, and billing workflows. It helps engineering and finance teams turn token, API, and feature events into billable usage across managed-cloud or self-hosted deployments. Use it when a product needs finance-grade chargeback and customer usage controls, not only LLM traces.

Graph-relational database with EdgeQL query language, formerly EdgeDB

Gel (formerly EdgeDB) is a graph-relational database that combines the relational model with graph database traversal capabilities through its EdgeQL query language. Built on PostgreSQL, it eliminates the object-relational impedance mismatch with a type system that maps directly to application data models. Features built-in migrations, authentication, and an interactive web UI.

First-party MCP server for querying and managing Grafana

Grafana's official Model Context Protocol server, letting AI assistants search dashboards, query datasources, and manage alerts/incidents on your Grafana instance through a standard MCP interface.

ACP skill definitions giving coding agents HuggingFace ML superpowers

Hugging Face Skills is the official collection of ACP skill definitions that give AI coding agents access to HuggingFace ML capabilities. The 13 skills cover LLM fine-tuning with TRL, vision model training, dataset management, model evaluation, and cloud job submission on HF infrastructure. Compatible with Claude Code, Codex, Gemini CLI, and Cursor via a single npx command.

Open-source post-building layer for agents — tracing, evals, and online monitoring

Judgeval is the open-source post-building layer for AI agents from Judgment Labs, providing OpenTelemetry-based tracing, hosted and custom evaluation scorers, and online behavior monitoring for LLM-powered applications. Instrument any function with a single decorator, score live production traffic against faithfulness and instruction-adherence checks, and feed real-world failures back into reinforcement learning or supervised fine-tuning loops.

CNCF Sandbox chaos engineering framework for Kubernetes resilience

Krkn is a CNCF Sandbox chaos engineering tool that tests Kubernetes cluster resilience by injecting controlled failures. It simulates pod kills, node failures, network partitions, CPU/memory pressure, and zone outages. Krkn-AI adds AI-powered scenario generation that suggests chaos experiments based on cluster topology. Supports CI/CD integration for automated resilience testing in deployment pipelines.

Sentry-style observability for AI agent conversations

Latitude is an agent observability platform for teams that need to inspect LLM traces, conversations, issues, and evaluation feedback in one workflow. Its public repo and docs position it as a Sentry-style monitor for AI agents, with semantic search, issue detection, annotations, MCP-assisted fixes, and cloud or self-hosted deployment paths for production debugging.

Engineering intelligence for DORA metrics and workflow automation

LinearB is a software engineering intelligence platform trusted by over 3,000 engineering leaders to track DORA metrics, cycle time broken into four phases (coding, pickup, review, deploy), and developer workflow patterns. It connects to Git repos and project management tools, benchmarks team performance against 8.1M+ pull requests from 4,800 organizations, and automates workflow improvements via gitStream — a policy-as-code engine for PR routing, labeling, and review automation.

Open-source AI-powered log analysis by Salesforce

LogAI is an open-source log analysis platform by Salesforce Research that uses deep learning to detect anomalies in large-scale system logs. It provides research-backed autonomous log troubleshooting capabilities, applying ML models to identify patterns, cluster log events, and surface anomalies that would be invisible in manual log review across high-volume production environments.

IBM-backed ContextForge gateway for federating MCP, A2A, REST, and gRPC APIs

MCP Context Forge is IBM’s Apache-2.0 ContextForge project for operating a gateway, registry, and proxy across MCP servers, A2A agents, REST APIs, and gRPC services. It centralizes discovery, authentication, policy controls, federation, and observability, with deployment paths through PyPI, Docker, and Kubernetes.

Observability data accessible to AI agents via MCP

Netdata's MCP integration exposes infrastructure monitoring, discovery, and root-cause analysis capabilities to AI agents. Built into the 78K+ star Netdata monitoring platform, it lets agents query real-time metrics, explore system health, investigate incidents, and generate observability reports through the Model Context Protocol.

AI-generated git commit messages in 1 second

OpenCommit is a CLI tool that generates meaningful git commit messages using LLMs in about one second. It analyzes staged changes and produces conventional commit-style messages following your team's conventions. Supports OpenAI, Anthropic Claude, Ollama local models, and other providers. 7,200+ GitHub stars, MIT licensed, winner of GitHub 2023 Hackathon. Works as a CLI command, git hook, or GitHub Action for automated commit message generation.

Lightweight eval library for LLM applications

OpenEvals is a lightweight evaluation library from the LangChain team for testing LLM application quality using LLM-as-judge patterns. It provides pre-built prompt sets and evaluation functions that score model outputs against criteria like accuracy, relevance, coherence, and safety without requiring complex infrastructure. Available as both Python and JavaScript packages, OpenEvals complements OpenAI Evals with a simpler, framework-agnostic approach to quality measurement in agentic workflows.

OpenTelemetry-native observability for LLM applications with evals and GPU monitoring

OpenLIT is an open-source AI engineering platform that provides OpenTelemetry-native observability for LLM applications. It combines distributed tracing, evaluation, prompt management, a secrets vault, and GPU telemetry in a single self-hostable stack. With 50+ integrations across LLM providers and frameworks, it lets teams monitor AI applications using their existing observability backends like Grafana, Datadog, or Jaeger.

Real-time usage metering and billing for AI, API, and DevOps products

OpenMeter is an Apache-2.0 open-source metering platform that ingests usage events, aggregates them into meters, and supports limits, entitlements, and usage-based billing. It fits AI, API, and DevOps products that need token-level reporting, customer usage visibility, or finance-ready chargeback data. Teams can self-host the core platform or evaluate OpenMeter's managed offering.

AI testing and evaluation for agents and LLM apps

RagaAI Catalyst is a comprehensive Python SDK for observability, monitoring, and evaluation of LLM and agentic applications. Provides agent tracing with execution graph visualization, self-hosted dashboard with analytics, synthetic data generation, multi-metric evaluation framework, and guardrail management. Built for teams running production RAG systems and AI agents who need systematic testing, debugging, and performance optimization workflows.

AI-powered production incident resolution

Resolve AI automates production incident investigation, diagnosis, and remediation acting as an AI SRE that participates in every on-call rotation. Autonomously investigates incidents pursuing multiple hypotheses in parallel, validates against real evidence, creates code snippets and drafts PRs, generates post-mortems, and onboards new teammates with instant answers about code and infrastructure. Drives 5x faster MTTR and 87% faster incident investigations.

AI-native observability for multi-agent systems

Sazabi is an AI-native observability platform designed for fast-moving engineering teams building with LLMs and multi-agent systems. Backed by leaders from Vercel and LangChain, it provides multi-agent tracing, tool-call visualization, and latency analysis for complex agentic workflows. Focuses on helping developers debug the complete path of requests through interconnected agents and tool calls.

Session reports for Claude Code and Codex runs

Spotlight by Backplanes turns completed Claude Code and Codex sessions into concise reports for engineering, security, and spend review. The CLI installs on macOS, Linux, or WSL 2, watches sessions after they finish, redacts PII and credentials locally before upload, then summarizes files touched, commands run, external domains reached, scope drift, risky actions, and next-session improvements.

Open-source LLM gateway with built-in optimization and A/B testing

TensorZero is an open-source LLMOps platform in Rust that unifies an LLM gateway, observability, prompt optimization, and A/B experimentation in a single binary. It routes requests across providers with sub-millisecond P99 latency at 10K+ QPS while capturing structured data for continuous improvement. Supports dynamic in-context learning, fine-tuning workflows, and production feedback loops. Backed by $7.3M seed funding, 11K+ GitHub stars.

Open-source observability and self-healing layer for AI agents

TraceRoot is a YC S25-backed open-source observability platform purpose-built for AI agents and LLM apps. It combines OpenTelemetry-compatible tracing with an agentic debugging runtime that reads your source code, correlates failures with recent commits, and proposes fix PRs automatically. BYOK support spans seven LLM providers; the entire stack runs self-hosted via Docker Compose, with TraceRoot Cloud available for managed deployments.

AI documentation tool with video-to-docs conversion

Trupeer is an AI documentation platform that converts screen recordings and videos into structured technical documentation with AI-generated voiceovers. It automates the creation of step-by-step guides, API documentation, and SaaS help articles by analyzing video content and generating written docs with annotated screenshots. Useful for teams creating onboarding materials and product documentation.

Persistent memory layer for AI coding agents — keeps Claude Code, Codex, Cursor, and any MCP agent in context across sessions

agentmemory is an open-source MCP server that gives AI coding agents persistent, cross-session memory. Built on hybrid vector-graph search, it achieves 95.2% recall on the LongMemEval-S benchmark while using up to 92% fewer context tokens than naive context injection. Works out of the box with Claude Code, Codex, Cursor, Windsurf, Cline, OpenCode, Kilo Code, Hermes, and any MCP client through 51 MCP tools plus 12 hooks and 4 skills.

AI-powered test generation agent for automated code coverage improvement

qodo-cover (formerly Cover Agent) is an open-source AI agent that automatically generates meaningful unit tests to improve code coverage. It analyzes existing code and test patterns to produce tests that follow project conventions and target uncovered branches. Uses an iterative approach where generated tests are verified by running them, discarding those that fail. MIT licensed with over 5,300 GitHub stars.