# Observability

Prompt registry, observability, and evaluation workflows for LLM applications.

PromptLayer is a prompt management, observability, and evaluation platform for LLM applications. Teams use its Prompt Registry, visual editor, request logs, Tables, evaluations, Tool Registry, and Skill Collections to version prompts, replay requests, compare variants, run datasets, and ship prompt changes without redeploying code. Pricing starts with Free $0 for 5 users and 2.5K requests/month, Pro $49/month, Team $500/month, and Enterprise custom.

See where your AI coding tokens actually go

Open-source TUI dashboard and CLI that shows where your AI coding tokens actually go, broken down by task type, tool, model, MCP server, and project. CodeBurn reads local session data directly from Claude Code, Codex, Cursor, OpenCode, Pi, and GitHub Copilot — no wrapper, proxy, or API keys — and layers on one-shot success rates so you can see whether the AI nails work first try or burns budget on edit/test/fix retries. Ships with a macOS menu bar widget and CSV/JSON export.

Agentic application security from prompt to cloud

Cycode is an AI-native application security platform that converges AST, SSCS, and ASPM into a single solution with the Maestro AI orchestrator managing multi-agent security workflows. It provides native SAST, SCA, secrets detection, IaC scanning, and container security alongside ConnectorX integration with 100+ third-party tools. Cycode's AI Exploitability Agent reduces false positives by 94%, and the Context Intelligence Graph maps risk across code, pipelines, and runtime environments.

Postgres sync engine for local-first and real-time applications

ElectricSQL is a sync engine that keeps local application state synchronized with PostgreSQL in real-time. It enables local-first architectures where apps work offline with instant responsiveness, syncing data bidirectionally when connectivity is available. Supports partial replication with shape-based subscriptions to sync only relevant data subsets to each client.

Usage metering and billing infrastructure for AI, API, and SaaS products

Flexprice is an AGPL-3.0 open-source platform for real-time usage metering, usage-based pricing, credits, entitlements, and billing workflows. It helps engineering and finance teams turn token, API, and feature events into billable usage across managed-cloud or self-hosted deployments. Use it when a product needs finance-grade chargeback and customer usage controls, not only LLM traces.

Graph-relational database with EdgeQL query language, formerly EdgeDB

Gel (formerly EdgeDB) is a graph-relational database that combines the relational model with graph database traversal capabilities through its EdgeQL query language. Built on PostgreSQL, it eliminates the object-relational impedance mismatch with a type system that maps directly to application data models. Features built-in migrations, authentication, and an interactive web UI.

First-party MCP server for querying and managing Grafana

Grafana's official Model Context Protocol server, letting AI assistants search dashboards, query datasources, and manage alerts/incidents on your Grafana instance through a standard MCP interface.

ACP skill definitions giving coding agents HuggingFace ML superpowers

Hugging Face Skills is the official collection of ACP skill definitions that give AI coding agents access to HuggingFace ML capabilities. The 13 skills cover LLM fine-tuning with TRL, vision model training, dataset management, model evaluation, and cloud job submission on HF infrastructure. Compatible with Claude Code, Codex, Gemini CLI, and Cursor via a single npx command.

CNCF Sandbox chaos engineering framework for Kubernetes resilience

Krkn is a CNCF Sandbox chaos engineering tool that tests Kubernetes cluster resilience by injecting controlled failures. It simulates pod kills, node failures, network partitions, CPU/memory pressure, and zone outages. Krkn-AI adds AI-powered scenario generation that suggests chaos experiments based on cluster topology. Supports CI/CD integration for automated resilience testing in deployment pipelines.

Engineering intelligence for DORA metrics and workflow automation

LinearB is a software engineering intelligence platform trusted by over 3,000 engineering leaders to track DORA metrics, cycle time broken into four phases (coding, pickup, review, deploy), and developer workflow patterns. It connects to Git repos and project management tools, benchmarks team performance against 8.1M+ pull requests from 4,800 organizations, and automates workflow improvements via gitStream — a policy-as-code engine for PR routing, labeling, and review automation.

IBM-backed ContextForge gateway for federating MCP, A2A, REST, and gRPC APIs

MCP Context Forge is IBM’s Apache-2.0 ContextForge project for operating a gateway, registry, and proxy across MCP servers, A2A agents, REST APIs, and gRPC services. It centralizes discovery, authentication, policy controls, federation, and observability, with deployment paths through PyPI, Docker, and Kubernetes.

Observability data accessible to AI agents via MCP

Netdata's MCP integration exposes infrastructure monitoring, discovery, and root-cause analysis capabilities to AI agents. Built into the 78K+ star Netdata monitoring platform, it lets agents query real-time metrics, explore system health, investigate incidents, and generate observability reports through the Model Context Protocol.

AI-generated git commit messages in 1 second

OpenCommit is a CLI tool that generates meaningful git commit messages using LLMs in about one second. It analyzes staged changes and produces conventional commit-style messages following your team's conventions. Supports OpenAI, Anthropic Claude, Ollama local models, and other providers. 7,200+ GitHub stars, MIT licensed, winner of GitHub 2023 Hackathon. Works as a CLI command, git hook, or GitHub Action for automated commit message generation.

Lightweight eval library for LLM applications

OpenEvals is a lightweight evaluation library from the LangChain team for testing LLM application quality using LLM-as-judge patterns. It provides pre-built prompt sets and evaluation functions that score model outputs against criteria like accuracy, relevance, coherence, and safety without requiring complex infrastructure. Available as both Python and JavaScript packages, OpenEvals complements OpenAI Evals with a simpler, framework-agnostic approach to quality measurement in agentic workflows.

Real-time usage metering and billing for AI, API, and DevOps products

OpenMeter is an Apache-2.0 open-source metering platform that ingests usage events, aggregates them into meters, and supports limits, entitlements, and usage-based billing. It fits AI, API, and DevOps products that need token-level reporting, customer usage visibility, or finance-ready chargeback data. Teams can self-host the core platform or evaluate OpenMeter's managed offering.

AI testing and evaluation for agents and LLM apps

RagaAI Catalyst is a comprehensive Python SDK for observability, monitoring, and evaluation of LLM and agentic applications. Provides agent tracing with execution graph visualization, self-hosted dashboard with analytics, synthetic data generation, multi-metric evaluation framework, and guardrail management. Built for teams running production RAG systems and AI agents who need systematic testing, debugging, and performance optimization workflows.

AI-powered production incident resolution

Resolve AI automates production incident investigation, diagnosis, and remediation acting as an AI SRE that participates in every on-call rotation. Autonomously investigates incidents pursuing multiple hypotheses in parallel, validates against real evidence, creates code snippets and drafts PRs, generates post-mortems, and onboards new teammates with instant answers about code and infrastructure. Drives 5x faster MTTR and 87% faster incident investigations.

AI-native observability for multi-agent systems

Sazabi is an AI-native observability platform designed for fast-moving engineering teams building with LLMs and multi-agent systems. Backed by leaders from Vercel and LangChain, it provides multi-agent tracing, tool-call visualization, and latency analysis for complex agentic workflows. Focuses on helping developers debug the complete path of requests through interconnected agents and tool calls.

Open-source LLM gateway with built-in optimization and A/B testing

TensorZero is an open-source LLMOps platform in Rust that unifies an LLM gateway, observability, prompt optimization, and A/B experimentation in a single binary. It routes requests across providers with sub-millisecond P99 latency at 10K+ QPS while capturing structured data for continuous improvement. Supports dynamic in-context learning, fine-tuning workflows, and production feedback loops. Backed by $7.3M seed funding, 11K+ GitHub stars.

Open-source observability and self-healing layer for AI agents

TraceRoot is a YC S25-backed open-source observability platform purpose-built for AI agents and LLM apps. It combines OpenTelemetry-compatible tracing with an agentic debugging runtime that reads your source code, correlates failures with recent commits, and proposes fix PRs automatically. BYOK support spans seven LLM providers; the entire stack runs self-hosted via Docker Compose, with TraceRoot Cloud available for managed deployments.

AI documentation tool with video-to-docs conversion

Trupeer is an AI documentation platform that converts screen recordings and videos into structured technical documentation with AI-generated voiceovers. It automates the creation of step-by-step guides, API documentation, and SaaS help articles by analyzing video content and generating written docs with annotated screenshots. Useful for teams creating onboarding materials and product documentation.

Persistent memory layer for AI coding agents — keeps Claude Code, Codex, Cursor, and any MCP agent in context across sessions

agentmemory is an open-source MCP server that gives AI coding agents persistent, cross-session memory. Built on hybrid vector-graph search, it achieves 95.2% recall on the LongMemEval-S benchmark while using up to 92% fewer context tokens than naive context injection. Works out of the box with Claude Code, Codex, Cursor, Windsurf, Cline, OpenCode, Kilo Code, Hermes, and any MCP client through 51 MCP tools plus 12 hooks and 4 skills.

AI-powered test generation agent for automated code coverage improvement

qodo-cover (formerly Cover Agent) is an open-source AI agent that automatically generates meaningful unit tests to improve code coverage. It analyzes existing code and test patterns to produce tests that follow project conventions and target uncovered branches. Uses an iterative approach where generated tests are verified by running them, discarding those that fail. MIT licensed with over 5,300 GitHub stars.