TraceRoot

LLM application observability and evaluation platform

LangSmith is LangChain's platform for debugging, testing, evaluating, and monitoring LLM applications in production. Provides detailed tracing of every step in LLM chains and agent workflows, dataset management for regression testing, prompt versioning, and automated evaluation with custom metrics. Features an annotation queue for human feedback, online monitoring dashboards, and integration with LangChain, LangGraph, and any LLM framework via the Python/JS SDK. Essential for production LLM ops.

Open-source LLM engineering platform for observability

Langfuse is an open-source LLM engineering platform with 29K+ GitHub stars for tracing, evaluating, and monitoring AI applications. Acquired by ClickHouse, it provides detailed traces of LLM calls, prompt management with versioning, dataset-based evaluation, user feedback collection, and cost tracking. Framework-agnostic with native integrations for LangChain, LlamaIndex, OpenAI SDK, and Vercel AI SDK. Offers both self-hosted deployment and a managed cloud service.

Open-source toolkit for building AI SRE incident response agents

OpenSRE is Tracer Cloud’s open-source public-alpha Python toolkit for building AI SRE agents that investigate and respond to production incidents. It ships 60+ tools across observability, databases, incident management, communications, deployment and protocol integrations, plus simulation/evaluation workflows for benchmarking agent accuracy before live pager use.

Open-source post-building layer for agents — tracing, evals, and online monitoring

Judgeval is the open-source post-building layer for AI agents from Judgment Labs, providing OpenTelemetry-based tracing, hosted and custom evaluation scorers, and online behavior monitoring for LLM-powered applications. Instrument any function with a single decorator, score live production traffic against faithfulness and instruction-adherence checks, and feed real-world failures back into reinforcement learning or supervised fine-tuning loops.

Open-source natural-language browser automation framework for AI agents

An AGPL-3.0 TypeScript framework that drives a real browser from natural-language tasks — the self-hostable OSS framework from Hyperbrowser, distinct from the vendor's paid cloud browser product.

JavaScript framework for building and visualizing multi-agent workflows on a Kanban board

KaibanJS is an MIT-licensed JavaScript framework for defining AI agents, tasks, tools, and teams, then orchestrating their work through a Kanban-inspired runtime and visual board. It can run inside Node.js, React, or Next.js projects, supports custom UIs and headless workflows, and provides real-time task-state visibility for multi-agent applications.

Better Stack is a hosted observability and incident-management platform that combines uptime monitoring, on-call workflows, status pages, logs, traces, metrics, error tracking, session replay, and an AI SRE interface. It is aimed at teams that want one SaaS control plane for telemetry and incident response.

HyperDX is the ClickStack UI for ClickHouse-backed observability. It provides a frontend for exploring logs, traces, metrics, session replay, dashboards, and alerts, with an OpenTelemetry-centered deployment path for teams that want a self-hosted or ClickHouse-aligned observability stack.

SigNoz is an OpenTelemetry-native observability platform for collecting and correlating logs, metrics, and traces. Teams can self-host it or use SigNoz Cloud, with dashboards, alerting, query workflows, and enterprise controls for cloud-native and AI application telemetry.

Multi-agent model API that orchestrates frontier models behind one OpenAI-compatible endpoint

Sakana Fugu is a hosted model-provider API that exposes a learned multi-agent system as one OpenAI-compatible model. It dynamically routes coding, code review, research, and reasoning tasks across a frontier-model pool, with Fugu for lower-latency work and Fugu Ultra for harder workloads where answer quality matters more than cost or speed.