TraceRoot

LLM application observability and evaluation platform

LangSmith is LangChain's platform for debugging, testing, evaluating, and monitoring LLM applications in production. Provides detailed tracing of every step in LLM chains and agent workflows, dataset management for regression testing, prompt versioning, and automated evaluation with custom metrics. Features an annotation queue for human feedback, online monitoring dashboards, and integration with LangChain, LangGraph, and any LLM framework via the Python/JS SDK. Essential for production LLM ops.

Open-source LLM engineering platform for observability

Langfuse is an open-source LLM engineering platform with 29K+ GitHub stars for tracing, evaluating, and monitoring AI applications. Acquired by ClickHouse, it provides detailed traces of LLM calls, prompt management with versioning, dataset-based evaluation, user feedback collection, and cost tracking. Framework-agnostic with native integrations for LangChain, LlamaIndex, OpenAI SDK, and Vercel AI SDK. Offers both self-hosted deployment and a managed cloud service.

Open-source toolkit for building AI SRE incident response agents

OpenSRE is an open-source Python toolkit from Tracer Cloud for building AI SRE agents that investigate and respond to production incidents. It ships with connectors to Prometheus, Grafana, Kubernetes and incident platforms, plus a simulation harness that replays past incidents so teams can benchmark agent accuracy before trusting it on live pager rotations.

Open-source post-building layer for agents — tracing, evals, and online monitoring

Judgeval is the open-source post-building layer for AI agents from Judgment Labs, providing OpenTelemetry-based tracing, hosted and custom evaluation scorers, and online behavior monitoring for LLM-powered applications. Instrument any function with a single decorator, score live production traffic against faithfulness and instruction-adherence checks, and feed real-world failures back into reinforcement learning or supervised fine-tuning loops.

Filesystem-first framework for durable AI agents

Eve is Vercel's filesystem-first TypeScript framework for building durable AI agents as ordinary project files. It combines Markdown instructions and skills, typed tools, channels, connections, subagents, schedules, sandboxes, and evals with Vercel's agent runtime so teams can ship deployable agents without hand-rolling orchestration. The current beta fits Vercel-native backend agent projects.

Open-source agentic browser that runs local AI agents in your browsing workflow.

BrowserOS is a privacy-first, open-source agentic browser for running AI assistants locally inside real browsing sessions instead of handing every task to a remote cloud browser.

Microsoft’s open-source toolkit for adding policy enforcement, identity, sandboxing, and audit controls to production AI agents.

Agent Governance Toolkit is an open-source Microsoft project for teams moving AI agents from demos into controlled production workflows. It focuses on runtime policy enforcement, zero-trust identity, sandboxed execution, and reliability patterns around autonomous agents, giving security and platform teams a governance layer around tool calls and agent actions rather than another prompt-only guardrail.

Microsoft’s pytest-native red teaming framework for turning AI agent safety findings into CI tests.

RAMPART is an open-source Microsoft framework for safety and security testing of agentic AI applications. It brings red-team findings into a pytest-native workflow so teams can turn prompt injection, unsafe tool use, and behavioral boundary failures into repeatable regression tests. The strongest aicoolies angle is developer workflow: RAMPART makes agent safety part of CI/CD instead of a one-off security review.

Local-first personal AI agent with memory trees, desktop integrations, and private workspace context.

OpenHuman is an open-source, local-first personal AI agent from TinyHumans. It combines a desktop app, persistent memory trees, Obsidian-compatible storage, OAuth integrations, and local model support into a private assistant harness. It is most interesting for users who want agentic workflows and long-term memory without handing every context detail to a fully cloud-hosted assistant.

MCP-native personal context vault for keeping AI agents aligned with your work, voice, and projects.

Unabyss is a personal context headquarters for AI agents. It syncs sources such as email, Slack, Notion, Drive, meetings, and professional profiles into structured context files that can be served to MCP-capable clients. The strongest angle is not generic note taking; it is permissioned, reusable context for Claude, Cursor, custom agents, and other tools that otherwise need the same background explained repeatedly.