Testing & QA

Official debugger for MCP server development

MCP Inspector is the official interactive developer tool from the Model Context Protocol team for testing, debugging, and validating MCP servers. It provides a visual interface to inspect available tools, test transport configurations, export configs for different clients, and verify protocol compliance during MCP server development.

Reliable end-to-end testing

Cross-browser E2E testing framework by Microsoft supporting Chromium, Firefox, and WebKit with one API. Features auto-waiting, tracing with timeline/screenshots/DOM snapshots, codegen for recording tests, and parallel execution. Component testing for React, Vue, Svelte. Built-in API testing, network mocking, and mobile emulation. Known for reliability and speed vs Selenium/Cypress. 70K+ GitHub stars, rapidly becoming the E2E standard.

Apache-2.0 Python framework for repeatable LLM, RAG, agent, MCP, and safety evaluation workflows.

DeepEval is an Apache-2.0 Python framework for evaluating LLM apps, RAG systems, agents, MCP workflows, and safety behavior with repeatable test cases. It works locally and in CI/CD, then connects to Confident AI for hosted reports, observability, red teaming, and governance when teams need shared evidence instead of ad-hoc prompt reviews and manual QA.

Automated code review for any linter on CI

reviewdog is an open-source automated code review tool that integrates any linter or static analysis tool with GitHub, GitLab, Bitbucket, and Gitea pull requests. Parses output in errorformat, Checkstyle XML, SARIF, and JSON formats to post inline review comments on changed lines only. Works with GitHub Actions, Travis CI, CircleCI, GitLab CI, and Jenkins. Supports 40+ languages through universal linter adapter architecture.

Open-source LLM engineering platform for observability

Langfuse is an open-source LLM engineering platform with 29K+ GitHub stars for tracing, evaluating, and monitoring AI applications. Acquired by ClickHouse, it provides detailed traces of LLM calls, prompt management with versioning, dataset-based evaluation, user feedback collection, and cost tracking. Framework-agnostic with native integrations for LangChain, LlamaIndex, OpenAI SDK, and Vercel AI SDK. Offers both self-hosted deployment and a managed cloud service.

Plain English QA testing for rapid MVPs

Stably enables developers to create QA tests in plain English using a no-code editor, with AI ensuring tests remain valid as the application evolves through self-healing locators and assertions. It lowers the barrier to high-quality QA for startups by eliminating the need for scripting knowledge, automatically adapting test steps when UI elements change position or structure.

Open-source sandboxes and SDKs for AI agents that control desktops

Open-source computer-use infrastructure for agents that need to drive desktop environments in the background. CUA includes Cua Driver, Sandbox, Run, Bench, and Verified Data across Linux, Windows, macOS, and Android, with MCP and CLI surfaces for screenshots, accessibility trees, keyboard/mouse actions, shell commands, task evaluation, and fleet execution.

Visual testing for Storybook

Chromatic is a Storybook-first visual testing and UI review platform for design systems and frontend teams. It publishes Storybook, captures component snapshots, reviews pull-request diffs, and supports interaction tests, accessibility checks, TurboSnap, SteadySnap, Playwright/Cypress workflows, and Storybook MCP context.

AI-native end-to-end testing with plain English test authoring

Momentic is an AI-native testing platform that lets teams write end-to-end tests in plain English. It features auto-healing test selectors that adapt to UI changes, instant mobile device emulators, built-in visual regression testing, and AI-powered flaky test handling. Backed by $15M Series A from Standard Capital, it eliminates brittle test maintenance through intelligent element identification and self-repairing test flows.

LLM testing and evaluation toolkit

Promptfoo is an OpenAI-owned open-source toolkit for evaluating, red-teaming and securing LLM applications. It supports config-driven prompt/model tests, CI regression gates, red-team scans, guardrails, model security workflows, MCP Proxy, code scanning and evaluations across prompts, agents and RAG pipelines.

UK AI Security Institute framework for LLM safety evaluations

Inspect AI is an MIT-licensed framework from the UK AI Security Institute for running large language model evaluations, including tool use, multi-turn dialogue, model-graded scoring, and reusable evaluation tasks.

Property-based API fuzz testing from OpenAPI and GraphQL schemas

Schemathesis automatically generates test cases from OpenAPI and GraphQL schemas to find crashes, validation errors, and specification violations in APIs. It uses property-based testing and fuzzing techniques to explore edge cases that manual test writing misses. CLI tool and Python library with CI/CD integration. 3.4K+ GitHub stars with support for authentication, custom checks, stateful testing, JUnit XML, and Allure reports.

AI-powered web browser automation with Playwright

Stagehand is an open-source browser-agent SDK from Browserbase that combines deterministic browser automation with AI primitives such as act(), extract(), observe(), and agent(). Instead of relying only on brittle selectors, developers can use natural-language actions, Zod-backed structured extraction, page observation, action caching, and Browserbase cloud-browser infrastructure for production web automation.

Blazing fast Vite-native testing

Next-generation testing framework powered by Vite for blazing-fast execution. Jest-compatible API with native ESM support, TypeScript out of the box, and in-source testing capabilities. The new default test runner for Vite-based projects, offering significantly faster test execution and a modern developer experience compared to Jest for projects using modern JavaScript tooling.

Headless browser cloud built for AI agents

Browserbase is cloud infrastructure that runs headless Chromium browsers on demand for AI agents and automation workflows, exposing Playwright, Puppeteer, and Selenium endpoints with built-in session replay, residential proxies, CAPTCHA solving, and stealth fingerprints. It also hosts Stagehand and a Model Gateway, letting teams build browser-using agents without maintaining their own fleet of Kubernetes-managed Chromium instances.

Headless browsers in Docker for automation at scale

Browserless is a headless browser-as-a-service platform that deploys Chrome, Firefox, and WebKit in Docker containers for web scraping, testing, and AI agent automation. It provides Puppeteer and Playwright-compatible APIs, a built-in MCP server for connecting AI assistants to browser automation, screenshot and PDF generation, and connection pooling for high-concurrency workloads. Available as self-hosted source-available software or managed cloud.

API and browser monitoring as code

Checkly is an active reliability platform for developers and agents that combines Playwright browser checks, API checks, synthetic monitoring, monitoring-as-code, traces, status pages, alerts, and Rocky AI analysis. It helps teams run code-defined production checks from global and private locations instead of relying only on pre-deploy tests.

AI-powered API testing with traffic recording

Keploy is an open-source AI-powered testing platform that generates API, integration, and unit tests by recording real network traffic via eBPF. It captures API calls, database queries, and streaming events, then replays them as deterministic tests with auto-generated mocks — no code changes needed. Works across any language or framework with CI/CD pipeline integration and popular testing framework support including JUnit, PyTest, Jest, and Go-Test.

Autonomous AI pentester for web apps and APIs

Shannon is an autonomous white-box AI pentesting tool for web applications and APIs. It analyzes authorized source code, identifies attack vectors, attempts proof-by-exploitation, and produces remediation-ready reports. Shannon Lite is AGPL-3.0 for local use, while Shannon Pro is the commercial Keygraph platform for continuous security testing.

Cloud browser automation via MCP for scalable testing

Browserbase MCP Server gives MCP clients a hosted or self-hostable browser through Browserbase and Stagehand. It exposes tools for starting sessions, navigating, acting, observing, extracting, and taking screenshots, with a hosted Streamable HTTP endpoint for easiest setup and local STDIO/Docker options for teams that want to run the Apache-licensed server themselves.

One tool for intercepting, mocking, and replaying HTTP — acquired by BrowserStack

Requestly is a BrowserStack-backed API client, HTTP interceptor, mock server, and session replay tool for frontend and QA teams. Its current product is commercial/API-client led, while the legacy interceptor/open-source code is AGPLv3. The free plan covers individual workflows, and Pro lists at $12/user/month monthly or $9/user/month annually for collaborative QA and frontend debugging teams.

LLM evaluation and tracking with RAG triad metrics

TruLens is an open-source framework for evaluating and tracking LLM experiments with feedback functions, RAG triad metrics (answer relevance, context relevance, groundedness), and Honest/Harmless/Helpful evaluations. Features a unified Metric API for systematic evaluation of RAG pipelines and AI agents. 3,200+ GitHub stars, MIT licensed. Snowflake partnership adds enterprise integration. Supports LangChain, LlamaIndex, and custom LLM applications.

Autonomous Java unit test generation in CI

Diffblue Testing Agent orchestrates verified regression unit test generation for Java and Python projects through existing AI coding platforms such as GitHub Copilot CLI and Claude Code. It measures baseline coverage, generates tests, verifies that they compile and pass, and charges for net new coverage lines added rather than per seat or API call.

API development and testing platform

The most popular API platform used by 30M+ developers worldwide. Build, test, and document APIs with collections, environments, automated testing, and team collaboration. Features mock servers, monitoring, API design tools, and a vast public API network. The industry standard for API development workflows from prototyping through production testing and documentation.

Microsoft’s pytest-native red teaming framework for turning AI agent safety findings into CI tests.

RAMPART is an open-source Microsoft framework for safety and security testing of agentic AI applications. It brings red-team findings into a pytest-native workflow so teams can turn prompt injection, unsafe tool use, and behavioral boundary failures into repeatable regression tests. The strongest aicoolies angle is developer workflow: RAMPART makes agent safety part of CI/CD instead of a one-off security review.

Kubernetes-native testing for coding agents

Signadot is a Kubernetes-native validation platform that lets developers test agent-generated code in isolated preview environments mimicking production. It creates lightweight sandboxes within existing clusters where code changes can be validated against real microservice dependencies, ensuring that AI-generated modifications do not break cross-service interactions before merging to main.

Autonomous AI agent for end-to-end software testing

TestSprite is an AI testing agent that autonomously generates test plans, writes test code, executes tests, and reports results without human intervention. Ranked #1 on Product Hunt for Developer Tools in 2025, it handles test creation from natural language requirements through execution and regression tracking. Credit-based SaaS model with 121K+ monthly visits.

Open-source LLMOps platform for prompt management and evaluation

Agenta is an open-source LLMOps platform that combines prompt engineering playgrounds, prompt version management, LLM evaluation, and observability in a unified interface. It supports 50+ LLM models with side-by-side prompt comparison, A/B testing, human evaluation workflows, and OpenTelemetry-native tracing. Self-hostable with 4,000+ GitHub stars.

Open-source observability for AI agents

Laminar is an open-source observability platform for AI agents providing tracing, evaluation, and analytics for LLM applications. It integrates with Vercel AI SDK, LangChain, OpenAI, and Anthropic with a single line of code. Features include OpenTelemetry-native SDKs, an extensible evaluation framework with CI/CD support, SQL access to traces and metrics, and a visual debugging timeline for agent reasoning and actions.

Cross-browser testing in the cloud

Cloud testing platform with access to 3,500+ real browser combinations and 30,000+ real devices for automated cross-browser testing. Integrates with Selenium, Cypress, and Playwright. Offers visual testing, accessibility checks, and Percy integration for screenshot comparison, enabling teams to catch UI regressions across every browser and device combination without maintaining local infrastructure.

Test anything that runs in a browser

JavaScript end-to-end testing framework that runs tests directly in the browser with real-time reloading and an interactive test runner UI. Features automatic waiting, time-travel debugging with DOM snapshots at each step, network request stubbing/interception, screenshot and video recording, and a dashboard service for CI analytics. Supports component testing for React, Vue, Angular, and Svelte. Known for developer experience and reliable test execution. 49K+ GitHub stars.

AI quality testing for bias, drift, and vulnerabilities

Giskard is an open-source testing framework for evaluating AI model quality, detecting bias, data drift, and security vulnerabilities. It provides automated test generation for LLMs and tabular models, scanning for issues like hallucination, prompt injection susceptibility, stereotypical outputs, and data leakage. Integrates with CI/CD pipelines for continuous model validation before deployment.

Unified API, performance, and contract testing DSL

Karate is an open-source testing framework that unifies API testing, performance testing, UI automation, and contract testing in a single BDD-style DSL. Write tests in plain Gherkin-like syntax without any Java knowledge. Built-in assertions, data-driven testing, parallel execution, and HTML reports. 8,900+ GitHub stars, MIT licensed. Mature, actively maintained project with commercial support options for comprehensive API quality assurance.

LLM application observability and evaluation platform

LangSmith is LangChain's platform for debugging, testing, evaluating, and monitoring LLM applications in production. Provides detailed tracing of every step in LLM chains and agent workflows, dataset management for regression testing, prompt versioning, and automated evaluation with custom metrics. Features an annotation queue for human feedback, online monitoring dashboards, and integration with LangChain, LangGraph, and any LLM framework via the Python/JS SDK. Essential for production LLM ops.

Browser automation with AI vision — no XPath or DOM parsing needed

Skyvern automates browser-based workflows using LLMs and computer vision instead of brittle XPath or CSS selectors. It understands web pages visually, navigating forms, clicking buttons, and extracting data like a human would. Achieved 85.85% success rate on WebVoyager benchmark and SOTA on WRITE tasks for RPA. 21,000+ GitHub stars, AGPL-3.0 licensed. Skyvern Cloud offers managed usage-based hosting for teams that prefer not to self-host the infrastructure.

AI-powered data quality for ML datasets

Cleanlab is a data-centric AI library that automatically detects and fixes label errors, outliers, and data quality issues in machine learning datasets. It works with any ML model and any data type including text, images, tabular, and audio by analyzing model predictions to identify mislabeled examples, near-duplicates, and ambiguous data points. Cleanlab helps teams improve model accuracy by cleaning training data rather than tuning model architecture.

AI code integrity platform for test generation and quality

Qodo, formerly CodiumAI, is an AI code integrity platform focused on reviewing, testing, and improving code quality across the development lifecycle. It provides AI-powered code reviews, automated test generation, and context-aware suggestions that span IDE, pull request, and CI/CD workflows. Qodo distinguishes itself from general-purpose AI coding assistants by focusing on quality assurance rather than code generation alone.

Evaluation framework for RAG pipelines

RAGAS is an Apache-2.0 open-source evaluation framework with 14K+ GitHub stars that provides standardized metrics for assessing RAG pipeline quality. It measures faithfulness, answer relevancy, context precision, and context recall to identify whether retrieval, generation, or both are failing. It is framework-agnostic, supports LLM-as-judge evaluation, and its README discloses minimal anonymized Open Analytics with a RAGAS_DO_NOT_TRACK opt-out.

Delightful JavaScript testing

Jest is Meta's open-source JavaScript and TypeScript testing framework with batteries-included support for unit, integration, and snapshot tests. Zero-config for most projects, it bundles its own test runner, assertion library, mocking utilities, code coverage, and parallel execution. The de facto standard for React, Node.js, Next.js, and modern JS apps, and a common target for AI-generated test suites.

Framework for evaluating LLM and agent performance

OpenAI Evals is an open-source framework and benchmark registry for evaluating LLM performance on custom tasks. It provides infrastructure for writing evaluation prompts, running them against models, and recording results in a structured format for comparison. The hosted Evals API on the OpenAI platform adds managed run tracking, dataset management, and programmatic access to evaluation pipelines. With 17,700+ GitHub stars, it serves as a foundation for systematic LLM quality measurement.

Open-source codeless test automation with NLP-based test creation

Testsigma is an open-source codeless test automation platform where tests are written in plain English using NLP-based interpretation. It supports web, mobile, and API testing with self-healing test maintenance that adapts to UI changes automatically. Apache 2.0 licensed Community Edition is free with full functionality. Cloud edition adds parallel execution, integrations, and team management. Active development with regular releases through 2026.

AI test generation from production traffic

Tusk is a Y Combinator W24-backed AI testing platform that converts real production traffic into unit and API tests, catching regressions in 43% of PRs. Its Drift SDK records live API traces with just 10 lines of code, then AI generates executable test cases covering thousands of edge cases from actual user behavior, auto-maintaining suites as application logic evolves without manual script writing.

Browser automation framework

The original browser automation framework with multi-language support for Java, Python, JavaScript, and C#. Drives end-to-end testing across all major browsers via the WebDriver protocol. Despite newer alternatives, Selenium remains the industry standard for large-scale automated browser testing, with the largest community and most extensive tooling ecosystem.

AI codeless test automation across platforms

ACCELQ Autopilot is an AI-powered codeless test automation platform with an Adaptive Relevance Engine that suggests next test steps and self-heals element locators across web, mobile, API, and desktop applications. It provides a unified testing environment that eliminates the need for scripting knowledge while maintaining the depth and flexibility required for enterprise-grade test automation.

Browser automation CLI built for AI agents by Vercel Labs

Agent Browser is a Rust-based browser automation CLI designed specifically for AI agent workflows rather than traditional testing. Developed by Vercel Labs, it provides semantic element selection through a refs system, accessibility tree snapshots, session persistence, and authentication vaults. Unlike Playwright or Puppeteer which target test automation, Agent Browser optimizes for token efficiency and deterministic element selection that gives LLMs reliable browser interaction capabilities.

AI autopilot for existing test cases

AgentQ runs existing manual test cases on autopilot using AI to navigate applications and verify expected behavior without requiring test script rewriting. It bridges the gap between manual test plans and full automation by taking existing test documentation and executing it automatically, serving teams with large manual test backlogs who need immediate automation without migration effort.

LLM vulnerability scanner and red teaming kit

Agentic Security is an open-source vulnerability scanner for LLM agent workflows that tests AI systems against jailbreaks, fuzzing, and multimodal attacks. It probes weaknesses across text, image, and audio inputs through multi-step jailbreak simulations, randomized stress testing, and reinforcement learning-powered adaptive attacks. The toolkit connects directly to LLM APIs for high-volume real-world attack scenarios, helping developers identify and patch safety gaps before deployment.

Cloud browser infrastructure for AI agents

Anchor Browser provides secure cloud-managed browser infrastructure for computer-use agents. Deploy humanized Chromium instances that access any website while maintaining bot-detection evasion and authentication support. Features OmniConnect for authentication lifecycle management, Web Action Cache for deterministic workflows, and built-in VPN infrastructure. Includes free tier and paid plans supporting millions of concurrent browser sessions for scalable agent automation.