Tools

LLM evaluation and tracing by Comet

Opik is an open-source LLM evaluation and tracing platform by Comet ML for debugging, testing, and monitoring AI applications. Provides detailed traces of LLM calls with latency, token usage, and cost tracking. Features automated evaluation with built-in and custom metrics, dataset management for regression testing, and production monitoring dashboards. Integrates with the broader Comet ML experiment tracking ecosystem. Available as both self-hosted open-source and managed cloud service.

Meta's open-source LLM security suite with Llama Guard and CodeShield

PurpleLlama is Meta's open-source suite of tools for evaluating and improving LLM safety. It includes Llama Guard models for input/output content safety classification, LlamaFirewall for multi-layer defense, CodeShield for insecure code detection, and CyberSecEval benchmarks for measuring LLM security. Llama Guard 4 supports multimodal safety across text and images. 4,100+ GitHub stars, backed by Meta AI with 44+ contributors.

Observability platform purpose-built for Python and Pydantic AI apps

Pydantic Logfire is an observability platform built by the Pydantic team specifically for Python AI applications. It provides structured logging, distributed tracing, and metrics with native understanding of Pydantic models, FastAPI, and AI framework data types. Auto-instruments OpenAI, Anthropic, LangChain, and other LLM providers. Built on OpenTelemetry for vendor-neutral data export. Offers a managed cloud dashboard with a generous free tier for development and small-scale production use.

Build software with AI

Browser-based IDE and cloud platform with a built-in AI agent that builds full-stack applications from natural language prompts. Supports 50+ languages with instant hosting, multiplayer collaboration, built-in databases (PostgreSQL, key-value), secrets management, and one-click deployment. Used by 30M+ developers for prototyping, learning, and shipping apps. Features Ghostwriter AI for code completion. Popular for hackathons, education, and rapid MVP development.

Read/write MCP bridge between AI coding agents and Figma

Talk to Figma MCP is an MIT-licensed bridge from Grab that connects Cursor, Claude Code, and other MCP-capable agents to Figma through a local MCP server, WebSocket bridge, and Figma plugin. Unlike read-only context servers, it can inspect selections, create or modify nodes, update text in bulk, and automate design operations, so teams should review permissions before enabling write access.

ByteDance's open-source LLM coding agent with multi-provider support

Trae Agent is ByteDance's open-source software engineering agent that autonomously resolves GitHub issues, fixes bugs, and implements features using any LLM provider. It supports OpenAI, Anthropic, Doubao, Azure, Ollama, and Gemini backends, making it one of the most provider-flexible coding agents available. With over 11,000 GitHub stars and a modular research-friendly architecture, it offers a strong alternative to Western-centric coding agents.

Zero Trust runtime security for Kubernetes and AI

AccuKnox provides Zero Trust runtime threat prevention for Kubernetes and cloud workloads with an AI-powered prompt firewall to prevent LLM injection attacks. Built on the open-source KubeArmor project, it manages Kubernetes identities via policy-as-code, enforces runtime security policies, and provides real-time workload protection for AI-native infrastructure environments.

Observability and lifecycle management for AI agents

AgentOps is an observability platform for monitoring, debugging, and managing AI agents. It provides session replay, event timelines, tool-call and LLM tracing, cost tracking, and dashboards for multi-agent workflows. Integrations span OpenAI, CrewAI, AutoGen, LangChain, LangGraph, LlamaIndex, Google ADK, OpenAI Agents, xAI, and 400+ LLMs/frameworks.

Microsoft's conversational multi-agent framework

AutoGen is an open-source programming framework from Microsoft Research for building AI agents and facilitating cooperation among multiple agents to solve complex tasks through multi-turn conversations. Pioneered conversable agents that interact, use tools, and involve humans in the loop for multi-agent workflows. v0.4 features a redesigned async event-driven architecture with stronger observability, flexible collaboration patterns, and reusable components.

Cross-browser testing in the cloud

Cloud testing platform with access to 3,500+ real browser combinations and 30,000+ real devices for automated cross-browser testing. Integrates with Selenium, Cypress, and Playwright. Offers visual testing, accessibility checks, and Percy integration for screenshot comparison, enabling teams to catch UI regressions across every browser and device combination without maintaining local infrastructure.

Code health analysis and technical debt quality gates

CodeScene analyzes code health, technical debt, and delivery risks by examining code patterns and change frequency, then integrates as CI/CD quality gates on pull requests. It scales AI-assisted development safely by measuring whether generated code maintains architectural integrity, with on-premises deployment options and pricing that starts free for open-source projects.

AI observability with security posture management

Coralogix uses AI to provide actionable insights across logs and traces with a dedicated AI-SPM dashboard for tracking prompt injections and data leaks in AI applications. Its pay-per-use model with no upfront fees integrates security posture management directly into the observability stack, making it uniquely positioned for teams running both traditional and AI-powered production workloads.

AI code review that groups and orders PR changes

Cubic is an AI-powered code review platform used by teams at cal.com and n8n that organizes complex PRs by grouping files logically (Backend → API → UI) and provides an AI walkthrough of changes. At paid/team pricing should be verified in Cubic’s current purchase flow; it solves the jumpiness problem in large pull requests where reviewers lose context hopping between unrelated files, offering one-click fixes and custom rule enforcement.

Test anything that runs in a browser

JavaScript end-to-end testing framework that runs tests directly in the browser with real-time reloading and an interactive test runner UI. Features automatic waiting, time-travel debugging with DOM snapshots at each step, network request stubbing/interception, screenshot and video recording, and a dashboard service for CI analytics. Supports component testing for React, Vue, Angular, and Svelte. Known for developer experience and reliable test execution. 49K+ GitHub stars.

Self-evolution engine for AI agents with auditable updates

Evolver is an open-source self-evolution engine for AI agents that turns run logs into auditable, reviewable updates via its Genome Evolution Protocol. Instead of ad hoc prompt tweaking, teams collect traces and Evolver proposes versioned diffs to prompts, tools and workflows that engineers can approve, reject or roll back like code.

DeepSeek's optimized attention kernel for Multi-Head Latent Attention

FlashMLA is DeepSeek's MIT-licensed CUDA kernel library for optimized attention in DeepSeek-V3 and DeepSeek-V3.2-Exp style inference. It includes dense MLA decoding plus sparse attention kernels for DeepSeek Sparse Attention, with README-reported H800/CUDA metrics up to 3000 GB/s, 660 TFLOPS, and sparse 640/410 TFlops paths. It has 12.7K+ GitHub stars.

Design and publish production websites

Visual website builder combining intuitive design tools with one-click publishing. Create responsive, animated sites without writing code, with React-based output under the hood. Includes built-in CMS, forms, and analytics. Popular among designers and startups who want pixel-perfect websites with interactions and animations that rival custom-coded sites.

Sandboxes for coding agents — Linux VMs, Git, and deploys in one box

Freestyle is YC-backed sandbox infrastructure built for AI coding agents, shipping secure Linux VMs with nested virtualization, Git servers, and one-click web deploys. It lets agents run real workloads, branch repos, and deploy apps under short-lived identities while billing only for active compute. Used in production by vly.ai, Rork, and Vibeflow.

AI quality testing for bias, drift, and vulnerabilities

Giskard is an open-source testing framework for evaluating AI model quality, detecting bias, data drift, and security vulnerabilities. It provides automated test generation for LLMs and tabular models, scanning for issues like hallucination, prompt injection susceptibility, stereotypical outputs, and data leakage. Integrates with CI/CD pipelines for continuous model validation before deployment.

GitHub Copilot in your terminal

Brings GitHub Copilot to the terminal, giving developers AI assistance for shell commands, error explanations, and code generation directly from the command line. Supports natural language queries to generate complex shell commands, explains error messages in plain English, and integrates with the broader GitHub Copilot ecosystem including model selection and premium request management.

xAI conversational AI with real-time X/Twitter data

Conversational AI by xAI with real-time access to X (Twitter) data and web search. Excels at current events, humor, and unfiltered responses. SuperGrok plan includes advanced reasoning, image generation, and DeepSearch capabilities. Positioned as the AI assistant most connected to real-time social discourse, with unique strength in analyzing trending topics and public sentiment from the X platform.

Modern task queue and workflow orchestration built on PostgreSQL

Hatchet is an open-source task queue and workflow orchestration platform designed as a modern alternative to Celery and BullMQ. Built on PostgreSQL for durability, it handles background jobs, AI agent workflows, RAG pipelines, and GPU task scheduling with TypeScript and Python SDKs. YC W24 batch with 7,400+ GitHub stars, MIT licensed. Supports fan-out, rate limiting, retries, and real-time observability through a web dashboard.

Scalable browser infrastructure for AI agents

Hyperbrowser is a cloud browser platform for AI agents and automation, providing managed Chrome sessions through Playwright, Puppeteer, CDP, REST, Python, and Node.js SDKs. Docs cover Stagehand, stealth/proxy options, ad blocking, recordings, scraping APIs, and credit pricing without promising universal CAPTCHA or anti-bot bypass.

Unified API, performance, and contract testing DSL

Karate is an open-source testing framework that unifies API testing, performance testing, UI automation, and contract testing in a single BDD-style DSL. Write tests in plain Gherkin-like syntax without any Java knowledge. Built-in assertions, data-driven testing, parallel execution, and HTML reports. 8,900+ GitHub stars, MIT licensed. Mature, actively maintained project with commercial support options for comprehensive API quality assurance.

Spec-driven agentic IDE and CLI by AWS

Spec-driven agentic AI IDE from AWS, built on Code OSS (the VS Code foundation), that transforms ad-hoc prompting into a structured workflow by auto-generating requirements documents, design specs, and implementation plans before writing code. Kiro keeps AI-written code tied to explicit acceptance criteria defined up front.

LLM application observability and evaluation platform

LangSmith is LangChain's platform for debugging, testing, evaluating, and monitoring LLM applications in production. Provides detailed tracing of every step in LLM chains and agent workflows, dataset management for regression testing, prompt versioning, and automated evaluation with custom metrics. Features an annotation queue for human feedback, online monitoring dashboards, and integration with LangChain, LangGraph, and any LLM framework via the Python/JS SDK. Essential for production LLM ops.

Multi-agent simulation and LLM evaluation platform

Maxim AI is an evaluation and monitoring platform for LLM applications with multi-agent simulation capabilities. Features automated test generation from production traces, custom evaluation metrics, regression detection, and real-time monitoring dashboards. Supports simulating multi-turn conversations for testing chatbots and agents at scale. Provides prompt versioning, A/B testing, and integration with CI/CD pipelines for continuous quality assurance of AI outputs across deployment cycles.

Data and AI observability for enterprise teams

Monte Carlo is the leading data and AI observability platform using ML to monitor pipelines, warehouses, and lakes for quality issues. It detects freshness delays, volume anomalies, schema changes, and distribution shifts before they impact analytics. With 500+ deployments at Nasdaq, Honeywell, and Roche, it provides automated root cause analysis, field-level lineage, and incident management. Available on AWS and Azure Marketplace.

Visual IDE for React that designers can use

Onlook is an open-source visual editor for React and Next.js with 25.9K+ GitHub stars that lets designers edit real code-backed interfaces on an infinite canvas. It connects visual changes, AI chat, component and layer inspection, and versioned code workflows so teams can prototype and refine product UI without treating design files as the source of truth.

OpenTelemetry-native LLM observability instrumentation

OpenLLMetry by Traceloop is an open-source instrumentation library with 7,000+ GitHub stars that adds OpenTelemetry-native tracing to LLM and AI agent applications. It captures detailed traces of model calls including latency, token usage, costs, and error rates, exporting data to any OpenTelemetry-compatible backend like Grafana, Datadog, or Jaeger for vendor-neutral AI observability.

Open-source AI-powered PR review with slash commands

PR-Agent by Qodo is the most widely adopted open-source AI code review tool with 11,500+ GitHub stars and 207 contributors. It provides automated PR description generation, intelligent line-by-line code review, and actionable improvement suggestions through intuitive slash commands like /describe, /review, and /improve, integrating natively with GitHub Actions, GitLab CI, Bitbucket, and Azure DevOps.

Open-source AI coding agent with self-hosted deployment option

Refact.ai is an open-source/on-premise oriented AI coding agent for VS Code and JetBrains that supports BYOK model routing, codebase understanding, developer-tool integrations and self-hosted deployment. Its public site now warns that Refact Cloud is shutting down soon, so teams should treat hosted availability as a migration risk and validate the current enterprise support path.

Browser automation with AI vision — no XPath or DOM parsing needed

Skyvern automates browser-based workflows using LLMs and computer vision instead of brittle XPath or CSS selectors. It understands web pages visually, navigating forms, clicking buttons, and extracting data like a human would. Achieved 85.85% success rate on WebVoyager benchmark and SOTA on WRITE tasks for RPA. 21,000+ GitHub stars, AGPL-3.0 licensed. Skyvern Cloud offers managed usage-based hosting for teams that prefer not to self-host the infrastructure.

Official Slack MCP server for approved workspace search, messaging, canvas, and user-context actions.

Slack MCP Server is Slack’s official remote MCP layer for giving approved AI clients workspace context and controlled actions. It lets agents search messages, files, users, and channels, draft or send messages, read threads, manage canvases, and authenticate through Slack OAuth while workspace admins approve integrations and normal Slack rate limits still apply.

AI-powered pull request summaries and code review

WhatTheDiff is an AI tool that generates human-readable pull request summaries and suggests code improvements. It analyzes code diffs to explain what changed and why in plain language, helping reviewers understand PRs faster. Integrates with GitHub and supports automated refactoring suggestions through a /wtd command. Useful for teams wanting to improve PR review speed and maintain changelog quality.

Enterprise AI coding assistant with deep codebase understanding

AI coding assistant for professional software development that uses a proprietary Context Engine to maintain live understanding of your entire stack — code, dependencies, architecture, and git history. Achieved a record-breaking SWE-bench score and a 70% win rate over GitHub Copilot in head-to-head evaluations. Agent plans, creates files, and ships features with a 200K-token context and 89% multi-file refactor accuracy. First AI coding assistant with ISO/IEC 42001 certification.

AI-powered data quality for ML datasets

Cleanlab is a data-centric AI library that automatically detects and fixes label errors, outliers, and data quality issues in machine learning datasets. It works with any ML model and any data type including text, images, tabular, and audio by analyzing model predictions to identify mislabeled examples, near-duplicates, and ambiguous data points. Cleanlab helps teams improve model accuracy by cleaning training data rather than tuning model architecture.

Open-source dev environment management with AI integration

Daytona is secure, elastic infrastructure for running AI-generated code in isolated sandboxes. It gives agents and developer workflows programmable environments with dedicated kernel, filesystem, network, vCPU, memory, and disk, backed by OCI/Docker compatibility, SDK/API access, and under-90ms sandbox startup. The project has 72,000+ GitHub stars and is AGPL-3.0 licensed.

Autonomous AI software engineer by Cognition

Devin is Cognition's managed AI software engineer for delegating engineering tasks to cloud and desktop agents. It can plan work, navigate codebases, write and run code, test changes, open PRs, review/autofix issues, and collaborate through GitHub, GitLab, Bitbucket, Linear, Jira, Slack, and Teams. Current Devin surfaces include Devin Cloud, Devin Desktop, Devin CLI, Devin Review, Windows VM support, DeepWiki, Ask Devin, and team/enterprise controls.

AI assistant built into JetBrains IDEs

Native AI assistant integrated into all JetBrains IDEs with code completion, chat, and context-aware suggestions. Powered by multiple LLMs with deep IDE integration for refactoring, analysis, and documentation. Leverages JetBrains' rich code understanding to deliver more accurate suggestions than standalone AI tools, especially for Java, Kotlin, and Python projects.

Run LLMs as a single portable executable file

Llamafile by Mozilla packages a complete LLM — model weights, inference engine, and OpenAI-compatible API server — into a single executable file that runs on Mac, Windows, Linux, FreeBSD, and OpenBSD with no installation. Built on llama.cpp and Cosmopolitan Libc for cross-platform portability, it delivers GPU-accelerated inference when available and falls back to optimized CPU execution. Supports GGUF models with a built-in web chat UI and REST API for integration.

Unified AI code review and application security platform

Panto AI is a unified AI-driven code review and application security platform that combines static analysis, secrets detection, dependency scanning, and infrastructure-as-code security into a single PR workflow. It contextualizes issues based on repository structure and code history to deliver actionable insights directly within pull requests on GitHub, GitLab, and Bitbucket with zero-configuration onboarding and compliance-ready reporting for SOC 2, ISO, and PCI-DSS.

AI code integrity platform for test generation and quality

Qodo, formerly CodiumAI, is an AI code integrity platform focused on reviewing, testing, and improving code quality across the development lifecycle. It provides AI-powered code reviews, automated test generation, and context-aware suggestions that span IDE, pull request, and CI/CD workflows. Qodo distinguishes itself from general-purpose AI coding assistants by focusing on quality assurance rather than code generation alone.

Evaluation framework for RAG pipelines

RAGAS is an Apache-2.0 open-source evaluation framework with 14K+ GitHub stars that provides standardized metrics for assessing RAG pipeline quality. It measures faithfulness, answer relevancy, context precision, and context recall to identify whether retrieval, generation, or both are failing. It is framework-agnostic, supports LLM-as-judge evaluation, and its README discloses minimal anonymized Open Analytics with a RAGAS_DO_NOT_TRACK opt-out.

MIT-licensed autonomous coding-agent reference, now superseded for many new uses by mini-swe-agent.

SWE-agent is an MIT-licensed autonomous coding-agent reference from Princeton and Stanford researchers that takes GitHub issues and attempts fixes with a bring-your-own language model. Its agent-computer interface remains foundational for repository navigation, editing, and test execution. The README now says development has shifted to mini-swe-agent, which supersedes SWE-agent and is generally recommended going forward.

Serverless vector database with pay-as-you-go API pricing

Upstash Vector is a managed serverless vector database for RAG, semantic search, and embedding lookup. It is separate from the existing Upstash platform record in the aicoolies catalog: this slug covers the Vector product line, not the broader Redis, Kafka, or QStash platform.

The modern terminal with AI

GPU-accelerated terminal built in Rust, now evolved into an Agentic Development Environment (ADE) used by 700K+ developers. Features block-based output navigation, AI command suggestions via the Oz orchestration engine, multi-line editing with syntax highlighting, and a built-in code editor with LSP support. Available on macOS, Linux, and Windows. Includes Warp Drive for sharing workflows, real-time session collaboration, and BYOK support for OpenAI, Anthropic, and Google API keys.

Open-source, local-first Notion alternative with built-in AI

AppFlowy is an open-source workspace platform offering documents, wikis, databases, and project management as a privacy-first alternative to Notion. Built with Rust and Flutter for native performance, it runs fully offline with local data storage by default. AI features include writing assistance, summarization, and translation powered by configurable LLM providers. 72.5K+ GitHub stars, backed by a growing contributor community with self-hosted and cloud deployment options.