# Python

Open-source platform for the complete machine learning lifecycle.

MLflow is an open-source platform for managing the end-to-end machine learning lifecycle. Covers experiment tracking, model packaging, model registry, and deployment. Created by Databricks and now a Linux Foundation project. Integrates with TensorFlow, PyTorch, scikit-learn, Hugging Face, and all major ML frameworks.

GPU-accelerated open-source vector database

Milvus is an open-source vector database with 45K+ GitHub stars for billion-scale similarity search. Features GPU-accelerated indexing, hybrid search combining vector and scalar filtering, multi-tenancy, partitioning, and horizontal scaling. Supports HNSW, IVF, DiskANN, and GPU index types. SDKs for Python, Java, Go, and Node.js. Zilliz Cloud offers a managed version. A production-grade foundation for RAG pipelines and recommendation systems at enterprise scale.

Turn scripts into workflows, UIs, and APIs at scale

Windmill is an open-source workflow engine and developer platform built in Rust that turns scripts in Python, TypeScript, Go, Bash, SQL, and other languages into auto-generated UIs, API endpoints, workflows, data pipelines, AI agents, and scheduled jobs. The project publishes performance benchmarks against Airflow/Prefect/Temporal, supports Docker/Kubernetes self-hosting, and offers paid enterprise features.

Free AI code completion supporting 70+ languages

AI-native code editor (now operating as Windsurf) built on VS Code with the Cascade agentic assistant for multi-file editing, terminal execution, and codebase-wide context. Codeium supports 70+ programming languages with lightning-fast completion, Cascade Memories for customizable AI behavior, semantic indexing, and automatic linter-error fixes — combining completion plus full agent in one cohesive IDE.

Source-available LLM app development platform

Source-available LLM application development platform combining a visual no-code canvas with backend capabilities for building AI workflows, RAG pipelines, and agent systems from prototype to production. Integrates hundreds of models from dozens of providers, with PDF/PPT ingestion, ReAct agents with 50+ tool integrations, and multi-step orchestration. Used by both technical and non-technical teams to ship GenAI apps like chatbots and Q&A systems.

MCP, ACP and Skills support for building production coding agents — interactive or automated.

fast-agent is an Apache-licensed Python framework for building and running LLM agents with full MCP (Model Context Protocol) and ACP support. It ships with an interactive shell mode, Skills management, and multi-model routing — making it a practical platform for coding agents, workflow automation, and agent evaluation across Claude, Codex, HuggingFace, and local models.

Lightweight multi-modal agent framework

Fast, lightweight Python framework for building multi-modal AI agents, formerly known as Phidata. Includes built-in memory, knowledge bases, tools, and reasoning capabilities with 40K+ GitHub stars. Designed for developers who want to build production-ready agents quickly with minimal boilerplate, supporting structured outputs and multi-agent coordination out of the box.

Automated code quality

Codacy is a managed code quality, security and AI-guardrails platform for GitHub, GitLab and Bitbucket teams. It scans pull requests and repositories for quality, coverage and security issues while adding AI Inventory, AI Guardrails, AI Risk Hub, AI Reviewer and Verity beta surfaces for AI-assisted engineering.

Tool infrastructure for AI agents

Composio connects AI agents to 1,000+ app toolkits with managed auth, delegated user connections, sessions, tool search, MCP gateway support, CLI workflows, and sandboxed workbench execution. It targets developers building Claude, Codex, Cursor, LangChain, CrewAI, OpenAI Agents SDK, and custom agent workflows that need authenticated business actions without hand-rolling every API integration.

High-performance open-source web crawler optimized for AI pipelines

Crawl4AI is an open-source Python web crawler built for AI and data-pipeline use cases. It produces LLM-ready Markdown, supports structured extraction, Playwright/browser automation, deep/adaptive crawling, proxy/security controls, anti-bot fallback patterns, and multiple output formats. With 68K+ GitHub stars and Apache-2.0 licensing, it is a strong local/self-hosted option for RAG datasets and agent data collection.

Open-source vulnerability management aggregator

DefectDojo is an open-source vulnerability management platform with 4.7K+ GitHub stars that aggregates findings from 200+ security tools into a single view for ranking, triaging, and tracking remediation. It serves as the operating system for security teams by normalizing data from SAST, DAST, container scanners, and dependency checkers into a unified workflow with deduplication and metrics.

Open-source ML and LLM monitoring with 100+ metrics

Evidently AI is an open-source platform with 100+ pre-built metrics for monitoring data quality, model performance, and data drift in AI/ML pipelines. Available under Apache 2.0 with a cloud version, it helps teams detect when production data shifts away from training distributions, LLM output quality degrades, or feature pipelines introduce anomalies that silently degrade model accuracy.

Rust-native multi-agent orchestration for production

GraphBit is a Rust-native, multi-agent orchestration framework built for production. It targets the gap between Python-first frameworks like LangGraph and the operational expectations of enterprise systems — predictable memory, low latency, deterministic concurrency, and the ability to embed an agent runtime in services that already run Rust without dragging in a Python interpreter.

Framework for LLM applications

The most widely-used framework for building LLM-powered applications, available in Python and JavaScript. Provides abstractions for chains, agents, RAG, memory, tool usage, and structured output. Integrates with 100+ LLM providers, vector stores, document loaders, and tools. LangSmith offers tracing and evaluation. LangGraph enables stateful, multi-agent workflows with cycles. 100K+ GitHub stars. The de facto standard for LLM application development despite growing alternatives like LlamaIndex.

Unified API proxy for 100+ LLMs

Drop-in OpenAI-compatible proxy supporting 100+ LLM providers with load balancing, spend tracking, rate limiting, and fallback routing. Acts as a unified gateway for all your AI model calls, letting teams switch between providers, enforce budgets, and add reliability layers without changing application code. Essential infrastructure for multi-model AI architectures.

Alibaba's agent framework built for the Qwen model family

Qwen-Agent is Alibaba's Apache-2.0 framework for building AI agents around the Qwen model family. It supports tool use, planning, memory, RAG, Code Interpreter, Browser Assistant, MCP extras, custom tools, and Qwen Chat backend patterns with Qwen3/Qwen3.5 examples. Best fit for teams standardizing on Qwen rather than a generic multi-agent router, with 16.5K+ GitHub stars.

LLM observability and evaluation by Weights & Biases

W&B Weave is the LLM observability and evaluation toolkit from Weights & Biases. It provides automatic tracing of LLM calls with full input/output logging, cost and latency tracking, evaluation pipelines with custom scorers, and a trace explorer for debugging multi-step agent workflows. Integrates with OpenAI, Anthropic, LangChain, and CrewAI via simple Python/TypeScript decorators.

MCP server for controlling Windows desktops through UIAutomation

Windows-MCP is an open-source MCP server for giving AI agents structured access to Windows desktop automation. It focuses on UIAutomation, snapshots, input control, and Windows-specific app workflows, making it different from general filesystem or shell MCP servers.

Open-source LLMOps platform for prompt management and evaluation

Agenta is an open-source LLMOps platform that combines prompt engineering playgrounds, prompt version management, LLM evaluation, and observability in a unified interface. It supports 50+ LLM models with side-by-side prompt comparison, A/B testing, human evaluation workflows, and OpenTelemetry-native tracing. Self-hostable with 4,000+ GitHub stars.

Multi-agent AI framework

Python framework for orchestrating autonomous AI agents that collaborate to accomplish complex tasks. Define agents with specific roles, goals, and backstories, then organize them into crews with sequential or parallel task execution. Supports tool usage (web search, file I/O, API calls), memory, delegation between agents, and human-in-the-loop input. Works with OpenAI, Anthropic, local models, and more. 25K+ GitHub stars. Leading multi-agent framework alongside LangGraph and AutoGen.

The LLM anti-framework for typed AI apps

Mirascope is an open-source Python and TypeScript toolkit for building LLM applications that prioritizes type safety, composability, and 100% test coverage. Positioned as the 'anti-framework,' it provides fine-grained control over LLM interactions using familiar language constructs rather than rigid abstractions, supporting all major providers through a unified interface.

LLM evaluation and tracing by Comet

Opik is an open-source LLM evaluation and tracing platform by Comet ML for debugging, testing, and monitoring AI applications. Provides detailed traces of LLM calls with latency, token usage, and cost tracking. Features automated evaluation with built-in and custom metrics, dataset management for regression testing, and production monitoring dashboards. Integrates with the broader Comet ML experiment tracking ecosystem. Available as both self-hosted open-source and managed cloud service.

Observability platform purpose-built for Python and Pydantic AI apps

Pydantic Logfire is an observability platform built by the Pydantic team specifically for Python AI applications. It provides structured logging, distributed tracing, and metrics with native understanding of Pydantic models, FastAPI, and AI framework data types. Auto-instruments OpenAI, Anthropic, LangChain, and other LLM providers. Built on OpenTelemetry for vendor-neutral data export. Offers a managed cloud dashboard with a generous free tier for development and small-scale production use.

Observability and lifecycle management for AI agents

AgentOps is an observability platform for monitoring, debugging, and managing AI agents. It provides session replay, event timelines, tool-call and LLM tracing, cost tracking, and dashboards for multi-agent workflows. Integrations span OpenAI, CrewAI, AutoGen, LangChain, LangGraph, LlamaIndex, Google ADK, OpenAI Agents, xAI, and 400+ LLMs/frameworks.

Microsoft's conversational multi-agent framework

AutoGen is an open-source programming framework from Microsoft Research for building AI agents and facilitating cooperation among multiple agents to solve complex tasks through multi-turn conversations. Pioneered conversable agents that interact, use tools, and involve humans in the loop for multi-agent workflows. v0.4 features a redesigned async event-driven architecture with stronger observability, flexible collaboration patterns, and reusable components.

DeepSeek's optimized attention kernel for Multi-Head Latent Attention

FlashMLA is DeepSeek's MIT-licensed CUDA kernel library for optimized attention in DeepSeek-V3 and DeepSeek-V3.2-Exp style inference. It includes dense MLA decoding plus sparse attention kernels for DeepSeek Sparse Attention, with README-reported H800/CUDA metrics up to 3000 GB/s, 660 TFLOPS, and sparse 640/410 TFlops paths. It has 12.7K+ GitHub stars.

AI quality testing for bias, drift, and vulnerabilities

Giskard is an open-source testing framework for evaluating AI model quality, detecting bias, data drift, and security vulnerabilities. It provides automated test generation for LLMs and tabular models, scanning for issues like hallucination, prompt injection susceptibility, stereotypical outputs, and data leakage. Integrates with CI/CD pipelines for continuous model validation before deployment.

LLM application observability and evaluation platform

LangSmith is LangChain's platform for debugging, testing, evaluating, and monitoring LLM applications in production. Provides detailed tracing of every step in LLM chains and agent workflows, dataset management for regression testing, prompt versioning, and automated evaluation with custom metrics. Features an annotation queue for human feedback, online monitoring dashboards, and integration with LangChain, LangGraph, and any LLM framework via the Python/JS SDK. Essential for production LLM ops.

Data and AI observability for enterprise teams

Monte Carlo is the leading data and AI observability platform using ML to monitor pipelines, warehouses, and lakes for quality issues. It detects freshness delays, volume anomalies, schema changes, and distribution shifts before they impact analytics. With 500+ deployments at Nasdaq, Honeywell, and Roche, it provides automated root cause analysis, field-level lineage, and incident management. Available on AWS and Azure Marketplace.

OpenTelemetry-native LLM observability instrumentation

OpenLLMetry by Traceloop is an open-source instrumentation library with 7,000+ GitHub stars that adds OpenTelemetry-native tracing to LLM and AI agent applications. It captures detailed traces of model calls including latency, token usage, costs, and error rates, exporting data to any OpenTelemetry-compatible backend like Grafana, Datadog, or Jaeger for vendor-neutral AI observability.

Open-source AI-powered PR review with slash commands

PR-Agent by Qodo is the most widely adopted open-source AI code review tool with 11,500+ GitHub stars and 207 contributors. It provides automated PR description generation, intelligent line-by-line code review, and actionable improvement suggestions through intuitive slash commands like /describe, /review, and /improve, integrating natively with GitHub Actions, GitLab CI, Bitbucket, and Azure DevOps.

Open-source AI coding agent with self-hosted deployment option

Refact.ai is an open-source/on-premise oriented AI coding agent for VS Code and JetBrains that supports BYOK model routing, codebase understanding, developer-tool integrations and self-hosted deployment. Its public site now warns that Refact Cloud is shutting down soon, so teams should treat hosted availability as a migration risk and validate the current enterprise support path.

Evaluation framework for RAG pipelines

RAGAS is an Apache-2.0 open-source evaluation framework with 14K+ GitHub stars that provides standardized metrics for assessing RAG pipeline quality. It measures faithfulness, answer relevancy, context precision, and context recall to identify whether retrieval, generation, or both are failing. It is framework-agnostic, supports LLM-as-judge evaluation, and its README discloses minimal anonymized Open Analytics with a RAGAS_DO_NOT_TRACK opt-out.

AI-native data application framework with SQL generation and agents

DB-GPT is an open-source AI-native data app framework combining SQL generation, database chat, RAG, and multi-agent orchestration for data-centric workflows. It supports natural language to SQL conversion, automated data analysis, and custom data app development. Integrates with MySQL, PostgreSQL, SQLite, and more. 19,000+ GitHub stars, MIT licensed. Positioned as an alternative to MindsDB for teams building AI-powered data applications and internal database tools.

Self-evolving local computer agent with a reusable skill tree

GenericAgent is a minimal, self-evolving autonomous agent from a 3.3K-line seed and ~3K core loop that gives LLMs system-level control of a local computer. It writes files, runs shell commands, browses the web, and uses keyboard/mouse/screen/mobile tools, while skill crystallization saves successful runs into a reusable skill tree that cuts token cost on repeats.

AI agent testing and LLM evaluation platform

LangWatch is an AI agent testing, LLM evaluation, and observability platform for production AI teams. It combines traces, evaluations, scenario simulations, guardrails, prompt management, and Optimization Studio/DSPy workflows, with cloud and self-managed options for teams that need release-quality feedback loops.

Federated SQL and semantic search across live data for AI agents

MindsDB Query Engine is a self-hosted, SQL-compatible federated query layer for AI agents. It connects live databases, warehouses, SaaS apps, documents, and vector stores without requiring a separate ETL pipeline; adds knowledge bases for semantic and hybrid retrieval; and exposes the result through SQL, MCP, HTTP, and MySQL/PostgreSQL-compatible interfaces. It is separate from MindsHub, the same team's hosted agent workspace.

Framework for evaluating LLM and agent performance

OpenAI Evals is an open-source framework and benchmark registry for evaluating LLM performance on custom tasks. It provides infrastructure for writing evaluation prompts, running them against models, and recording results in a structured format for comparison. The hosted Evals API on the OpenAI platform adds managed run tracking, dataset management, and programmatic access to evaluation pipelines. With 17,700+ GitHub stars, it serves as a foundation for systematic LLM quality measurement.

Prompt registry, observability, and evaluation workflows for LLM applications.

PromptLayer is a prompt management, observability, and evaluation platform for LLM applications. Teams use its Prompt Registry, visual editor, request logs, Tables, evaluations, Tool Registry, and Skill Collections to version prompts, replay requests, compare variants, run datasets, and ship prompt changes without redeploying code. Pricing starts with Free $0 for 5 users and 2.5K requests/month, Pro $49/month, Team $500/month, and Enterprise custom.

Open-source toolkit for building AI SRE incident response agents

OpenSRE is Tracer Cloud’s open-source public-alpha Python toolkit for building AI SRE agents that investigate and respond to production incidents. It ships 60+ tools across observability, databases, incident management, communications, deployment and protocol integrations, plus simulation/evaluation workflows for benchmarking agent accuracy before live pager use.

AI-powered code review and refactoring assistant

Automated code reviewer that integrates with GitHub PRs. Detects code quality issues, suggests refactoring, and auto-generates reviews. Python specialist with growing JS/TS support and 1.8k+ GitHub stars. Enforces team coding standards consistently and helps developers learn best practices through inline suggestions on every pull request.

Open-source AI code review with hybrid AST-LLM

Kodus is an open-source AI code review platform that combines AST analysis with LLM reasoning to reduce noisy comments. Kody learns team standards, supports GitHub, GitLab, Bitbucket, and Azure DevOps, is model-agnostic across Claude, GPT, Gemini, Llama and OpenAI-compatible endpoints, and offers free/community, $10/dev monthly or $8/dev annual Teams, and Enterprise options.

OpenTelemetry-based observability SDK for LLM applications

Traceloop is an LLM reliability platform built around OpenLLMetry, an Apache-2.0 OpenTelemetry instrumentation layer for GenAI applications. It traces calls across OpenAI, Anthropic, vector databases, LangChain, LlamaIndex, and other frameworks, then sends data to OTel-compatible backends or Traceloop Cloud. Current positioning adds monitoring, evaluation dashboards, CI/CD integration, prompt management, and enterprise/on-prem options.

AI test generation from production traffic

Tusk is a Y Combinator W24-backed AI testing platform that converts real production traffic into unit and API tests, catching regressions in 43% of PRs. Its Drift SDK records live API traces with just 10 lines of code, then AI generates executable test cases covering thousands of edge cases from actual user behavior, auto-maintaining suites as application logic evolves without manual script writing.

Open-source RAG-based text-to-SQL engine

Vanna AI is an MIT-licensed text-to-SQL and SQL-agent framework with 23.6K+ GitHub stars. Its current Vanna 2.0 story adds user-aware agents, access control, audit logs, streaming UI components, and optional hosted admin features for teams that need natural-language database access without locking into one LLM or database. The original repo is now archived, so verify the current Vanna 2.0 path before adoption.

Security scanner for AI model files

ModelScan by Protect AI is an open-source tool that scans machine learning model files for malicious or unsafe code before they are loaded into production. Supporting formats like Pickle, HDF5, and SavedModel, it detects hidden code execution, deserialization attacks, and supply chain threats in the AI/ML model artifact pipeline, integrating into CI/CD as a critical security gate.

Browser automation framework

The original browser automation framework with multi-language support for Java, Python, JavaScript, and C#. Drives end-to-end testing across all major browsers via the WebDriver protocol. Despite newer alternatives, Selenium remains the industry standard for large-scale automated browser testing, with the largest community and most extensive tooling ecosystem.

Prompt fuzzing tool for LLM security testing

ps-fuzz by Prompt Security is a security testing tool with 680+ GitHub stars that fuzzes system prompts against dynamic LLM-based attack scenarios including jailbreaks, prompt injection, and data extraction attempts. It helps developers harden their GenAI applications by simulating adversarial attacks in a controlled environment, turning LLM security into a testable and reproducible quality gate.