Best tools for Data Engineering

The GitHub of ML — model hub, datasets, and inference

Open-source platform for building, sharing, and deploying machine learning models and datasets. Hosts 500k+ models, 100k+ datasets, and Spaces for interactive demos. The central hub of the open-source AI ecosystem, providing model discovery, inference APIs, and collaborative tools that make it the GitHub of machine learning for researchers and developers worldwide.

Serverless Postgres

Serverless Postgres platform separating storage and compute for branching, autoscaling, read replicas, instant restore, and scale-to-zero workloads. Neon works with standard PostgreSQL clients and ORMs, supports extensions such as pgvector, and sits inside a broader Neon backend platform with Auth, Data API, Functions, Object Storage, and AI Gateway features.

The open-source Firebase alternative

Open-source Firebase alternative providing a full backend-as-a-service on PostgreSQL. Auto-generated REST and GraphQL APIs from your schema, real-time subscriptions, built-in auth with 20+ social providers and Row Level Security, S3-compatible file storage with CDN, and Deno-powered Edge Functions. Visual dashboard with SQL editor and table editor. Supports pgvector for AI apps. Self-hostable or managed with a generous free tier. 75K+ GitHub stars.

Turn websites into LLM-ready structured data

Firecrawl is a Y Combinator-backed API that crawls websites and converts them into clean, LLM-ready Markdown or structured JSON. Handles JavaScript rendering, pagination, sitemaps, and anti-bot measures automatically. Designed for RAG pipelines, AI agents, and data extraction workflows. Features batch crawling, scheduled scraping, webhook notifications, and custom extraction schemas. Processes content for direct ingestion into vector databases and LLM context windows.

High-performance vector database written in Rust for similarity search at scale.

Qdrant is a high-performance vector similarity search engine and database written in Rust. Designed for production-grade AI applications with advanced filtering, payload indexing, and distributed deployment. Supports billion-scale vector collections with sub-second query times. Popular choice for RAG, recommendation systems, and anomaly detection.

Fully managed vector database built for AI applications at production scale.

Pinecone is a leading managed vector database designed for high-performance similarity search at scale. Purpose-built for AI applications including RAG, recommendation systems, and semantic search. Offers managed serverless infrastructure with automatic scaling, filtering, hybrid retrieval, and namespacing. No infrastructure management required.

Embedded vector database for multimodal AI with petabyte scale

LanceDB is an open-source embedded vector database built on the Lance columnar format for multimodal AI. It delivers near in-memory performance from disk with zero-copy architecture, supporting vector search, full-text search, and SQL. Native SDKs for Python, TypeScript, and Rust integrate with LangChain, LlamaIndex, and DuckDB. Backed by a $30M Series A, used by Harvey AI and Runway, with 18,000+ GitHub stars.

Fully managed RAG-as-a-Service platform for enterprise AI applications

Ragie is a managed retrieval-augmented generation platform that handles document ingestion, indexing, and retrieval so developers can build grounded AI applications without managing vector databases or chunking pipelines. It connects to Google Drive, Notion, Slack, Confluence, and other enterprise data sources with simple APIs for hybrid search and entity extraction.

High-recall Postgres vector search at billion scale

VectorChord is a Postgres extension from the supervc-stack/VectorChord project that brings high-recall vector search to PostgreSQL. As the spiritual successor to pgvecto.rs, it combines IVF indexes with RaBitQ quantization to deliver Pinecone-class performance at billion-vector scale while keeping all data inside a single Postgres database — no separate vector store, no two-system sync, no rewrites when the workload grows.

Vector similarity search for PostgreSQL

pgvector is an open-source PostgreSQL extension with 22K+ GitHub stars adding vector similarity search to your existing Postgres database. Store embeddings alongside relational data, perform exact and approximate nearest neighbor search using L2, inner product, cosine, and L1 metrics. Supports HNSW and IVFFlat indexes for fast similarity queries at scale. Eliminates the need for a separate vector database by bringing vector capabilities into existing PostgreSQL infrastructure.

Open data platform wrapping any SQL database with instant APIs

Directus is a database-first open data platform that wraps any existing SQL database with instant REST and GraphQL APIs, a no-code admin dashboard, and built-in authentication. Unlike traditional headless CMS platforms that impose their own data model, Directus introspects your existing schema and works on top of it — supporting PostgreSQL, MySQL, SQLite, MS SQL, MariaDB, and CockroachDB out of the box.

Git for data — version-controlled SQL database with branch, merge, and diff

Dolt is a SQL database that implements Git-style version control directly on structured data. Table changes can be staged, committed, branched, merged, diffed, and reverted through SQL workflows and a Git-like CLI. It speaks the MySQL wire protocol so existing MySQL clients, ORMs, and tools can connect with minimal driver changes. Dolt is used for AI training data management, reproducible analytics, collaborative data editing, and agent-memory experiments.

Open source business intelligence and analytics

Metabase is an open-source business intelligence and embedded analytics platform for teams that want self-service dashboards, SQL workflows, and customer-facing analytics without adopting a heavy BI suite. It supports visual querying, saved questions, alerts, database connectors, cloud or self-hosted deployment, and embedding paths that now require careful plan, permission, and license review.

Adaptive web scraping library with anti-bot evasion and smart selectors

Scrapling is a Python web scraping library that uses adaptive selectors and anti-bot evasion techniques to extract data from websites reliably. It generates selectors that survive website layout changes by understanding element context rather than relying on brittle CSS paths. Features stealth browser automation, automatic retry logic, and proxy rotation. 65K+ GitHub stars.

Open-source vector database for AI-native applications and semantic search.

Weaviate is an open-source vector database purpose-built for AI applications. Supports vector, keyword, and hybrid search with built-in vectorization modules for OpenAI, Cohere, Hugging Face, and more. Used for RAG pipelines, semantic search, recommendation engines, and multimodal search. Written in Go for high performance.

Open-source embedding database — the AI-native way to store and query embeddings.

Chroma is an open-source embedding database designed for simplicity and developer experience. Runs in-memory, as a Python library, or as a client-server deployment. Popular for prototyping RAG applications, local development, and lightweight vector search. Integrates natively with LangChain, LlamaIndex, and OpenAI.

Modern data orchestration for ML and analytics

Dagster is an open-source data orchestration platform with 15K+ GitHub stars combining pipeline scheduling with software-defined assets, built-in data quality checks, and a modern developer experience. Defines data assets declaratively rather than imperatively. Features asset lineage visualization, partitioned processing, sensor-based triggers, comprehensive testing, and integrated observability. A modern alternative to Airflow for teams wanting asset-centric orchestration.

Distributed search and analytics engine for all types of data.

Elasticsearch is the world's most popular open-source search and analytics engine, powering search experiences for companies like Wikipedia, GitHub, Netflix, and Uber. Built on Apache Lucene, it provides near-real-time search, structured and unstructured data analysis, and machine learning capabilities. Part of the Elastic Stack (ELK), it handles log analytics, application search, security analytics, and observability at scale. Supports vector search for AI/RAG applications.

Web scraping and crawling via MCP for AI agents

Firecrawl MCP Server is the official MCP integration for Firecrawl, giving Cursor, Claude, Windsurf, and other MCP clients scrape, crawl, map, search, extract, and agent-style web research tools. It now supports a hosted remote endpoint, keyless rate-limited scrape/search/interact use, API-key/OAuth access for the full tool set, and self-hosted Firecrawl deployments.

AI-native database for hybrid RAG retrieval

Infinity is an AI-native database from InfiniFlow that unifies dense vectors, sparse vectors, tensors, and full-text search in a single engine. Built for retrieval-augmented generation (RAG) at scale, it powers hybrid search workflows where lexical matching, semantic similarity, and reranking all happen against one storage layer instead of four loosely coupled services.

Open-source platform for the complete machine learning lifecycle.

MLflow is an open-source platform for managing the end-to-end machine learning lifecycle. Covers experiment tracking, model packaging, model registry, and deployment. Created by Databricks and now a Linux Foundation project. Integrates with TensorFlow, PyTorch, scikit-learn, Hugging Face, and all major ML frameworks.

GPU-accelerated open-source vector database

Milvus is an open-source vector database with 45K+ GitHub stars for billion-scale similarity search. Features GPU-accelerated indexing, hybrid search combining vector and scalar filtering, multi-tenancy, partitioning, and horizontal scaling. Supports HNSW, IVF, DiskANN, and GPU index types. SDKs for Python, Java, Go, and Node.js. Zilliz Cloud offers a managed version. A production-grade foundation for RAG pipelines and recommendation systems at enterprise scale.

OpenTelemetry-native observability with AI tracing, logs, traces, metrics, and session replay — self-hosted in 90 seconds.

Traceway is an open-source, OpenTelemetry-native observability platform that combines logs, traces, metrics, exceptions, session replay, and AI tracing in a single self-hosted system. MIT licensed with no open-core restrictions, it deploys in 90 seconds via Docker Compose and accepts OTLP/HTTP from any OTel SDK without a Collector or per-language vendor SDK.

DiskANN-powered vector search extension for PostgreSQL

pgvectorscale is an open-source PostgreSQL extension from Timescale that complements pgvector with DiskANN-based approximate vector search. It is useful for teams that want faster embedding retrieval while keeping vectors, filters, and application data inside the Postgres ecosystem instead of adopting a separate hosted vector database.

Serverless vector and full-text search on object storage

turbopuffer is a serverless vector and full-text search engine built on object storage and vendor-positioned as roughly 10x cheaper than traditional vector databases. Used by Anthropic, Cursor, Notion, and Atlassian for production search workloads. Official site reports 4T+ documents, 10M+ writes/s, and 25k+ queries/s in production systems. Funded by Thrive Capital.

High-performance open-source web crawler optimized for AI pipelines

Crawl4AI is an open-source Python web crawler built for AI and data-pipeline use cases. It produces LLM-ready Markdown, supports structured extraction, Playwright/browser automation, deep/adaptive crawling, proxy/security controls, anti-bot fallback patterns, and multiple output formats. With 68K+ GitHub stars and Apache-2.0 licensing, it is a strong local/self-hosted option for RAG datasets and agent data collection.

Open-source ML and LLM monitoring with 100+ metrics

Evidently AI is an open-source platform with 100+ pre-built metrics for monitoring data quality, model performance, and data drift in AI/ML pipelines. Available under Apache 2.0 with a cloud version, it helps teams detect when production data shifts away from training distributions, LLM output quality degrades, or feature pipelines introduce anomalies that silently degrade model accuracy.

Production-grade browser automation with AI self-healing and Playwright code ownership

Intuned is a code-first browser automation platform that turns natural language prompts into production-ready Playwright code, deploys it, and self-heals it when target sites change. Supports TypeScript and Python with Anthropic Computer Use, OpenAI CUA, Stagehand, Browser-Use, and Gemini Computer Use integrations. Built-in stealth, captcha solving, auth session management, and scheduled runs with concurrency control. No vendor lock-in—you own the code.

Open-source search engine with vector and hybrid retrieval

OpenSearch is an Apache-2.0 distributed search engine with native vector-search support for teams that want BM25, filters, aggregations, and k-NN retrieval in the same search stack. It is distinct from Elasticsearch in the aicoolies catalog: OpenSearch is the AWS-backed open fork with its own docs, plugin path, and serverless deployment options.

MySQL-compatible serverless database

Relational database platform for MySQL and Postgres with Vitess-backed MySQL scale, PlanetScale Postgres, query insights, deploy-request workflows, and Database Traffic Control. It fits production teams that need managed relational performance, safe schema changes, replicas, and database expertise rather than a simple hobby database.

Hybrid search and ML ranking engine at scale

Vespa is an open-source serving engine with 6K+ GitHub stars for hybrid search combining vector similarity, BM25 text ranking, and structured filtering in a single query. Built by Yahoo for web-scale, it handles billions of documents with millisecond latency. Features real-time indexing, ML model serving, tensor computation, and ACID-compliant writes. Supports custom ranking models, query federation, and geographic search. Used for recommendation systems, personalization, and RAG.

Library for efficient similarity search and clustering of dense vectors at billion-scale.

FAISS is Meta AI Research's open-source library for efficient similarity search and clustering of dense vectors. It implements approximate nearest-neighbor algorithms designed to scale to billions of vectors, with optimized indexes that fit in RAM and GPU acceleration for the largest workloads. Engineering teams use FAISS as the retrieval primitive underneath custom RAG pipelines, recommendation systems, and large-scale embedding search infrastructure.

Data and AI observability for enterprise teams

Monte Carlo is the leading data and AI observability platform using ML to monitor pipelines, warehouses, and lakes for quality issues. It detects freshness delays, volume anomalies, schema changes, and distribution shifts before they impact analytics. With 500+ deployments at Nasdaq, Honeywell, and Roche, it provides automated root cause analysis, field-level lineage, and incident management. Available on AWS and Azure Marketplace.

AI-powered data quality for ML datasets

Cleanlab is a data-centric AI library that automatically detects and fixes label errors, outliers, and data quality issues in machine learning datasets. It works with any ML model and any data type including text, images, tabular, and audio by analyzing model predictions to identify mislabeled examples, near-duplicates, and ambiguous data points. Cleanlab helps teams improve model accuracy by cleaning training data rather than tuning model architecture.

Evaluation framework for RAG pipelines

RAGAS is an Apache-2.0 open-source evaluation framework with 14K+ GitHub stars that provides standardized metrics for assessing RAG pipeline quality. It measures faithfulness, answer relevancy, context precision, and context recall to identify whether retrieval, generation, or both are failing. It is framework-agnostic, supports LLM-as-judge evaluation, and its README discloses minimal anonymized Open Analytics with a RAGAS_DO_NOT_TRACK opt-out.

Serverless vector database with pay-as-you-go API pricing

Upstash Vector is a managed serverless vector database for RAG, semantic search, and embedding lookup. It is separate from the existing Upstash platform record in the aicoolies catalog: this slug covers the Vector product line, not the broader Redis, Kafka, or QStash platform.

Edge-native vector database for Workers and AI applications

Cloudflare Vectorize is Cloudflare’s managed vector database for Workers and edge AI applications. It is distinct from the existing Cloudflare Workers tool page: Workers is the compute runtime, while Vectorize is the embedding index and vector-query layer used to add semantic retrieval to Cloudflare-hosted apps.

AI-native data application framework with SQL generation and agents

DB-GPT is an open-source AI-native data app framework combining SQL generation, database chat, RAG, and multi-agent orchestration for data-centric workflows. It supports natural language to SQL conversion, automated data analysis, and custom data app development. Integrates with MySQL, PostgreSQL, SQLite, and more. 19,000+ GitHub stars, MIT licensed. Positioned as an alternative to MindsDB for teams building AI-powered data applications and internal database tools.

Federated SQL and semantic search across live data for AI agents

MindsDB Query Engine is a self-hosted, SQL-compatible federated query layer for AI agents. It connects live databases, warehouses, SaaS apps, documents, and vector stores without requiring a separate ETL pipeline; adds knowledge bases for semantic and hybrid retrieval; and exposes the result through SQL, MCP, HTTP, and MySQL/PostgreSQL-compatible interfaces. It is separate from MindsHub, the same team's hosted agent workspace.

Open-source RAG-based text-to-SQL engine

Vanna AI is an MIT-licensed text-to-SQL and SQL-agent framework with 23.6K+ GitHub stars. Its current Vanna 2.0 story adds user-aware agents, access control, audit logs, streaming UI components, and optional hosted admin features for teams that need natural-language database access without locking into one LLM or database. The original repo is now archived, so verify the current Vanna 2.0 path before adoption.

English-to-SQL with 100K+ users

AI2SQL converts plain English into optimized SQL queries with automatic schema detection, serving over 100,000 users worldwide. It features SQL explanation for learning, a formula generator for spreadsheets, and extensions for Slack and Chrome, providing a comprehensive ecosystem for non-technical users and developers who want to query databases faster using natural language input.

ELT platform with 350+ data connectors

Airbyte is an ELT platform with 350+ pre-built connectors for syncing data from any source to warehouses, lakes, and AI pipelines. It handles incremental syncs, schema evolution, and change data capture with a connector builder for custom integrations. Used by DoorDash, Replit, and thousands of data teams. Over 15,000 GitHub stars and $150M+ in funding.

AWS's fully managed machine learning service for building, training, and deploying ML models.

Amazon SageMaker is AWS's comprehensive ML platform covering data labeling, notebook environments, model training, hyperparameter tuning, model hosting, and MLOps pipelines. Supports all major ML frameworks. Offers SageMaker Studio as an integrated IDE. Used by enterprises for production-scale ML workloads.

Workflow orchestration platform for data pipelines

Apache Airflow is an open-source workflow orchestration platform with 39K+ GitHub stars for authoring, scheduling, and monitoring data pipelines as Python DAGs. Used by 80K+ organizations for ETL, ML training, and data transformation. Features dynamic pipeline generation, extensive operator library for AWS/GCP/Azure, task dependencies, retries, SLA monitoring, a rich web UI with Gantt charts, and pluggable executors from local to Kubernetes. The industry standard for pipeline orchestration.

Distributed event streaming platform

Apache Kafka is the industry standard open-source event streaming platform with 30K+ GitHub stars, used by 80% of Fortune 100 companies. Handles trillions of events per day with high throughput, low latency, and built-in fault tolerance. Features real-time stream processing via Kafka Streams, connectors for 100+ data sources, exactly-once semantics, and multi-datacenter replication. Foundation for event-driven architectures, real-time pipelines, and data integration at scale.

Open-source data curation platform for LLM fine-tuning

Argilla is an open-source platform for curating and annotating data for LLM fine-tuning and RLHF workflows. It provides collaborative annotation interfaces for text classification, ranking, and preference labeling with integrated quality metrics. Part of the Hugging Face ecosystem, Argilla supports direct dataset publishing to the Hub and integrates with major training frameworks for seamless model improvement pipelines.

Chat with your database in natural language

AskYourDatabase lets users chat with their databases using natural language, automatically generating SQL queries and visualizations from conversational questions. It autonomously creates dashboards and charts from queried data, serving as a self-serve reporting tool that reduces the business intelligence workload on engineering teams by letting non-technical users access data directly.

Database schema management for modern teams

Database schema management and migration tool by Ariga with a declarative schema-as-code approach. Features automatic migration planning, CI/CD integration, and drift detection to keep database schemas in sync across environments. Supports PostgreSQL, MySQL, MariaDB, and SQLite, making database changes as reviewable and version-controlled as application code.