Best tools for Data Engineering

Open-source SQL editor and DB manager

Beekeeper Studio is a modern, open-source SQL editor and cross-platform database GUI for MySQL, PostgreSQL, SQLite, SQL Server, MariaDB, CockroachDB, Amazon Redshift, Oracle, and LibSQL. Focuses on a clean interface, fast multi-tab editor, saved queries, data export, and schema browsing — a lightweight alternative to DBeaver or DataGrip for everyday database work on Windows, macOS, and Linux.

Instant database schema diagrams from a single query

ChartDB is an open-source database diagramming tool that converts SQL queries into instant visual entity-relationship diagrams. It supports PostgreSQL, MySQL, SQL Server, MariaDB, SQLite, CockroachDB, and ClickHouse, enabling database engineers to visualize schemas, plan migrations, and generate DDL scripts across different SQL dialects using AI assistance. Available as a free cloud app at chartdb.io or self-hosted via Docker deployment.

Real-time analytics OLAP database

ClickHouse is an open-source column-oriented database built for real-time analytical queries on massive datasets. Its columnar storage with advanced compression and vectorized query execution using SIMD instructions deliver exceptional performance for aggregations and scans. It handles billions of rows per second, supports SQL with analytical extensions, and scales horizontally for petabyte-scale data warehousing and real-time dashboards.

Knowledge graph memory engine for AI agents

Cognee is an open-source knowledge engine that builds persistent memory for AI agents by combining vector search with graph databases. It ingests data from 38+ source formats, structures information into a knowledge graph with embeddings, and enables semantic and relational queries through its ECL pipeline. Its cognitive science-inspired architecture provides superior cross-document entity identification compared to traditional RAG approaches.

The reactive backend for modern apps

Reactive backend-as-a-service with real-time sync, TypeScript-native queries and mutations, automatic caching, and built-in file storage. No SQL required — define your backend logic in TypeScript and Convex handles the database, real-time subscriptions, and serverless functions. Ideal for apps that need instant data updates without complex WebSocket infrastructure.

Full-stack framework for building AI copilots with generative UI

CopilotKit is an open-source full-stack framework for building AI-native applications with generative user interfaces. It provides React and Angular SDKs that enable agents to dynamically generate and render UI components, synchronize state between frontend and backend in real time, and implement human-in-the-loop workflows. Supports integration with LangChain, LangGraph, CrewAI and protocols including AG-UI, MCP, and A2A for standardized agent interaction.

Production-grade web scraping and browser automation library

Crawlee is an open-source web scraping and browser automation library for Node.js and Python that handles the hard parts of building reliable crawlers. It manages proxy rotation, request queuing, automatic retries, session management, and fingerprint spoofing out of the box. Supports Puppeteer, Playwright, Cheerio, and HTTP-based crawling with a unified API. Built by Apify, it includes persistent storage, autoscaling concurrency, and TypeScript-first design for production deployments.

Universal database tool

DBeaver is a free, open-source universal database GUI and SQL client supporting 100+ engines including PostgreSQL, MySQL, Oracle, SQL Server, SQLite, MongoDB, Cassandra, and Redis. Offers a consistent cross-platform interface for schema browsing, visual query building, SQL execution, data editing, ER diagrams, and data migration between heterogeneous databases, all in a single Java-based desktop app.

3D reconstruction without camera parameters

DUSt3R is Naver's breakthrough 3D reconstruction method that generates dense 3D scenes from unconstrained image pairs without known camera intrinsics or extrinsics. It casts pairwise reconstruction as pointmap regression, removing hard geometric constraints of projective camera models. Supports multi-view alignment, depth estimation, visual localization, and extends to MASt3R and MUSt3R for large-scale applications.

Git-based version control for ML data and pipelines

DVC (Data Version Control) is a free open-source tool that brings Git-like version control to datasets, ML models, and experiment pipelines. It stores pointer files in Git while keeping large data in remote storage like S3, GCS, or Azure. Features include reproducible ML pipelines with DAG-based dependency tracking, experiment management, metrics comparison, and a VS Code extension for visual experiment tracking.

High-performance data engine for multimodal AI workloads

Daft is a high-performance distributed data engine designed specifically for AI and multimodal workloads. It processes structured data alongside images, audio, video, and embeddings natively, outperforming Spark and Polars on AI-specific data pipelines. Built in Rust with a Python API, Daft handles the data engineering challenges unique to machine learning workflows.

Open-source BI tool for data visualization

DataEase is an open-source business intelligence tool that enables anyone to perform data analysis and build visualizations through a drag-and-drop interface without coding. It connects to MySQL, PostgreSQL, Elasticsearch, ClickHouse, and other data sources, providing interactive dashboards that can be shared via links or embedded in applications. DataEase offers chart templates, calculated fields, and role-based access control for team collaboration.

JetBrains database IDE

JetBrains' professional database IDE with intelligent SQL assistance, schema navigation, data editor, and admin tools for PostgreSQL, MySQL, Oracle, SQL Server, MongoDB, Redis, and more. Features context-aware completion, error detection, explain plan visualization, data export, and version control for schema changes. Part of JetBrains All Products pack or standalone. The premium choice for database development.

Open-source metadata platform for data discovery

DataHub is an open-source metadata platform for data discovery, governance, and observability, originally developed at LinkedIn. It provides a centralized catalog with 80+ integrations for data warehouses, lakes, dashboards, and ML platforms. DataHub offers real-time metadata ingestion, column-level lineage tracking, automated quality checks, and fine-grained access policies. Used by 3,000+ organizations in production. Apache 2.0 licensed with 11.8K+ GitHub stars.

AI data runtime for multimodal datasets and vector search

Deep Lake is an open-source AI data runtime from Activeloop for storing, versioning, and querying multimodal data and embeddings. It fits teams building RAG, training, evaluation, or dataset-heavy agent workflows that need a bridge between vector search, structured metadata, and large image, text, audio, or video collections.

Deep learning optimization for distributed training

DeepSpeed is Microsoft's open-source deep learning optimization library that makes distributed training and inference easy, efficient, and effective. Its ZeRO optimizer eliminates memory redundancies across data-parallel processes, enabling training of models with trillions of parameters. DeepSpeed supports 3D parallelism combining data, pipeline, and tensor parallelism, along with mixed precision training, gradient checkpointing, and CPU/NVMe offloading for memory-constrained environments.

ByteDance multimodal document image parser

Dolphin is ByteDance's multimodal document parsing model that handles intertwined text, tables, formulas, and figures in complex documents. Using a two-stage analyze-then-parse approach with a Swin Transformer vision encoder and MBart decoder, it performs layout analysis and parallel element parsing with heterogeneous anchor prompts. Dolphin-v2 adds document-type awareness for invoices, papers, and forms.

In-process analytical SQL database

DuckDB is a high-performance analytical database that runs as an in-process SQL OLAP engine. Unlike traditional client-server databases, DuckDB embeds directly within your application, similar to SQLite but optimized for analytical queries. It supports complex SQL including window functions, CTEs, and nested types while processing columnar data with vectorized execution. DuckDB reads Parquet, CSV, JSON, and Arrow formats natively and integrates with Python and R data science workflows.

Multimodal data labeling and curation for production AI

Encord is a data labeling and curation platform for teams building production AI systems with complex multimodal data. It supports image, video, audio, DICOM medical imaging, and 3D point cloud annotation with AI-assisted labeling, advanced ontology management, and quality assurance workflows. Features active learning for prioritizing high-value samples and integrates with major ML frameworks.

Open-source feature store for machine learning

Feast is an open-source feature store that manages and serves ML features for both training and online inference. It prevents training-serving skew by providing consistent feature access across offline and real-time environments. Feast supports batch materialization from data warehouses, real-time feature retrieval, on-demand transformations, and integrates with major data platforms including BigQuery, Snowflake, Redshift, and DynamoDB.

Fast memory-efficient GPU attention kernels

FlashAttention is a fast and memory-efficient exact attention implementation that reduces GPU memory usage from quadratic to linear in sequence length. Created by Tri Dao, it achieves 3-4x speedups over baseline implementations through IO-aware tiling that minimizes HBM reads and writes. Versions include FlashAttention-2 with improved parallelism, FlashAttention-3 optimized for Hopper H100 GPUs, and FlashAttention-4 targeting Hopper and Blackwell architectures.

MCP server for secure database tooling with AI agents

Google GenAI Toolbox is an open-source MCP server from Google that specializes in easy, fast, and secure database tools for AI agents. It provides structured database access through the Model Context Protocol, enabling agents to query, inspect schemas, and manage data across supported databases with built-in security controls.

Build real-time temporal knowledge graphs for AI agents

Graphiti is an open-source Python framework by Zep for building temporally-aware knowledge graphs for AI agents. It continuously integrates conversations, business data, and external information into queryable graphs with bi-temporal tracking. The hybrid retrieval combines semantic search, BM25 keywords, and graph traversal for sub-300ms queries without LLM calls at retrieval time.

Data quality validation framework for Python

Great Expectations is an open-source Python framework for validating, documenting, and profiling data quality. Teams define expectations as expressive unit tests for their data using an intuitive API, then validate datasets against those rules in CI/CD pipelines or production workflows. It connects to pandas, Spark, and SQL sources, generates data documentation automatically, and integrates with orchestrators like Airflow and Prefect for continuous data quality monitoring.

Synthetic data generation platform for privacy and ML

Gretel is a synthetic data platform that generates realistic, privacy-preserving datasets for ML training, testing, and data sharing. It supports tabular, text, and time-series data with configurable privacy guarantees including differential privacy. Features include data augmentation for imbalanced datasets, PII detection and anonymization, and API/SDK access for pipeline integration with BigQuery, Snowflake, and Databricks.

Agent memory system that learns, not just remembers

Hindsight is an agent memory system that enables AI agents to learn from experience rather than just store conversations. It organizes memories into three biomimetic categories: World knowledge for facts, Experiences for agent events, and Mental Models for learned understanding. The system provides retain, recall, and reflect operations backed by a temporal knowledge graph with parallel retrieval strategies including semantic, keyword, graph traversal, and temporal search.

AI Lakehouse with Feature Store for real-time ML

Hopsworks is a data-intensive AI platform combining a Python-centric Feature Store with MLOps capabilities for production ML systems. Provides sub-millisecond feature retrieval powered by RonDB, dual offline and online storage for batch and real-time inference, experiment tracking, model registry, and deployment pipelines. Available as managed cloud on AWS, Azure, and GCP, self-hosted on Kubernetes, or serverless platform.

Cloud-native POSIX filesystem on object storage

JuiceFS is a high-performance distributed POSIX filesystem built on object storage like S3 and metadata engines like Redis or MySQL. It enables seamless data sharing across thousands of clients with low latency and elastic throughput. JuiceFS ships with a Kubernetes CSI driver, Hadoop SDK compatibility, and FUSE mount support for AI training, big data analytics, and shared storage workloads. Apache 2.0 licensed with 13K+ GitHub stars.

Entity-based synthetic data generation for enterprise

K2view is an enterprise data platform that generates synthetic data using an entity-based micro-database architecture. It ensures referential integrity across complex multi-relational datasets by treating each business entity as a self-contained unit. Used for privacy-compliant test data generation, data masking, and AI training data creation in financial services, telecom, and healthcare industries.

SQL query builder for Node.js

Flexible SQL query builder for Node.js supporting PostgreSQL, MySQL, SQLite, Oracle, and MSSQL. Chainable API for constructing queries with parameterized bindings. Features schema builder, migration system, seed files, transaction support, and connection pooling. Foundation for ORMs like Bookshelf.js and Objection.js. One of the most established query builders in the Node.js ecosystem.

Open-source MLOps platform for Kubernetes

Kubeflow is a CNCF open-source MLOps platform with 14,000+ GitHub stars for deploying and managing machine learning workflows on Kubernetes. It provides notebooks for experimentation, scalable training pipelines with distributed computing support, model serving with autoscaling, and comprehensive pipeline orchestration for teams running AI/ML workloads in cloud-native environments.

Type-safe SQL query builder for TS

Type-safe SQL query builder for TypeScript that provides auto-completion and compile-time type checking without the abstraction overhead of a traditional ORM. Write queries that map directly to SQL while getting full TypeScript inference from your database schema. Supports PostgreSQL, MySQL, and SQLite with migrations, transactions, and raw SQL escape hatches. Lightweight with zero dependencies. Ideal for developers who want SQL control with TypeScript safety. Growing Knex.js alternative.

Open-source multi-type data labeling platform

Label Studio is an open-source data labeling tool by HumanSignal supporting images, text, audio, video, and time series. It offers ML-assisted pre-labeling, customizable XML-based annotation interfaces, multi-user review workflows, and REST API access. Used for computer vision, NLP, speech, and LLM fine-tuning including RLHF annotation pipelines.

Data factory for AI teams and model training

Labelbox is a comprehensive data platform for AI teams handling reinforcement learning, evaluations, robotics, and human feedback workflows. Core capabilities include RL data generation with knowledge work rubrics, custom evaluations for private benchmarks and model comparisons, robotics data with full-stack video and trajectories, and an expert network of 1.5M+ knowledge workers including 50K+ PhDs. Trusted by 80% of leading AI labs for production data operations.

Lightning-fast Rust-powered code editor

Lapce is an open-source code editor written in Rust, delivering sub-millisecond response times through native GPU rendering via wgpu. It features modal editing inspired by Vim, built-in LSP support for intelligent code completion, and a WASI-based plugin system supporting Rust, C, and AssemblyScript extensions. Includes integrated remote development over SSH, a built-in terminal, and split-pane layouts. Its rope-based text architecture ensures efficient handling of large codebases.

Modern data pipeline orchestration with built-in AI

Mage AI is an open-source data pipeline orchestration tool positioned as a modern alternative to Apache Airflow. It provides a visual pipeline editor, native AI integrations for generating pipeline code, real-time streaming support, and built-in data quality checks. Mage handles batch and streaming workloads with a developer-friendly notebook-style interface and deploys to any cloud provider.

Embedding-first search and discovery engine for AI-powered product experiences.

Marqo is an open-source tensor search engine that combines embedding generation and vector search in a single API, removing the need to manage separate embedding pipelines and vector databases. Built for product discovery and multi-modal search, it lets teams index text, images, and structured data together, returning ranked results based on semantic similarity rather than keyword overlap.

No-code AI web scraping platform with visual workflow builder

Maxun is a no-code web scraping platform that uses AI to extract structured data from websites through a visual workflow builder. Users point and click on the data they want to extract, and Maxun generates resilient scraping workflows that handle pagination, authentication, and dynamic content. Features anti-bot detection avoidance, scheduled runs, and API access for integration. Over 15,300 GitHub stars.

Declarative code-first ELT data integration

Meltano is a declarative, code-first data integration engine with 500+ Singer connectors for building ELT pipelines. It replaces custom API integration code with configuration-driven pipeline definitions that live in version control alongside application code. Integrates with dbt for transformation, supports scheduling and monitoring through a unified CLI, and powers production pipelines at scale.

SQL-native memory infrastructure for AI agents and applications

Memori is an AI memory engine that provides persistent, queryable memory for agents and applications using SQL-native storage. It stores structured memories with semantic search, temporal awareness, and relationship tracking, enabling AI systems to remember user preferences, past interactions, and contextual facts across sessions. With 12,900 GitHub stars, it offers a database-native approach to the agent memory problem.

High-performance S3-compatible object storage

MinIO is a high-performance, S3-compatible object storage server designed for AI, machine learning, and data-intensive workloads. Written in Go, it delivers industry-leading throughput for both read and write operations while maintaining full compatibility with the Amazon S3 API. MinIO includes an embedded web console for bucket management, a command-line client, and supports erasure coding, bitrot protection, and encryption at rest for enterprise-grade data durability.

Private, self-hosted research notebooks with flexible AI models, source chat, and podcasts

Open Notebook is an MIT-licensed, self-hosted alternative to NotebookLM for collecting sources, chatting over research, generating reusable transformations, and producing multi-speaker podcasts. Its Docker stack keeps notebook data under the user's control while supporting 18-plus model providers, including local Ollama and LM Studio workflows.

Open-source financial data platform for quants, analysts, and AI agents

OpenBB is an open-source financial data platform that normalizes data from 100+ providers into a unified Python SDK, REST API, and Excel Add-in. It serves as the open-source alternative to Bloomberg Terminal for developers building fintech applications, quantitative research pipelines, and AI-powered financial analysis tools. With over 65,000 GitHub stars and SOC 2 Type II certification, it is one of the most popular open-source developer tools for financial data.

AI-ready PDF parser with benchmark-leading accuracy

OpenDataLoader PDF is a high-performance parser that extracts structured, AI-ready data from PDFs with industry-leading 0.907 benchmark accuracy. Combines deterministic local processing with optional AI hybrid mode for complex layouts, OCR support across 80+ languages, formula extraction in LaTeX, chart descriptions, and built-in prompt injection filtering. Available as Python, Node.js, and Java SDKs for seamless RAG pipeline and data preparation integration.

Lightweight open-source browser-based database GUI

Outerbase Studio is a lightweight, open-source database GUI that runs directly in your browser. It supports PostgreSQL, MySQL, and SQLite with a modern interface featuring intelligent query editor with auto-completion, multi-query execution, advanced data editing with staging and preview, and high-performance table rendering for thousands of rows. Also available as an Electron desktop client for databases requiring specialized drivers.

Data versioning and pipeline automation for ML

Pachyderm is a data versioning and pipeline automation platform that provides Git-like version control for datasets with automatic data lineage tracking. Acquired by HPE, it enables reproducible ML workflows by connecting data versioning to containerized processing pipelines. Features include automatic provenance tracking, incremental processing, and deduplication for efficient storage of large datasets.

State-of-the-art OCR toolkit supporting 100+ languages from Baidu

PaddleOCR is an open-source OCR toolkit from Baidu's PaddlePaddle ecosystem with over 73,000 GitHub stars. It provides ultra-lightweight and high-accuracy text detection and recognition for 100+ languages including CJK, Arabic, and Indic scripts. The toolkit offers pre-trained models, easy deployment via pip, and server/edge inference options for document digitization workflows.

Vectorless, reasoning-based RAG that reads documents like a human expert — no vector DB, no chunking.

PageIndex is a vectorless, reasoning-based RAG system that builds hierarchical tree indexes from long documents and uses LLMs to navigate them like a human expert would. Instead of chunking text and comparing embeddings, it constructs a table-of-contents-style structure and reasons its way to the right sections — no vector database required. Available as an open-source Python package, cloud API, MCP server, and chat platform.