Best tools for Data Engineering

Conversational data analysis with natural language queries over databases

PandasAI enables natural-language queries against databases, data lakes, CSVs, and parquet files using LLMs and RAG pipelines. With 23,400+ GitHub stars, it bridges the gap between database tools and AI by letting developers and analysts interact with data conversationally, supporting SQL, PostgreSQL, and various file formats.

Real-time ETL and RAG engine with Python API and Rust core

Pathway is an open-source Python ETL framework with a high-performance Rust engine for stream processing, real-time analytics, and RAG pipelines. It handles both batch and streaming data in a unified API, enabling live-updating vector indexes, real-time document processing, and AI agent memory that refreshes as new data arrives. 63,000+ GitHub stars, used for production RAG systems where static vector databases create stale context. Apache 2.0 licensed with enterprise cloud options.

Declarative multimodal AI data infrastructure

Pixeltable is a declarative data infrastructure for multimodal AI that stores video, audio, images, and documents as first-class column types. Define Python computed columns for inference and transformations, and Pixeltable auto-orchestrates execution with incremental updates. Built-in vector search eliminates the need for separate vector databases while supporting RAG and semantic search workflows.

Lightning-fast DataFrame library in Rust

Polars is an extremely fast DataFrame library written in Rust that provides a powerful query engine for data manipulation in Python, Node.js, and R. Built on Apache Arrow columnar format, Polars delivers performance that outpaces Pandas by 10-100x on common operations through parallel execution and SIMD optimizations. It features lazy evaluation with automatic query optimization, streaming for out-of-core processing, and an expressive API for filtering, joining, and aggregating datasets.

Modern PostgreSQL client for Mac

Native macOS client for PostgreSQL with a clean, intuitive interface designed for simplicity. Features a visual table editor for inline data editing, SQL query editor with syntax highlighting, table structure inspector, favorites for saved queries, and SSH tunnel support. Known for its polished Apple-native UI and ease of use compared to heavier tools. Ideal for developers and designers who need quick PostgreSQL access without enterprise complexity.

Modern workflow orchestration for data pipelines

Prefect is an open-source workflow orchestration framework with 18K+ GitHub stars providing a Python-native approach to building, scheduling, and monitoring data pipelines. Turns any Python function into a schedulable, observable workflow with decorators. Features automatic retries, caching, concurrency controls, event-driven triggers, and a modern dashboard. Easier to adopt than Airflow with less boilerplate. Prefect Cloud provides managed orchestration with team collaboration features.

Open-source PII detection and anonymization for AI data flows

Presidio is an MIT-licensed privacy framework for identifying and anonymizing personally identifiable information in text, images, and structured data. It can act as a de-identification layer around LLM prompts, logs, RAG corpora, and customer-data workflows.

Governed analytics with a unified semantic layer

Querio turns plain English into SQL queries with a focus on governed analytics, connecting to live data warehouses and enforcing a unified semantic layer across the organization. It ensures that AI-generated queries remain consistent with business logic definitions, preventing the common problem where different teams get different answers to the same question from the same data.

All-in-one multimodal RAG framework

RAG-Anything is an all-in-one multimodal RAG framework from the University of Hong Kong that processes text, images, tables, and equations through a unified pipeline built on LightRAG. It constructs multi-modal knowledge graphs by extracting multimodal entities and establishing cross-modal relationships. The VLM-Enhanced Query mode integrates visual content into large language models for deeper document understanding beyond plain text retrieval.

Deep document understanding RAG engine

RAGFlow is an open-source RAG engine with 76K+ GitHub stars that provides deep document understanding for building knowledge-based AI applications. Optimizes chunking for 20+ document types including PDFs, Word docs, presentations, and images using layout-aware parsing. Features template-based chunking strategies, citation with source references, multi-recall retrieval combining keyword and semantic search, and a visual knowledge base management interface with drag-and-drop document upload.

Container-native local AI model serving with Podman

RamaLama is an open-source tool that containerizes AI model inference using Podman or Docker, eliminating host system configuration complexity. It auto-detects GPUs (NVIDIA, AMD, Intel, Apple Silicon), pulls models from HuggingFace, Ollama, and OCI registries, and runs them in isolated rootless containers with read-only mounts and network isolation. Developed under the Containers project (Red Hat ecosystem), it brings familiar container workflows to local LLM serving.

Kafka-compatible streaming platform, no JVM required

Redpanda is a Kafka-compatible streaming data platform written in C++ using the Seastar framework. It eliminates the need for ZooKeeper and the JVM, delivering up to 10x lower tail latencies and significantly reduced operational complexity. Redpanda ships as a single binary with a built-in schema registry, HTTP proxy, and message broker. It supports the Kafka wire protocol, so existing producers, consumers, and tools work without code changes. Backed by $165M+ in funding with 12.0K GitHub stars.

React meta-framework for enterprise CRUD apps

Refine is an open-source React meta-framework for building data-intensive enterprise applications like admin panels, dashboards, and internal tools. Its headless architecture separates business logic from UI, supporting Ant Design, Material UI, Mantine, and Tailwind CSS. Connects to 15+ backends including REST, GraphQL, Supabase, Firebase, Strapi, and Hasura with built-in authentication and role-based access control. Deploys on Next.js, Remix, React Native, and Electron.

High-performance S3-compatible object storage built in Rust

RustFS is an open-source distributed object storage system built entirely in Rust, offering 2.3x faster performance than MinIO for small object payloads. It provides full S3 API compatibility, enabling seamless migration from MinIO, Ceph, and AWS S3 with existing SDKs and CLI tools. Released under Apache 2.0 license, it avoids MinIO's restrictive AGPL terms. Features include distributed architecture, erasure coding, WORM compliance, encryption via RustyVault, and a web management console.

Multi-database AI query generator and optimizer

SQLAI.ai is a comprehensive AI toolkit for generating, optimizing, and explaining SQL queries across any database engine including both SQL and NoSQL systems. Supporting schemas with 600+ tables, it handles the full query lifecycle from natural language generation to performance optimization and cross-database migration, starting at $5/month as one of the most affordable AI SQL tools available.

LLM-powered web scraping with graph-based extraction pipelines

ScrapeGraphAI is a Python library that uses LLMs and graph-based logic to build automated, self-healing web scraping pipelines. Developers describe desired data in natural language and ScrapeGraphAI constructs a processing graph that extracts structured information from any website. It supports multiple LLM providers, achieves 96%+ accuracy on semantic extraction benchmarks, and adapts to layout changes automatically. Over 20,000 GitHub stars.

AI-native state store with hybrid vector and full-text search

SeekDB is an open-source AI-native state store from the OceanBase ecosystem that combines MySQL-compatible data access with hybrid vector and full-text retrieval. It targets agent and AI application teams that need embedded or server deployment, copy-on-write style sandboxes, and searchable state without gluing together several separate storage layers.

Promise-based Node.js ORM

Mature, promise-based ORM for Node.js supporting PostgreSQL, MySQL, MariaDB, SQLite, and SQL Server. Features model definitions with validations, eager/lazy loading, transactions, migrations, raw queries, and lifecycle hooks. Supports soft deletes, scopes, and virtual fields. One of the oldest and most battle-tested Node.js ORMs, widely used in enterprise apps though increasingly succeeded by Prisma and Drizzle in new projects.

Browser-based AI SQL editor with modern UX

Sheeta.ai is a browser-based AI SQL editor designed to modernize the data workflow with a simple, intuitive interface and high-accuracy query generation. It addresses the UX gaps of traditional database tools like DBeaver by providing a clean, modern interface with AI-powered query suggestions, schema visualization, and result exploration built for the contemporary data stack.

Data-centric AI platform for programmatic data labeling

Snorkel AI is a data-centric AI platform that enables programmatic labeling of training data through labeling functions rather than manual annotation. Spun out of Stanford AI Lab, it lets teams write Python functions that encode domain heuristics to label data at scale, with the platform combining weak labels into high-quality training sets. Used by Fortune 500 companies for text, image, and structured data labeling.

AI schema generator from natural language

Structa is an AI tool that turns natural language descriptions into production-ready database schemas and SQL in under 45 seconds, focused on PostgreSQL and Supabase. Recognized as an editor's choice for AI schema generation, it is the fastest idea-to-schema tool available, generating optimized table structures, relationships, indexes, and constraints from plain English project requirements.

Reusable computer vision tools for developers

Supervision is an open-source Python toolkit by Roboflow providing reusable CV utilities for detection, tracking, annotation, and dataset management. It works with any model including YOLO and Hugging Face via a standardized Detections class. Features include 20+ annotators, ByteTrack object tracking, zone counting, speed estimation, and dataset conversion between COCO, YOLO, and Pascal VOC formats.

Multi-model database for the AI era — document, graph, vector, and relational in one

SurrealDB is a multi-model database that natively combines document, graph, relational, key-value, and vector storage in a single engine. It eliminates the need for separate databases by handling structured queries, graph traversals, full-text search, and vector similarity in one SQL-like query language called SurrealQL. Built in Rust for performance and safety, it supports real-time subscriptions, row-level permissions, and embedded or distributed deployment modes.

Open-source library for generating synthetic tabular data

Synthetic Data Vault (SDV) is an MIT-backed open-source Python library for generating synthetic tabular, relational, and time-series data. It learns statistical patterns from real datasets and produces synthetic versions that preserve distributions, correlations, and referential integrity. Supports single-table, multi-table, and sequential data with built-in privacy and quality metrics.

Modern database management GUI

Native database management GUI for macOS, Windows, and Linux that provides a fast, clean interface for working with PostgreSQL, MySQL, SQLite, SQL Server, Redis, CockroachDB, and more. Features an intuitive query editor with auto-completion, visual table editor for inline data editing, import/export in multiple formats, SSH tunnel support, and connection management. Known for its speed and polished UI compared to heavier tools like DBeaver. Popular among Mac developers.

Enterprise feature platform for real-time ML

Tecton is an enterprise feature platform for building and serving ML features at scale. Created by the team behind Feast, it provides managed feature engineering, real-time feature computation from streaming data, feature monitoring, and a unified feature store with offline/online consistency. Used by production ML teams to eliminate training-serving skew and accelerate model deployment cycles.

Managed Postgres platform with 200+ extensions as pre-built stacks

Tembo is a managed PostgreSQL platform that packages 200+ Postgres extensions into purpose-built stacks for specific workloads. Stacks include OLAP analytics, vector search, message queues, geospatial, and machine learning, turning PostgreSQL into a specialized database for each use case. Eliminates the need for separate Redis, Elasticsearch, or Kafka instances alongside Postgres.

Hugging Face's open-source inference server for embeddings, rerankers, and classifiers

Text Embeddings Inference is Hugging Face's Apache-2.0 server for high-throughput embedding, reranking, and sequence-classification models. TEI packages token-based dynamic batching, optimized Transformers kernels, Safetensors loading, OpenAI-compatible embedding endpoints, Prometheus metrics, and configurable OpenTelemetry tracing in deployable CPU and GPU images.

Secure text-to-SQL with schema-only processing

Text2SQL.ai is a secure text-to-SQL platform that uses a schema-only model where actual database data never leaves the developer's infrastructure. It provides an affordable solution for teams building internal analytics tools with a public API for embedding SQL generation into custom applications, prioritizing data privacy for organizations in regulated industries.

Google's pretrained foundation model for zero-shot time-series forecasting

TimesFM is a pretrained time-series foundation model from Google Research that performs zero-shot forecasting on diverse datasets without task-specific training. It handles univariate and multivariate time series across domains including finance, logistics, energy, and infrastructure monitoring with accuracy competitive against traditional statistical methods like ARIMA and Prophet.

SQLite for production

Edge-hosted distributed database built on libSQL (an open-source fork of SQLite) designed for low-latency data access worldwide. Features multi-region replication, embedded replicas that sync to your application server for microsecond reads, database branching for development workflows, and point-in-time recovery. Ideal for edge computing, serverless functions, and mobile apps. Compatible with SQLite ecosystem tooling. Generous free tier with 9GB storage and 500 databases.

ORM for TypeScript and JavaScript

Full-featured ORM for TypeScript and JavaScript supporting Active Record and Data Mapper patterns. Works with PostgreSQL, MySQL, MariaDB, SQLite, Oracle, SQL Server, and MongoDB. Features decorator-based entity definitions, migrations, relations (one-to-one, many-to-many), lazy/eager loading, query builder, transactions, and caching. Supports both Node.js and browser runtimes. 34K+ GitHub stars. Mature but losing mindshare to Prisma and Drizzle for new TypeScript projects.

Fast embeddable vector search engine

USearch is a high-performance vector search engine implementing HNSW algorithms for approximate nearest neighbor queries across C++, Python, JavaScript, Rust, Java, Go, and more. It supports user-defined distance metrics, memory-mapped persistence for datasets larger than RAM, and filtered search with predicates. Used by YugabyteDB and ScyllaDB as their production vector indexing backend.

Cloud-native distributed vector search engine built for Kubernetes with automatic indexing and horizontal scaling.

Vald is a highly scalable distributed approximate nearest neighbor (ANN) vector search engine designed for cloud-native, Kubernetes-based architectures. Maintained by LY Corporation and listed in the CNCF Landscape, it uses the NGT algorithm (developed at Yahoo Japan), supports automatic incremental index backup, and handles billion-scale datasets across loosely coupled microservice components that scale horizontally via Helm.

One vector-search extension across PostgreSQL, DuckDB and SQLite

MIT-licensed vector-search extension for PostgreSQL, DuckDB and SQLite that shares one graph-index core with PQ/RaBitQ quantization, persistent indexes and metadata filtering; SQLite packages cover Linux, macOS, iOS, Android and WASM, so it runs inside existing databases instead of as a separate vector service.

Enterprise RAG framework by Tencent

WeKnora is a Tencent-developed LLM-powered knowledge management and Q&A framework for enterprise document understanding and semantic retrieval. Supports 10+ document formats including PDF, Word, Excel, and images with seamless IM platform integration for WeCom, Feishu, Slack, and Telegram. Offers Quick Q&A mode using RAG pipelines and Intelligent Reasoning mode with ReACT agents for complex multi-step reasoning tasks across organizational knowledge bases.

Serverless database with search and AI built-in

Serverless database platform that combines Postgres, full-text search, analytics, and AI features in a single service. Built-in vector search for AI applications, branching for safe schema changes, and a spreadsheet-like UI for data exploration. Designed for developers who want powerful database capabilities without managing separate services for search, analytics, and embeddings.

In-process vector database — the SQLite of vector DBs

Zvec is an open-source in-process vector database from Alibaba designed as the SQLite of vector search. It runs as an embedded library directly inside applications without requiring external servers, delivering 8,000+ QPS with high recall rates. Zvec supports dense and sparse embeddings, multi-vector queries, and combined semantic plus structured filtering. Built on Alibaba's proven Proxima engine, it provides a lightweight alternative to server-based vector databases for local AI workflows.

SQL-based data transformation framework

dbt (data build tool) is an open-source SQL transformation framework with 10K+ GitHub stars that lets analytics engineers transform data in their warehouse using select statements. Brings software engineering practices to data — version control, testing, documentation, and CI/CD for SQL. Supports Snowflake, BigQuery, Redshift, Databricks, PostgreSQL, and more. Features Jinja templating, incremental models, snapshots, and a package hub of reusable transformations.

Python library for declarative data loading that LLMs can generate

dlt (data load tool) is a Python library for building data pipelines with declarative, schema-aware loading that is simple enough for LLMs to generate correctly. It extracts data from APIs, databases, and files, normalizes nested structures, handles schema evolution, and loads into warehouses and lakes. Supports 30+ destinations including BigQuery, Snowflake, DuckDB, and PostgreSQL. Over 5,200 GitHub stars.

Header-only C++ implementation of HNSW for fast approximate nearest-neighbor search.

hnswlib is a header-only C++ library implementing the Hierarchical Navigable Small World (HNSW) graph algorithm for approximate nearest-neighbor search, with Python bindings and a tiny dependency footprint. Originally developed by the nmslib team, it has become the default HNSW implementation embedded inside many vector databases and search products. Engineers use it directly when they want HNSW retrieval without pulling in a heavyweight vector DB.

Git-like version control for data lakes and object storage

lakeFS is an open-source platform that brings Git-like branching, committing, and merging to data lakes and object storage. It works on top of S3, GCS, Azure Blob, and MinIO, enabling teams to create isolated data branches for experimentation, run CI/CD for data pipelines, and maintain full data lineage. Acquired DVC in 2025, uniting data version control for both small and enterprise-scale workloads.

ModelScope's fine-tuning framework supporting 600+ models

ms-swift is ModelScope's open-source framework for fine-tuning over 600 large language and multimodal models. It supports SFT, DPO, RLHF, LoRA, QLoRA, and full fine-tuning with a web UI and CLI interface. Optimized for the Chinese AI ecosystem with native ModelScope Hub integration alongside Hugging Face support. Over 13,500 GitHub stars.

PostgreSQL administration tool

The most popular open-source administration and management tool for PostgreSQL, used by millions of DBAs and developers worldwide. Features a powerful SQL query editor with auto-completion, visual query builder (graphical EXPLAIN), server dashboard with real-time monitoring, backup/restore wizards, ERD generator for schema visualization, and user/role management. Runs as a web application accessible via browser. Supports PostgreSQL 12+ and is available on Windows, macOS, Linux, and Docker.

BM25 full-text search extension for PostgreSQL

pg_textsearch is a PostgreSQL extension from Timescale that adds BM25 relevance-ranked full-text search directly inside Postgres. Using the same ranking algorithm as Elasticsearch and Lucene, it provides search-engine quality results without requiring a separate search cluster — particularly valuable for developers building RAG pipelines on PostgreSQL who want semantic-quality ranking alongside pgvector.

Vector search extension for SQLite that runs anywhere

sqlite-vec is a lightweight vector search extension for SQLite written in pure C with zero dependencies. It brings nearest-neighbor search capabilities directly into SQLite databases, enabling AI applications to store and query embeddings without running a separate vector database. The extension works everywhere SQLite runs including Linux, macOS, Windows, WebAssembly in browsers, and even Raspberry Pi devices. Sponsored by Mozilla Builders, Fly.io, and Turso.

CI-friendly database documentation generator

tbls is an open-source database documentation tool that automatically generates schema documentation in Markdown, with built-in linting to enforce documentation standards and coverage metrics for tables and columns. It supports 13+ databases including PostgreSQL, MySQL, BigQuery, Snowflake, MongoDB, and ClickHouse. Designed for CI integration with GitHub Actions support, tbls runs schema diff detection and documentation enforcement as part of automated pipelines.

Production-grade reinforcement learning framework for LLM training

verl is an open-source reinforcement learning framework designed specifically for training and aligning large language models. Built for production use with support for distributed training across multiple GPUs and nodes, it implements RLHF, DPO, and other alignment algorithms that make LLMs follow instructions, avoid harmful outputs, and generate higher quality responses. Over 580 contributors and 20,000 GitHub stars signal strong adoption.