aicoolies logo
Pathway logo
Pathway logo

Pathway

Real-time ETL and RAG engine with Python API and Rust core

open sourceupdated Jul 9, 2026

Pathway is an open-source Python ETL framework with a high-performance Rust engine for stream processing, real-time analytics, and RAG pipelines. It handles both batch and streaming data in a unified API, enabling live-updating vector indexes, real-time document processing, and AI agent memory that refreshes as new data arrives. 63,000+ GitHub stars, used for production RAG systems where static vector databases create stale context. Apache 2.0 licensed with enterprise cloud options.

Pathway bridges the gap between data engineering and AI agent orchestration by providing a unified framework for real-time data processing. Traditional RAG pipelines suffer from stale context — documents are indexed once and queries hit outdated embeddings. Pathway solves this with streaming ETL that continuously processes incoming data, updates vector indexes in real-time, and serves fresh context to LLM applications without batch reprocessing jobs.

The architecture features a Python API for developer ergonomics backed by a Rust execution engine for performance. You write transformation logic in familiar Pandas-like syntax while Pathway handles parallelism, fault tolerance, and incremental computation under the hood. Built-in connectors support Kafka, S3, PostgreSQL, Google Drive, SharePoint, and dozens of other data sources. The LLM integration layer includes document parsers, embedding generators, and vector index maintenance — a complete real-time RAG stack in one framework.

With 63,000+ GitHub stars, Pathway is one of the most-starred AI data infrastructure projects. It is Apache 2.0 licensed with a managed cloud offering for production deployments. The project is particularly relevant for teams building AI agents that need continuously updated context — competitive intelligence monitoring, real-time document analysis, and live knowledge bases that evolve alongside the data they represent.

Pricing

Free open-source (Apache 2.0); Enterprise cloud available

Platforms

Python library with Rust engine; Docker, Cloud managed

Categories

Tags

Use Cases

Kestra logo

Kestra

Declarative orchestration for data, AI, and infra

Kestra is an open-source orchestration platform that uses declarative YAML to define event-driven and scheduled workflows for data pipelines, infrastructure automation, and AI workloads. With over 1,200 plugins, it connects to databases, cloud services, APIs, and SaaS tools without custom glue code. Kestra reached version 1.0 LTS with agentic AI capabilities, SDKs for Python, TypeScript, Java, and Go, and SOC 2 compliance. Clients include Leroy Merlin, Huawei, Tencent, and Decathlon.

freemiumOpen Source
LangFlow logo

LangFlow

Visual framework for building multi-agent AI apps

LangFlow is an open-source visual framework for building multi-agent AI apps with drag-and-drop. Built on LangChain, it lets developers compose chains, agents, and RAG pipelines by connecting modular components visually. Features real-time interaction, Python customization, one-click deployment, and export to LangChain code. Supports all major LLM providers, vector stores, and tools. With 146K+ GitHub stars, it bridges visual prototyping and production deployment.

Open Source
Mastra logo

Mastra

TypeScript AI agent framework

TypeScript-native framework for building AI agents and workflows with great developer experience. Provides primitives for agents with tool calling, RAG pipelines, workflow orchestration with branching/parallel steps, and integration connectors. First-class TypeScript support with type-safe tool definitions. Local dev server with playground UI for testing. Growing as a LangChain alternative for TypeScript developers building AI apps.

Open Source

Related Tools

computed discovery: shared active categories · kept separate from editor-verified Alternatives

KTransformers parent kvcache-ai logo

KTransformers

Heterogeneous CPU-GPU inference and SFT for large MoE models

Open-source framework for running and fine-tuning large Mixture-of-Experts models with heterogeneous CPU-GPU execution, optimized kernels, limited VRAM and SGLang or LLaMA-Factory integrations.

Open Source
vLLM Production Stack parent vLLM logo

vLLM Production Stack

Official Kubernetes and Helm reference stack built on the vLLM inference engine

Official vLLM reference implementation for scaling the existing inference engine on Kubernetes with Helm, request routing, KV-cache offload, autoscaling and Prometheus/Grafana observability.

Open Source
Dynamo logo

NVIDIA Dynamo

Distributed inference orchestration above vLLM, SGLang and TensorRT-LLM

Open-source, datacenter-scale orchestration layer that coordinates vLLM, SGLang and TensorRT-LLM across nodes with disaggregated serving, KV-aware routing, multi-tier cache management and automatic scaling.

Open Source
GPUStack logo

GPUStack

Open-source GPU control plane for scalable AI model serving

Open-source GPU cluster manager that configures vLLM, SGLang, TensorRT-LLM or custom engines, serves models through compatible APIs, and provisions SSH-accessible GPU instances across on-premises, Kubernetes and cloud environments.

Open Source
Mooncake logo

Mooncake

Disaggregated KV cache storage and transfer for LLM serving

Open-source infrastructure for disaggregated LLM serving that pools KV caches across prefill and decode workers, with high-performance transfer, distributed storage and integrations for vLLM and SGLang.

Open Source
LMCache logo

LMCache

Reusable KV cache infrastructure for scalable LLM inference

Open-source KV cache management layer that persists, offloads and reuses model key-value caches across requests and serving engines to reduce repeated prefill work and improve inference throughput.

Open Source

Used in Stacks

FAQ

What is Pathway?

Pathway is an open-source Python ETL framework with a high-performance Rust engine for stream processing, real-time analytics, and RAG pipelines. It handles both batch and streaming data in a unified API, enabling live-updating vector indexes, real-time document processing, and AI agent memory that refreshes as new data arrives. 63,000+ GitHub stars, used for production RAG systems where static vector databases create stale context. Apache 2.0 licensed with enterprise cloud options.

Is Pathway free?

Yes — Pathway is open source and free to use. Free open-source (Apache 2.0); Enterprise cloud available

Is Pathway open source?

Yes — Pathway is open source.

What are the best Pathway alternatives?

The top editor-verified Pathway alternatives are Kestra, LangFlow, Mastra.