Kreuzberg

ETL for LLMs — preprocess any document format

Unstructured is an open-source ETL library that preprocesses and transforms documents from diverse formats into clean, structured data ready for LLM ingestion and RAG pipelines. It handles PDF, HTML, Word, PowerPoint, and many other file types through partitioning, cleaning, and chunking operations. The library offers connector-based architecture for integrating with various data sources and destinations, making it a key component in document processing workflows for AI applications.

Get your documents ready for gen AI

Docling is an open-source document processing toolkit by IBM Research that converts complex documents into structured formats optimized for generative AI applications. It parses PDF, DOCX, PPTX, XLSX, HTML, images, audio, and LaTeX with advanced PDF understanding including layout analysis, reading order detection, and table structure recognition. Docling exports to Markdown, HTML, JSON, and DocTags, and integrates natively with LangChain, LlamaIndex, and other AI frameworks for RAG workflows.

Convert any file to Markdown for LLM pipelines

MarkItDown is a lightweight Python utility by Microsoft that converts files into clean Markdown optimized for LLM pipelines and text analysis. It supports PDF, Word, Excel, PowerPoint, HTML, images with OCR, audio with transcription, and text formats like CSV, JSON, and XML. The tool preserves document structure including headings, tables, lists, and links while keeping output token-efficient. It offers a CLI, a four-line Python API, Docker support, and a plugin architecture for extensions.

All-in-one embeddings database with RAG, search, and agent capabilities

txtai is a self-contained AI search and RAG platform that combines vector embeddings, semantic search, LLM pipelines, and agent workflows in a single Python library. It handles embedding generation, similarity search, extractive QA, summarization, translation, and custom pipelines without external dependencies. Runs locally with over 12,400 GitHub stars and Apache 2.0 license.

Distributed AI compute engine for scaling Python and ML workloads

Ray is an open-source distributed computing framework built for scaling AI and Python applications from a laptop to thousands of GPUs. It provides libraries for distributed training, hyperparameter tuning, model serving, reinforcement learning, and data processing under a single unified API. Ray's public site highlights OpenAI and other enterprise users. Maintained by Anyscale with Apache-2.0 open-source licensing.

Unified framework for fine-tuning 100+ large language models

LLaMA-Factory is an open-source toolkit providing a unified interface for fine-tuning over 100 LLMs and vision-language models. It supports SFT, RLHF with PPO and DPO, LoRA and QLoRA for memory-efficient training, and continuous pre-training. The LLaMA Board web UI enables no-code configuration, while CLI and YAML workflows serve advanced users. Integrates with Hugging Face, ModelScope, vLLM, and SGLang for model deployment.

Official MCP server for GitHub repo operations

GitHub MCP Server is the official Model Context Protocol server from GitHub that connects AI assistants to repositories, issues, pull requests, workflows, and code search. It exposes 100+ operations with toolset filtering, permission scoping, and audit logging, available in both remote-hosted and self-hosted Docker deployment modes.

Up-to-date docs for AI code editors via MCP

Context7 is an MCP server developed by Upstash that injects up-to-date, version-specific documentation directly into AI code editors and coding assistants. By typing 'use context7' in prompts, developers get accurate library documentation instead of hallucinated or outdated API references. It pulls from official source documentation and serves it through the Model Context Protocol, solving the common problem of LLMs generating code with incorrect or nonexistent API calls.

Apache-2.0 Python framework for repeatable LLM, RAG, agent, MCP, and safety evaluation workflows.

DeepEval is an Apache-2.0 Python framework for evaluating LLM apps, RAG systems, agents, MCP workflows, and safety behavior with repeatable test cases. It works locally and in CI/CD, then connects to Confident AI for hosted reports, observability, red teaming, and governance when teams need shared evidence instead of ad-hoc prompt reviews and manual QA.

Turn websites into LLM-ready structured data

Firecrawl is a Y Combinator-backed API that crawls websites and converts them into clean, LLM-ready Markdown or structured JSON. Handles JavaScript rendering, pagination, sitemaps, and anti-bot measures automatically. Designed for RAG pipelines, AI agents, and data extraction workflows. Features batch crawling, scheduled scraping, webhook notifications, and custom extraction schemas. Processes content for direct ingestion into vector databases and LLM context windows.