Promptfoo

Programming — not prompting — LLMs

Declarative framework from Stanford University for programming language models rather than prompting them. DSPy treats LLM interactions as programmable modules with input-output signatures and uses optimization algorithms to automatically compile these modules into effective prompts or fine-tuned weights, replacing brittle prompt strings with structured, modular AI software.

Type-safe LLM function builder

BAML is a domain-specific language by BoundaryML for building reliable AI workflows and agents through schema engineering. It turns prompt engineering into a structured, type-safe discipline by letting developers declaratively define function schemas, validate LLM responses, and version prompts without fragile JSON parsing or boilerplate. BAML reframes prompt engineering as schema definition, making AI workflows testable and maintainable across models.

Structured LLM outputs with validation

Instructor is the most popular Python library for extracting structured, validated data from large language models, with over 3 million monthly downloads and ports across Python, TypeScript, Go, Ruby, Elixir, and Rust. It uses Pydantic models to define output schemas and automatically handles validation, retries, and error correction when the LLM output does not match. Instructor patches existing client libraries instead of replacing them, preserving full access to the underlying API.

Open-source LLMOps platform for prompt management and evaluation

Agenta is an open-source LLMOps platform that combines prompt engineering playgrounds, prompt version management, LLM evaluation, and observability in a unified interface. It supports 50+ LLM models with side-by-side prompt comparison, A/B testing, human evaluation workflows, and OpenTelemetry-native tracing. Self-hostable with 4,000+ GitHub stars.