Continuous evaluation architecture
This stack turns evaluation into a release workflow rather than a one-time benchmark. Promptfoo runs prompt and provider matrices, DeepEval expresses code-native assertions, RAGAS measures retrieval behavior where context is involved, Langfuse links offline cases to production traces, and garak adds a focused security probe stage. The pipeline separates fast pull-request checks from larger scheduled and pre-release suites so teams get useful feedback without making every code change wait for the full test catalog.
Use it for applications whose prompts, models, retrieval settings, or tool behavior change often enough that manual spot checks no longer protect releases. Define an evaluation contract before choosing thresholds: critical business rules, semantic quality dimensions, RAG metrics, safety cases, cost limits, and latency budgets. A green result means the candidate met that documented contract on the tested slices; it does not prove universal model quality or eliminate the need for production monitoring.
Build the regression matrix
Promptfoo is the matrix layer for testing prompts, models, providers, variables, and assertions from version-controlled configuration. Keep the core suite small and deterministic enough to run on every pull request, then add representative semantic graders where exact checks cannot express the requirement. Pin evaluation inputs and record provider and model identifiers so a remote default change is visible. Compare candidate results with a stored baseline instead of relying only on absolute pass rates.
DeepEval complements the matrix with Python and pytest-friendly metrics, custom rubrics, conversational tests, and application-specific validators. Use deterministic assertions for schemas, citations, prohibited actions, required tool events, and business invariants; reserve LLM judges for semantic qualities that have been calibrated against human labels. A failing judge should retain the input, output, rubric, judge configuration, and reason so an engineer can reproduce and challenge the decision.
Add RAG and production evidence
RAGAS supplies context precision, context recall, faithfulness, and answer relevancy for retrieval-backed paths. Run those metrics only when the test case includes the evidence needed to interpret them, and keep difficult retrieval negatives alongside ordinary questions. Dataset versions should track source snapshots, chunking, embedding, and reranker settings because the same prompt can regress when the corpus or retrieval layer changes.
Langfuse connects production traces to the offline suite. Sample incidents, low-confidence outputs, costly traces, and representative successful runs into reviewed datasets, then canary a release against real traffic slices before broad rollout. Link scores to prompt, model, retriever, and application versions. Production feedback should improve the regression suite, while privacy and retention rules limit which prompts, contexts, and outputs are safe to keep.
Run security and release gates
garak adds a focused pre-production probe stage for jailbreak, prompt-injection, disclosure, and other model-facing weaknesses relevant to the application. Run a narrow trusted subset on release candidates and a broader campaign on a schedule, recording scanner configuration and target version. Triage and reproduce severe findings before treating them as confirmed; scanner errors and provider refusals should not silently reduce coverage or produce a misleading pass.
Structure CI in tiers: lint and deterministic checks on every commit, Promptfoo and DeepEval regression tests on pull requests, RAGAS cases when retrieval code or data changes, and garak plus the large evaluation set before release. Block on critical policy violations, statistically meaningful regressions, missing results, or unexplained cost and latency jumps. Any bypass needs a named owner, reason, expiry, and follow-up issue.
Operate the feedback loop
Review the suite as a product. Remove checks that teams repeatedly ignore, repair rubrics with high disagreement, expand slices after real incidents, and watch evaluation cost and duration. Store machine-readable artifacts for trend analysis and a concise human summary for release review. When a threshold changes, make the rationale and baseline comparison part of the code review so evaluation policy cannot drift invisibly.
The budget varies with provider calls, judge usage, retrieval evaluation, trace retention, CI minutes, and security probing. Start with twenty to fifty high-value cases and one production canary before scaling to thousands. The full stack is unnecessary for a stable low-risk prototype, but it becomes valuable when multiple teams ship model or prompt changes and need a shared, auditable answer to whether the candidate is safer and more reliable than the current release.