Evaluation architecture and handoffs
This stack separates RAG quality into five accountable layers instead of asking one dashboard for a single accuracy score. RAGAS measures retrieval and answer behavior, DeepEval turns expectations into test assertions, Langfuse connects scores to traces and datasets, Arize Phoenix helps diagnose retrieval and embedding failures, and GitHub Actions makes the checks repeatable before release. Each component owns a distinct stage, so a failing score can be traced to data, retrieval, generation, or deployment rather than treated as an unexplained model problem.
The design is best for teams that already have a working retrieval pipeline and need regression evidence as prompts, chunking, embeddings, rerankers, or source documents change. It does not create a trustworthy golden dataset automatically, and no LLM-as-judge metric should be treated as ground truth without calibration. Human-reviewed examples, failure labels, and explicit acceptance thresholds remain the control surface; the tools automate evaluation and diagnosis around that evidence.
Build the dataset and metric layer
Start with RAGAS on a narrow dataset that includes the user question, retrieved context, generated answer, and any known reference answer. Faithfulness, context precision, context recall, and answer relevancy expose different failure modes, so combine them only after each metric has a documented decision rule. Keep difficult negatives, ambiguous questions, missing-source cases, and citation failures in the suite because an average score can hide exactly the incidents that matter in production.
Use DeepEval to express release-facing assertions in the same workflow as application tests. Deterministic checks should cover schema, citations, required phrases, latency budgets, and tool-call shape; judge-backed checks can handle relevance, groundedness, and rubric-based quality. Pin judge prompts and evaluation settings, record the model used for each run, and rerun a stable calibration subset whenever the judge changes. That discipline prevents a judge upgrade from looking like an application improvement.
Trace and diagnose retrieval failures
Langfuse is the trace and dataset bridge. Capture the prompt, retrieved passages, model response, latency, token usage, and evaluation scores on a shared trace identifier, then promote representative failures into versioned datasets. Production sampling should be deliberate: retain enough context to investigate regressions without storing secrets or personal data by default. Scores are most useful when they remain linked to the exact prompt, retriever, model, and release that produced them.
Arize Phoenix adds a retrieval-analysis workspace for examining embeddings, document relevance, span behavior, and clusters of weak results. Use it when a RAGAS score says retrieval deteriorated but the cause is unclear: source drift, chunk boundaries, embedding mismatch, metadata filters, or reranking can produce similar symptoms. Phoenix does not replace the release suite; it shortens the path from a failed metric to a concrete retrieval experiment and a reproducible fix.
Make evaluation a CI release gate
GitHub Actions should run a fast deterministic and small judge-backed suite on every pull request, then reserve larger datasets and red-team-style cases for scheduled or pre-release jobs. Cache dependencies, cap concurrency, and separate code failures from external model-provider failures so a temporary API problem does not silently become a pass. Upload machine-readable results and a human summary as artifacts, and require the responsible team to review any accepted threshold change in the same pull request.
A practical gate compares the candidate branch with a stored baseline rather than demanding perfect absolute scores. Block releases on severe groundedness or citation failures, statistically meaningful regressions, missing evaluation data, or unexplained cost and latency jumps. Allowing a bypass may be necessary for incidents, but the exception should name an owner, reason, expiry, and follow-up dataset. The goal is a visible engineering decision, not an opaque green badge.
Rollout, cost, and failure modes
Roll out the stack in stages: instrument one RAG route, curate a small human-reviewed set, add RAGAS and DeepEval locally, connect Langfuse traces, use Phoenix on recurring retrieval failures, and only then enforce GitHub Actions gates. Track dataset versions and false-positive rates alongside scores. If engineers routinely ignore a metric, either repair its rubric or remove it; accumulating low-trust checks makes the pipeline slower without improving release confidence.
The budget range varies because the open-source components can be self-hosted while model calls, embeddings, trace retention, CI minutes, and managed hosting create real operating cost. This stack is justified when retrieval changes frequently or a wrong answer carries support, compliance, or revenue risk. A small internal prototype with a stable corpus may need only a compact offline dataset and one test runner until its usage and failure cost warrant the full observability loop.