Judge architecture and boundaries
This stack treats an LLM judge as a measured component rather than an automatic source of truth. DeepEval defines rubrics and judge-backed metrics, Opik organizes datasets and experiment runs, Langfuse attaches scores to production traces, Arize Phoenix supports analysis of weak or drifting judgments, and LangSmith offers an optional managed path for teams already operating LangChain workflows. The tools can share evaluation evidence, but the rubric, calibration set, and release policy remain owned by the application team.
Use the stack when human review is too slow to cover every prompt, conversation, or agent run but important quality dimensions still require semantic judgment. Do not use it to hide an undefined product requirement behind a numeric score. Exact-match checks, schema validation, business rules, security policies, and human escalation should remain deterministic wherever possible. The judge is appropriate for qualities such as relevance, completeness, style, groundedness, or task success only after those qualities are written as observable criteria.
Design rubrics and calibration sets
DeepEval provides the metric construction layer for G-Eval-style rubrics, DAG-based decisions, conversational checks, and custom criteria. Write one narrow rubric per decision and include both positive and negative examples. A useful rubric names the input evidence, allowed assumptions, severity levels, and conditions that force a failure. Avoid combining factuality, tone, safety, and usefulness into one prompt because a single score cannot explain which requirement the candidate violated.
Calibrate every judge against a human-labeled set before using it as a gate. Measure agreement by class and inspect disagreements rather than relying only on an overall correlation number. Test position order, answer length, self-preference, verbosity, and reference leakage because judge models can reward superficial signals. Preserve the judge model, prompt, temperature, rubric version, and label provenance with each run so a later score change can be separated from an application change.
Run datasets and controlled experiments
Opik is the experiment ledger for datasets, candidate outputs, feedback scores, and repeated evaluation runs. Version examples instead of editing them in place, keep production failures beside intentionally constructed edge cases, and tag slices such as language, customer segment, workflow, or risk level. Slice-level results matter because a stable global average can conceal a regression for long contexts, non-English prompts, tool failures, or a high-value customer workflow.
Use Opik to compare prompts, models, retrieval settings, and judge configurations under the same dataset contract. Require every experiment to state a hypothesis and a promotion rule before results are visible, which reduces the temptation to select whichever score looks best. When judge and candidate models share a provider or model family, include an independent judge or human audit slice to detect self-preference and correlated blind spots.
Score and investigate production behavior
Langfuse connects judge outputs to production traces, prompt versions, releases, cost, and latency. Sample traces by risk and uncertainty rather than evaluating every request blindly, then route low-confidence or severe failures to human review. Production scores should be stored with the judge configuration and input evidence, not as context-free labels. That linkage lets a team reproduce the decision and determine whether the problem came from the application, the evaluator, or missing context.
Arize Phoenix is the analysis layer when judge results drift or disagree with users. Group failures by prompt version, trace span, embedding behavior, model, latency, or dataset slice, then inspect clusters rather than individual anecdotes. Phoenix can reveal whether a drop aligns with a retrieval change or a new traffic pattern, but it cannot establish fairness or correctness by itself. Human audit sets remain necessary for sensitive domains and for any rubric tied to compliance or customer-impact decisions.
Operate, govern, and know when to stop
LangSmith is an optional operations layer for teams already invested in LangChain tracing, datasets, evaluators, and CI workflows. It can consolidate managed runs and review queues, but adding it beside Opik and Langfuse should solve a specific ownership or integration need. Otherwise, choose one primary experiment store and one production trace system to avoid duplicated datasets, conflicting scores, and unclear retention policies. The stack is a pattern, not a requirement to run every component at maximum scope.
Roll out with one rubric, one human-labeled calibration set, and one production workflow. Review disagreement samples weekly, rotate judges only with side-by-side evidence, and monitor evaluation cost and latency. The budget varies with judge calls, trace volume, retention, and managed services. Stop expanding the judge layer when deterministic checks or targeted human review are cheaper and clearer; a smaller trusted evaluator is better than a broad scoring system that product and engineering teams cannot explain.