OpenEvals is LangChain's open-source starter kit for LLM-as-judge evaluation: prebuilt evaluators for correctness, conciseness, and RAG, plus create_llm_as_judge for your own rubrics. This guide builds a QA eval, with a playable diagram and a six-stage roadmap.
Pick a flow and press Play. Each step lights up one piece, from inputs to a scored verdict, so you can see how an LLM-as-judge turns a rubric into a pass or a score.
An OpenEvals eval is a judge function. create_llm_as_judge takes a rubric prompt and a model and returns an evaluator; you call it with inputs and outputs, and it returns a score plus a comment explaining why.
You adopt OpenEvals one judge at a time. Run a prebuilt judge, write a custom rubric, add RAG evaluators, then wire it into pytest and LangSmith.
OpenEvals is LangChain's open-source starter kit for evaluating LLM applications with an LLM-as-judge. Where a test framework gives you a runner and an assertion, OpenEvals gives you the thing in the middle: a small, well-built judge function that reads an input, an output, and optionally a reference answer, applies a rubric, and hands back a score with its reasoning. You call it from wherever you already run tests, pytest, a CI script, a LangSmith experiment, and assert on the result.
The reason a QA engineer should care is that the judge is the hard part of testing LLM output, and OpenEvals ships it pre-built and honest. A hand-rolled "ask GPT if this is correct" prompt is easy to write and quietly unreliable: it drifts, it is inconsistent, and nobody calibrated it. OpenEvals' prebuilt prompts for correctness, conciseness, hallucination, and RAG groundedness are tuned, printable, and editable, and the create_llm_as_judge factory turns any rubric of your own into the same clean evaluator shape. You get a trustworthy judge without owning the prompt engineering.
Three situations where it lands for a test team:
| QA situation | The brittle way | The OpenEvals way |
|---|---|---|
| Checking a generated answer | Exact-match against one wording | CORRECTNESS_PROMPT judges meaning against a reference |
| A RAG box that invents facts | Spot-check that answers sound right | A groundedness judge fails answers the context does not support |
| A product-specific rule | Nobody can automate "did it propose a next step" | A custom rubric via create_llm_as_judge, asserted in pytest |
It is a pip install, it needs no account, and it deliberately stays small: a few evaluators, a factory, and a return shape you can assert on. It also plugs into LangSmith for tracking scores over time, and has a sibling package, agentevals, for judging agent trajectories. For an SDET, the value is that it turns the vague act of "reviewing AI output" into a function call with a boolean or a score, which is the first step to making it a test.
OpenEvals has one central idea and two supporting ones. The central idea is the evaluator: a function you build once from a rubric and a model, then call many times with inputs and outputs. Everything else is a way to make that function or a way to consume its result.
You make an evaluator with create_llm_as_judge, giving it a prompt (the rubric) and a model (the judge):
from openevals.llm import create_llm_as_judge from openevals.prompts import CORRECTNESS_PROMPT correctness = create_llm_as_judge( prompt=CORRECTNESS_PROMPT, feedback_key="correctness", model="openai:o3-mini", )
The prompt is the rubric. The prebuilt ones live in openevals.prompts: CORRECTNESS_PROMPT, CONCISENESS_PROMPT, HALLUCINATION_PROMPT, and a set for RAG such as groundedness, helpfulness, and retrieval relevance. They are plain f-strings, and each declares its own variables: correctness takes {inputs}, {outputs}, and {reference_outputs}; conciseness takes only {inputs} and {outputs}; hallucination optionally adds {context}; the RAG prompts take {context} (groundedness pairs it with {outputs}, retrieval relevance with {inputs}). That matters for testers: you can print(CORRECTNESS_PROMPT), read exactly what the judge is being asked, and edit it. Nothing is hidden. A custom prompt can be your own f-string, a LangChain ChatPromptTemplate, or a function that returns messages.
The model is the judge, given as an init_chat_model string like "openai:o3-mini" or "openai:gpt-4o", or as any LangChain chat model object passed via judge=, which is how you point it at a local model. The feedback_key names the result, so when you run several evaluators, correctness and conciseness stay distinct in a report.
Calling the evaluator is the moment of judgement, and the return shape is the thing you assert on:
result = correctness(
inputs="Why did CheckoutTest fail?",
outputs="A NullPointerException at line 42 when the cart was empty.",
reference_outputs="NPE in CheckoutTest line 42, empty cart.",
)
# {"key": "correctness", "score": True, "comment": "The output matches ..."}
A dict with a key, a score (a boolean by default, or a 0-to-1 float with continuous=True), and a comment carrying the judge's chain-of-thought. That is the entire architecture: build an evaluator from a rubric and a model, call it, read score and comment. The two supporting ideas are the integrations, pytest for gating and LangSmith for tracking, and both consume that same dict.
You adopt OpenEvals one judge at a time. The six stages below are gates in order: a prebuilt judge you have not read is not ready to customise, and a custom rubric you have not calibrated is not ready to block a merge.
Stage 1, install. pip install openevals and set the judge model's key in the environment. Ship signal: from openevals.llm import create_llm_as_judge imports cleanly.
Stage 2, a prebuilt judge. Build an evaluator from CORRECTNESS_PROMPT and run it on one real output with a reference answer. Ship signal: a score and a comment you agree with, for an output you know.
Stage 3, a custom rubric. Write your own prompt for a product-specific property, because "proposes a next action" is never a prebuilt. Ship signal: your rubric passes good outputs and fails a deliberately bad one.
Stage 4, RAG evaluators. If your feature retrieves context, add the groundedness and helpfulness judges so an answer the context does not support fails even when it reads well. Ship signal: a hallucinated answer scores low on groundedness.
Stage 5, wire to pytest. Call the evaluators inside test functions and assert on score. Ship signal: a red pytest run when the judge disagrees.
Stage 6, LangSmith and CI. Run the suite in the pipeline to gate merges, and log results to LangSmith to see quality over time. Ship signal: a slow slide in correctness becomes a visible trend, not a surprise.
The stages are small; the order is the discipline. By the time a judge can block a merge, you have read its prompt, calibrated it, and seen it fail on purpose.
Getting to a first verdict is a few lines. Install the package and set a key for the judge model.
pip install openevals # the judge model reads its key from the environment export OPENAI_API_KEY="sk-..."
Then build a correctness judge and run it on one interaction from a feature you own. The example below checks a triage bot's summary against what a human would have said.
# first_judge.py from openevals.llm import create_llm_as_judge from openevals.prompts import CORRECTNESS_PROMPT correctness = create_llm_as_judge( prompt=CORRECTNESS_PROMPT, feedback_key="correctness", model="openai:o3-mini", ) result = correctness( inputs="Summarise why the login suite went red.", outputs="LoginTest timed out 3 of 20 runs with no code change on its path; looks flaky.", reference_outputs="An intermittent timeout with no related change, i.e. a flake.", ) print(result["score"]) # True print(result["comment"]) # the judge's reasoning
python first_judge.py
The comment is the habit-forming part. It is not a bare boolean; it is the judge explaining why the output does or does not match the reference, which is exactly the note a reviewer would leave. Change outputs to something confidently wrong, "a real bug in the login form", rerun, and watch score flip to False with a comment naming the disagreement. That edit-run-read loop is how you learn what a judge is actually sensitive to before you let it gate anything.
Before moving on, print the rubric itself: print(CORRECTNESS_PROMPT). Reading the exact instructions the judge receives is the difference between trusting a black box and trusting a checklist you have seen.
reference_outputs when you want correctness judged against a known answer; it is optional, and without it the judge grades factual accuracy on its own. For your first judges, use interactions where you already know the right answer, so a disagreement tells you about the judge, not the data.Let us build something a team keeps: a small suite that judges a triage assistant on two properties, whether its verdict is correct and whether it is concise enough to post to Slack, and runs as ordinary pytest. Two evaluators, two feedback keys, one test per case.
# test_triage_judges.py import pytest from openevals.llm import create_llm_as_judge from openevals.prompts import CORRECTNESS_PROMPT, CONCISENESS_PROMPT JUDGE = "openai:o3-mini" correctness = create_llm_as_judge( prompt=CORRECTNESS_PROMPT, feedback_key="correctness", model=JUDGE) conciseness = create_llm_as_judge( prompt=CONCISENESS_PROMPT, feedback_key="conciseness", model=JUDGE) CASES = [ ("LoginTest failed 3 of 20 runs, timeout, no related change.", "Flaky: intermittent timeout, no code change. Quarantine.", "A flake from a timing issue; quarantine, do not file."), ("CheckoutTest NPE at line 42 on every run since the cart refactor.", "Real bug: NPE on empty cart introduced by the cart refactor. File it.", "A genuine regression from the cart refactor; file a bug."), ] @pytest.mark.parametrize("inputs,outputs,reference", CASES) def test_triage_is_correct_and_concise(inputs, outputs, reference): c = correctness(inputs=inputs, outputs=outputs, reference_outputs=reference) assert c["score"], c["comment"] k = conciseness(inputs=inputs, outputs=outputs) assert k["score"], k["comment"]
pytest test_triage_judges.py -v
Three details make this a real suite rather than a demo. The evaluators are built once at module level, so the rubric and model are fixed for every case. Each assert passes the judge's comment as the failure message, so a red test prints the reasoning instead of a bare AssertionError. And parametrize turns a list of cases into one test each, exactly the pattern you use for any table-driven test; adding a case is adding a tuple.
Note that conciseness does not take a reference: not every rubric needs one. Correctness compares against a known answer, so it does; conciseness judges the output on its own terms. Matching the fields you pass to what the rubric actually reads is the small piece of craft here. From this point, every LLM property you care about is the same move: pick or write a rubric, build an evaluator, assert on score.
The prebuilt prompts cover the universal properties. The properties that matter to your product, "does the triage note propose a concrete next step", "does the release summary mention every breaking change", are yours to write, and create_llm_as_judge makes that a short prompt rather than a project.
A custom rubric is an f-string using the same placeholders the prebuilts use:
from openevals.llm import create_llm_as_judge NEXT_STEP_PROMPT = """You are grading a QA triage note. A good note names a likely cause AND proposes one concrete next action (quarantine, file a bug, rerun, or escalate). <input> {inputs} </input> <output> {outputs} </output> Does the output propose a concrete next action? Answer with reasoning.""" next_step = create_llm_as_judge( prompt=NEXT_STEP_PROMPT, feedback_key="next_step", model="openai:o3-mini", )
By default the evaluator returns a boolean score, which is the strictest thing to assert on and the right default for a gate. When a property is a matter of degree, ask for a graded score instead:
quality = create_llm_as_judge(
prompt=NEXT_STEP_PROMPT,
feedback_key="next_step_quality",
model="openai:o3-mini",
continuous=True, # score becomes a float from 0.0 to 1.0
)
r = quality(inputs="...", outputs="...")
assert r["score"] >= 0.7, r["comment"]
Two more levers keep a custom judge consistent. Reasoning is on by default (use_reasoning=True), which makes the judge think before it scores and puts that thinking in comment; leave it on, it is worth the tokens. And you can add few-shot examples of good and bad outputs to the prompt, which is the single most effective way to stop a rubric drifting between runs. The feedback_key is not decoration either: name each judge clearly, because once you have four of them the key is how a LangSmith report or a pytest log tells them apart.
For retrieval-augmented features, reach for the RAG prompts before writing your own. Groundedness checks that the answer is supported by the retrieved context, which catches the confident, well-written, wrong answer that erodes trust in a RAG box; helpfulness checks the answer actually serves the question; retrieval relevance checks the retriever fetched the right chunks in the first place. They are the three RAG failure modes, each as a ready evaluator.
Because an OpenEvals evaluator is a plain function that returns a dict, gating CI is just running the pytest file that calls it. No plugin, no runner; the evaluators run inside ordinary tests and a false score is an ordinary failure.
# .github/workflows/llm-judges.yml name: llm-judges on: [pull_request] permissions: contents: read jobs: judges: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-python@v5 with: python-version: "3.12" - run: pip install openevals pytest - run: pytest tests/judges -v env: OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
A judge that returns False fails the job, and the pull request goes red with the judge's comment in the log, because you passed it as the assertion message. As with every LLM eval, split the suite: a small, fast set of judges on every pull request, and the full set on a nightly schedule so cost and latency never block a merge.
The second integration is LangSmith, and it is where OpenEvals goes from gate to instrument. LangSmith runs experiments over a dataset and accepts OpenEvals evaluators directly as the scoring functions, logging every score and comment against each example. Instead of a single pass or fail, you get a trend: correctness this week versus last, per case, per model. A quality slide that no single CI run would flag shows up as a line drifting down. If your team already traces with LangSmith, the evaluators plug into that same view; the sibling LangSmith for QA guide in this series covers the tracing side in depth.
An LLM judging an LLM is a powerful test, and it brings non-determinism, cost, and fallibility into the suite. OpenEvals gives you the levers; you still have to pull them on purpose.
Flaky verdicts. A judge can say True on one run and False on the next for a borderline output. Stabilise it: pin the judge model and version so a silent upgrade does not shift every verdict, keep reasoning on so the judge thinks before it scores, add a few-shot examples of good and bad outputs to the rubric, and prefer boolean scores for gates. When you do use continuous=True, set the threshold with headroom rather than on the edge. And for any property a rule can check, format, length, the presence of a required field, check it in plain Python before the judge ever runs; no judge, no flake.
Do not over-trust the judge. A score is a model's opinion. Before a judge blocks merges, calibrate it: label a sample of outputs by hand, run the judge, and check it agrees. When it passes something you would fail, tighten the rubric or add a counter-example. The printable prompt is the tool for this; read it, edit it, rerun. A judge everyone trusts and nobody checked is worse than no judge, because it launders bad output as approved.
Keep test data inside your boundary. Every judge call sends the inputs, outputs, and references to the judge model. If that is a hosted API, real transcripts, customer data in a RAG context, and internal stack traces leave your perimeter. For sensitive fixtures, pass a local model through judge= so nothing leaves the machine, or scrub the data first. And treat a genuinely flaky judge like a flaky test: quarantine the case, file it, and fix the rubric. Never let a team learn to ignore red.
No, it runs inside them. It provides the judge functions; pytest, or any runner, still owns execution and reporting. Your deterministic tests are untouched, and the judges add a way to assert on LLM output next to them.
OpenEvals is deliberately small: a factory for LLM-as-judge evaluators plus tuned prebuilt prompts, from LangChain, with first-class LangSmith integration. DeepEval is a fuller pytest-native framework with a large metric library. promptfoo is config-first and built for comparing prompts and providers in a matrix. Pick OpenEvals when you want a clean judge function you can drop into existing tests or LangSmith experiments.
Pin the judge model, keep reasoning on, add few-shot examples to the rubric, prefer boolean scores for gates and headroom thresholds for continuous ones, and check anything rule-checkable in plain code first. Those together turn a jittery judge into a stable gate.
No. OpenEvals is open-source and runs locally; you pay only for the judge model calls. LangSmith is optional, for tracking scores over time.
Yes. Pass a local LangChain chat model, for example one served by Ollama, through judge=, and nothing leaves your machine. That is the right setup when fixtures contain real product data.
Start with five to ten real interactions where you know the right verdict, and one prebuilt judge. Calibrate that before adding rubrics or cases. A small, trusted set that gates CI beats a large one nobody has checked.
The six stages, in order, are the map to pin above your desk:
| Stage | What you build | Ship signal |
|---|---|---|
| 1. Install | pip install openevals, judge key in env | the import runs clean |
| 2. Prebuilt judge | create_llm_as_judge + CORRECTNESS_PROMPT | a score and comment you agree with |
| 3. Custom rubric | your own prompt with {inputs} {outputs} | passes good cases, fails a bad one |
| 4. RAG evaluators | groundedness, helpfulness judges | a hallucinated answer scores low |
| 5. Wire to pytest | evaluators inside tests, assert on score | a red run when the judge disagrees |
| 6. LangSmith + CI | pytest in the pipeline, scores logged over time | a quality slide becomes a visible trend |
Read the order as a dependency chain. Stage 2 proves a judge can verdict at all. Stage 3 makes it say something specific to your product. Stage 4 grounds it against sources so confident nonsense fails. Stage 5 turns it into a test, and only at stage 6 do you let it block a merge and trend it, because by then you have read its prompt and seen it fail on purpose. Rushing to CI with an uncalibrated judge is how a team learns to ignore red.
Series siblings: DeepEval for QA and promptfoo for QA. See also LangSmith for QA, LangGraph for QA, and the AI for QA hub.