LangSmith for QA: trace, evaluate, and gate your LLM tests

Watch a run become a regression test

Pick a flow and press Play. Each step lights up one part of the observability loop, from a live agent run to a CI gate that blocks a regression before it ships.

The roadmap

LangSmith pays off fast and compounds. Turn on tracing, learn to read one, build a dataset, write evaluators, gate CI, then monitor and close the loop.

1. What LangSmith is, and why QA should care

LangSmith is the observability, evaluation, and monitoring platform built by the team behind LangChain, but the name is misleading in one useful way: you do not need LangChain to use it. A single decorator and two environment variables will stream traces from a plain OpenAI or Anthropic call, a hand-rolled RAG pipeline, or an agent loop you wrote from scratch. That vendor-neutral posture is exactly why LangSmith for QA belongs in a test engineer's toolkit and not only a developer's. It treats "an LLM feature" the way you already treat a service under test: something with inputs, outputs, a version, and a quality bar you can measure.

Think of the product as three surfaces stacked on one data model. Observability is the trace: every LLM call, tool invocation, retrieval step, token count, latency figure, and dollar cost, captured as a replayable tree. Evaluation is the harness: datasets of inputs plus evaluators that score outputs, run on demand or inside CI. Monitoring is the long tail: production runs, human and automated feedback, and dashboards that watch quality drift after you ship. The primitives connect. A trace you captured while triaging a bug becomes a dataset example, which feeds an evaluator, which gates a deploy, which you then monitor in production. One loop, four nouns: runs, datasets, evaluators, feedback.

Why non-determinism breaks your assertions

Every test you have shipped assumes a stable oracle. You call the system, you compare the result to a known-good value, green or red. LLM features quietly delete that assumption. Run the same prompt twice and the token stream differs. Bump the temperature above zero and it differs more. The provider silently rolls a new model snapshot, your retrieval index gets re-embedded, an upstream teammate edits the system prompt, and suddenly a paragraph that was correct on Tuesday reads differently on Wednesday while still being correct. An assertEquals("expected paragraph", output) against that surface is not a test, it is a flaky-test generator. You will either pin it so tightly that every legitimate improvement turns the build red, or so loosely that real regressions sail through.

The honest response is to stop asserting on exact strings and start asserting on properties: is the answer faithful to the retrieved context, does it cite the right document, is it under the length budget, does it refuse the unsafe request, is it at least as good as the previous version. Those are semantic oracles, and building, versioning, and running them at scale is precisely the job LangSmith exists to do. QA has always owned oracles and test data; LLM apps just made the oracle problem harder, which pushes ownership toward the people who think about it hardest.

The test harness and the oracle, in one tool

Getting data flowing is the whole onboarding. Decorate the function you want visibility into with @traceable, set LANGSMITH_TRACING to true, and point an API key at your workspace. The older LANGCHAIN_TRACING_V2 and LANGCHAIN_API_KEY names are still honored, so mixed docs will not trip you up. Name the project per environment (ci, staging, prod) so runs land in separate buckets you can filter and compare.

# pip install -U langsmith
import os
from langsmith import traceable

os.environ["LANGSMITH_TRACING"] = "true"
os.environ["LANGSMITH_API_KEY"] = "lsv2_your_key"
os.environ["LANGSMITH_PROJECT"] = "checkout-agent-ci"

@traceable
def answer(question) -> str:
    # plain OpenAI, Anthropic, or your own pipeline
    return run_my_llm(question)

That handles observability. The harness-and-oracle half is a dataset plus an evaluate() call. A dataset is a versioned collection of examples, each an input and (optionally) a reference output, created with client.create_dataset and populated with client.create_examples. An evaluator is a function that scores one output; return a plain bool and LangSmith uses the function name as the score key. The harness runs your system over every example, applies the evaluators, and records the result as an experiment you can open, diff, and share.

from langsmith import Client, evaluate

client = Client()

# the oracle: name becomes the score key in the UI
def exact_answer(outputs, reference_outputs) -> bool:
    return outputs["answer"].strip() == reference_outputs["answer"].strip()

# the harness: run the system over a golden dataset
evaluate(
    lambda inputs: {"answer": answer(inputs["question"])},
    data="regression-golden-set",
    evaluators=[exact_answer],
    experiment_prefix="pr-1234",
)

Exact match is the trivial case shown for clarity. Swap it for a custom evaluator that checks JSON schema conformance, or an LLM-as-judge that scores faithfulness, and the shape of the call does not change. That stability is the point: one harness spans string checks, structural checks, and judged checks.

The primitives, mapped to what you already know

None of this is a new discipline, it is your existing vocabulary with different nouns. The table below is the Rosetta stone most QA engineers need before the rest of this series clicks into place.

Read that top to bottom and the ownership question answers itself. Datasets are test data governance. Evaluators are assertion design. Experiments are test runs with history. Feedback is production observability. Every column is work QA already does; LangSmith just gives it a home that survives non-determinism.

From dashboard to regression gate

A tracing dashboard is nice for debugging, but LangSmith for QA earns its keep when evaluate() runs on pull requests and can turn the build red. The pattern is direct: keep a versioned golden dataset, run the harness in CI with an experiment_prefix tied to the PR or commit, read the aggregate scores back through the Client, and fail the step when a metric drops below its threshold. That converts "the AI feels worse this week" into a specific number attached to a specific change, which is the only form of feedback a release process can act on.

Regressions between two candidate versions get a sharper tool. Pairwise evaluation compares outputs from two experiments head to head with evaluate_comparative, so instead of asking "did prompt B score 0.81," you ask "did prompt B beat prompt A on the same inputs," which is both easier for a judge to answer reliably and closer to how you actually decide whether to ship. Layer production create_feedback signals and monitoring dashboards on top and you close the loop: real user thumbs-down become tomorrow's dataset examples, and yesterday's fix stays fixed because the gate remembers it.

That is the map. LangSmith is observability, evaluation, and monitoring for non-deterministic systems, and QA is the discipline that already owns fixtures, oracles, and gates. The sections ahead go deep on each primitive, tracing and projects, building datasets, writing custom and LLM-as-judge evaluators, running pairwise comparisons, and wiring regression gates into CI, and the sibling guides in this LangChain-family series connect the same harness to the frameworks you may be testing around it. Treat this page as the shared vocabulary the rest of them assume.

2. The LangSmith architecture for testers

Before you can trust LangSmith for QA work, you need a mental model that maps its objects onto the vocabulary you already own: suites, fixtures, oracles, and regression runs. LangSmith is not a magic quality button. It is an observability and evaluation layer that records exactly what your LLM app did, lets you freeze those recordings into test cases, and then re-runs candidate versions of the app against those cases using programmable oracles. Once you see the pieces as a testing harness rather than a pretty dashboard, the whole thing clicks. This section walks the architecture object by object, in the order a tester actually meets them, and shows how a single production request becomes a permanent regression test.

Keep this translation table next to your keyboard. Every LangSmith noun has a QA counterpart you already understand.

Runs and traces: the execution log

The atomic unit is the run. A run is a single recorded invocation of a function, a model call, a retriever, or a tool, with its inputs, outputs, latency, token counts, and any raised error already captured. A trace is the tree of runs for one end-to-end request, so a chatbot turn that plans, retrieves three chunks, and calls the model is one trace made of several nested runs. You produce runs by switching tracing on and decorating the functions you care about with @traceable.

# .env for the app under test
LANGSMITH_TRACING=true
LANGSMITH_API_KEY=ls-...
LANGSMITH_PROJECT=qa-regression

from langsmith import traceable

@traceable(run_type="chain")
def answer_ticket(question: str) -> str:
    # inputs and the return value are auto-captured as a run
    ...

The run_type field ("chain", "llm", "tool", "retriever") only affects how the run is rendered, so use it to keep triage readable. If you call OpenAI directly, wrap the client with wrap_openai from langsmith.wrappers and every model call becomes a child run with prompts and token usage attached. A run is your execution log with all the forensic detail already collected, which is exactly the artifact you always wished a flaky end-to-end test had left behind after it failed at 2am.

Projects: the suite boundary

Every run lands in a project, set by LANGSMITH_PROJECT or per-call configuration. Think of a project as a suite or an environment bucket. Keep prod-traffic separate from ci-regression and from a throwaway local-dev project, the same way you would not let a load-test run pollute your smoke-suite report. Projects give you the filtered views that make triage fast: sort by error, by latency percentile, or by a feedback key, then drill into the offending trace. When teams say they run LangSmith for QA in CI, they mean each pull request writes its evaluation runs into a dedicated experiment project so results never mix with live user traffic.

Datasets and examples: fixtures and the golden set

A dataset is a named collection of examples, and an example is one test case: an inputs dict plus an optional outputs dict holding the reference (expected) answer. This is your fixture file, except it lives inside LangSmith, is versioned, and is queryable. You build one with the Client, either by hand-writing cases or by importing a CSV of known-good pairs.

from langsmith import Client
client = Client()

ds = client.create_dataset(dataset_name="triage-golden",
                             description="labeled support tickets")
client.create_examples(
    inputs=[{"question": "I was charged twice this month"}],
    outputs=[{"label": "billing"}],
    dataset_id=ds.id,
)

How a run becomes a test case

This is the most valuable loop in the whole tool. A real production trace that exposed a bug becomes a permanent regression example. In the UI you open the failing run and click Add to Dataset, editing the output to the corrected answer before you save. Programmatically, you read the run, then copy its captured inputs (and your fixed expected output) into a new example on the golden dataset. Either path turns a one-time incident into a test that can never silently regress again. It is the same discipline as writing a failing repro before you fix a defect, and it means your dataset grows from the exact places your app actually broke rather than from a QA engineer's imagination alone.

Evaluators: assertions, judges, and pairwise oracles

Evaluators are your oracles. They inspect an experiment's outputs against the reference and emit a score. LangSmith supports three flavors, and mature suites use all three because no single oracle fits every output.

Heuristic (code-based) evaluators are ordinary Python functions: exact match, regex, JSON-schema validity, a latency budget, or "does the answer cite a real document id". The current custom-evaluator signature receives inputs, outputs, and reference_outputs and can return a bool, a number, or a dict. You pass a target function and the dataset to evaluate(), which runs the app over every example and applies each oracle.

from langsmith import evaluate

def exact_label(inputs, outputs, reference_outputs) -> bool:
    return outputs["label"] == reference_outputs["label"]

def target(inputs: dict) -> dict:
    return {"label": answer_ticket(inputs["question"])}

evaluate(target, data="triage-golden",
         evaluators=[exact_label], experiment_prefix="triage")

LLM-as-judge evaluators handle open-ended outputs where there is no single string to diff, like summary quality, tone, or groundedness. LangChain's openevals package ships prebuilt judges that plug straight into evaluate() and return a score plus a written rationale you can read during triage.

from openevals.llm import create_llm_as_judge
from openevals.prompts import CORRECTNESS_PROMPT

judge = create_llm_as_judge(prompt=CORRECTNESS_PROMPT,
    feedback_key="correctness", model="openai:gpt-4o-mini")
# then: evaluate(target, data="triage-golden", evaluators=[judge])

Treat a judge as a fuzzy assertion, not a hard gate. It is itself a nondeterministic model, so pin a low temperature, spot-check its scores against human labels, and prefer a pass threshold over a single strict equality. Pairwise evaluation sidesteps the noise entirely: instead of scoring one output in isolation, you compare two experiments and ask which is better. Use evaluate_comparative over two prior experiment names when absolute scores drift but relative preference stays stable, which is common for prompt tweaks and model swaps.

from langsmith.evaluation import evaluate_comparative

evaluate_comparative(["triage-v1", "triage-v2"],
                     evaluators=[preference])

Feedback and monitoring: the production signal

Feedback is a score attached to any run after the fact, from a human reviewer, a thumbs-up widget, or an automation. You write it with client.create_feedback(run_id, key="human_verdict", score=0, comment="wrong label"). In production, LangSmith monitoring charts these keys over time and can run online evaluators against live traffic, so a regression shows up as a dip on a graph rather than as an angry support ticket. The loop closes when you pull the low-scoring runs back down with client.list_runs(project_name="prod-traffic", limit=50), promote the worst offenders into triage-golden, and rerun your suite.

The eval loop

Assemble the pieces and the architecture is just a regression loop wearing new nouns: trace the app to capture runs, curate the interesting ones into a dataset, run evaluate() with heuristic and judge oracles over each candidate version, compare experiments (including pairwise) to pick a winner, gate the merge in CI on the scores, monitor production feedback, then promote real failures back into the dataset. That last arrow is what makes LangSmith for QA compounding instead of static: every incident permanently hardens the suite.

3. The LangSmith-for-QA adoption roadmap

You do not flip a switch and suddenly have a tested LLM pipeline. LangSmith earns its place the same way a Selenium grid or a Playwright CI job once did: one small, boring, reversible step at a time. The roadmap below is six stages, each shippable in an afternoon, and each one leaves you strictly better off than the stage before it. Map them onto the visual roadmap above and treat them as a ladder, not a leap. A team practicing LangSmith for QA rarely needs all six on day one, but every stage assumes the one beneath it is already in place, so resist the urge to skip ahead. Stages one through three build the substrate (traces and a labeled dataset); four through six turn that substrate into an actual quality gate and a feedback loop.

Stage 1: Turn on tracing

Nothing downstream works until runs are being recorded, so tracing is the foundation. It is controlled entirely by environment variables, which means the first change touches your CI config and secrets, not your application logic. Set the tracing flag, an API key, and a project name so runs land in a bucket you can actually find later during triage.

export LANGSMITH_TRACING="true"
export LANGSMITH_API_KEY="lsv2_..."
export LANGSMITH_PROJECT="checkout-agent-ci"  # one project per surface

If you already build with LangChain or LangGraph, that is the entire job: every chain, tool, and model call is now traced with zero code change. For plain Python, wrap the functions you care about with the @traceable decorator, and nested calls link into one tree automatically. The legacy LANGCHAIN_TRACING_V2 variable still works, but new code should prefer the LANGSMITH_ names to match current docs.

from langsmith import traceable

@traceable(run_type="retriever")
def retrieve(question: str):
    ...                            # your vector search

@traceable(run_type="llm")
def generate(question: str, docs):
    ...                            # your model call

@traceable(run_type="chain")
def answer_ticket(question: str) -> str:
    docs = retrieve(question)      # a real child run, because retrieve is @traceable too
    return generate(question, docs)

Stage 2: Read a trace to debug

A trace is a tree: the root run, then every retrieval, tool call, and model invocation nested underneath, each carrying its exact inputs, outputs, latency, and token count. This is your oracle for a system that ships no stack trace. When an agent returns a wrong answer, you triage it the way you triage a flaky end-to-end test: open the run, walk down the tree, and find the single step where the state went bad. In practice it is rarely the model itself. It is a retriever that returned empty documents, a tool that received a malformed argument, or a prompt template that silently dropped a variable. Being able to see the intermediate state is what turns "the LLM is flaky" (untestable) into "step three returned nothing on this input" (a bug with an owner). You can also pull any run back down by id from the Client for a scripted post-mortem or to attach it to a ticket.

from langsmith import Client

client = Client()
run = client.read_run(run_id)
print(run.inputs, run.outputs, run.error)

Stage 3: Build a dataset from real runs

Production is the best test-case generator you will ever have. Every failure a user hits is a regression case waiting to be captured, and unlike synthetic prompts it is guaranteed to be in-distribution. Query the runs that errored or drew a thumbs-down, then promote them into a dataset that becomes your golden set. This is the QA move that makes everything after it possible: without labeled examples there is simply nothing to evaluate against, and no coverage to measure.

from langsmith import Client

client = Client()
runs = list(client.list_runs(
    project_name="checkout-agent-ci",
    is_root=True,
    error=True,          # start with the ones that blew up
))
ds = client.create_dataset("checkout-regressions")
client.create_examples(
    dataset_id=ds.id,
    inputs=[{"question": r.inputs["question"]} for r in runs],
    outputs=[{"answer": "TODO: label"} for r in runs],
)

Label the expected outputs by hand or in the UI. A hundred real, labeled failures beats a thousand random synthetic ones, exactly the way a curated regression suite beats blind fuzzing.

Stage 4: Write evaluators

An evaluator is just an assertion for non-deterministic output. LangSmith accepts plain Python functions and binds their parameters by name: ask for outputs and reference_outputs and you receive the model output and the dataset label. Return a bool, a number, or a dict, and the function name becomes the feedback key. Cheap, deterministic checks come first, because they are your fast, flake-free layer.

def exact_match(outputs: dict, reference_outputs: dict) -> bool:
    return outputs["answer"].strip() == reference_outputs["answer"].strip()

def cites_source(outputs: dict) -> bool:
    return "http" in outputs["answer"]

For fuzzy checks like "is this answer factually correct", string comparison is useless, so you reach for LLM-as-judge. The openevals package ships prebuilt judges you plug straight into evaluate, keyed so their scores land alongside your custom ones.

from openevals.llm import create_llm_as_judge
from openevals.prompts import CORRECTNESS_PROMPT

correctness = create_llm_as_judge(
    prompt=CORRECTNESS_PROMPT,
    feedback_key="correctness",
    model="openai:gpt-4o-mini",
)

results = client.evaluate(
    target,                                  # your app under test
    data="checkout-regressions",
    evaluators=[exact_match, cites_source, correctness],
    experiment_prefix="rag-v2",
)

When two versions score too close to call, run evaluate_comparative over two finished experiments for a pairwise, A/B judgment instead of trusting a hair-thin absolute-score gap. That mirrors the sibling LangChain-for-QA guide in this series, where head-to-head comparison settled a prompt-versus-prompt tie the raw numbers could not.

Stage 5: Gate CI on eval scores

Now the payoff. An evaluation you run by hand is a demo; an evaluation your pipeline runs on every pull request is a quality gate. Call evaluate inside CI, pull the results into a dataframe, and fail the build when the aggregate drops below your line. Because each run carries an experiment_prefix, every commit produces its own comparable experiment in the LangSmith UI, so a reviewer can diff two SHAs example by example.

results = client.evaluate(target, data="checkout-regressions",
                          evaluators=[correctness])
df = results.to_pandas()
mean = df["feedback.correctness"].mean()
assert mean >= 0.9, f"correctness {mean:.2f} below the 0.90 gate"

Teams that prefer pytest can skip the dataframe entirely: mark a test with @pytest.mark.langsmith, log outputs through the langsmith.testing module, and let ordinary assertions both fail CI and stream results to LangSmith at once. Either way, LangSmith for QA stops being a dashboard someone remembers to check and becomes a wall your regressions cannot walk through unnoticed.

Stage 6: Monitor production and close the loop

CI only catches what your dataset already knows about. Production surfaces everything else: the phrasings, edge cases, and adversarial inputs no one thought to write down. Attach online evaluators to your live project so every real trace is scored the moment it lands, and wire user signals back as feedback keyed to the run that produced them.

client.create_feedback(
    run_id=run_id,
    key="user_thumbs",
    score=0,
    comment="quoted the wrong refund window",
)

Grab the run_id at request time with get_current_run_tree() inside your traced function so the feedback attaches to the exact run, not a guess. Now the flywheel turns: monitored production failures become new dataset examples (back to Stage 3), evaluators grow to cover them (Stage 4), and CI defends the fix forever (Stage 5). That closed loop is the entire point of LangSmith for QA. Coverage is no longer a fixed suite you wrote once and let rot; it compounds every time a real user finds a hole you had not imagined.

4. Tracing your first agent run

You cannot assert on what you cannot see. A classic test suite hands you a stack trace when something breaks, and Playwright hands you a trace file with a DOM snapshot for every step. An LLM agent gives you neither by default: it emits one final string and quietly swallows every decision that produced it. That opacity is the first wall you hit doing LangSmith for QA work, and tracing is how you get through it. A LangSmith trace is the missing artifact, a step by step recording of every model call, tool invocation, retrieved document, and intermediate value inside a single agent run. Once you have it, a non-deterministic failure stops being a ghost story and becomes an object you can inspect, filter, and triage.

Turn tracing on with two environment variables

The fastest path costs no code changes. If your agent is built on LangChain or LangGraph (both covered in the sibling guides in this series), setting the right environment variables auto-captures every chain, tool, and model run. You need a tracing switch and an API key, plus an optional project name to group the runs into one view.

export LANGSMITH_TRACING="true"
export LANGSMITH_API_KEY="lsv2_pt_your_key_here"
export LANGSMITH_PROJECT="qa-agent-regression"  # one project = one run view

Those are the current names. The older LANGCHAIN_TRACING_V2 and LANGCHAIN_API_KEY are still honored, so mixed docs will not fail on you. A useful convention for QA is one project per environment: point local runs at qa-agent-dev, your CI job at qa-agent-ci, and production at qa-agent-prod by overriding LANGSMITH_PROJECT per context. Now every run lands in a bucket you can search, the same way you would tag a Playwright suite by environment. The moment tracing is on, LangSmith records the full run tree without you writing a single line of instrumentation.

Wrap plain functions with @traceable

Not every agent is a LangChain object. Plenty of real systems are raw OpenAI or Anthropic SDK calls glued together with custom Python tools. For those, the @traceable decorator from the langsmith package turns any function into a traced run, and wrap_openai auto-traces the model call underneath it with token counts attached.

from openai import OpenAI
from langsmith import traceable
from langsmith.wrappers import wrap_openai

client = wrap_openai(OpenAI())

@traceable(run_type="tool")
def lookup_order(order_id) -> dict:
    return {"status": "shipped", "eta": "2 days"}

@traceable(run_type="chain", name="support_agent")
def run_agent(question):
    order = lookup_order("A-1001")
    reply = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": question}],
    )
    return reply.choices[0].message.content

The run_type argument classifies each run so the trace tree reads correctly: "chain" for the orchestrating call, "tool" for the lookup, and the wrapped client emits an "llm" run on its own. Other valid types include "retriever", "prompt", and "parser". A matching wrap_anthropic exists if your model lives on Claude. Because the decorator nests runs by call stack, calling run_agent once produces a three-node tree: the agent, its tool, and its model call, each with its own inputs and outputs.

What a trace actually shows

Open any run in the LangSmith UI and you get a tree on the left and a detail pane on the right. For a QA engineer, each field maps to a test concern you already care about.

This is the payoff of LangSmith for QA: the same run that answered a user is now a fully labeled test artifact. You are not reading logs and reconstructing what happened, you are looking at a structured record of it.

Reading a trace to debug a flaky agent

Here is where tracing earns its keep. Consider a support agent that passes nine times out of ten and fails the tenth with a wrong order status. You rerun it and it is green, so it never reproduces on your laptop. This is the classic flaky failure that wastes triage hours. Open the failing run in LangSmith and expand the tree. The root run is red. One level down, the lookup_order tool run shows inputs order_id "A-1001" but an empty outputs object and a 4,900 ms latency where sibling runs took 200 ms. That is your answer: the tool timed out, returned nothing, and the model confabulated a plausible status from an empty result. No amount of re-reading the system prompt would have surfaced that. The trace turned a non-deterministic failure into a concrete, fixable defect.

To make this repeatable, attach a test id and commit sha to every run through the langsmith_extra argument, which any traceable function accepts at call time.

run_agent(
    "where is my order?",
    langsmith_extra={
        "metadata": {"test_id": "T-42", "git_sha": "9f3a90d"},
        "tags": ["ci", "nightly"],
    },
)

Now filter the project by test_id to line up the passing and failing runs side by side. LangSmith renders both trees; scan for the first node where inputs or the chosen tool diverge. For agents that flake, the divergence usually starts at the planning step, where the model picked a different tool or emitted malformed JSON arguments that the next tool silently rejected. Once you know the divergence point, you know where to add a guardrail, a retry, or a schema check. For programmatic triage in CI, the Client pulls the same runs back so a failing pipeline can list its own red runs.

from langsmith import Client

client = Client()
for run in client.list_runs(
        project_name="qa-agent-regression", error=True):
    print(run.name, run.total_tokens, run.error)

With tracing in place you have the raw material for everything that follows. The next step in a real LangSmith for QA workflow is to stop inspecting runs one at a time and start scoring them in bulk against saved inputs, which is exactly what datasets and the evaluate() function do in the sibling guide.

5. Datasets and evaluators: the oracle

Tracing tells you what happened. It cannot tell you whether what happened was correct. In QA terms a trace is a log line and an evaluator is the assertion, and without the assertion you are just watching your agent fail in high resolution. This is the section where LangSmith for QA stops being observability and becomes a real test harness: a dataset that plays the role of your oracle, and evaluators that play the role of your assertions. The workflow mirrors any suite you have ever written. Collect representative inputs with known good outputs, express a pass or fail rule, run the system under test against every case, and read the aggregate. The only twist is that some of your assertions are themselves language models, which means you have to treat the judge as flaky test infrastructure and pin it down.

Build the dataset from real traces

A dataset in LangSmith is just a named collection of examples, and an example is an input paired with an optional reference output. The most honest source of examples is the traffic you already captured through @traceable, because production inputs carry the edge cases your imagination never would. The Client exposes create_dataset to make the container and list_runs to pull the raw material, and create_examples to seed cases in bulk. Point list_runs at your production project, filter to root runs so you get whole requests rather than internal sub-steps, and hand the inputs and outputs straight into the new dataset.

from langsmith import Client

client = Client()

ds = client.create_dataset(
    dataset_name="qa-agent-golden",
    description="Root traces pulled from prod, curated as the oracle",
)

# pull real traffic: the last 50 root traces from the prod project
runs = list(client.list_runs(
    project_name="qa-agent-prod",
    is_root=True,
    limit=50,
))

client.create_examples(
    inputs=[r.inputs for r in runs],
    outputs=[r.outputs for r in runs],
    dataset_id=ds.id,
)

Evaluator 1: an LLM-as-judge for correctness

Some questions have no regex. Whether a support answer is actually correct, on topic, and grounded in the reference is a judgement call, and that is what LLM-as-judge evaluators are for. A LangSmith evaluator is any function whose parameter names it declares from a fixed set: inputs, outputs, reference_outputs, plus run and example if you want the raw objects. Declare only what you need. Return a dict with a key (the feedback name) and a numeric score, and optionally a comment so a failing case explains itself in the UI. Wrap the judge client with wrappers.wrap_openai so the judge's own calls get traced too, which matters when the judge itself misbehaves.

from langsmith import wrappers
from openai import OpenAI

judge = wrappers.wrap_openai(OpenAI())

def correctness(inputs: dict, outputs: dict, reference_outputs: dict) -> dict:
    prompt = (
        f"Question: {inputs['question']}\n"
        f"Reference: {reference_outputs['answer']}\n"
        f"Candidate: {outputs['answer']}\n"
        "Reply PASS or FAIL, then one short reason."
    )
    verdict = judge.chat.completions.create(
        model="gpt-4o-mini",
        temperature=0,
        messages=[{"role": "user", "content": prompt}],
    ).choices[0].message.content
    return {
        "key": "correctness",
        "score": 1 if verdict.startswith("PASS") else 0,
        "comment": verdict,
    }

If you would rather not hand-roll the prompt, the openevals package ships create_llm_as_judge, which returns an evaluator with the same shape and a tested correctness rubric behind it. Either way, the judge is now the flakiest component in your suite, so treat it like one: pin temperature to zero, force a two-way PASS or FAIL verdict instead of a vague 1 to 5 scale, and version the judge prompt in the LangSmith Prompt Hub so a scoring change is a reviewable diff and not a silent drift in your pass rate.

Evaluator 2: a heuristic that never lies

Not every assertion needs a model, and the ones that do not are the backbone of a stable suite. A regex or an exact-match check is deterministic, costs nothing, and never has a bad day. Use heuristics for the things that are objectively true or false: a leaked API key in the output, a refusal string that should never appear, a JSON field that must match exactly. Heuristic evaluators can skip the dict entirely and return a bare bool, which LangSmith records as a 0 or 1 under a feedback key named after the function.

import re

FORBIDDEN = re.compile(r"\b(as an ai|i cannot help|api[_-]?key)\b", re.I)

def no_leak(outputs: dict) -> dict:
    text = outputs.get("answer", "")
    return {"key": "no_leak", "score": 0 if FORBIDDEN.search(text) else 1}

def exact_match(outputs: dict, reference_outputs: dict) -> bool:
    return outputs.get("answer", "").strip() == reference_outputs["answer"].strip()

Run evaluate() and read the experiment

With a dataset and a list of evaluators, evaluate() is your test runner. Pass the target (any callable that takes an inputs dict and returns outputs, or a chain), the dataset name as data, the evaluators, and an experiment_prefix that names this run the way a CI job name would. The call fans out across examples with max_concurrency, scores each run with every evaluator, and logs the whole thing back to LangSmith as an experiment you can open, sort, and filter. The return value is iterable and converts to a DataFrame, so your pass rate is one aggregation away.

from langsmith import evaluate

results = evaluate(
    lambda inputs: run_agent(inputs["question"]),
    data="qa-agent-golden",
    evaluators=[correctness, no_leak, exact_match],
    experiment_prefix="baseline-prompt-v1",
    max_concurrency=4,
)

df = results.to_pandas()
print(df["feedback.correctness"].mean())

This is the part of LangSmith for QA that feels most like home. The experiment page is your test report: one column per evaluator, a row per example, and the failing cells linking straight to the full trace so you can see exactly which tool call or retrieval step produced the wrong answer. Sort by the correctness score, read the five worst comments, and you have your triage list. Run the same dataset again after a change and LangSmith lines the experiments up side by side, so a regression is a red cell you can click, not a vibe.

Pairwise: which prompt actually won?

Absolute scores answer "is this good enough." They are weak at "is version two better than version one," because a judge scoring runs in isolation drifts and compresses toward the middle. Pairwise evaluation fixes that by showing the judge both candidates for the same input and asking it to pick. In the SDK you run evaluate() once per prompt variant to produce two experiments, then feed those experiment names to evaluate_comparative with a comparative evaluator that receives the list of runs and returns a scores array ranking them. Here a deterministic length tiebreaker stands in for the judge to keep the mechanics visible.

from langsmith.evaluation import evaluate_comparative

def prefer_shorter(runs: list, example) -> dict:
    a, b = runs
    ta, tb = a.outputs["answer"], b.outputs["answer"]
    ranks = [1, 0] if len(ta) <= len(tb) else [0, 1]
    return {"key": "preferred", "scores": ranks}

evaluate_comparative(
    ["baseline-prompt-v1", "concise-prompt-v2"],
    evaluators=[prefer_shorter],
)

In practice you swap the length rule for an LLM judge, or the prebuilt pairwise evaluator from openevals, and let it read both answers against the reference. The comparative experiment then reports a win rate: concise-prompt-v2 preferred on 68 of 100 cases, with every disagreement one click from the two full traces. That is the honest way to promote a prompt change, and it is why pairwise belongs in the same repo as your golden dataset.

6. Regression testing and production monitoring

A dataset and a set of evaluators are only worth building if they run again. The whole promise of LangSmith for QA is that the same suite you used to pick a prompt becomes a gate that runs on every change, and that the runs your students generate in production quietly refill that suite with the cases you never thought to write. This section wires the pieces from the previous sections (the Client, a dataset, custom and LLM-as-judge evaluators) into a repeatable regression loop, then extends it into live monitoring so you catch drift before a student does.

Think of it the way you already think about a Selenium or Playwright suite. The dataset is your fixture set, the evaluators are your oracles, evaluate() is the test runner, and the experiment score is the assertion. The difference is that the system under test is non-deterministic, so a single green run means nothing. You gate on an aggregate over many examples, and you watch that aggregate move across commits.

Run the dataset and evaluators on every change

The regression suite is one call. Point evaluate() at the named dataset, pass every evaluator, and tag the run with the git SHA so you can trace a score back to a diff. Reuse both oracle styles: a cheap deterministic check for anything you can grade exactly, and an LLM-as-judge for the fuzzy properties.

from langsmith import Client, evaluate

client = Client()

def answer_correct(inputs, outputs, reference_outputs):
    # deterministic oracle: exact match on the graded field
    return outputs["label"] == reference_outputs["label"]

results = evaluate(
    lambda inputs: run_app(inputs["question"]),
    data="qa-regression-v1",
    evaluators=[answer_correct, judge_helpfulness],
    experiment_prefix="ci",
    metadata={"git_sha": sha},
    max_concurrency=4,
)

The judge_helpfulness evaluator is an ordinary function that calls a model and returns a keyed score. Keeping the return shape as a dict with key and score means each evaluator lands as its own feedback column, so you can gate on one oracle and merely watch another.

def judge_helpfulness(inputs, outputs, reference_outputs):
    verdict = judge_model.invoke(judge_prompt.format(
        question=inputs["question"], answer=outputs["text"]))
    return {"key": "helpfulness", "score": verdict["score"]}

Make it a CI gate

evaluate() returns an experiment-results object you can turn into a dataframe, so the gate is a two-line assertion at the end of your CI job. Read the pass rate off the feedback column, compare it to a threshold you agreed on, and fail the build when it slips. Because the run is tagged with the SHA and shows up as an experiment in the LangSmith UI, a red build comes with a clickable list of exactly which examples regressed, not just a number.

df = results.to_pandas()
score = df["feedback.answer_correct"].mean()
assert score >= 0.90, f"regression: {score:.2f} below 0.90 gate"

If you prefer to live inside your existing test runner, install the langsmith[pytest] plugin and mark a normal test with @pytest.mark.langsmith. Each test case is recorded as a tracked example, and expect(...) assertions surface as feedback in the UI, so a developer running pytest locally and CI running the suite write to the same history. Either path gives you the property that matters: no prompt, model, retrieval, or chain change merges without the oracle set voting on it first.

Monitor production runs and capture feedback

This is where LangSmith for QA stops being a pre-merge tool and becomes an observability one. Turn on tracing in the production worker with two environment variables, and every request becomes a run grouped under a project you can filter, chart, and mine.

# production runtime env
LANGSMITH_TRACING=true
LANGSMITH_PROJECT=qa-app-prod
LANGSMITH_API_KEY=ls__...

A trace on its own tells you what happened but not whether it was good. That signal comes from feedback. Attach it to a run with create_feedback: a thumbs control in the product, a downstream heuristic (did the student retry the same question ten seconds later?), or a later human review all become the same object.

client.create_feedback(run_id, key="thumbs", score=0,
                       comment="answer contradicted the docs")

Now the run store is queryable like a log aggregator with an opinion attached. You can pull the errored runs, the slow ones, or the ones a user rated down, all with list_runs. The following table maps the prod signals a QA owner watches to the LangSmith surface that carries them and the action each should trigger.

Feed failures back into the dataset

Every production failure is a test case you were missing. The loop is: find the bad runs, harvest their inputs, and grow the dataset so the next evaluate() covers them. Query the root runs that errored or carry negative feedback, then write each one back as an example.

for run in client.list_runs(project_name="qa-app-prod",
                             error=True, is_root=True):
    client.create_example(
        inputs=run.inputs,
        dataset_id=ds.id,   # reference filled in during review
    )

Note the missing outputs. A run that errored or got a thumbs-down does not carry a trustworthy answer, so you capture the input and leave the reference blank, then let a human supply the correct label. LangSmith annotation queues are built for exactly this: route flagged prod runs to a queue, a reviewer assigns the gold answer, and the example lands in the dataset fully formed. You can automate the routing with an online evaluator (an automation rule) that samples a percentage of incoming traces, scores them with one of your evaluators or a rule, and drops anything below threshold into the queue. That turns your regression suite into a living fixture set that grows from real student traffic instead of your imagination.

Catch drift before students do

Drift is the slow kind of regression: nothing broke in a single commit, but the score has bled from 0.92 to 0.84 over a month of model updates and prompt tweaks. Two LangSmith features catch it. First, the experiment comparison view lines up every run of the same dataset so you can literally watch the trend and open the examples that flipped. Second, pairwise (comparative) evaluation asks a judge which of two versions answered a given input better, which is far more reliable than trusting two absolute scores.

from langsmith import evaluate_comparative

evaluate_comparative(
    ["ci-abc123", "ci-def456"],  # two experiment names
    evaluators=[prefers_grounded_answer],
)

Guard the judge from drifting too. If your LLM-as-judge prompt lives in the Prompt Hub, pull a pinned commit rather than the moving latest so the oracle itself stays constant while the app changes underneath it. A judge that silently gets more lenient will hide a real regression, which is the QA equivalent of a flaky assertion that quietly starts passing.

Put together, the flow is one continuous loop: evaluate() gates the merge, tracing and feedback watch production, failed runs and annotation queues refill the dataset, and comparative eval plus the monitoring charts flag drift for the next round. That loop, and not any single run, is what makes LangSmith for QA a real regression system instead of a one-off benchmark you ran once and screenshotted. Wire it into CI the same day you build the first dataset, because an oracle set that only runs when someone remembers is exactly as good as a Playwright suite nobody scheduled.

7. Putting LangSmith eval gates in CI

An LLM feature is just another system under test, and the cruelest thing about it is that it degrades quietly. A prompt tweak, a model version bump, or a refactor of your retrieval step will not throw an exception. It will simply start answering a few more questions wrong, and nobody notices until a student complains. A unit test asserts on exact outputs, which does not survive contact with a non-deterministic model. What you want instead is a gate that runs your app over a fixed set of cases, scores the answers, and refuses to let the pull request merge when quality drops. This is where LangSmith for QA stops being a tracing and debugging aid and becomes a real quality gate, the same way a Playwright suite guards your UI.

The shape of an eval gate

Every LangSmith eval gate is the same four moving parts: a target function that produces outputs, a pinned dataset of inputs plus reference answers, one or more evaluators that score each output, and a threshold you assert against. The evaluate() function ties them together. It calls your target once per example, runs the evaluators, uploads the whole thing as an experiment, and hands back a results object you can aggregate. Wrap that in a plain pytest test so it runs under the same command your team already trusts.

import os, json
from langsmith import Client, evaluate

client = Client()   # reads LANGSMITH_API_KEY from the env

def run_qa_agent(inputs: dict) -> dict:      # the target under test
    return {"answer": my_app(inputs["question"])}

def correctness(inputs, outputs, reference_outputs) -> dict:
    hit = outputs["answer"].strip() == reference_outputs["answer"].strip()
    return {"key": "correctness", "score": int(hit)}

def test_qa_agent_no_regression():
    results = evaluate(
        run_qa_agent,
        data="qa-regression-v3",           # dataset by name; add as_of= to pin an exact version
        evaluators=[correctness],
        experiment_prefix="ci-qa-agent",
        metadata={"commit": os.environ.get("GITHUB_SHA", "local")},
        max_concurrency=4,
    )
    df = results.to_pandas()
    score = df["feedback.correctness"].mean()
    assert score >= 0.85, f"correctness {score:.3f} below gate 0.85"

The target takes the example's inputs dict and returns an outputs dict. The custom evaluator uses the current (inputs, outputs, reference_outputs) signature and returns a dict with a key and a numeric score. Because correctness here is binary, the mean is just the pass rate, and the assert turns a pass rate below 0.85 into a failing test. If you would rather not depend on the to_pandas() column naming, iterate the results object directly: each row is a dict, and row["evaluation_results"]["results"] holds the individual EvaluationResult objects with their own key and score attributes. Same numbers, more control.

Pin the dataset so the gate is deterministic

A gate is only meaningful if the thing it measures against does not move under you. If a teammate adds ten easy examples to the dataset the night before your PR, your score jumps for reasons that have nothing to do with your code, and the comparison to last week is meaningless. LangSmith datasets are versioned, so treat a dataset version like a frozen fixture: tag it, and reference that tag with the as_of argument on list_examples(). Pass the resulting examples straight into evaluate() as the data argument.

from langsmith import Client, evaluate
client = Client()

# freeze to a tagged version so the gate is comparable across PRs
examples = list(client.list_examples(
    dataset_name="qa-regression",
    as_of="ci-baseline",          # a version tag, or a datetime
))
results = evaluate(run_qa_agent, data=examples, evaluators=[correctness])

Now dataset growth is a deliberate act. When you want the gate to cover new cases, you add examples with create_examples(), move the ci-baseline tag forward, and bump the baseline number in the same PR so the change is reviewable. The dataset stops being an invisible dependency and becomes versioned test data, exactly like the golden files your integration suite already checks in.

Compare against a baseline, not just a floor

A hard floor catches catastrophes but sleeps through slow bleed. A model that was answering 92 percent correctly and now sits at 86 is still above an 0.85 floor, yet it just lost six points of accuracy that your students will feel. The fix is to gate on two things: an absolute floor for the disasters and a delta against a known-good baseline for the regressions. The simplest honest baseline is a number committed next to your tests, updated only in a reviewed PR.

# committed baseline: tests/baselines.json -> {"correctness": 0.88}
with open("tests/baselines.json") as f:
    baseline = json.load(f)["correctness"]

score = df["feedback.correctness"].mean()
assert score >= 0.85, "below absolute floor"
assert score >= baseline - 0.02, f"regressed {baseline:.3f} -> {score:.3f}"

If you prefer a live baseline over a checked-in file, LangSmith already stores every experiment's aggregate feedback. Because evaluate() records each run under a project named from your experiment_prefix, you can read a previous run's numbers with client.read_project(project_name=prev_experiment).feedback_stats, which returns per-key stats including the average. The committed number is easier to reason about in review, so start there and reach for feedback_stats only when you want the baseline to track your main branch automatically.

Wire it into GitHub Actions

With the test written, CI is trivial. Trigger the workflow on pull_request, install the SDK, hand the job your LangSmith key as a secret, and run pytest. A non-zero exit from the failing assert marks the check red, and branch protection does the rest by blocking the merge. The whole point of LangSmith for QA in CI is that a red build, not a Slack complaint three days later, is what tells you a prompt change made answers worse.

# .github/workflows/eval-gate.yml
name: eval-gate
on: pull_request
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -U langsmith pytest pandas
      - name: run gate
        env:
          LANGSMITH_TRACING: "true"
          LANGSMITH_API_KEY: ${{ secrets.LANGSMITH_API_KEY }}
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: pytest tests/test_eval_gate.py -q

Setting LANGSMITH_TRACING to true means the traced internals of your app, every @traceable step and every model call, get logged under the experiment. That is what makes a red build actionable: the metadata={"commit": ...} you passed links the experiment straight back to the PR, and one click takes a reviewer from the failing check to the exact examples that regressed and the full trace of each wrong answer. The gate tells you it broke, and the trace tells you where, which collapses the triage step that usually eats the afternoon. If you are still on the older environment names, LANGCHAIN_TRACING_V2 and LANGCHAIN_API_KEY are the equivalent variables and behave the same way.

Pairwise gates and the flaky-judge problem

Absolute scores are the right default, but some qualities are easier to judge as a comparison than on an absolute scale. Is answer A more helpful than answer B is a question an LLM judge answers far more reliably than rate this answer from 0 to 1. LangSmith supports comparative, pairwise evaluation for exactly this: you run your candidate and a baseline over the same dataset as two experiments, then a comparative evaluator picks a winner per example, and you gate on the win rate. Use it for the fuzzy dimensions like tone and completeness, and keep the binary correctness floor for the objective ones.

Two cautions before you trust any of this in CI. First, an LLM-as-judge evaluator (whether hand-written or built from the openevals helpers) is a metered model call, so a large dataset scored on every PR burns tokens and can trip rate limits. Keep a small, fast deterministic gate on every push, and run the expensive judged suite nightly or on a labeled subset. Second, the judge is itself a flaky oracle, and you should treat it like any flaky test: pin its model and temperature, sample enough examples that the mean is steady, and spot-check the judge's own verdicts against a handful of human labels before you let it block merges.

Put together, this is LangSmith for QA doing the same job your Playwright suite does for the UI: a pinned dataset stands in for fixtures, evaluators stand in for assertions, and evaluate() plus a threshold turn subjective answer quality into a pass or fail signal a reviewer can act on. Start with one binary correctness gate on a pinned dataset, get it green and merged, then layer on the baseline delta and the pairwise checks as the feature earns them. The gate that ships first and runs on every PR beats the perfect eval harness that never leaves a notebook.

8. Guardrails, cost, and safety

Everything the earlier sections gave you (full tracing with @traceable, golden datasets, an LLM-as-judge, live monitoring, and feedback) turns into a liability the instant you run it at production volume with no limits. A trace is a network call plus stored payload; a judge is a nondeterministic model grading another nondeterministic model; a dataset is an oracle that quietly rots. Using LangSmith for QA well is less about adding more evaluators and more about putting a contract around the ones you have: cap the cost, keep personal data out of the payload, keep the oracle honest, and refuse to let one noisy run fail a build. Treat this section the way you treat flaky-test triage, as the boring discipline that keeps the signal trustworthy.

Sample traces before they sample your budget

Tracing every production request at full payload is the fastest way to a surprise invoice and a slow app. You almost never need 100 percent of prod runs to spot a regression; a representative sample is enough for trend monitoring, and you keep full fidelity where it actually matters (CI experiments, incident windows). LangSmith supports head sampling through a sampling-rate setting: a float between 0.0 and 1.0 that decides the fraction of traces the SDK ships. Note the SDK rebranded its environment prefix, so the older LANGCHAIN_ names still work as aliases for the newer LANGSMITH_ ones.

# Log 10% of production traffic, keep 100% in CI
export LANGSMITH_TRACING="true"
export LANGSMITH_TRACING_SAMPLING_RATE="0.1"

The QA framing: sampling is your coverage-versus-cost dial. In CI you want deterministic, complete traces so a failing experiment is fully reproducible, so run there at rate 1.0. In prod you want a cheap, steady signal, so sample down. Keep the two projects separate (a distinct LANGSMITH_PROJECT for CI versus prod) so a sampled production number is never mistaken for a full evaluation result. Do not sample your evaluation runs themselves; evaluate() against a dataset is a bounded, deliberate cost you actually want fully recorded.

Keep PII out of the trace (redaction)

A trace captures inputs and outputs verbatim, which means student emails, support-ticket bodies, resumes, and API keys land in LangSmith unless you stop them at the source. Redact before the payload leaves the process, not after. The blunt instrument is the boolean env switch (LANGSMITH_HIDE_INPUTS / LANGSMITH_HIDE_OUTPUTS), which drops the fields entirely. The precise instrument is a callable passed to the Client: it receives the inputs or outputs and returns a scrubbed version, so you keep the shape of the data for debugging while masking the sensitive parts.

from langsmith import Client
import re

def scrub(inputs):
    text = str(inputs)
    text = re.sub(r"[\w.]+@[\w.]+", "[EMAIL]", text)
    return {"redacted": text}

client = Client(hide_inputs=scrub, hide_outputs=lambda o: {"redacted": "***"})

For anything more than a couple of patterns, the SDK also ships a create_anonymizer helper that applies a list of regex replacement rules across the whole payload, which is easier to review than a hand-rolled scrubber. Treat redaction like a security test, not a nice-to-have: add a fixture that sends a known fake email and SSN through the traced path, then assert (via the Client reading the run back) that the stored input contains the mask and not the raw value. That single test is your oracle for "no PII leaked into observability," and it belongs in the same suite as the rest of your LangSmith for QA checks.

Dataset hygiene: your oracle can rot

An evaluation is only as trustworthy as the dataset behind it, and datasets decay in predictable ways. Three rules keep them clean. First, no leakage: if an example also appears in your prompt as a few-shot demonstration, the model has seen the answer and your score is inflated. Keep few-shot exemplars in a separate list from the graded set. Second, use splits so you are not grading train on train; LangSmith lets you tag examples into named splits (for instance a small curated test split) and read just that split back at evaluation time. Third, version deliberately. Datasets are versioned, and you can read a specific historical version with the as_of parameter when listing examples, so a CI run can pin the exact oracle it was written against instead of silently drifting as teammates add rows.

Promotion discipline matters just as much. It is tempting to pipe every interesting production trace straight into the golden set, but auto-promoted rows carry unreviewed labels and often PII. Route them through an annotation queue for a human to confirm the expected output first, the same way you would not merge a generated test without reading its assertion. Keep golden sets small, curated, and captioned with metadata (source, difficulty, the bug they regress-test); a tight 200-row set you trust beats a 20,000-row set nobody has audited.

The flaky judge problem

LLM-as-judge is itself a flaky test. Even at temperature 0 the same input can score differently across runs, and the judge has its own biases (it rewards longer, more confident, more nicely formatted answers regardless of correctness). Do not treat one judged score as ground truth. The first defense is repetition: evaluate() takes a num_repetitions argument that runs each example several times so you can look at the mean and the spread rather than a single sample.

# Smooth judge variance by grading each example five times
results = evaluate(
    target,
    data="qa-golden-set",
    evaluators=[correctness_judge],
    num_repetitions=5,
)

The second defense is calibration. Before you trust a judge, correlate it against human labels on a sample: log human verdicts with client.create_feedback(run_id, key="human_correct", score=1) and check that the judge agrees often enough to be useful. A judge that disagrees with humans a third of the time is a broken oracle, and tightening its rubric prompt is the fix, not adding more runs. When you are comparing two prompts or two models, prefer pairwise evaluation (LangSmith's comparative flow, for example evaluate_comparative) over comparing two absolute scores; asking the judge "which of these two is better" is more stable than asking it to put an absolute number on each in isolation.

Never gate CI on a single run

The cardinal anti-pattern is wiring a build to fail the moment one experiment dips below a hardcoded score. Between model nondeterminism, judge variance, and a small dataset, a single number will cross any fixed line by chance, and you will train your team to rerun until green, which destroys the signal. Gate on aggregates and trends instead: use the mean across num_repetitions, require the score to hold across the whole test split rather than any one example, and compare against a stored baseline experiment (has this PR regressed relative to main) rather than an absolute threshold pulled from thin air. If you must have a hard gate, make it a wide guardrail (catastrophic drop, not a two-point wobble) and let the fine-grained scores be reported, not blocking. This is the same lesson as UI test flakiness: you quarantine and investigate a noisy signal, you do not let it randomly redden the pipeline.

Anti-patterns at a glance

9. FAQ and roadmap recap

We have covered a lot of ground, so before the questions, here is the whole journey on one screen. Every section in this guide mapped to one stage of a six-stage roadmap for adopting LangSmith for QA, and each stage answers a question you already ask of any test suite: can I see it, can I organize it, can I pin down expected behavior, can I gate on it, can I compare two candidates, and can I watch it in production. If you skimmed the middle, this recap plus the six answers below is the load-bearing summary.

Stage one, tracing: set LANGSMITH_TRACING=true (older code uses LANGCHAIN_TRACING_V2=true) plus LANGSMITH_API_KEY, or decorate functions with @traceable, so every model call lands as a run. Stage two, read a trace: open a run and walk its tree to the step where the state went bad, the way you read a failing end-to-end test. Stage three, datasets: promote real, triaged runs into a golden set with create_dataset and create_examples. Stage four, evaluators: write LLM-as-judge and heuristic scorers and run them with evaluate() for an offline gate. Stage five, gate CI: fail the build when the score drops below your baseline, so a regression never merges. Stage six, monitoring: dashboards, create_feedback, and online evaluators keep the system honest once real traffic hits it.

Read the table as a maturity ladder, not a checklist you must clear in one sprint. Turning on tracing already beats print debugging, and most teams get their first real win the moment stage three hands them a dataset they can replay against every change. The sibling guides in this series go deeper on each layer; this page is the QA-first tour through all of them.

Is LangSmith better than Langfuse for a QA team?

They solve the same shape of problem (tracing, datasets, evaluation, prompt management) with different trade-offs, so "better" depends on your constraints. LangSmith comes from the LangChain team and gives you the least friction if you already run LangChain or LangGraph, plus a managed evaluate() harness, first-class LLM-as-judge support, and the prompt hub. It is a hosted SaaS with a self-managed option on higher tiers. Langfuse is open-source at its core, which many teams pick when they want to self-host by default, avoid per-trace pricing, or keep everything inside their own infrastructure without an enterprise contract. For a QA team the honest decision rule is this: if you value a batteries-included eval harness and tight LangChain integration, LangSmith is the shorter path; if open-source control and zero-cost self-hosting rank higher, evaluate Langfuse. Both let you attach human and automated feedback to runs, so neither locks you out of proper regression testing. Do not choose on a benchmark you saw in a tweet; run a one-week spike on your own traces and datasets, then decide.

Do I need LangChain to use LangSmith?

No, and this is the single most common misconception. LangSmith traces any Python or JavaScript function through the @traceable decorator, and it ships thin wrappers like wrap_openai so a raw provider SDK call shows up as a run with inputs, outputs, latency, and token counts. If you have a homegrown pipeline built on the OpenAI SDK, the Anthropic SDK, or plain HTTP requests, you can adopt LangSmith for QA without rewriting anything on LangChain.

from langsmith import traceable
from langsmith.wrappers import wrap_openai
from openai import OpenAI

client = wrap_openai(OpenAI())   # raw OpenAI SDK, no LangChain

@traceable                        # any function becomes a run
def answer(q: str) -> str:
    r = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": q}],
    )
    return r.choices[0].message.content

Set LANGSMITH_TRACING=true and LANGSMITH_API_KEY, and the run appears. The evaluate() function is equally framework-neutral: it accepts any callable target, so your existing agent is testable as-is.

How reliable is LLM-as-judge for pass and fail decisions?

Useful, but never treat it as an oracle. An LLM judge is non-deterministic and carries known biases: position bias in comparisons, verbosity bias toward longer answers, and self-preference toward its own model family. It can also drift silently when the judge model is updated under you. Concrete mitigations that fit LangSmith practice: pin the judge model and set temperature to zero; make the judge return a structured score plus written reasoning against an explicit rubric so you can audit it; calibrate against a human-labeled subset by logging both with create_feedback and checking agreement; prefer pairwise eval for "which is better" because relative judgments hold up better than absolute scores; and always pair the judge with deterministic custom evaluators (exact match, regex, JSON schema validity, latency ceilings). The safe pattern is to gate CI on the deterministic checks and let the judge cover the softer dimensions like helpfulness or tone, surfacing borderline runs for human triage rather than auto-merging on a model's say-so.

How much does LangSmith cost, and how do traces add up?

Pricing is trace-based with a per-seat component. There is a free developer tier with a monthly trace allowance, then paid plans with a larger allowance and overage charged per trace beyond it, plus enterprise pricing for self-hosting and higher limits. Because every traced run in production counts, cost scales with volume, so the levers that matter for a QA budget are operational, not contractual. Sample production tracing instead of tracing one hundred percent of traffic. Keep heavy evaluation runs pointed at datasets rather than replaying live traffic. Separate a dev project from a prod project so experiment runs do not consume the production allowance. Use tags and metadata to find and prune noisy or duplicated traces. Remember that evaluate() also spends real model API dollars on every target and judge call, which is a separate line item from LangSmith's trace charge. Check the current pricing page for exact figures before you commit, since tiers change.

Can I self-host LangSmith behind our firewall?

Yes. LangSmith offers a self-managed deployment (Kubernetes or Docker) on enterprise plans, and a hybrid option where the control plane is managed but your trace data stays inside your own network. For QA teams handling regulated data, PII in prompts, or strict data-residency rules, this is often the deciding factor. The important practical detail is that your instrumentation code does not change: you point the SDK at your instance through the LANGSMITH_ENDPOINT environment variable (older code uses LANGCHAIN_ENDPOINT) and supply the matching API key. The same @traceable functions, datasets, and evaluate() calls run unmodified. The managed cloud is the fastest way to start and evaluate the tool; if compliance later demands it, you migrate the endpoint rather than the test code.

What should actually go into a QA dataset?

Treat the dataset as your regression corpus, curated the way you would curate a test suite. Include representative happy-path inputs with known-good reference outputs; real production failures you have triaged from traces, promoted into examples so a fixed bug can never silently return; edge and adversarial cases such as prompt injection, empty input, oversized input, and wrong-language input; and property-based cases where the correct answer is a rule rather than a fixed string, paired with a custom evaluator that checks valid JSON, a required citation, or absence of PII. Build it once with create_dataset, then add rows with create_examples, and version it as coverage grows.

from langsmith import Client
client = Client()

ds = client.create_dataset("qa-golden-set", description="regression corpus")
client.create_examples(
    inputs=[{"q": "reset my password"}, {"q": ""}],
    outputs=[{"a": "Go to Settings -> Security"}, {"a": "Please enter a question"}],
    dataset_id=ds.id,
)

Resist the urge to bulk-generate thousands of near-duplicate rows. A small, diverse, deliberately chosen set catches more real regressions and runs faster, which is what turns running LangSmith for QA into honest regression testing instead of a vibe check. Start with the failures that already hurt, add the edge cases you fear, and let production traces feed the rest over time.

Series siblings: LangChain for QA and LangGraph for QA. See also RAG for QA, AI Agents for QA, and the AI for QA hub.