QA AI Agents,
built node by node.
A hands-on series where we build production-style QA agents in Langflow — no glue code. Each agent reads real engineering data, reasons over it with an LLM, and returns strict JSON your tools can consume.
One pattern, three agents
Every agent in this series follows the same backbone: get the data → make it readable → tell the model what to decide → let it reason → force the shape → ship the result. Learn it once on Agent 1, and Agents 2 and 3 are variations on the same theme.
| Agent | Reads | Decides | Status |
|---|---|---|---|
| Bug Triage | A Jira ticket | Severity, priority, impact, RCA | Live |
| Flaky Test Analyzer | Two Playwright runs | Flaky vs real failure, reruns | Live |
| API Contract Validator | Your cURL requests | Status + schema conformance, pass/fail | Live |
Langflow basics,
from zero to Hello Groq.
Before we build the three QA agents, get comfortable with Langflow itself: what it is, how to install it, the components you'll wire together, and two starter flows — a plain Hello World and a Hello Groq that actually calls an LLM.
What is Langflow?
Langflow is an open-source, visual builder for LLM and agent apps. You drag components (nodes) onto a canvas and connect their ports; each wire carries a typed value (a Message, Data, or a LanguageModel). No glue code — the flow is the program.
| What it gives you | Why it matters for QA |
|---|---|
| Visual drag-and-drop canvas | See the whole pipeline as a sentence: input → reason → output |
| Built-in Playground | Run and test a flow instantly, no deploy |
| Every flow is also an API endpoint | Call your agent from CI, a script, or another tool |
| Import / export as JSON | Share a flow with students in one file (that's what our 3 agents are) |
| Python under the hood | Drop into a custom component when a node doesn't exist |
How to install Langflow
Python 3.10–3.13. Use a fresh virtual environment so Langflow's dependencies stay isolated. Pick one of the three ways below, then open http://127.0.0.1:7860.
# 1) fresh virtual env (recommended) python -m venv .venv source .venv/bin/activate # Windows: .venv\Scripts\activate # 2) install Langflow pip install langflow # 3) start it (opens the canvas on :7860) langflow run
# uv is a fast Python package manager # install: https://docs.astral.sh/uv/ # run Langflow without a manual venv uvx langflow run # or install into a project uv pip install langflow langflow run
# no Python setup needed - just Docker docker run -it -p 7860:7860 langflowai/langflow:latest # then open http://127.0.0.1:7860
Core components of Langflow
Components are grouped in the left palette. You'll build almost everything in this masterclass from these six families. Drag one out, and its input/output ports tell you what it accepts and emits.
1 · Inputs & Outputs
2 · Prompts
3 · Models (LLMs)
4 · Data & Loaders
5 · Processing & Helpers
6 · Agents, Memory & Vector Stores
Hello World — your first flow
The smallest possible flow: take input, echo it back. Two nodes, one wire. This proves Langflow runs and teaches you how to connect ports.
Drag it from Inputs. It emits whatever you type as a Message.
Drag it from Outputs. Connect Chat Input's Message port into it.
Build it
New Flow → Blank Flow. Drag Chat Input and Chat Output onto the canvas. Drag a wire from the Input's output dot to the Output's input dot.
Run it
Click Playground (top right). Type hello world and send. The Output echoes it back. That's a working flow.
Hello Groq — call a real LLM
Now put a model in the middle. Add the Groq component between input and output, paste a free API key, and your flow starts thinking.
e.g. "Explain flaky tests in one line."
From Models. Paste your GROQ_API_KEY and pick a model. Wire the input Message into its Input.
Connect Groq's Message output here.
Get a free Groq key
Sign in at console.groq.com → API Keys → Create. Copy it once (you can't see it again).
Add & wire the Groq node
Drag Groq between Chat Input and Chat Output. Paste the key into Groq API Key. Pick a model such as llama-3.3-70b-versatile or openai/gpt-oss-120b.
Run it
Open the Playground, ask your question, and Groq answers. You just built an LLM app with three nodes.
# Better than pasting the key into the node: # set it as an environment variable / Langflow global variable export GROQ_API_KEY="gsk_your_key_here" # then reference it in the Groq component's API key field # as a global variable -> export the flow WITHOUT the secret
The overall flow — one shape, every agent
Hello Groq is the seed. Add a Prompt to set the instructions and a Structured Output to force JSON, and you have the exact backbone all three QA agents use.
Chat Input, an API Request (Jira), or a File (Playwright report).
The instructions + the fields you want back.
Low temperature so verdicts stay stable.
Re-extract the answer against a fixed JSON schema.
Show JSON, or write back to Jira / Slack / CI.
Ready? Jump into Agent 01 · Bug Triage and watch this backbone come alive on a real Jira ticket.
Bug Triage AI Agent
Hand it one Jira issue key. It fetches the ticket, reads it with Groq, and returns severity, priority, impact areas, and a root-cause hypothesis — as strict JSON.
What we're solving
A QA lead opens a fresh Jira ticket and decides by hand: how bad is this, how fast must it be fixed, what does it touch, and what likely caused it. Across a backlog that's hours of repetitive judgement — and it drifts between people.
We want to hand the agent one issue key, have it fetch the ticket, reason over the description, and return a consistent verdict in a machine-readable shape — ready to feed dashboards, auto-label tickets, or route to a team.
Five required outputs map to a straight-line pipeline. Each node has one job; the chip colour is the data type on the wire.
Operator types one key, e.g. PROJ-123. The only manual input.
Calls Jira Cloud REST v3 /issue/{key} with Basic auth; returns summary, description, priority, components, labels.
Maps nested Jira fields into a clean text block. Models reason better over prose than deep JSON.
Wraps the ticket with the triage brief and the five fields to decide. {issue} is filled by the node above.
Runs the triage thinking on Groq's OpenAI-compatible endpoint. Temperature 0.1 so verdicts stay stable.
Takes the prose and a LanguageModel handle, re-extracts against a fixed schema. Free text becomes reliable fields.
Renders the JSON in the Playground. In production, swap for a node that writes back to Jira or posts to Slack.
Severity and priority are different axes. Severity = how broken. Priority = how urgently we fix it. The model picks from a closed set on each, so output stays consistent.
Severity — technical impact
Priority — business urgency
This text lives inside the Prompt Template node — the entire reasoning spec. {issue} is replaced at runtime with the parsed ticket.
You are a senior bug triage engineer. Analyze the Jira issue below and produce a structured triage.
JIRA ISSUE:
{issue}
Assess and decide ALL of the following:
1. SEVERITY - technical impact. One of: Blocker, Critical, Major, Minor, Trivial.
2. PRIORITY - business urgency. One of: P0, P1, P2, P3, P4.
3. IMPACT_AREAS - modules, journeys, or systems affected.
4. ROOT_CAUSE_ANALYSIS - best hypothesis of the underlying cause.
5. JUSTIFICATION - one or two sentences on the severity/priority call.
Be decisive. Base every conclusion only on the issue content. Do not invent stack traces or logs.Import the JSON, or rebuild from scratch. Either way you only fill two secrets.
Get the flow into Langflow
Download the JSON below. In Langflow: project → New Flow → Import → pick the file. All seven nodes appear pre-wired.
Create a Jira API token
At id.atlassian.com → Security → API tokens. Base64-encode email:token and put it in the API Request Authorization header as Basic <value>.
Point the URL at your Jira
Replace YOUR_DOMAIN in the API Request URL with your site. Keep the ?fields=… query — it keeps the response small.
Add your Groq key
Get a key at console.groq.com (gsk_…). Paste into the Groq node. Model is preset to openai/gpt-oss-120b.
Run it
Open Playground, type PROJ-123, send. Watch the data move node to node; the triage JSON lands in chat.
Test the agent like you'd test any classifier — on the edges where its judgement is hardest.
Severity ≠ priority: feed a cosmetic typo on a public page. Verify it returns low severity but high priority — not both low.
Sparse ticket: a one-line description with no repro. The RCA field should say what's unknown, not invent a stack trace.
Schema conformance: run 10 tickets. Every output must contain all five keys with valid enum values for severity/priority.
Auth failure: use a bad Jira token. Verify the flow surfaces the API error instead of triaging an empty body.
Determinism: same ticket, 5 runs. At temp 0.1 the severity/priority verdict should not drift.
What a run returns
"severity": "Critical",
"priority": "P1",
"impact_areas": ["Checkout", "Payment gateway", "Order email"],
"root_cause_analysis": "Null order ID reaching the payment callback — likely a race between cart-clear and webhook receipt.",
"justification": "Revenue path broken for a subset of users with a fragile workaround, so Critical severity and P1 urgency."
}
Flaky Test Analyzer
Give it two Playwright runs of the same suite. It separates genuinely flaky tests from real, reproducible bugs — and tells you exactly which ones to rerun.
Flaky, or actually broken?
A suite goes red. Half the failures are real bugs; the other half are flaky tests that pass on the next run. Teams waste hours re-running everything and lose trust in the suite. The two need opposite responses: bugs go to engineering, flakes get quarantined or rerun.
We hand the agent two Playwright JSON reports — Build 1 and Build 2 of the same suite — and it tells us which tests are flaky, which are consistently failing, and what to do with each.
Two file inputs feed a custom comparison node, then the same reason → shape → ship backbone as Agent 1.
The Playwright JSON report from the first run. Carries status, retries, ok per test.
The JSON report from the second run of the same suite.
Indexes both reports by test name, then classifies each: flaky (mixed / retry), consistent failure (both), or stable. The deterministic core — done in code, not by the LLM.
Gives the model the strict flaky-vs-bug definitions and asks for per-test cause hypotheses + rerun advice.
Adds the reasoning the diff can't: why each test is flaky, likely root causes for the real failures.
Schema: flaky_tests, consistent_failures, rerun_recommendation, counts, summary.
In production, post to Slack or open Jira tickets for the consistent failures automatically.
The heart of the agent. This runs inside the Flaky Diff custom component — pure Python, deterministic, no LLM.
def build_comparison(self):
r1 = self._index(self.run1) # {test_title: {status, retries}}
r2 = self._index(self.run2)
all_tests = sorted(set(r1) | set(r2))
flaky, consistent_fail, retry_flaky, stable = [], [], [], []
for t in all_tests:
a = r1.get(t, {}).get("status", "absent")
b = r2.get(t, {}).get("status", "absent")
ra = r1.get(t, {}).get("retries", 0)
rb = r2.get(t, {}).get("retries", 0)
if {a, b} == {"passed", "failed"}: # mixed across runs
flaky.append(t) # -> FLAKY
elif a == "failed" and b == "failed": # failed in both
consistent_fail.append(t) # -> REAL BUG
elif a == b == "passed" and (ra or rb): # passed, but retried
retry_flaky.append(t) # -> FLAKY
elif a == b == "passed":
stable.append(t) # -> STABLE
return report(flaky, retry_flaky, consistent_fail, stable)| Build 1 | Build 2 | Verdict | Action |
|---|---|---|---|
pass | fail | Flaky (mixed) | Rerun / quarantine |
fail | pass | Flaky (mixed) | Rerun / quarantine |
pass (retry) | pass | Flaky (retry) | Investigate wait/timing |
fail | fail | Consistent failure | File as a bug |
pass | pass | Stable | None |
The reliability brief inside the Prompt Template node. The deterministic diff has already classified everything — the model adds the why.
You are a senior test reliability engineer. You are given a comparison of two Playwright runs (Build 1 and Build 2) of the same suite.
COMPARISON REPORT:
{comparison}
Definitions you MUST follow:
- FLAKY = non-deterministic result: passed in one build and failed in the other, OR passed only after a retry. Flaky tests need a rerun / quarantine, not a code fix.
- CONSISTENT FAILURE = failed in BOTH builds. A real, reproducible bug, NOT flaky. Needs a fix.
Produce:
1. FLAKY_TESTS - names + one-line hypothesis of flake cause (timing, data, parallelism, network...).
2. CONSISTENT_FAILURES - tests failing in both builds, each with a probable root cause.
3. RERUN_RECOMMENDATION - which to rerun (flaky) vs send to engineering (bugs).
4. SUMMARY - counts + one sentence on suite health.
Base everything only on the comparison data. Do not invent test names.One secret to fill (Groq), plus the custom component code is already embedded in the flow.
Generate two Playwright reports
Add the JSON reporter: reporter: [['json', { outputFile: 'results.json' }]] in playwright.config.ts. Run your suite twice (e.g. across two builds) to get build1.json and build2.json.
Import the flow
Download the JSON below → Langflow → Import. The Flaky Diff custom component ships inside it, code included.
Upload both reports
In the two File nodes, upload build1.json to Build 1 and build2.json to Build 2.
Add your Groq key
Paste your gsk_… key into the Groq node. Model preset to openai/gpt-oss-120b.
Run it
Open Playground and run. The report lists flaky tests with cause hypotheses, consistent failures with root causes, and a clear rerun recommendation.
Test the analyzer against the failure modes that would mislabel a test.
Mixed = flaky: a test that passes in build 1, fails in build 2. Verify it lands in flaky, never consistent_failures.
Both = bug: a test failing in both builds. Verify it's a consistent failure and the rerun advice sends it to engineering, not a rerun.
Retry pass: a test that passed only after a retry in one build. Verify it's flagged flaky even though both builds are green.
New / removed tests: a test present in only one build. Verify it doesn't crash the diff or get miscounted as failed.
Count integrity: flaky_count + consistent_failure_count must match the arrays. The model must not invent or drop tests.
Malformed JSON: feed a truncated report. Verify the File/diff step fails loudly rather than silently triaging nothing.
What a run returns
"flaky_tests": [
"profile loads — pass→fail across builds, likely a race on async profile fetch",
"search returns results — passed on retry, probable missing wait for results render"
],
"consistent_failures": [
"checkout completes — failed in both builds, payment callback returns null order ID"
],
"rerun_recommendation": "Rerun profile + search; file checkout as a bug — do not rerun.",
"flaky_count": 2,
"consistent_failure_count": 1,
"summary": "1 real bug in checkout; 2 flaky tests inflating the red. Suite health is recoverable."
}
API Contract Validator
Paste your cURL requests. It authenticates, threads the token through every protected call, fires them for real, and validates each response against a JSON Schema — then reports the verdict.
Does the API keep its promises?
An API has a contract: each endpoint should return a known status code and a response that matches an agreed shape — the right fields, the right types. When a deploy quietly changes a field from integer to string, or drops a required key, downstream consumers break. We want to catch that automatically.
The user pastes the cURL requests for an end-to-end flow — like Restful Booker's auth → create → read → update → delete. The agent runs the whole chain for real, carrying the auth token through the protected calls, and validates every response against a JSON Schema. Out comes a clear pass/fail contract report.
Two LLM stages bracket one deterministic runner: parse the cURLs → run + validate for real → explain the result.
The user pastes the requests to validate, one per line — auth call first, then the chain.
Asks the model to turn each cURL into a structured request with its contract expectations (status + schema kind).
Tolerates messy, real-world cURL syntax and produces a clean structured plan. A good LLM job.
An ordered list of {method, url, headers, body, expects}.
Authenticates, extracts the token, injects Cookie: token= on PUT/PATCH/DELETE, fires every request with requests, validates each response with jsonschema. Deterministic — no LLM.
Asks the model to explain the real results readably — but it cannot overturn a verdict.
Turns pass/fail data into a plain-English contract report with next steps.
overall_passed, counts, per_call, next_steps.
In CI, hit the flow's /run endpoint and fail the pipeline if overall_passed is false.
The deterministic core, inside the custom component. Real requests traffic, real cookie threading, real jsonschema validation.
def run_validation(self) -> Data:
session = requests.Session()
results, token = [], None
# 1) AUTH FIRST - get the token. Restful Booker returns 200 even on
# bad credentials, so assert the token EXISTS, not just the status.
auth = session.post(f"{self.base_url}/auth",
json={"username": self.username, "password": self.password},
timeout=self.timeout)
token = auth.json().get("token")
results.append({"name": "AUTH POST /auth", "actual_status": auth.status_code,
"passed": auth.status_code == 200 and token is not None})
# 2) Run the rest of the plan IN ORDER, threading the token through.
for req in self._plan():
method, url = req["method"].upper(), req["url"]
headers = dict(req.get("headers") or {})
headers.setdefault("Accept", "application/json")
# cookie auth ONLY on the mutating calls - exactly Restful Booker's contract
if method in ("PUT", "PATCH", "DELETE") and token:
headers["Cookie"] = f"token={token}"
resp = session.request(method, url, headers=headers,
json=req.get("body"), timeout=self.timeout)
expects = req.get("expects", {})
status_ok = resp.status_code == expects.get("status", 200)
# REAL schema validation - jsonschema, not the LLM guessing
schema_ok, errors = self._validate(resp.json(), expects.get("schema_kind", "none"))
results.append({"name": req["name"], "expected_status": expects.get("status"),
"actual_status": resp.status_code, "status_ok": status_ok,
"schema_ok": schema_ok, "errors": errors,
"passed": status_ok and schema_ok})
return Data(data={"results": results})The Restful Booker contract the agent validates against. Status codes and response shapes the live API actually returns.
| Call | Auth | Expect | Validates |
|---|---|---|---|
POST /auth | — | 200 + token | token is a string |
POST /booking | — | 200 | bookingid + booking object |
GET /booking/{id} | — | 200 | full booking shape |
PUT /booking/{id} | Cookie: token= | 200 | updated booking shape |
DELETE /booking/{id} | Cookie: token= | 201 | status only |
The booking schema
firstname string (required) lastname string (required) totalprice integer (required) depositpaid boolean (required) bookingdates object (required) checkin string YYYY-MM-DD checkout string YYYY-MM-DD additionalneeds string (optional)
The cURL-parser brief inside the first Prompt Template. The model's only job here is turning messy cURLs into a clean plan — it never makes a call.
You convert raw cURL commands into a clean JSON request plan for an API contract test runner.
CURL COMMANDS:
{curls}
For EACH cURL, extract: name, method, url, headers, body, and expects (status = expected HTTP code, schema_kind = auth | booking | create | none).
Rules:
- /auth -> status 200, schema_kind 'auth'.
- POST /booking -> status 200, schema_kind 'create' (response is wrapped: bookingid + booking).
- GET/PUT/PATCH /booking/{id} -> status 200, schema_kind 'booking'.
- DELETE /booking/{id} -> status 201, schema_kind 'none'.
- Do NOT add a Cookie/token header yourself - the runner injects the auth token automatically.
Return ONLY the structured plan. Do not execute anything.One secret to fill (Groq). The custom component code and the schemas ship inside the flow.
Import the flow
Download the JSON below → Langflow → Import. The API Contract Validator custom component, with its Restful Booker schemas, is embedded.
Allow the custom component to run
The validator runs Python (requests + jsonschema). If your Langflow has code execution locked down, approve/trust the component on first import. jsonschema ships with most Langflow installs; if you hit an ImportError, add it to your instance's requirements.txt.
Add your Groq key
Both Groq nodes use the same gsk_… key, preset to openai/gpt-oss-120b.
Paste your cURLs
The Chat Input ships with a full Restful Booker example (auth → create → get → update → delete). Replace it with your own API's cURLs, or run the demo as-is.
Run it
Open Playground and send. The runner authenticates, validates the chain for real, and the report lists each call's expected vs actual status and any schema failures.
Test the validator against the failures that matter most — the ones where a naive check would pass something broken.
Status mismatch: point a call at an endpoint that returns 500. Verify the report marks it failed with expected-vs-actual, not a pass.
Type drift: validate a response where totalprice comes back as a string. The schema check must fail and name the field.
Missing field: drop a required key like lastname. Verify schema_ok is false with a clear "required property" error.
Token threading: confirm the Cookie: token= header appears only on PUT/PATCH/DELETE, never on auth, POST, or GET.
Bad-credentials trap: use a wrong password. Verify auth fails on missing token even though Restful Booker returns 200.
DELETE status: verify a successful delete expects 201, not 200 — the report must not flag the correct 201 as a failure.
No hallucinated pass: break the network mid-run. The failed call must surface the real error, never a fabricated success.
What a run returns
"overall_passed": false,
"passed_count": 4,
"total": 5,
"per_call": [
"AUTH POST /auth — 200, token received — PASS",
"POST /booking — expected 200, got 200, schema valid — PASS",
"GET /booking/42 — expected 200, got 200, schema valid — PASS",
"PUT /booking/42 — FAIL: totalprice returned as string, expected integer",
"DELETE /booking/42 — expected 201, got 201 — PASS"
],
"next_steps": "Fix the PUT response type contract for totalprice before release; all other endpoints conform."
}
The Shared Stack
Every agent in this series is built from the same small set of pieces. Learn these once.
Six steps, every time
| Step | Langflow piece | Job |
|---|---|---|
| Get the data | API Request / File / Webhook | Pull the raw engineering data in |
| Make it readable | Parse Data / Custom Component | Flatten or pre-process for the model |
| Say what to decide | Prompt Template | The reasoning spec + variables |
| Let it reason | Groq Model · gpt-oss-120b | The actual LLM thinking |
| Force the shape | Structured Output | Prose → strict typed JSON |
| Ship it | Chat Output / API Request | Display or write back |
Why Groq + gpt-oss-120b
Groq's OpenAI-compatible endpoint runs the open gpt-oss-120b model at high speed and low cost — ideal for classification and triage where you call it often and want fast, stable answers. Temperature stays low (0.1) so verdicts don't drift between runs. Swap to llama-3.3-70b-versatile if you prefer.
The one concept beginners miss
Reason in code, judge with the LLM
Where logic is deterministic — counting failures, comparing two runs, exact classification — do it in a custom component, not the prompt. Hand the model only the parts that need judgement. It's cheaper, exact, and testable. Agent 2's Flaky Diff is the template for this. Agent 3 goes further with a two-LLM sandwich: the model parses input at the front and explains results at the back, while a deterministic runner in the middle does everything that must actually be true.