The Testing Academy · Masterclass · Advanced RAG
Two retrievers. One grounded answer.
Build an Advanced RAG flow for QA and SDET work that searches the same knowledge with two open embedding models, reranks their combined evidence, and answers only from cited context. Everything runs inside one local Docker network.
Map the system
Start with the quality goal, then trace files and questions through two independent retrieval spaces into one evidence-bound answer.
Outcomes · what this flow teaches
What this flow teaches.
Search twice, rerank once, and make every answer prove where it came from.
The flow teaches four practical ideas: keep documents and inference local, use two embedding spaces to reduce one model's blind spots, rerank the merged candidates, and refuse unsupported claims. For QA engineers, each stage creates an observable checkpoint instead of one opaque chatbot response.
Treat the pipeline like a test system. Inputs are versioned documents and questions, intermediate results are inspectable chunks, and the output contract requires citations such as [Context 2].
Given: indexed QA knowledge
When: a tester asks a supported question
Then: retrieve from BGE-M3 and Nomic
And: rerank the combined candidates
And: cite at least one [Context N]
Given: no supporting passage
Then: state that evidence is missing
And: do not invent an answerWhy it mattersMore nodes are useful only when they expose or reduce a specific failure. This design makes retrieval coverage, ranking, and grounding independently testable.
Architecture · see the whole system move
See the whole system move.
Files fan out into separate vector collections. A question searches both, then one local model selects the evidence.
During ingestion, the same overlapping chunks are embedded by BGE-M3 and Nomic. Their vector dimensions differ, so each model writes to its own persistent Chroma collection.
During questioning, both collections return candidates, Qwen removes duplicates and keeps the strongest passages, then the grounded prompt produces a cited answer.
Why it mattersThe architecture exposes where relevance was gained or lost. A bad answer can be traced to chunking, one retriever, reranking, or generation.
Build the flow
Start the local runtime, import the graph, build two indexes, and understand the distinct ingestion and question paths.
Setup · three containers, one network
Three containers. One network.
Ollama serves models, Langflow runs the graph, and named volumes preserve models and indexes.
Inside a container, localhost points back to that container. A shared Docker network lets Langflow reach Ollama by the hostname ollama.
Pull BGE-M3, Nomic, and Qwen before importing the flow. Named volumes keep model files and both indexes across container restarts.
# network and Ollama docker network create advanced-rag docker volume create ollama-data docker run -d --name ollama \ --network advanced-rag -p 11434:11434 \ -v ollama-data:/root/.ollama ollama/ollama:latest # models docker exec ollama ollama pull bge-m3 docker exec ollama ollama pull nomic-embed-text docker exec ollama ollama pull qwen2.5:7b # Langflow docker volume create langflow-data docker run -d --name langflow \ --network advanced-rag -p 7860:7860 \ -v langflow-data:/app/langflow \ -e LANGFLOW_CONFIG_DIR=/app/langflow \ langflowai/langflow-all:1.10.2 docker ps --filter network=advanced-rag curl http://localhost:11434/api/tags
Why it mattersMost local connection failures are naming failures. A shared network and persistent volumes make restarts predictable without rebuilding models or indexes.
Import · from JSON to first answer
From JSON to first answer.
Import the graph, load controlled knowledge, run both Chroma nodes once, then use Playground for repeated questions.
Open Langflow at http://localhost:7860, create a project, and import langflow.json. Upload supported source files through the reader, then run each Chroma branch so both receive the same chunks encoded in their own space.
In Playground, inspect selected context before trusting prose. A supported answer should cite [Context N]. An unsupported answer should say which evidence is missing.
1. Open http://localhost:7860 2. New Project -> Import -> langflow.json 3. Read Knowledge Files -> choose PDF, CSV, TXT, MD, or DOCX 4. Run Chroma BGE once 5. Run Chroma Nomic once 6. Playground -> ask -> inspect [Context N] citations
Why it mattersIngestion is a controlled build step, not question-time work. Reusing indexes makes tests faster and prevents accidental re-embedding from changing results.
Pipeline · index once, retrieve often
Index once. Retrieve often.
Indexing turns documents into durable vectors. Retrieval turns one question into ranked evidence.
The ingestion path reads files, splits at 900 characters with 150 overlap, embeds twice, and persists two collections. The question path embeds the same question twice, retrieves six candidates per collection, reranks, and passes at most five passages to the answer prompt.
Why it mattersSeparating the paths gives QA stable fixtures. Freeze one index, replay a fixed question set, and attribute score changes to retrieval or generation.
Components · every node has one job
Every node has one job.
Readers load, splitters focus, embedders map, stores retrieve, parsers expose, and prompts constrain.
Parser nodes turn both Chroma result sets into readable lists. The rerank prompt compares evidence explicitly, while the grounded answer prompt receives only passages that survived selection.
Keep reranking deterministic at temperature 0 and generation low at 0.1. Preserve raw retrieval outputs for assertions and debugging.
Split Text chunk_size = 900, overlap = 150 Ollama Embeddings A model = bge-m3:latest Ollama Embeddings B model = nomic-embed-text:latest Chroma A collection = advanced_rag_bge_m3 Chroma B collection = advanced_rag_nomic Retriever A / B search_type = MMR, results = 6 Local Reranker model = qwen2.5:7b, temperature = 0 Grounded Answer cite = [Context N], temperature = 0.1
Why it mattersSingle-purpose nodes create clean test seams. Swap one embedding model, compare candidates, or tighten the prompt without redesigning the whole flow.
Test and tune
Understand the vector spaces, trace a realistic QA question, change only meaningful knobs, and diagnose failures from the outside in.
Embeddings · different maps of the same meaning
Different maps of the same meaning.
BGE-M3 offers broad multilingual coverage. Nomic adds a compact, independent semantic view.
Embeddings turn text into coordinates, but two models place passages in different neighbourhoods. One can surface MFA bypass and lockout while the other finds credential stuffing and recovery flows.
BGE-M3 produces 1024-dimensional vectors and Nomic produces 768-dimensional vectors. Never mix them in one collection. Retrieve text from both first, then compare that text in the reranker.
Why it mattersRetriever diversity can improve recall without giving generation more freedom. The reranker narrows the larger candidate pool before any answer is written.
Demo · simulate one question
Simulate one question.
Ask: What are the highest-risk login test scenarios?
BGE-M3 may return session timeout, MFA bypass, lockout policy, and localization noise. Nomic may return credential stuffing, MFA recovery, account lockout, and remember-me behaviour.
The reranker removes overlap and weak evidence. The final QA answer prioritizes MFA bypass, credential stuffing, and account lockout, each tied to a selected context block.
Why it mattersA polished answer can hide weak retrieval. Assert candidate relevance, shortlist quality, citations, and final usefulness as separate checks.
Tuning · four knobs that matter
Four knobs that matter.
Chunk size, overlap, candidate count, and final context count control most retrieval tradeoffs.
Smaller chunks improve precision but can lose surrounding meaning. Overlap protects boundary sentences but creates duplicates. A larger candidate pool improves recall, while a smaller final context keeps the prompt focused.
Keep fixed QA questions with expected sources. Change one knob, rebuild both collections when chunking changes, then compare recall, rank, citation correctness, latency, and groundedness.
Why it mattersWithout a fixed evaluation set, tuning becomes anecdotal. A repeatable corpus and question suite turns retrieval changes into evidence-backed decisions.
Troubleshooting · fast checks before debugging
Fast checks before debugging.
Check container state, network membership, installed models, collection dimensions, and raw retrieval in that order.
If Langflow cannot reach Ollama, confirm both containers share advanced-rag and use http://ollama:11434 inside the flow. For a dimension mismatch, rebuild only the affected collection.
When answers are vague, inspect Parser A and Parser B. If evidence is absent, tune retrieval. If good evidence survives but the answer ignores it, tighten the grounded prompt.
docker ps -a docker network inspect advanced-rag docker network connect advanced-rag langflow docker network connect advanced-rag ollama docker exec ollama ollama list docker logs -f ollama # keep dimensions isolated /app/langflow/chroma_bge_m3 /app/langflow/chroma_nomic # inspect in this order question -> Chroma A/B -> Parser A/B -> reranker -> answer
Why it mattersDebug the first stage where expected data disappears. This avoids rewriting prompts when the real defect is a stopped container, missing model, or empty index.