A test case generator that earns its way past every human gate.
Most LLM test-case generators dump 50 generic cases from a Jira blurb and call it done. This is the architecture I'd build instead: multi-source ingestion, staged approvals, business-aware mapping, explicit coverage guardrails, and a path back into the test management tool that doesn't bypass the human.
"Cheap to generate test cases. Expensive to generate the right ones. The whole architecture exists to bend that ratio."
"Generate test cases for this Jira ticket" is the wrong API
Every off-the-shelf test-case generator I have evaluated starts in the same place: paste a story, get a list. It is fast, it is impressive in a demo, and it is wrong for a real product. The failure modes are predictable.
- No business context. The LLM does not know what an MVNE is, what an eSIM activation flow looks like, or which states the order can legally transition through. It writes English that sounds like a test case.
- Single-source. The Jira description is the least informative artifact. The Confluence spec, the Figma flow, and the existing test library are where the real intent lives.
- One coverage axis. Ask an LLM for test cases and you get five happy paths. Ask for negative cases and you lose the happy paths. There is no built-in obligation to return a full matrix.
- No gate, no audit. Output goes straight into a Google Doc or, worse, straight into the TCM. The human reviewer has no anchor point to push back on.
- Expensive. Every run re-ingests the same Confluence pages, re-summarises the same product glossary, re-pays for the same context tokens.
The fix is not a better prompt. The fix is treating this as a pipeline with explicit stages, explicit gates, and a memory layer — the same shape you would design for any production data system. That is the proposal below.
The pipeline, end to end
Three approval gates. One knowledge base. One business-skill layer that the whole pipeline reads from. Generation is cheap; gates are where the human spends attention.
What each stage actually does
Connectors for Jira, Confluence, Figma, and the existing TCM pull raw artefacts on demand. Every fetch is hashed and logged through a pre/post hook so we can answer "which version of the spec produced this test set?" months later. Nothing in this stage talks to an LLM — it is a boring data layer, and that is the point.
Normalisation strips Jira/Confluence formatting noise. Chunking respects semantic boundaries (sections, Figma frames, acceptance criteria), not arbitrary token windows. Summaries are written once and cached; the same Confluence page should never get re-summarised three times in a week.
The knowledge base is RAG-style: source-backed, refreshable, citation-aware. It holds the things that change — current pricing rules, current state machines, current API contracts.
The business skill is the opposite: stable, hand-curated product literacy. What an order is, what an activation is, which flows are eligibility-gated, which ones never can be. The LLM reads it the same way a new QA hire reads onboarding docs — once, before doing anything useful. This is the layer that "teach your LLMs the business, not the trivia" is really about.
The LLM produces a structured list: "this story implies the following testable requirements, mapped to these sources, with these unknowns." It is not yet writing test cases. It is showing its reading of the story.
The human reviewer can edit, delete, or add requirements inline. The interface writes those edits back into the KB so future runs on adjacent stories inherit the correction. This is the highest-leverage gate — every later stage is downstream of getting this right.
Cheap output, expensive review. The model emits only the title and one-line intent for each proposed case, alongside a confidence score and a coverage tag (positive / negative / edge / regression). No steps yet — steps at this stage are a waste of tokens because the human will rewrite half of them anyway.
The interface lets the reviewer reorder, drop, edit, or add titles, and shows the coverage matrix live. If any axis is empty, the gate refuses to advance.
Only the blueprint that survived Gate 2 gets expanded. Steps, expected results, preconditions, test data references — formatted to the team's house style (enforced by a formatter guardrail, not by hope). The final gate is a normal QA review: the reviewer reads what they will own.
Before anything lands in AIO / Xray / Zephyr / TestRail, the dedup step embeds the new cases and compares them against the existing TCM corpus. Anything above a similarity threshold gets flagged for the human to merge, supersede, or skip. The import itself is a one-click action — but only one click, because the rest of the pipeline already earned the trust.
The guardrails that make this safe to leave running
Generation gets boring fast. Guardrails are where the engineering lives.
An LLM asked for "test cases" tends to return all-positive, or all-negative if you nudge it. The pipeline asks for each axis separately — positive, negative, edge, regression — and refuses to advance Gate 2 if any axis is empty. Coverage is a contract, not a vibe.
Every output carries a confidence score and a source citation. Low-confidence items get a visual badge in the UI; the human is allowed to disagree, but only by clicking through a "yes I read this carefully" affordance. No silent approvals.
Step output is validated against the team's TCM schema before it ever reaches the reviewer. Missing expected results, malformed step IDs, or non-existent test data references fail closed — the LLM is asked to repair, not the human.
Embeddings against the existing TCM catch near-duplicates before import. Reviewer sees the existing case, the proposed case, and a diff; the action is merge / supersede / skip, never blind insert.
Every approve / edit / reject at every gate is captured as an episode. Over time the system knows: "for this product area, reviewer X always rewrites the negative cases this way." That feedback nudges future generations — and is auditable.
Logging is structured, not freeform. Every LLM call, every source fetch, every gate decision is a typed event with cost, latency, model, and source hashes. Reproducibility and cost attribution come for free.
Why the staged design is also the cheap design
Token cost is dominated by two things: how much context you feed in, and how often you re-feed it. The architecture attacks both.
- Context caching at the ingest layer means the same Confluence page contributes once per change, not once per run.
- Summarisation happens at chunk time and is cached against the source hash. Stage 1 reads the summary; only Stage 3 dereferences back to the full chunk when expanding steps.
- Cheap stages early. Stage 2 emits titles + one-liners — tiny outputs. The model is not asked to write 400-line test cases that the human will throw away.
- Knowledge base is reusable across runs. Business-skill content is loaded once per session, not re-derived from raw sources.
The coverage guardrail is the other half of the cost story. Generating one "balanced" prompt that returns thirty cases and re-running it three times because positive/negative/edge are missing is more expensive than three targeted prompts that each return ten. Asking the right question once beats asking the wrong question three times.
What "advanced" looks like once the base pipeline is trusted
- Live KB editing from the gate UI. When a reviewer corrects a requirement, the edit becomes a KB patch in the same click. No "I'll update the wiki later."
- Round-trip blueprint editing. Reviewer can drag, edit, or write new titles; the system re-runs only the affected step generations, not the whole batch.
- One-click TCM import via API to AIO / Xray / Zephyr — but only after Gate 3 and the dedup pass.
- Episodic memory feeds back into the business skill. Patterns the team approves repeatedly become first-class rules; patterns they reject repeatedly become first-class anti-patterns.
- Coverage check is bidirectional. Not just "did we generate positive / negative / edge" — also "does the existing TCM already cover this for the requirement, or is there a real gap?"
The solution-architect bet
The instinct with Gen AI is to wire a single model call to a single input and admire the output. That ships fast, demos well, and rots in production. The bet behind this design is the opposite: treat the LLM as one stage in a system, not the system itself.
Everything that makes the architecture interesting — staged gates, business-skill memory, coverage matrix, episodic feedback, dedup, audit hooks — is what would be in the design doc for any production data pipeline. The LLM is the cheap, replaceable component in the middle. The expensive parts are the contracts around it.
That is the framing I would bring to building this for real. AI accelerates. Humans still decide — and the architecture is what makes "humans still decide" cheap enough to actually do.
This is an architecture proposal, not a shipped system. Pieces of it exist in production work I have done — the qa-agent skill stack handles the guardrails-and-gates layer, the AIO TCC skill handles the format-enforcement layer, and the multimodal triage pipeline is a worked example of staged LLM gates over multi-source input. The piece I would build next is the requirement-mapping gate and the coverage-matrix contract, because that is where the trust earns the rest of the pipeline.
The post
Shared on LinkedIn — read the reactions and the discussion thread.