← back to work
case study · /multimodal-triage

Multimodal ticket triage.

A Gemini 3 pipeline that takes a Jira epic (or a single ticket) with a video attachment + a one-line summary, extracts repro steps, validates them via Playwright MCP, and adds a fully-grounded RCA comment back on the ticket. The whole point is to assist — the agent never fixes prod, it only writes down what it sees.

shipped gemini 3 multimodal playwright MCP JAM MCP sentry · victoria logs · github JIRA API
↗ medium ↗ linkedin
// 01 · the input

Customers send video and one line.

The customer support flow looks like this:

  1. A customer hits something weird.
  2. They screen-record. They write a one-line summary in Jira.
  3. The CS team gets the ticket. They have to: watch the video, transcribe the steps, file the proper bug.
  4. That's where wrong information gets in — "user tapped X" when they actually tapped Y, missed a step, summarized too aggressively.

So we built the agent to do steps 3 and 4. It watches the video, extracts the steps, replays them, and writes them back as a Jira comment. CS still drives. The agent just stops typos.

// 02 · architecture

The pipeline.

Starts at a Jira epic (or a single ticket — the pipeline adapts). Ends with a grounded RCA comment back on the ticket.

01
Jira API · fetch input epic or single ticket
If the input is an epic, expand it into the list of child tickets. If it's a single ticket, skip this expansion layer entirely — adapt to available context.
↓
02
Per-ticket fetch attachment URLs + description
Pull all attachments and the description. Branch by attachment type: video, .har, or text-only.
↓
03
Gemini 3 multimodal video → steps
For video tickets: pass the attachment + the description to Gemini 3 with a prompt asking for canonical Jira steps. Output is a structured list (one action per step).
↓
04
Playwright MCP execute & validate steps
Run the extracted steps against the staging / pre-prod environment via Playwright MCP. Capture screenshots at each step and a final reproduce flag (reproduces / cannot reproduce / blocked).
↓
05
Grounded RCA sentry · victoria logs · github MCP
Cross-reference the reproduced behavior with Sentry events, Victoria Logs, and GitHub MCP (recent commits to the affected component). Look for a single, well-bounded code change that explains the regression.
↓
06
Write back to Jira comment only — never autonomous actions
Post a structured comment with: extracted steps, repro result, suspected commit / Sentry event, and confidence level. The human reviews and either uses it as-is or rejects.

Hard constraint: the agent never writes to prod. If the issue affects production, it stops at "add the comment" and explicitly flags the human-needed bit.
// 03 · adaptability

How it handles the input variations.

The point isn't to fill the process gap. It's to assist the existing process to adapt situationally.

branch A · standard path
Input: Epic with linked tickets, each ticket has a video
Full pipeline. Fan out to each linked ticket, run video → steps → Playwright → RCA → comment for each.
branch B · solo ticket
Input: Single Jira ticket sent directly (not via epic)
Skips the epic-expansion layer entirely. Treats the one ticket as the unit of work. adapt to available context.
branch C · HAR file
Input: Ticket has an attached .har
Skips the video → steps extraction. Goes straight to logs (Victoria + Sentry) and GitHub commits via MCP for the RCA. Faster path when network evidence is already on the ticket.
branch D · description-only
Input: Text description, no video, no HAR
Reads the description for repro pointers. Searches Sentry for similar patterns by error signature. Falls back to a "needs more info" comment if nothing matches.
// 04 · constraints

What it doesn't do.

  • Doesn't fix anything. The agent only adds repro steps + RCA evidence. No PRs, no Jira status changes, no prod traffic.
  • Doesn't act on prod issues. If the reproduction confirms a live prod bug, the agent escalates by leaving the comment with severity tagged — a human pages on-call.
  • Doesn't pretend to know. Confidence level is in the comment. "High confidence: this is commit abc123. Low confidence: similar Sentry pattern but no clear cause." The CS lead knows whether to trust.
  • Doesn't replace CS. The CS team still owns customer comms, prioritization, and decision-making. The agent removes the "transcribe a 4-minute video into steps" toil.
// 05 · rollout

How it scaled across teams.

Started as a personal Gemini 3 pipeline I ran from my own AI Studio. Worked. Got copied. Got requested. Eventually wrapped it through JAM MCP so any team's customer support / PM could trigger it from a Jam recording with one-click.

That's the lift — one-click ticket-with-video → one-click triaged-with-repro-steps. Across teams, with the same agent.

// 06 · related

Adjacent.

LLM evaluation rubrics
How I picked Gemini 3 multimodal for this pipeline over Claude / GPT.
read→
Slack bots
The triage-from-Slack flow that built on this pipeline.
read→
Why HITL is non-negotiable
An essay on why I never let this agent take autonomous actions.
read→