LLM evaluation rubrics.
How I picked the models behind the QA agent stack. Browser-action accuracy, tool-call correctness, minimum tool calls, cost per task, and token usage — logged via runtime hooks. Eight model families compared.
Four metrics, weighted by use case.
Browser-action accuracy
Given a UI task, does the agent click the right thing? Scored on a held-out suite of 50 tasks per agent surface.
Tool-call correctness
When the agent has access to typed MCP tools, does it pick the right one with the right args? Catches "hallucinated shell commands" before they run.
Minimum tool calls
Penalize agents that explore by trial-and-error. The right model gets to the answer in fewer calls, which scales cost.
Cost per task
Real dollars on the eval suite. Includes input + output tokens, all retries, and any vision-token costs.
What we tested.
US-based and US-accessible LLMs. The list shifted over the year — this is the latest cut.
Hooks for the truth.
The published model cards lie a little. Or rather, they tell the truth about idealized cost — not your cost. Your cost includes retries, tool-call loops the agent runs when it's unsure, vision tokens when the model auto-decides to look at a screenshot.
So I wired runtime hooks at the agent runtime layer:
- Every tool call logged with model, input-token count, output-token count.
- Every retry counted separately.
- Every "decided to look at an attached image" charged at vision token rate.
- Per-task aggregated cost surfaced in the evaluation dashboard.
That's how you find out the agent that scored best on accuracy was actually costing 3x the runner-up because it kept loop-retrying on edge cases. Made the model picks easier.