Visual Diff Analyzer.
A semantic visual-regression tool layered over Playwright screenshots. Built to catch the silent CSS regressions that a CMS-driven site ships every other release, without burning the team on false positives. Runs against pre-prod and prod only — the only environments where the CMS publishes copy.
What it looks like.
Side-by-side diff with regions flagged at the semantic component level — not by raw pixel-by-pixel noise.
// replace this with your real screenshots once recorded · the toolbar & flagging UI are real
CMS regressions ship faster than humans can spot them.
The product is a CMS-driven e-commerce site for an MVNE. Marketing publishes copy, pricing tiles, and component variants directly. Engineering ships JS and CSS updates on a separate cadence. The combinations multiply.
What kept biting us:
- Color tokens getting bumped on the wrong CSS variable, breaking contrast on one page.
- Pricing-tile copy overflowing the container at certain breakpoints.
- Hero CTAs losing margin after a Tailwind config refactor.
- Loading states with the wrong skeleton color after a theme update.
None of these break functional tests. Playwright happily clicks through the broken page. The client only finds out when a customer screenshots a bug into Slack.
Off-the-shelf visual diff tools (Percy, Chromatic) gave us either too much noise (anti-aliasing pixel differences flagged as regressions) or required a level of CI buy-in we didn't have.
Diff at the component level. Wait for the page to stabilize.
Two insights, in order:
1. Pixel diff is the wrong unit. What humans care about is "did the Plans tile change," not "did 47 pixels at (302, 218) shift to (302, 219)." So the analyzer groups raw pixel deltas back up to semantic components by reading data-test-id boundaries from the rendered DOM at screenshot time, and clusters spatially-adjacent deltas into one flagged region.
One regression. One review. Not 47.
2. Take the screenshot when the page is actually ready. Most "flaky visual tests" aren't bug detection — they're false positives from screenshots taken mid-render. The analyzer waits for:
- Skeleton loaders to be gone. Watch for any element with
[data-skeleton]or[aria-busy="true"], then wait until none remain in the DOM for two consecutive RAF frames. - Spinners and indeterminate loaders to settle. Same pattern with role-based selectors.
- Network idle, but bounded. Bounded so that an analytics ping doesn't keep us in waiting purgatory.
- A 2-frame layout-stability check. The page must not shift more than 1px between two consecutive RAFs.
Only then does the analyzer capture. This single piece (skeleton-stabilization) cut our false-positive rate by roughly half all on its own.
→ Full case study on the skeleton-loader stabilization pattern
Run a real diff on your own images.
Drop a baseline and a candidate. The tool computes a pixel-level diff in your browser — no server, no upload. The semantic-grouping layer isn't here (it needs the rendered DOM at capture time) but the pixel engine is real.
// images must be the same dimensions · runs in <100ms for ~1024×768 · nothing leaves your browser
How it's wired.
data-test-id, its bounding rect, and component path. This is the semantic backbone for grouping.
→ case-skeleton-loader for the full writeup.
{component, rect, deltaPct, severity}. 47 raw pixel deltas → 1 flagged region.
-
Capture · Playwright
Stable screenshot capture with deterministic font/animation pinning, fixed viewport, and a 2-frame settle delay. Pre-baked CSS that disables all animations.
-
Boundary extract · DOM snapshot
Before screenshot, walk the rendered DOM and emit a JSON map of every element with a
data-test-id, its bounding rect, and its component path. This is the semantic backbone for grouping. -
Pixel diff · pixelmatch
Standard pixel-diff but with anti-aliasing tolerance turned up. Produces a raw delta mask.
-
Group · semantic clusterer
Walk delta mask, find connected components, intersect them with the DOM boundary map. Output is a list of
{component, rect, deltaPct, severity}. -
Review · web UI
Side-by-side / onion-skin / flicker viewer. Approve baseline directly from the UI. Approvals are committed by the user's GitHub identity, not the CI bot.
-
Gate · CI
Any region above severity threshold blocks the merge. Below threshold, results post as a comment but don't block.
How a spec calls it.
// tests/visual/plans.spec.ts import { test, expect } from '@playwright/test'; import { visualCheck } from '../helpers/visual-diff'; test('plans page · semantic visual', async ({ page }) => { await page.goto('/plans'); await page.waitForSelector('[data-test-id="pricing-grid"]'); const result = await visualCheck(page, { name: 'plans', boundaries: 'data-test-id', // semantic anchor threshold: 0.018, // 1.8% delta ignoreComponents: ['live-counter'], // known dynamic }); expect(result.flagged, result.summary).toBe(false); });
> visualCheck() is the only import a spec needs · everything else (baseline storage, semantic grouping, CI gating) is in the helper.
What changed after this shipped.
Lessons.
- Start with the reviewer UI, not the diff engine. The thing that determines adoption is "how fast can a dev approve or reject this." I wasted 2 weeks tuning pixelmatch before realizing nobody could see the output.
- Component boundaries beat anything fancier. I tried bounding-box clustering, then connected-components in pixel space.
data-test-idrectangles won. Cheaper, faster, more useful. - Make baselines explicit. Auto-approving the first run "as baseline" sounds friendly but means you never have a real baseline. Force a human to click approve.
- CMS components need their own threshold. Marketing changes copy intentionally; the analyzer should know which components are "expected to vary" and warn-without-blocking on them.