Agentic Visual Testing Workflows in GitHub

Visual regression testing is one of those things that looks solved until a check fails. Detecting that two screenshots differ is relatively easy. Deciding whether the difference is a genuine regression, an intentional design change, a viewport problem, or harmless data variance is the expensive part.

That is why products such as Percy and Applitools are attractive. They provide much more than a pixel comparison: baseline management, review workflows, history, collaboration, and a polished way to decide what happens next. I wanted to investigate whether there was a useful GitHub-native alternative for teams that already have Playwright visual checks running in GitHub Actions.

The experiment was not “ask an LLM to approve screenshots”. It was “use Playwright to produce deterministic evidence, then ask a constrained vision-capable agent to triage the failed evidence with product context”.

The Question I Wanted to Answer

Could I keep Playwright's native visual comparisons as the objective test, then use an agentic workflow to make the failure more useful to a human reviewer? In particular, could the workflow compare the expected, actual, and diff images while also considering the information that already exists in the repository, Figma, and Jira?

The answer from my investigation is a qualified yes. The useful architecture is a two-stage process. Playwright remains responsible for detecting a mismatch. The agent is responsible for explaining and classifying that mismatch, using read-only evidence from the rest of the delivery process.

• Playwright runs the visual checks and produces native expected, actual, and diff screenshots.
• GitHub Actions collects the failed visual evidence and the metadata around each checkpoint.
• The analysis script groups the images into bounded multimodal requests.
• Read-only context from repository documentation, release history, Figma, Jira, and sanitized diagnostics helps the agent understand intent.
• A vision-capable Copilot model classifies each difference and recommends the next action.
• A downloadable HTML report keeps the images, findings, confidence, and assumptions together for review.

Why Build This Around GitHub?

This is not an attempt to claim that a few scripts reproduce every feature of a specialist visual testing platform. Those products solve a broad operational problem and can be the right choice, especially when you need large-scale baseline management and a mature review experience. The attraction of this approach is different: the evidence stays close to the code, the workflow runs where the tests already run, and the analysis can be shaped around the terminology and decision-making process of a particular team.

It also gives you a way to start small. The first version only needs to analyse failed visual checks and publish a report as a GitHub Actions artifact. There is no requirement to move every baseline or every test into a new platform before you can learn whether the workflow is useful.

The GitHub Actions Workflow

The workflow is deliberately conventional at the start. It runs a normal Playwright project, and only invokes the visual analysis after the visual test step has failed. A second step always attempts to create the downloadable report so that a failure still leaves behind something reviewable.

env:
  COPILOT_GITHUB_TOKEN: ${{ secrets.COPILOT_GITHUB_TOKEN }}
  COPILOT_MODEL: gpt-5.6-luna
  COPILOT_VISUAL_ANALYSIS_CONCURRENCY: 3
  PLAYWRIGHT_CAPTURE_HAR: 'true'

steps:
- name: Run Playwright tiles tests
  id: run-tiles-tests
  run: npx playwright test tests/tiles-tests --workers=6

- name: Analyze failed visual checks with Copilot
  if: ${{ failure() && steps.run-tiles-tests.conclusion == 'failure' }}
  run: node -r ts-node/register/transpile-only \
  ./scripts/analyze-tiles-visual-diffs.ts

- name: Create downloadable HTML visual report with Copilot
  if: ${{ always() && steps.run-tiles-tests.conclusion == 'failure' }}
  run: node -r ts-node/register/transpile-only \
  ./scripts/create-tiles-visual-report.ts

The concurrency setting is there to keep several bounded analyses moving without turning the model call into an uncontrolled fan-out. The HAR flag is useful because network behaviour can help explain a screenshot, although a network error should remain supporting evidence rather than proof that it caused the visual difference.

Collecting Playwright's Visual Evidence

Playwright already gives the workflow the most important evidence. For each failed checkpoint, the script locates the native screenshot triplet and adds the page and checkpoint metadata that the model will need later.

const diffs = entries
  .filter((entry) => entry.endsWith('-actual.png'))
  .map((entry) => {
    const actualPath = path.resolve(directory, entry);

    return {
      actualPath,
      expectedPath: actualPath.replace(
        /-actual\.png$/,
        '-expected.png',
      ),
      diffPath: actualPath.replace(
        /-actual\.png$/,
        '-diff.png',
      ),
      checkpoint: checkpointFromArtifactName(path.basename(actualPath)),
      page: pageFromCheckpoint(path.basename(actualPath)),
    };
  });

Each failure is represented by an expected, actual, and diff image triplet. That matters because the red pixels in a diff image are not enough on their own. The expected image shows the contract, the actual image shows what the user experienced, and the diff image focuses attention on the changed area.

The model receives the evidence that a reviewer would need, rather than a cropped diff with the surrounding context removed.

Bounding the Image Batches

A large visual suite can produce a surprisingly large prompt. Sending every screenshot from every failure in one request is expensive, hard to reason about, and more likely to hit attachment limits. The analysis script therefore bounds each batch by both the number of diffs and the total attachment size.

const exceedsBatchLimit =
  currentBatch.diffs.length > 0 &&
  (
    currentBatch.diffs.length >= maxDiffsPerAnalysisBatch ||
    currentBatch.attachmentBytes + attachmentBytes >
      maxAttachmentBytesPerAnalysisBatch
  );

if (exceedsBatchLimit) {
  batches.push(currentBatch);
  currentBatch = {
    diffs: [],
    attachments: [],
    attachmentBytes: 0,
  };
}

The workflow limits each request to avoid oversized multimodal prompts and retries individual diffs when necessary. This makes the failure mode more manageable: one awkward image or one unusable response does not invalidate the analysis for every other failure in the run.

Adding Product Context with MCP

A screenshot can tell you what changed, but it cannot tell you whether the change was intended. That is where the Model Context Protocol part of the experiment becomes interesting. I used MCP servers as a route to read-only context from systems that already contain the reasoning behind a UI change.

Figma data can provide design intent: the relevant screen or component, expected hierarchy, spacing decisions, and the current visual direction. Jira data can provide delivery intent: the ticket description, acceptance criteria, sprint changes, status, and links to the work that prompted the change. The repository can add recent commits, test metadata, release notes, and known assumptions.

I treated all of that as evidence, not authority. A Jira ticket being marked complete does not make a broken screenshot acceptable, and a Figma frame does not automatically explain a viewport-specific layout failure. The context helps the agent ask better questions and express a more useful confidence level.

In practice, the MCP results are narrowed into a `knowledgeContext` before they are included in the analysis prompt. That keeps the model focused on the relevant release and journey, avoids passing an entire project history into every request, and makes the report easier to audit. Sensitive tokens and unrelated account data should never be copied into the prompt or the generated artifact.

An Evidence-Aware Analysis Prompt

The prompt is intentionally opinionated. It tells the model what it is allowed to do, what it must compare, which classifications are valid, and what a useful finding needs to contain. It also explicitly stops the model from treating network diagnostics as a shortcut to a conclusion.

return `You are a visual regression triage analyst.

Analyze the attached native Playwright screenshot triplets.
Do not modify files, source code, tests, baselines, or workflows.

Compare expected, actual, and diff images for every failure.
Classify each change as exactly one of:

- IGNORE_VALUE_VARIANCE
- EXPECTED_UI_CHANGE
- INVESTIGATE
- REGRESSION

Use the supplied release history, repository documentation,
test metadata, and sanitized HAR diagnostics. Treat network
failures as supporting evidence, not proof of a visual cause.

For every failure, report:
- the affected page and browser/device
- what changed visually
- missing and newly appeared elements
- confidence and likely cause
- recommended next action

Repository documentation:
${knowledgeContext}

Visual diff metadata:
${diffDetails}`;

The prompt also embeds product assumptions, sprint UI changes, recent commits, test failures, and journey context. This is the difference between asking for a generic image caption and asking for a visual regression triage finding.

Calling a Vision-Capable Copilot Model

The model selection is capability-based rather than hard-coded to one available model. The session is created with a permission hook that denies tool execution, plus a system message that makes the report-only boundary explicit.

const client = new CopilotClient({
  githubToken: process.env.COPILOT_GITHUB_TOKEN!,
  useLoggedInUser: false,
  logLevel: 'error',
});

await client.start();

const availableModels = await client.listModels();
const candidateModels = availableModels.filter(
  (model) =>
    model.policy?.state !== 'disabled' &&
    model.capabilities?.supports?.vision === true,
);

const session = await client.createSession({
  model: candidateModels[0].id,
  onPermissionRequest: () => ({
    kind: 'denied-by-permission-request-hook',
    message: 'This CI session is report-only and cannot execute tools.',
  }),
  systemMessage: {
    mode: 'append',
    content:
      'Never edit files or execute destructive commands. ' +
      'Base conclusions on attached evidence.',
  },
});

const response = await session.sendAndWait(
  {
    prompt,
    attachments: imagePaths.map((filePath) => ({
      type: 'file',
      path: filePath,
    })),
  },
  180000,
);

This is an important distinction. The session can inspect and reason about the evidence, but it cannot edit source code, update baselines, execute arbitrary tools, or quietly turn a failed check green. The output is a recommendation for a human or a later, explicitly authorised workflow step.

Turning the Response into a Report

Free-form prose is useful for exploration but awkward to render consistently. I validate the response against a small structured finding model so that every diff has a corresponding result and the HTML report can render the same fields for every failure.

interface VisualReportFinding {
  id: string;
  test: string;
  project: string;
  page: string;
  checkpoint: string;
  classification: string;
  confidence: string;
  summary: string;
  why: string;
  missingElements: string[];
  newElements: string[];
  bugs: string[];
  recommendation: string;
}

The HTML report then embeds the original evidence alongside the analysis. That means a reviewer can move from the summary to a specific failure and see the expected, actual, and diff images without having to reconstruct the paths from a CI artifact directory.

const images = {
  expected: await imageData(diff.expectedPath),
  actual: await imageData(diff.actualPath),
  diff: await imageData(diff.diffPath),
};

await writeFile(
  outputPath,
  renderHtml(model, diffs, images, visualAnalysis),
  'utf8',
);

If the LLM is unavailable or returns unusable output, the workflow preserves the failure and generates a deterministic fallback report rather than silently approving the visual change. That fallback is not as insightful, but it is honest: the visual check failed, and the analysis was not available.

What the Findings Look Like

The per-failure view is where the workflow starts to feel practical. A finding includes the page, project, checkpoint, classification, confidence, summary, likely cause, missing and new elements, potential bugs, and a recommended action. It gives a reviewer a route forward instead of making them decode a diff image in isolation.

In the example shown above, the report called out that an iPhone 13 screenshot was 390 by 3430 pixels instead of the 390 by 3402 expected image. It found deterministic displacement in the lower card and controls, did not identify missing or newly appeared elements, and recommended inspecting responsive spacing and typography before deciding whether to update the baseline. That is a much more useful outcome than “visual test failed”.

Where the Agent Helps and Where It Does Not

The agent is good at joining up several kinds of evidence. It can notice that a change is isolated to a mobile project, connect a new spacing rule to a recent UI ticket, distinguish a missing cell from a simple value change, and describe why a baseline should be investigated rather than immediately accepted. It is particularly useful when a run contains many failures that are related but not identical.

It is not a replacement for stable visual assertions or a substitute for a product owner deciding what the interface should do. A model can misunderstand a screenshot, trust stale Jira data, or overfit to the context it is given. The classification is therefore a triage aid. The evidence, the test failure, and the final baseline decision remain visible and reviewable.

• Keep Playwright's visual assertion as the deterministic gate.
• Use read-only MCP context and pass only the relevant Figma, Jira, repository, and release information.
• Make the model report-only and deny tool execution in the CI session.
• Require one structured finding for every diff.
• Never auto-approve a baseline because the model was unavailable or uncertain.
• Keep the original images and the generated report as downloadable artifacts.

The Trade-offs

The main trade-off is that you are assembling a small product yourself. You need to decide how to retain baselines, how long to keep reports, how to expose artifacts to reviewers, how to handle credentials for MCP access, and how to control model cost and latency. You also need to consider whether screenshots and design context can safely be sent to the model you select.

A specialist visual testing service may still be the better investment when you need a mature cross-browser review workflow, broad collaboration, visual history, and support at organisational scale. The GitHub approach becomes compelling when you value control, already have the evidence in Playwright, and want the analysis to understand your own release process rather than a generic visual diff alone.

Agentic analysis should make a failed visual check more explainable, not make a failing check disappear.

Final Thoughts

This investigation changed how I think about the role of an LLM in visual testing. The interesting part is not asking it to perform the pixel comparison. Playwright is already very good at that. The interesting part is giving it the evidence and the surrounding delivery context needed to explain why the difference might matter.

With a bounded workflow, a vision-capable model, read-only Figma and Jira context through MCP, and a report that falls back safely when the model is unavailable, GitHub Actions can become a credible place to build this kind of visual triage. It is an alternative architecture rather than a universal replacement for Percy or Applitools, but it is a viable option worth investigating when your team already lives in GitHub and Playwright.