Who audits the collector? Turning a compliance evidence agent into shared infrastructure

A few weeks ago I wrote about how a single oversized customer audit — 300-plus evidence items, north of 900 files — pushed us to build an AI agent that collects compliance evidence. The feedback was generous, and some of the most thoughtful of it converged on a sharp question.

The moment an agent starts collecting audit evidence, the collection itself enters audit scope. So the next question is: how do I trust the collector? 

It's the right question. It's also the question that shaped how the tool evolved from a one-off into something we now run across the team.

Another learning was that simply collecting evidence was not enough. To maximise the impact, the explanation of what the evidence is, why it addresses which controls, and how it was collected, needed to be built into the result.

In this post, we explain how we address these things, and more! 

Evidence that explains itself

Historically, handing over evidence was only step one; step two was a run of meetings walking the auditor through it — what this screenshot is, why it satisfies the control, which system it came from, who owns that system, how the process actually runs. The screenshots were proof, but they weren't an explanation, so the explaining happened live, in the room (or google meet/zoom/teams…), item by item.

Now each item ships with its explainer: what was requested, what's provided, how the control is actually implemented, who does what, and how the whole thing ties back to the control objective — the narrative is written once in the playbook and reused on every package. The auditor can read an item end to end without us there to narrate it. We're in the middle of an audit cycle as I write this, and the difference is tangible: the auditor has been very enthusiastic, and the time we spend in evidence-walkthrough meetings has dropped by 60%. The agent made us faster at producing evidence while it made the evidence faster to consume, which is where a lot of the real audit time was hiding all along.

From a script to shared infrastructure

The first version was just a script. The pages to capture were a Python list with specific URLs hardcoded into it — this pull request, that dashboard, scroll to the bottom for the approval. It worked because it was built for one questionnaire on one laptop, and the person who wrote it was sitting right there.

That's fine for a fire drill. It's the wrong shape for infrastructure. A hardcoded script can't be reviewed in any meaningful way, can't be handed to a teammate, can't explain why it captured what it captured, and quietly rots the moment a URL changes.

So we rebuilt it as a shareable skill, living in a GitHub repository (TrustOps-Vanta-Evidence-Skill) in our org. A teammate clones it, runs ./install.sh — which runs preflight checks and symlinks the skill into Claude and Cursor — drops their Vanta API credentials in place, and they're running the same pipeline I am. Nothing about the evidence targets is hardcoded anymore. The agent figures out what to capture from two sources of truth: the GRC platform for what's pending, and a knowledge base for how we actually do it here.

The shape that emerged is two sources of truth feeding one agent, with a person on each end of the consequential decisions:

‍

┌─────────────────────────────┐
│  Vanta (GRC) — the pending list   =  SCOPE     │
│  Knowledge base (Git + Notion)    =  METHOD    │
└─────────────────────────────┘
                            │  agent reads both
                            ▼
        research  →  open browser  →  capture  →  explain
                            │
                            ▼
            uploads to Vanta as DRAFT  (never submits)
                            │
                            ▼
        human reviews & submits  =  accountable signature

Code in GitHub, instructions in Notion

The single most useful decision we made was to split the tool cleanly in two, along the line of who owns each part.

The mechanics live in GitHub: the Vanta API client, the screenshot capture script, the PDF generator, and the pipeline instructions the agent follows. These are code. They change through pull requests, and a CODEOWNERS file routes anything touching the scripts, the knowledge index, or the pipeline to a human reviewer. If you want to know exactly what the collector does, you read the diff.

The instructions for evidence collection — the per-control playbooks — live in Notion. For each category of control there's a page describing how we satisfy it: which systems are involved, what an auditor is really asking for, the narrative framing that fits, the URLs that work and the ones that don't. These are written for humans, owned by humans, and validated by the people who own those controls day to day.

Every playbook follows the same shape, which keeps them easy to read and easy to challenge:

‍

# Playbook — <control category>        (e.g. Change Management)

## What the auditor is really asking
Plain-language restatement of the ask.

## How we actually satisfy it
The real control story: which systems, who owns them, what runs when.

## Where to capture, in order
1. <system> → <page> → what the screenshot must show
2. ...

## URLs that work / don't
- works:  <url that lands on the authenticated view>
- avoid:  <url that dead-ends or shows a login>  — and why

## Narrative for the explainer
The framing that connects this evidence back to the control.

## Run history
- 2026-05-24 — 3 screenshots; AWS tile needed a re-SSO

This split is the first half of the answer to "how do I trust the collector." The collector isn't a black box that happens to produce screenshots. The method it follows is a documented, reviewed, version-controlled artifact — the code in one place, the human-readable playbooks in the other. Both are inspectable before the agent ever opens a browser. The agent is executing a procedure that a person could read, challenge, and sign off on independently of any single run.

A knowledge base that builds itself

Underneath the playbooks sits a small, deliberately boring layer of routing knowledge in the repo: a map of our systems and their URLs, the SSO tile for each service, a running list of known access blockers and their workarounds, and an index that points each control at its Notion playbook. The agent reads these instantly at the start of a run — no API calls — then fetches the full playbook from Notion only for the controls it's actually working on.

The index is just a pointer table. One entry maps a control to the systems it touches and the playbook that explains it:

‍

# knowledge/playbooks/index.yaml — read first, from Git, no API calls
categories:
  - id: change-management
    title: "Change Management & Edit Checks"
    controls:
      - Evidence of Edit Checks
      - List of System Changes
    systems: [github, semgrep, aws-cloudtrail]
    notion_page: 2f1c…       # → full playbook lives in Notion
    summary: "GitHub rulesets + PR workflow + SAST"

The part I like most is that this base grows on its own. After every run, the agent appends what it learned. A dashboard moved? The corrected URL goes in. A service stopped honouring the direct link and needs the SSO tile? That's recorded. A new platform quirk cost us twenty minutes to figure out? It's written down so it costs the next person zero. The rule is append-only — we never delete the history of what worked and what didn't, because that history is the context.

Two recent entries give the flavour: a Chrome update broke the screenshot library's connection method, so the workaround for driving the browser a different way is captured with the date it was found; and Google's download protections meant a particular file-fetch trick failed, so the working alternative is documented next to it. A learned blocker is just another small, dated record — the kind the agent appends after a run and a teammate can review in a pull request:

‍

# knowledge/access-issues.yaml  — append-only; checked before each capture
- system: GitHub
  issue: "Branch-protection settings page 404s for non-admins."
  workaround: "Capture the org Rulesets page (/rules) — visible to all members."
  discovered: 2026-05-24

- system: CDP Chrome / Playwright
  issue: > With Chrome 149.x, Playwright connect_over_cdp() fails immediately with "Protocol error (Browser.setDownloadBehavior): Browser context management is not supported." This breaks screenshot_capture.py capture/ensure-sso/check.
  workaround: > Bypass Playwright and drive the running CDP Chrome directly over the raw DevTools websocket (websocket-client lib): Target.createTarget + Target.attachToTarget(flatten) → Page.navigate / Page.printToPDF / Page.captureScreenshot at the page session level. See ~/.vanta/cdp_capture.py. Page-level CDP avoids the browser-level setDownloadBehavior call that fails.
  discovered: 2026-06-04

Dynamic, run-to-run learnings go to the Notion playbooks. Structural changes go to the repo as a commit — which means they show up as a reviewable change, not a silent mutation. The knowledge base gets smarter every week, and every step of getting smarter leaves a trail.

About that logged-in browser

Now the part that, understandably, makes people uneasy: the agent drives a browser that's logged in to real systems. Here is how it's actually bounded, because the framing matters as much as the controls.

The agent never touches your everyday browser. The capture step copies the relevant session files into a separate, throwaway profile and launches a distinct Chrome instance for automation, with a visible window. It carries no credentials of its own — no standing service account with god-mode across our SaaS estate. It borrows the live session of the human running it, and nothing more.

That last point is an important part of the security model. The agent's reach is exactly the operator's reach at that moment, and we keep the operator's reach deliberately small. Where a read-only role exists, that's what we use — our cloud console sessions, for instance, run on read-only access. The concern that not every SaaS tool implements a read-only admin role is real, and our answer is to let that constraint stop the agent rather than route around it. When a page genuinely needs elevated rights we don't have — an identity-provider policy screen that requires Super Admin, say — the agent doesn't find a clever workaround. It stops. The item is recorded as blocked and handed to the team that legitimately holds that access, and where we can, we substitute monitoring evidence from the SIEM that shows the same control is live. The agent escalating its own privileges is simply not a path that exists in the design.

And when it hits a login wall or an MFA prompt, it stops and asks. It detects the auth screen, pauses, names the system that needs a human, and waits in the visible window for a person to authenticate. A person is always in the loop at exactly the points where authentication and authorization are decided. The browser is logged in, but it's logged in as a specific, present human, on least-privilege roles, in a sandboxed instance that fails closed — not as an autonomous identity roaming the estate.

Authorized at the moment of execution

The agent doesn't decide what's in scope. The GRC platform does. Each run starts from the list of items the platform itself reports as pending, filtered to a framework or a single control, and the human chooses at triage whether to process all of them or a subset. The agent can't invent a target that isn't already an open ask in the system of record.

What it's authorized to see is settled the same way — by the live, least-privilege session it borrowed. If the operator isn't authorized for a page, that page is an auth wall, and the agent pauses rather than fabricating around it. Authorization at the moment of execution isn't asserted in a policy document; it's enforced by the fact that the agent has no access the named operator doesn't currently have. And the one action with real downstream consequences — submitting evidence to the auditor — is one the agent never takes. More on that below.

Provenance by construction

If you hand an auditor a pile of files, the first thing they'll want is to trace any single one back to where it came from. We built the tool so that trail is a property of how evidence is created, not something reconstructed afterward.

Every capture is born with a record attached. Screenshots land in a dated, control-scoped folder, and as each one is taken the tool writes an entry to a manifest that sits beside it: the source URL it navigated to, the capture time in UTC, the operator who ran it, and a SHA-256 checksum of the exact bytes. The explainer document that accompanies each package restates the auditor's exact ask, embeds the screenshots that answer it, explains in plain language how they satisfy the control, and prints that same per-item table — source, timestamp, checksum — on its own page. The PDF is then hashed into the manifest as well, so the whole package can be fingerprinted, not just its parts.

Concretely, that record is a small file written next to the evidence — one entry per item, including the explainer's own checksum:

‍

{
  "test_id": "CC8.1",
  "items": [
    {
      "file": "01_github_pr.png",
      "kind": "screenshot",
      "source_url": "https://github.com/acme/app/pull/1062",
      "captured_at": "2026-05-24T09:14:02Z",
      "operator": "n.barberis@acme.com",
      "sha256": "43be52f9…af06de49"
    },
    {
      "file": "evidence_CC8.1.pdf",
      "kind": "explainer_pdf",
      "captured_at": "2026-05-24T09:14:05Z",
      "operator": "n.barberis@acme.com",
      "sha256": "0d352c5c…17f0baf6"
    }
  ]
}

That makes any single file answerable on demand. Read its entry to see where it came from, when, and who pulled it; re-hash the file and compare it to the recorded checksum to prove it hasn't changed since capture. Reproducibility comes from the playbooks — because each control's strategy and working URLs are written down and kept current, a pull can be repeated and land in the same place — and the run closes with a summary mapping each control to its evidence and its destination. None of this is a separate audit-logging system bolted on afterward: the manifest, the checksums, the explainer, and the folder structure are all produced as a side effect of doing the work, and they travel with the evidence.

Built-in Accountability

Here is the line we will not cross: the agent uploads, but it does not submit. Every piece of evidence lands in draft. A person opens the platform, reviews the screenshots, reads the explainer, and decides what goes to the auditor — and the submission is recorded against that named person.

That's not a limitation we're apologising for. It's the point. The agent scales the repetitive work; a human carries the accountability, and the human's review is the opposable artifact at the end of the chain. An auditor doesn't have to trust the agent on faith, because the agent's output isn't the final word — a named reviewer's signature is, sitting on top of a method that was itself written down, reviewed, and version-controlled before the run began. Speed comes from the automation. Defensibility comes from the human at the end and the paper trail underneath.

Sample generated PDF

Created with mock data, but using the tool:

When a new framework lands

The real test of whether this is infrastructure or just a clever one-off is what happens when a brand-new framework shows up. One did — a big one, on the order of five hundred controls. The usual math for something that size is four to six months of scoping, collecting, and chasing.

This time we pointed the agent at it instead of a person. It pulled every control, test, and document for the new framework out of Vanta and cross-referenced them against what our existing certifications already prove — so the work became the genuine gap, not five hundred items from a standing start. Most of it was already covered. Then it turned that remainder into a plan the team could actually run: a project in Linear, with milestones and issues grouped by control area.

‍

Linear project: New framework — certification
├── Access control
│├──[]MFA enforced for in-scope systems ←run /get-vanta-evidence
│ └── [] Quarterly access reviews
├── Change management
│└── [] PR review + required checks   ← run /get-vanta-evidence
└── Logging & monitoring
 └── [] Audit log retention & coverage

From there the skill lives where the team already plans its work. You can point it at a Linear issue, or trigger it from Linear directly; it runs that control's tasks, drafts the evidence in Vanta, and reports back on the issue. None of the guarantees change — each item still carries its provenance, evidence still lands as a draft, and a named owner on each issue still reviews and submits. What changes is that the plan, the owners, and the status now live in the tool the team uses every day, which is its own form of accountability: who owns what, what's done, what's left, all tracked. A framework we'd have penciled in at four to six months came together in a few weeks — without loosening any of the controls that make the evidence defensible in the first place.

What we'd tell another team

The approach is replicable, and the lessons that travel aren't really about screenshots.

Separate the how-we-collect from the machinery, and put the how-we-collect somewhere humans naturally own and review. Give the agent exactly the access the operator has and no more, and let missing read-only roles stop it rather than tempt a workaround. Make provenance a property of how artifacts are created, not a report you generate later. Keep the consequential action — the submission, the thing with a signature on it — firmly in human hands. And make the agent write the explanation, not just capture the proof: an auditor who can read an item end to end is one who doesn't need a meeting to clear it.

A tool that's fast is easy to build and easy to distrust. A tool that's fast and can explain, on demand, where every artifact came from and who stands behind it — that's the one you can actually hand to your team, and to your auditor. That's the version we run now.

Where we go from here

We are just getting started. What began as a tool for our own audit burden is quickly evolving into something broader—an AI-assisted GRC platform that treats compliance not as a static, annual checklist, but as living, executable infrastructure. We are already mapping out how this architecture scales to meet the complexity of our entire stack. Keep an eye out for our next post, where we will share the design principles behind the platform and potentially open-source the core components so you can build and extend your own collector.