Running Cloudflare's security-audit skill on a production web app

AI agents are good at reading code. The hard part of using them for security review is that they are also good at sounding confident. Ask a model to "find vulnerabilities" and you get a list, every time, whether or not the bugs are real.

Cloudflare published a skill for AI coding agents that tries to fix exactly that: cloudflare/security-audit-skill (MIT). We vendored it unmodified into a work repo today and ran it as a full audit against one of our production web apps: a Next.js frontend that sits in front of several backend services and is deployed through GitLab CI to Kubernetes.

I'm keeping the app and the company out of this post. The leads are still being checked, and the point here is the method.

Two modes

The skill has two modes, and it is strict about which one it is in.

• Guidance mode (the default): answer a security question, review one file, triage one finding. Loading the skill does not start an audit or write files.
• Full audit mode: only when you explicitly ask for an audit, pen test or full review. Then it runs all six phases and writes a report.

If the request is ambiguous, it asks one question before starting. We asked for a full audit with the deep profile (the other options are quick and standard).

The six phases

1. Reconnaissance. Map the source, the trust boundaries, the build paths, and build a coverage ledger: a deterministic list of units (surface × boundary × attack class) that must each be covered.
2. Hunting waves. Isolated hunter agents each get units from the ledger, plus the attack-class guides that apply. After each wave a coverage critic looks for what was missed and adds work.
3. Candidate validation. Every candidate goes to a fresh verifier agent that did not find it.
4. Structured output. Everything goes into findings.json, checked against a JSON schema and two validator scripts (one for findings, one for the coverage claim).
5. Independent record verification. Another set of fresh agents re-checks every final record against the source.
6. Report. REPORT.md, FINDINGS-DETAIL.md and NEEDS-VALIDATION.md, derived from the JSON, so prose and data can't disagree.

The run is not allowed to stop in the middle. It either finishes with both validators passing, or it records itself as incomplete with the exact reason.

What it looks at

The core is a set of companion guides, picked per target. For our app it selected:

Guide	What it hunts
Web protocol and auth	authentication, sessions, authorization per route, CSRF/CORS, rate limits, proxy headers
Client-side	HTML injection sinks, unsafe URLs in window.open and links, CSP, framing, data sent to the browser
Cloud and deployment	Helm charts, secrets delivery, probes, service accounts, ingress
Supply chain and release	Dockerfile, lockfiles, CI includes, who can deploy what, unpinned tools in pipelines
Resource exhaustion	unbounded bodies, caches, expensive endpoints without limits
Data isolation and lifecycle	what data a holder of an ID can reach, PII in URLs and logs
AI and LLM	prompt injection and tool abuse (used for one CI unit only)

There are more guides it did not need here: memory safety, desktop/mobile IPC, RPC and messaging protocols.

The ledger ended with 44 units: 19 covered with nothing found, 22 with candidates, 1 blocked, and 2 out of scope because they were defined in another repo.

Safety rules

This is what made me comfortable running it on a real app:

• Source first, read-only. Code from the target only runs inside a sandbox with no network, a read-only copy of the source, writes only to a scratch folder, and limits on CPU, memory, processes, file size and time.
• Never touch anything live. No deployed endpoints, no CI system, no cloud account, no Vault, no cluster. If the answer depends on something outside the source, it says so instead of probing.
• Evidence is copied out carefully. Only allow-listed files, no symlinks, no hard links, size-checked twice.
• Dummy data only.

In our run node_modules was empty, so the app was never built. The agents tested logic by extracting single functions into small plain-Node harnesses, for example to prove how a rate-limit key is derived.

The evidence bar

This is the part I like most. A finding must name:

• the lower-trust actor,
• what input or action they control,
• which control is supposed to stop it,
• which boundary is crossed,
• who or what is affected, and the concrete result.

No boundary and result, no finding. Missing best practices are not vulnerabilities. Guessing how a proxy or cloud provider is configured is not allowed.

Anything that is a real, source-grounded lead but depends on a fact the agent cannot see becomes needs_validation. It has no severity, and it comes with the exact deciding fact, a bounded local next step, and a safe check the owner can do.

Severity is only given to confirmed findings, and it is anchored: critical means unauthenticated code execution or full data access; high means a security control is fully defeated with real consequences; and so on down.

Our result

About 110 agent runs: 4 reconnaissance, 20 hunters over 3 waves, 6 coverage critics, 23 candidate verifiers, 22 final-record verifiers and 3 re-verifiers. The final coverage critic came back clean, so it counts as a complete deep pass for that scope.

• 0 confirmed findings
• 22 prioritised leads (needs_validation), grouped as: sensitive request paths, CI/CD and cluster identity, brute-force and abuse controls, browser-side injection and headers, supply chain, availability
• About 50 hardening notes, plus a list of things the code already does well

Zero confirmed sounds like nothing happened. It is the opposite. The report says plainly that "no LOW findings were invented to fill this section". Every lead depends on something the repo can't show: how a backend service checks what the frontend forwards, how a load balancer handles a header, how CI variables are protected. That is an honest answer, and it gives us an exact checklist instead of a pile of maybes.

What we learned (in general terms)

None of these are specific to our app. They are patterns the skill is good at spotting, and they are worth checking in any similar setup:

• A thin proxy inherits all of its trust from the backend. When a frontend calls every backend with one shared service token, every business rule the browser enforces has to be enforced again server-side. If not, the browser check is just a suggestion.
• Rate limits are only as good as their key. A limit keyed on something the client can set is not a limit.
• Security headers that depend on runtime config can fail open. If a pod starts happily without its secrets, the safe defaults have to be in the code, not in the config.
• Your CI is part of your attack surface. Who can run a pipeline, which credentials those jobs hold, and what a merge-request pipeline can deploy matter as much as the app code. Shared CI templates included by a branch name change without review.
• Pin what you pull in at build time. Global installs, plugins installed from a default branch and scanner images at :latest are all code you run without reviewing.

Would I use it again?

Yes. It is slow and it uses a lot of tokens: a deep run is hours of agent time. But it behaves like a careful reviewer, not a scanner. It shows its coverage, it separates "proven" from "plausible", and every claim has a file and a line.

Next step for us is the boring part: working through the owner checks, closing the leads one by one, and then running it again. The skill reads its previous run's ledger, so the second pass builds on the first instead of starting over.