From vulnerability finding to exploit proof: inside our AI reproduction pipeline

The vulnpocalypse has left security teams with more vulnerability findings in their backlogs than they can manually reproduce. Bug bounty reports, SCA and SAST scanners, AI pen-test agents, and code security harnesses such as Codex Security keep adding to the queue. For every source, the question is whether an attacker can produce the claimed impact in the affected deployment.

That takes more than reading a claim or ranking it by severity. An engineer has to build the affected version, set up the conditions the finding assumes, and try the exploit.

Reproduction helps teams use their remediation capacity better in two ways: identify which findings are exploitable, then rerun the same exploit against a patch to confirm the fix.

We built an agent pipeline for both checks. It deploys the target in an isolated environment, runs an attacker agent against it, and checks the observed effect against the claimed impact. The exploit can then be rerun against a patched branch. Bug bounty reports were our first input. The SCA and SAST paths are rolling out in early access to Konvu customers.

This post explains how we split the work among agents, isolate the attacker, and verify the verdict. We walk through a Grafana CVE, then share the lessons and limits of that run for anyone building a similar system. It expands our OWASP AppSec Days France 2026 talk; an earlier post covers deployment in more detail.

The triage bottleneck: proving exploitability

Severity and reachability cannot prove an exploit works

Scanners and bug bounty platforms already dedupe and rank findings. That helps teams decide where to look. Reproduction still has to show whether the exploit works.

Reproducing one finding by hand takes 30 to 90 minutes of a senior engineer: read the claim, find the exact version, deploy it, configure it the way the finding assumes, then try the exploit. Most teams skip that and triage on the reported severity.

A severity score cannot show whether the exploit fires in an environment you control. Severity is what you were told. Exploitability is what you prove.

A vulnerable function running in your app does not mean an attacker can trigger it, as we argued in our post on reachability and exploitability. A successful reproduction turns that theory into observable proof. When the environment or evidence is incomplete, the verdict remains inconclusive.

Capable models and cheaper reasoning make reproduction practical

Two shifts made this worth building:

1. Models got good enough to do the reasoning. A current frontier model reads code it has never seen, forms a theory of the bug, and plans the steps to confirm it. That is the work a senior engineer does on a finding, and two years ago a model could not do it.
2. That reasoning got cheap. OpenAI shipped GPT-6 Luna in September 2026 at ten cents per million input tokens, about half its earlier prices. When an investigation costs cents instead of an engineer's afternoon, you stop saving it for the scary findings and run it on all of them.

Those shifts let us split the engineer's playbook into small agents. Each handles one step and produces an output the next stage can check.

Reproduction has two hard requirements: build a faithful copy of the target, then give an attacker room to act without exposing anything else.

How the reproduction pipeline builds, attacks, and verifies in isolation

From finding to running target: mapper, planner, and deployer

The diagram follows a bug bounty report through the pipeline. Program rules and the repository feed a mapper that builds a reusable threat model; deduplication turns incoming reports into a canonical report. Scanner findings join at the planner after source-specific qualification, described below.

The bug bounty path shown here has five stages. The mapper runs once per project; the remaining stages handle each report:

1. Mapper reads the target's code and program rules and builds a threat model of it: the assets, the roles, the attack surface. It is cached per project, so later reports reuse it instead of re-deriving it.
2. Dedup collapses near-duplicate reports into one canonical report, so each real bug is reproduced once, not fifty times.
3. Planner reads the canonical report against that model and writes the environment spec: the version to run, the config, the seed data, the prerequisites the exploit needs.
4. Deployer builds the app at the affected version, brings it up, seeds it, and health-checks it.
5. Attacker gets the running target and tries to fire the exploit.

The two dashed feeds are shared context: the codebase the planner reads, and a knowledge store of lessons from past runs, which we come back to below.

Each stage exits one of three ways: success (continue), stop (a clean halt), or fail (a crash or schema violation). A stop records why an agent could not reproduce the finding: environment_limitation, report_inaccurate, hardening, or not_exploitable.

The planner declares the exploit's prerequisites up front as a typed object (required_features, env_vars_needed, config changes, seed data). A lab can stop for a closed set of limitations: a paywalled feature, a managed service, third-party credentials, a physical device, or a non-lab OS. When the planner hits one, the schema validator records environment_limitation before we spend money on an attack. When the lab cannot build the required conditions, the verdict stays inconclusive.

Reliable handoffs: typed contracts, smoke tests, and retries

An agent is a model in a loop with tools. Left alone, it drifts, declares success early, and hallucinates a working exploit. The constraints around the model make its output checkable.

Four of them do most of the work:

1. Precise goals. One narrow subtask per agent. The planner plans, the provisioner provisions, and the deployer deploys.
2. Verifiable outcomes, checked deterministically. Before the deployer hands off, a smoke gate probes the health endpoint the planner declared, and re-runs the deploy up to two times to fix a broken seed. The deployer is not allowed to call the exploit endpoint itself, so a "ready" signal never leaks into the attack. To decide impact, a side-effect verifier records a baseline row count before the attack and re-reads it after, and will only promote a run from inconclusive to proven when the state actually changed.
3. Typed contracts. Every handoff between agents is a schema, validated with Pydantic. The planner writes plan.json with the prerequisites the exploit needs: required_features, env_vars_needed, the config changes, the seed data. The deployer writes target.json with the URL, credentials and services. If a stage emits something off-contract, it fails hard with a stage-specific schema error rather than passing garbage downstream.
4. Retry loops. Stages fix and retry against real feedback instead of failing on the first stumble.

The agents themselves are off the shelf. The environment builders run on the Claude Agent SDK; the offensive side uses a tool-use loop directly on the API. We spent our engineering effort on narrow goals, typed contracts, and deterministic checks between the model and the verdict.

The disposable lab: separate hosts and restricted egress

We give the attacker agent carte blanche inside the lab, including a bypass on its own tool approvals.

Four controls bound the attacker's blast radius:

1. Every run assumes a role into a separate AWS account dedicated to reproduction, tagged with the tenant it belongs to. The attacker cannot see production, ours or yours.
2. Every run gets its own EC2 instances: a target box, and a separate attacker box.
3. The attacker's egress is deny by default. It can reach the target hosts in target.json and the Anthropic API for model calls, but no other internet hosts.
4. Everything is torn down at the verdict. A cron reaper terminates anything tagged as a lab older than six hours as a backstop, and there is a hard cap on concurrent instances.

The attacker loop: escalation tiers, budgets, and stop conditions

A leader orchestrates the offensive agents and picks how hard to try.

It escalates through three pentester tiers:

1. Junior runs on a cheap model with a ten-turn budget for quick checks.
2. Medium gets a hundred turns and shell access for multi-step exploits that need recon.
3. Confirmed runs on a stronger model with two hundred turns for the ones that resist.

Each tier's model is overridable on its own, so you can point the confirmed tier at a bigger or a cheaper model without touching the other two.

An autonomous loop with a budget will happily burn all of it, so four guardrails cap each run:

1. A twenty-dollar cost circuit breaker spans the sub-agents.
2. A repetition detector injects a "try a different approach" nudge after the same tool call three times.
3. A finalize nudge fires once spend crosses three quarters of the budget.
4. A hard turn cap on the source-reading sub-agent keeps a scope-drift run from eating the whole budget.

Evidence-backed verdicts, including inconclusive

We derive the verdict from assertions, each marked proven, disproven, or inconclusive. An inconclusive assertion has to carry a reason. A pure function rolls them up: one or more proven assertions with the rest disproven means exploitable; all disproven means not exploitable; any inconclusive assertion makes the whole run inconclusive.

"We could not build the precondition" resolves to inconclusive, so an infrastructure gap cannot become a clean bill of health. We attach a CVSS score only when the rollup says exploitable. A failed run gets no number to argue over.

Treating findings as untrusted input

A bug bounty report is attacker-supplied text. Scanner findings and other artifacts also feed a model that can run commands. We treat the text as untrusted at every stage that reads it.

We handle injection fail-closed. If a stage detects an injection attempt in the canonical report or any artifact derived from it, the finding is rejected with reason injection_detected, a sticky sentinel fences the rest of the run, and the offending text is redacted in place. The check runs at the validator, the planner, the deploy reporter, the attacker and the final reporter, not just at the door.

Reusing lessons from previous runs

In the bug bounty flow, the planner, deployer and builder each read lessons from past runs before they start, scoped to the program. After every run they write new ones, and we keep the lessons from failed runs on purpose. A failed deploy teaches the system more than a clean one. Manual reproduction knowledge lives in people's heads and leaves when they do.

Applying the pipeline to other findings and fixes

The source changes what reaches the planner. In the bug bounty flow, program rules and repository context feed the mapper, while reports pass through deduplication. AI pen-test agents may supply proposed exploit paths; Codex Security and other code security harnesses can produce code-level findings. We still build the affected version and check the claimed effect. For SCA and SAST, we qualify the scanner finding before it reaches the planner.

An SCA or SAST finding becomes a canonical report with source_kind set to sca or sast and no program attached. Instead of program rules, it carries an immutable, signed snapshot of the finding and a pinned commit. The planner therefore tests the exact finding reported against the exact code revision that produced it.

What "prerequisites" means shifts with the source:

• For SCA, a vulnerable dependency in the lockfile is not enough. The qualification has to come back applicable: attacker-controlled input reaches the vulnerable function, through a real dependency path and entry-point trace, and the advisory supplies the mechanism. A finding that is only reachable, with no concrete impact and no expected observable, is blocked before a lab is ever built.
• For SAST, the reporter has to trace attacker control from an entry point, across the authorization boundary, to the sink, at the resolved commit.

For findings that pass qualification, the deployer builds the affected version in an isolated environment. The attacker agent then tries to fire the exploit.

Fix verification runs the same machine in reverse. A maintainer comments on the patch pull request, the engine re-runs the original exploit against the branch, and the verdict inverts: if the exploit still fires, the fix is incomplete. The comment is untrusted and write access to the repo is required, for the same reason the report is untrusted.

The scanner workflow is rolling out in early access to Konvu customers and currently runs in isolated labs only. Automatic ingestion and writing results back into the scanner remain out of scope for now.

What reproduction at scale taught us

Bug bounty programs for major open source projects were our first proving ground. We've triaged and reproduced hundreds of vulnerabilities and helped maintainers discard many more reports. Those runs shaped the pipeline's central rule: an exploitable verdict requires observable evidence from a reproduced attack. A run stays inconclusive when we cannot build or check the conditions the exploit requires.

The Grafana CVE below shows one run in detail: the target we built, the attack we attempted, the evidence behind the verdict, and the time and model calls it took.

One run in detail: Grafana CVE-2024-1313

We ran the pipeline against Grafana CVE-2024-1313, where an authenticated user in one organization can delete another organization's snapshot using its key. It is a broken object level authorization bug.

For this Grafana finding, the attacker agent used $0.44 in model calls and confirmed the bug on its first try. The exploit was simple: one endpoint was missing an authorization check. That let the leader keep the investigation in the cheapest, junior tier. More involved bugs need stronger tiers or more attempts, so they cost more.

For more complex findings, preparing the target is usually the slowest part. The pipeline must run the affected version with the configuration and data the exploit needs. We keep that work down by caching builds, reusing an environment across findings in the same project, and building only the parts the exploit requires.

Lessons and current limits

The $0.44 model cost and 17m 40s wall clock describe this Grafana run. We have not published an average cost or success rate across vulnerability types. We return inconclusive when the target cannot be configured as the exploit requires or when we cannot verify the claimed impact.

1. Make every handoff a typed contract, and fail the stage on a schema miss. One validated model per artifact (the plan, the infra, the target, the result) turns "the agent probably did the right thing" into an object you can check. A malformed field halts the run with a specific error instead of poisoning the next agent.
2. Split the agent that proves readiness from the agent that attacks. Our deployer health-checks the environment and probes the seed state, but it is not allowed to call the exploit path. A green "ready" can never be the exploit firing by accident. Different processes, different goals.
3. Calculate the verdict from each claim. Mark each claim as proven, disproven, or inconclusive. If any is inconclusive, the whole finding stays inconclusive. Otherwise, at least one proven claim means exploitable; all disproven means not exploitable. A pure function applies these rules to the evidence.
4. Give the offensive agent a disposable, egress-locked room. Use a dedicated account, per-run hosts, deny-by-default egress with an allowlist built from the target, and a reaper that kills stale labs.
5. Fail closed on input text. A bug bounty report is attacker-controlled, and scanner findings also go into a tool-using model. Scan for injection at every stage that reads them, and on a hit, reject and fence the rest of the run. Do not sanitize and continue.
6. Ship "inconclusive" as a first-class verdict. "Could not prove it" is not "safe." Track inconclusive runs to see which environments the system cannot build yet.