AWS Security Agent exploits 33 of 37 real-world CVEs

AWS ran its autonomous penetration‑testing agent, now part of AWS Continuum, against 37 real, recently disclosed CVEs (AWS Security Blog). Each flaw was deployed as a live, containerized target with a hidden flag at the intended sink.

From an unauthenticated or low‑privilege start, the agent received only an endpoint URL and a one‑line hint, with no pre‑built exploit or proof‑of‑concept. It then had to discover the vulnerability, craft an exploit, chain primitives when needed, and retrieve the flag.

The results show the agent captured 35 of the 37 flags. Of those, 33 flags—89.2 percent—were obtained end‑to‑end through the intended vulnerability, confirmed by a per‑sample integrity audit against each challenge’s canonical proof‑of‑concept.

During the audit, two captures were reclassified as off‑path, prompting AWS to harden those challenges and count them as unsolved. Additionally, two misses occurred where the agent exploited the bug but took the target down before the flag could be read.

The evaluation targets spanned web frameworks, a container registry, machine‑learning and AI platforms, an e‑commerce stack, a tRPC API, several media platforms, and an LDAP admin console. They covered eight vulnerability classes, from SQL injection and server‑side request forgery to multi‑step cross‑site scripting and admin‑cookie chains.

AWS noted that public CVE‑based agent benchmarks suffer contamination because frontier models ingest write‑ups and public proofs‑of‑concept into their training data. To mitigate this, the team created an internally developed capture‑the‑flag style benchmark sourced continuously from freshly disclosed vulnerabilities.

The challenges’ recency is intended to close the memorization path by tracking vulnerability classes that appear in production each quarter. Non‑destructive guardrails were also applied to keep production‑style engagements from disrupting target systems prematurely.

The performance of the AWS Security Agent against fresh, unmemorized CVEs suggests that autonomous penetration testing is reaching a practical maturity level for identifying complex vulnerabilities. By establishing a continuously updated benchmark, AWS appears to circumvent the data contamination limits plaguing static public suites, which likely provides a more realistic measure of agentic capability in security operations.

Spotted an error? See our corrections policy or email editorial@lordofclaude.com.