The agentic frontier
Changing
the Game.
Assessing Agentic AI’s Impact on Cyber
Competitions and Cyber Operations
17 serious flaws.
About two hours.
A tool-builder in our study pointed an AI bug-finding system at a large software codebase. It returned 17 high-severity findings, each validated with a working proof of concept.
lines of Ruby on Rails code
to surface the findings
validated high-severity findings
Finding a vulnerability is one task.
Completing an operation without being detected is another.
What agentic AI
is good at.
Agents perform best when the task is bounded, success can be checked, and source code or a clear starting point is available. Four strengths stand out.
White-box vulnerability discovery
White-box discovery means working with access to the full source code. The system behind the opening example had a defined codebase to inspect and could test each suspected flaw with a working proof of concept.
Exploit generation
Agents can turn vulnerabilities into working payloads quickly. One challenge author watched a model find a stack buffer overflow in a roughly 30 MB game executable in about ten seconds, then write the payload—work he estimated would have taken a person a week.
Large-scale forensics and data processing
Interviewees highlighted packet-capture triage, steganography, and other artifact-heavy work. Agents can search large volumes of material for anomalous signals, helping operators find the relevant evidence in a much larger body of data.
Obscure-technique research at speed and scale
Some challenges hinge on recognizing a technique that even experienced practitioners rarely encounter. Agents can research unfamiliar methods, match them to the problem, and explore many possible approaches quickly.
Where agents
still struggle.
Fast technical execution is only part of an operation. Interviewees described five recurring weaknesses where human direction and judgment remain important.
Stealth — “loud, not sneaky”
Agents can trigger detection systems, leave artifacts behind, and probe things they should leave alone. Interviewees were reluctant to trust them with lateral movement without close human oversight, especially against an alert and capable defender.
Novel problems requiring creativity
When there is no source code or established route to an answer, agents can fall back on guessing and brute force. In one quantum-related challenge, a person helped the model make progress by reframing the task as mathematics.
Reliability and reward hacking
Agents can invent evidence, defend an incorrect solution, or take shortcuts that miss the intended task. One false vulnerability finding fed into further analysis, producing wrong conclusions about which software versions were affected.
Mental models and tacit knowledge
Experienced operators infer how an unfamiliar system is likely to be configured and which paths matter. Agents can miss that context, pursue irrelevant attacks, and require repeated prompting even when they have the underlying technical skills.
Multi-step exploits
Individual steps may work while a long chain fails. One interviewee described agents struggling when an exploit chained 60–70 vulnerabilities together; others saw systems lose their original plan as dependencies accumulated. Participants said this weakness is receding.
These findings reflect publicly available, general-purpose models. Capabilities are improving, and specialist systems may perform differently; the report does not treat these weaknesses as permanent.
How practitioners
rated agentic AI.
Competition categories and phases of an operation are different comparisons. These charts present the report’s survey results on the same 1–5 scale.
Twenty agents.
One challenge.
An hour later: a solution.
One competitor described launching up to 20 agents on a difficult challenge and returning an hour later to a solution. Another built a tool that raced roughly 30 models and kept the first successful result.
“Before, I would come up with an idea for an exploit, but 80% of the work was (implementing) it. Now, AI just figures it all out.”— Interview participant
The operator’s work moves toward:
- Choosing the approachIdentify promising ideas and decide which avenues agents should pursue.
- Checking the resultDistinguish working evidence from false positives and shortcuts.
- Redirecting the effortCorrect the plan when agents stall, lose context, or risk exposing the operation.
What has to
happen next?
The report calls for better model access in government, coordinated support for private-sector defense, and training that rewards judgment as well as correct answers.
01Equip the missionThe best models.
At every classification level.
Federal operators need current models and enough compute in both unclassified and classified environments.
The best models.
At every classification level.
Federal operators need current models and enough compute in both unclassified and classified environments.
Ensure federal cyber operators have access to the latest models, backed by adequate compute across unclassified and classified environments.
- Develop infrastructure and security arrangements for rapid deployment.
- Improve procurement mechanisms as model capability advances.
- Give national-security operators the tools to match the speed and scale of adversaries.
02Prepare the ecosystemA Y2K-style effort
to prepare private-sector defense.
Smaller organizations may not be able to sustain an active defense against autonomous intrusions on their own.
A Y2K-style effort
to prepare private-sector defense.
Smaller organizations may not be able to sustain an active defense against autonomous intrusions on their own.
Orchestrate a Y2K-style initiative to help the private sector prepare, especially organizations unable to sustain an active defense against autonomous intrusions.
- Convene relevant actors and make preparedness visible.
- Remove obstacles to collective action and apply regulatory pressure where appropriate.
- Help smaller organizations access a scalable defensive ecosystem.
03Reinvent the training groundReinvent the basics, then teach people to lead teams of agents.
If an agent captures the flag, a novice can finish a challenge without learning why the exploit worked.
Reinvent the basics, then teach people to lead teams of agents.
If an agent captures the flag, a novice can finish a challenge without learning why the exploit worked.
Build a new generation of cyber competitions and training environments around human–AI teaming, while preserving routes into the profession and instruction in the fundamentals.
- Reward understanding, judgment, creativity, and stealth.
- Teach effective supervision of autonomous agents.
- Create a new gateway for the next generation of cybersecurity professionals.