CyberArena ranked 12 frontier AI agents across live attack-and-defense matches. GPT-5.6 Sol leads at 78.6, and its 83% flag-retention rate is what set it apart.
A model that can break into a server makes a good headline. A model that can stop the opponent next door from breaking into its own server, at the same moment, is a harder claim to make. A new benchmark put twelve frontier agents in that position and counted every capture, every patch and every second it took.
A Benchmark Where the Target Fights Back
CyberArena comes from 2077AI and UC Berkeley RDI. It was published on Hugging Face on November 3, with a companion paper on arXiv. Most cybersecurity benchmarks work like exams: one agent, one fixed target, one role. The authors argue that this setup has two weaknesses. The target never adapts, and fixed tasks saturate as models improve.
Cybench, the widely used professional-level CTF benchmark, shows why static scoring is attractive. It is reproducible and easy to grade. Still, a study of 22 frontier models on 23 Cybench challenges by Dreadnode found pass rates inflated by cheating: an average pass rate of 41.5% against a clean solve rate of just 26.1%.
CyberArena swaps the exam for a contest. Two agents get identical copies of a vulnerable service, and each attacks the other’s copy while patching its own. The ceiling rises with the opponent instead of stopping at 100%.
How a CyberArena Match Is Played and Scored
A match runs three 20-minute rounds with no hints. Before each round, the arena plants a fresh objective in each service. Both agents then attack and defend at the same time. At the bell, objectives rotate, so last round’s flags are worthless, but patches persist.
Every score comes from arena facilities the agents can’t touch. Across the campaign, agents called the arena’s game tools 53,511 times. A flag service verifies captures, and a health poller running outside the defender’s root access checks availability, so a fix that breaks a feature costs points. The final game score weighs three things equally: rounds captured, rounds defended and service availability, each worth a third of a 100-point scale.
The campaign submitted 760 matches, of which 466 were valid through batch 22. It covered 12 agents from ten model families on 40 ranked challenges, drawn from a catalog of 46. Thirty of those are built from real-world software, 28 of them around a published CVE, and 14 are ported from past attack-and-defense CTFs. Playing every pairing is impractical, so the arena selects matches in batches of 24 and fits a statistical model, calibrated on 53 held-out matches, to estimate how agents would fare against opponents they never met.
The live leaderboard has moved since the paper’s snapshot. It now shows 479 of 1,948 matches ranked, and the twelve expected scores still match the published figures.
What the Capture Model Reveals About Live Defense
The fitted capture model shows why live opponents change everything. Once defenders start patching, captures become rare unless the attacker already captured in the previous round. Put simply, an exploit that works against a static target can stop working in the middle of a match, and momentum matters.
This is the most useful insight in the paper, and a solo CTF can’t produce it, because the target never responds. In CyberArena, the best attacker captured the opponent’s flag in 70% of rounds, while the weakest managed 23%. On the defense side, the spread was wider, from 43% of rounds kept to 83%.
Attack and defense travel together, with a correlation of 0.85 across agents. The leader’s advantage, however, is not symmetric. GPT-5.6 Sol’s fitted attack ability (+0.46) is nearly identical to Claude Opus 5’s (+0.45). Its fitted defense ability is +0.86 against +0.23. That gap explains most of the leaderboard.
The Standings and Which Gaps Are Real
GPT-5.6 Sol leads at 78.6, with a 90% interval of 76.8 to 80.2. Claude Opus 5 follows at 73.8, and GLM-5.3 takes third at 68.5. Gemini 3.7 Flash scores 67.1, Grok 4.6 65.5 and Claude Opus 4.8 65.0. Qwen3.8 Max comes in at 62.3 and Gemini 3.5 Flash at 60.6. DeepSeek V4 Pro 0813 scores 59.5, MiniMax M3 59.1, Kimi K3 57.0 and HY4 Preview 56.3.
The top three gaps look meaningful. GPT-5.6 Sol leads Claude Opus 5 by 4.8 points, and Opus 5 leads GLM-5.3 by 5.3, with non-overlapping intervals. GPT-5.6 Sol also held first place at every one of 22 cumulative refits.
Other gaps call for more caution. A plain Bradley–Terry fit agrees with the ranking closely (Kendall’s τ = 0.85) but swaps the top two, with a p-value of 0.41 for that pair. Further down, places 3 through 6 sit within 3.5 points of each other with overlapping intervals, as do places 9 through 12, which span only 3.2 points. The order inside those clusters carries little weight.
The leader’s edge is defense. GPT-5.6 Sol kept its flag in 83% of rounds and Claude Opus 5 in 81%, against 43% to 61% for the rest. GPT-5.6 Sol also deployed its first fix fastest, with a median of 2.3 minutes against 3.9 to 12.3 for the others. In one audited match, it captured Claude Opus 4.8’s flag at 0.7 minutes and patched its own service at 1.2 minutes, while Opus 4.8 didn’t deploy its first patch until 18.9 minutes into a 20-minute round.
Cost adds another layer. GPT-5.6 Sol averaged $17.7 per match and Claude Opus 5 $30.6, while GLM-5.3 reached third place at $4.7. HY4 Preview was the cheapest at $1.2.
How Agents Fail When Someone Is Fighting Back
An automated audit read all 466 match trajectories and produced 2,397 findings across 673 candidate patterns. The most common error is the unverified fix claim, where an agent says a patch passed its checks while reading logs from before the restart. That appeared in 23% of audited matches. Partial patches, which close one entry point and leave another open, showed up in 21%. Over-blocking, which stops attacks but breaks the service, appeared in 8%.
Defenders that were hit did tend to respond. In rounds one and two where a defender lost its flag, it inspected the attacks fired at it in 67% of cases and restarted before the next round in 57%. Those that restarted kept their flag in 60% of the following rounds, against 9% for those that didn’t, though the authors caution that this is an association rather than proof the fixes worked.
Attacker-installed code stayed executable after repairs or restarts in 4% of matches. Both Gemini Flash entrants slept through active rounds in about a third of their matches, and because they share a harness, model and harness can’t be separated. Availability held up well overall, with health probes passing between 90.1% (MiniMax M3) and 97.7% (Claude Opus 5 and Qwen3.8 Max).
What the Results Mean for Security Claims
The limits matter. The study covers twelve model-and-harness combinations, pairwise single-service matches and challenge types the authors chose. The audit is automated, and the paper’s figures are a snapshot through batch 22. The live leaderboard keeps updating as more matches are ranked.
Even so, the practical lesson is clear. When a vendor says a model scores well on a security benchmark, the next question should be whether the target could fight back. A contest measures things an exam can’t: how fast an agent patches, whether it checks its own fixes and whether it breaks the service while repairing it. In CyberArena, those traits separated the top two from the rest of the field.
As more labs publish security claims for their models, live-opponent evaluations like this one are likely to matter more. Whether the top-two ordering survives more matches and more challenge types is the next thing to watch, and the leaderboard’s climb from 466 to 479 ranked matches means that test is already underway.
Disclosure: This is not trading or investment advice. Always do your research before buying any cryptocurrency or investing in any services.
Follow us on Twitter @themerklehash to stay updated with the latest Crypto, NFT, AI, Cybersecurity, and Metaverse news!