On the evening of April 13, 2026, the research arm of security company Winfunc finished the latest run of a new benchmark: N-Day-Bench. It tests exactly one thing — whether frontier language models can find real, already-disclosed vulnerabilities (N-Days) in real codebases, using only advisories published after each model’s knowledge cutoff. First results: OpenAI’s GPT-5.4 leads at 83.93, Z.ai’s GLM-5.1 scores 80.13, and Anthropic’s Claude Opus 4.6 scores 79.95 — the top three packed within four points — while Google’s Gemini 3.1 Pro trails the leader by more than 15.
The interesting part is less the ranking than the design. The oldest objection to security evals is contamination: a model memorizes vulnerabilities seen in training and “replays” the answer in the exam. N-Day-Bench’s answer is blunt — the test refreshes monthly, and only accepts advisories published after the tested models’ cutoffs, so recall does not help. Every trace is publicly browsable, and any score can be traced back to specific shell commands and code.
Why Post-Cutoff Bugs Matter
A frozen benchmark degrades into a memorization test; an unbounded rolling one makes runs incomparable. N-Day-Bench splits the difference with monthly editions anchored to cutoff timestamps: each edition’s pool contains only advisories published after the previous cutoff and on or before the current one, and the model roster is upgraded to each lab’s latest version and checkpoint. The site states that every model gets the same harness and the same context, with “no leeway for reward hacking.”
Case Selection: 1,000 Advisories, 47 Survive
Cases come from GitHub Security Advisories, scanned in reverse publication order via the GraphQL API. Qualification is strict: an advisory must explicitly reference a repository (not inferred from package metadata), the repo must exceed 10,000 stars, and the advisory must point to exactly one unambiguous fix commit. Selection proceeds in 500-advisory windows until all 50 slots fill; this month’s run scanned 1,000 advisories, accepted 47, and skipped 953. The harness checks out the fix commit’s sole parent — the pre-patch, vulnerable state — while the fix diff goes to the question writer and is withheld from the model under test. Ambiguous cases (multi-repo advisories, merge commits, unresolvable refs) are dropped rather than approximated, and a round-robin pick per repository keeps one popular project from dominating the set. The stated philosophy: a smaller, clean dataset beats a larger one built on guesses.
The Curator-Finder-Judge Pipeline
Three agents with asymmetric roles. The Curator, pinned to GPT-5.4 at medium reasoning, reads the advisory context and patch excerpts to produce a structured case with an answer key: expected vulnerability class, components, sink paths, and required evidence. The Finder — the model under test — gets the case plus a read-only bash shell over the checkout, up to 24 tool-use steps, and always starts from a known sink to test backward data-flow tracing rather than open-ended bug discovery. The Judge, also GPT-5.4, sees only a digest-based blind label. Sandbox rules: a read-only overlay filesystem mounted at /workspace, git shimmed to safe read-only stubs, a 12-second per-command timeout, and limits on call depth, total commands, and awk/sed loop iterations. Scoring uses a fixed five-dimension rubric: target alignment 30%, source-to-sink reasoning 30%, impact and exploitability 20%, evidence quality 10%, and overclaim control 10%.
First Scores: Top Three Within Four Points
GPT-5.4 leads at 83.93 (44 submissions, 1.07 average findings); GLM-5.1 follows at 80.13 with the highest findings-per-submission rate at 1.23; Claude Opus 4.6 posts 79.95 (43 submissions, 1.16); Moonshot’s Kimi K2.5 lands at 77.18 on 37 submissions; Gemini 3.1 Pro manages 68.50 with the lowest findings rate at 0.91. The run started at 5:03 PM on April 13 and finished at 8:53 PM the same day — about 3 hours 50 minutes, zero failures. A cluster this tight at the top suggests the constrained task of tracing backward from a known sink is largely handled by frontier models, and the differences show up in report quality more than in whether the bug is found at all. Gemini’s 15-point gap is the number its team should be investigating.
How to Read This Leaderboard
Three caveats. First, the Curator and Judge are both GPT-5.4 — which also competes and ranks first; blind labels reduce bias, but the site itself concedes it cannot rule out stylistic fingerprinting or length-correlated bias. Second, the Judge emits the entire score object in a single pass with no server-side weighted recomputation, trading mechanical consistency for flexibility. Third, backward tracing from a known sink is far easier than open-ended vulnerability discovery, so scores should not be extrapolated to real bug-bounty conditions. Even so, the combination — monthly refresh, full trace transparency, and a read-only harness — makes this one of the few credible continuous trackers of LLM security capability. And note the incentive: Winfunc sells autonomous code-auditing AI agents, so the leaderboard doubles as a public demo of its own product’s machinery.
Sources
AI-assisted summary compiled from the sources above, reviewed by a human before publishing.