🔍 Read the full analysis: FrontierCode: Before You Submit That AI-Generated Pull Request on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Artificial Analysis added the FrontierCode coding benchmark on October 12, 2026, and tested 23 models on repository tasks graded against maintainers’ standards. Claude Opus 5.5 led, but passed all blocking checks in 57.7% of attempts; results indicate that human review remains important, while model costs vary sharply.
Artificial Analysis published FrontierCode results on October 12, 2026, after testing 23 AI models on code changes drawn from real software repositories. Claude Opus 5.5 ranked first, but passed every blocking check in 57.7% of attempts—a result that underscores why AI-generated pull requests still need review before maintainers merge them.
Claude Opus 5.5 (max with fallback) scored 52.2% overall, ahead of GPT-6.1 Sol (max) at 49.1% and Claude Fable 5.1 (max with fallback) at 49.0%. The separate pass rate measures attempts that cleared all blocking checks: Opus reached 57.7%, GPT-6.1 Sol 54.7%, and Fable 5.1 54.3%. The figures come from Artificial Analysis’s evaluation, not a measure of how often these models succeed on every kind of coding task.
FrontierCode’s full benchmark contains 150 tasks; Artificial Analysis ran its 100-task Main set. The tasks come from 36 repositories, including Celery, Budibase, uppy and Mattermost, and were written by more than 20 maintainers. According to the source material, task authors spent more than 40 hours on each task. Models were run through the same open-source Stirrup agent harness, with up to 250 turns per attempt and three attempts per task.
Scoring covers hidden tests, command checks, whether an agent’s tests expose failures in the original code, adaptive tests, limits on files and diff size, and an LLM review against maintainers’ criteria. Failing any blocking check gives an attempt a zero; attempts that pass all blocking checks receive scores from 70% to 100%, depending on quality checks. The evaluation used GPT-6 Luna (medium) for the review step. The source says attempts flagged by Cognition’s trajectory scanner for looking up solutions also score zero.
Infographic 01 · Would the maintainer merge it?
The best coding model clears the bar 58% of the time
FrontierCode asks whether an open-source maintainer would actually merge an agent’s change. A patch that fails any blocking check scores zero, even if the tests pass. 23 models, same harness, three attempts per task.
FrontierCode score, 23 models
Score in %, with the share of attempts that cleared every blocking check below it. Reasoning effort as tested.
Not on the board yet: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of 13 Oct 2026. Nemotron 3 Ultra has no published pass rate.
Source: Artificial Analysis, FrontierCode launch analysis (12 Oct 2026) and FrontierCode leaderboard, read 13 Oct 2026. FrontierCode 1.1 by Cognition, Main set: 100 private tasks, three attempts each, Stirrup harness, 250-turn cap.
Powered by Thorsten Meyer AI
Why Passing Tests Is Not Enough
FrontierCode measures a practical question that conventional coding tests can miss: whether a change is suitable for a repository and acceptable to its maintainers. A patch can pass tests yet still make unrelated edits, add weak tests or create an unnecessarily large diff. By making some of those issues blocking or penalizing them, the benchmark tests more of the work involved in reviewing an AI-generated pull request.
The leading pass rate also puts a limit on what the results support. In this evaluation, even the top model failed at least one blocking check in 42.3% of attempts, calculated from its reported 57.7% pass rate. That does not establish the failure rate for all coding work or production repositories, but it does not support treating automated code generation as a substitute for review. Maintainers and teams considering coding agents have reason to inspect both patch quality and test coverage before relying on a model’s output.
The scores also do not map neatly to general intelligence rankings. The source comparison says Claude Sonnet 5.5 was second among the listed models on Artificial Analysis’s Intelligence Index but fifth on FrontierCode; Grok 4.7 placed 18th on FrontierCode in that comparison. Conversely, DeepSeek V4.1 Flash scored 44.0% and ranked seventh. A general-purpose leaderboard may not indicate which model produces the most acceptable repository changes.
As an affiliate, we earn on qualifying purchases.
How FrontierCode Grades Repository Changes
FrontierCode was developed by Cognition with open-source maintainers and announced in June 2026. Version 1.1 followed in July with rules against looking up solutions online, according to the supplied account. The benchmark’s tasks are private, intended to prevent models from having encountered the solutions in training.
Its scoring design differs from a simple pass-or-fail test suite. It assesses correctness but also checks regressions, test usefulness, scope and patch quality. Maintainers’ criteria inform an LLM-based code review, while blocking checks determine whether an attempt receives any score at all. Tomer Nosrati, a Celery maintainer, described the distinction in Cognition’s announcement: “Where others grade like a CI, FrontierCode grades like a tech lead.”
Artificial Analysis’s results show a tight cluster near the top: nine models from six labs scored between 43% and 52%. Xiaomi’s MiMo-V2.6-Pro was the best open-weights model listed, scoring 45.4%, followed by DeepSeek V4.1 Flash at 44.0% and Kimi K3 at 43.6%. The benchmark also reports costs: GPT-6 Luna (max) scored 41.9% at $0.09 per task, while Opus scored 52.2% at $10.15. Those are the source’s reported task costs, not a complete estimate of the cost of operating a coding workflow.
“Where others grade like a CI, FrontierCode grades like a tech lead.”
— Tomer Nosrati, a Celery maintainer, in Cognition’s announcement
software repository management software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Limits of the Published Results
The results do not establish how models perform across all repositories, programming languages or production workflows. The available account identifies a 100-task Main set and a 150-task full benchmark, but does not provide enough detail here to assess how representative the selected tasks are or how scores vary by repository and task type. The test conditions also include a particular agent harness, turn limit and review model, so results may differ under other setups.
Several models had no FrontierCode result as of October 13, according to the source: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview. Their absence means the published rankings do not cover every model readers may want to compare. The material does not give an update schedule, independent replication or evidence that benchmark performance directly predicts outcomes in a team’s own code review process.
As an affiliate, we earn on qualifying purchases.
Further Results and Team Testing
Readers can watch for results from models not listed in the October 13 account, along with any later FrontierCode evaluations or changes to the benchmark. Until those are available, the published table is a snapshot of 23 models tested by Artificial Analysis under its stated setup—not a final ranking of every coding model.
For software teams, the immediate next step is to compare models on representative tasks from their own repositories and review the resulting patches, tests and diffs. FrontierCode’s reported rates offer a way to compare performance under one benchmark, but they do not answer whether an individual change is safe or suitable to merge. That decision remains with the people responsible for reviewing the code.
As an affiliate, we earn on qualifying purchases.
Key Questions
What is FrontierCode?
FrontierCode is a coding benchmark developed by Cognition with open-source maintainers. It evaluates repository changes using tests, scope and quality checks, plus an LLM review based on maintainers’ criteria.
Which model ranked first?
Claude Opus 5.5 (max with fallback) led the Artificial Analysis results with a 52.2% score. It passed every blocking check in 57.7% of attempts.
Does a 57.7% pass rate mean the model is unreliable for every coding task?
No. It describes this model’s results on the tested FrontierCode tasks and setup. It is not a general success rate across all coding work, repositories or uses.
Why does the benchmark check more than whether tests pass?
A change can pass tests while still having weak tests, unnecessary edits or problems with its scope. FrontierCode adds checks for those issues and applies blocking criteria intended to reflect maintainers’ acceptance standards.
Can teams use the leaderboard to decide what to merge?
The results can inform model comparisons, but they do not certify individual patches. Teams still need to review AI-generated changes and test them in the context of their own repositories.
Source: ThorstenMeyerAI.com
Columbus Day / Indigenous Peoples' Day Picks
long weekend sales
As an affiliate, we earn on qualifying purchases.
