🔍 Read the full analysis: GPT-6 Astra Found Bugs In Claude’s Model Edit That Tests Had Missed on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
A ThorstenMeyerAI.com report says OpenAI’s GPT-6 Astra, running in Codex, reviewed an experiment in which Anthropic’s Claude Opus 5.5 edited a local model. The review identified reproducible problems in the benchmark rig and challenged several claims, while the reported legal score remained zero. The review did not independently repeat the full experiment, and the article says its findings do not establish that the edit improved legal performance.
A review by OpenAI’s GPT-6 Astra, running in Codex, found reproducible faults in the test rig for an experiment in which Anthropic’s Claude Opus 5.5 edited a local language model, according to a report updated November 9 by ThorstenMeyerAI.com. The findings matter because some problems could make a benchmark run appear successful or preserve results from a different setup, even when routine tests passed.
The report describes a deliberate division of roles: Claude ran the experiment on the local model Ornith 1.5, while GPT-6 Astra was asked to review the implementation rather than make a competing edit. The reviewer wrote separate checks; the Claude session made the resulting changes. The report says GPT-6 Astra did not inspect hidden benchmark prompts, topics, answers or reference solutions.
In its first review, conducted while the experiment was being built, GPT-6 Astra identified four reproducible faults. Resuming a run could retain answers from a different model or sampling setup; an unknown task ID could lead to a successful run with no evaluations; a command-line syntax conflict remained despite passing parser unit tests; and converting model weights changed values even when the edit strength was zero.
The article says the first conversion check found a difference in 105,315 of 1,048,576 reconstructed values in one matrix. After the initial fixes, the second review round had a different result: all 12 author-written tests passed, while four of the reviewer’s six provenance checks failed. In a separate measurement, a format conversion moved one matrix by 0.51% and changed 915,360 of its 1,048,576 values. The report says these figures concern weight conversion, not legal accuracy or an error rate.
Infographic 01 · The two review rounds
Green tests, real faults
GPT-6 Astra (OpenAI, in Codex) reviewed a Claude Opus 5.5 session's model-edit experiment while it was still being built. What it found, and what happened next.
Round 1 · 4 Nov · four faults
- Resuming a run could replace a different experimentAnswers were skipped even after model, settings or edit changed; in the reviewer’s test: 0 new inferences, old answer kept! Partly fixed
- A mistyped task ID gave a “successful” empty runZero evaluations, no error✓ Fixed
- Unit tests green, real command brokenCommas in the sampling syntax clashed with the model list; 3 parser tests passed, the real dry run failed✓ Fixed
- Conversion changed weights at edit strength zeroRe-quantising one real matrix: 0.84% change, 105,315 of 1,048,576 values differed! Partly fixed
Round 2 · 4 Nov, evening · the fixes re-checked
- Old result rows got a new historyAccepted after the requested seed was changed to 999✓ Solved another way
- Loop-detector settings not fingerprintedChanging them left the configuration “identical”✓ Solved another way
- Model configuration and chat template invisibleOnly file names, sizes and modification times were compared✓ Solved another way
- bf16 storage still changes the computationThe zero-strength test compared a conversion with itself→ Control added
✓ Fixed with a test that fails on the old code · ! Partly fixed in round 1, finished in round 2 · ✓ Solved another way: resuming was removed entirely, and every run records fingerprints of configuration, tokenizer and code · → Control added: a conversion-only copy with no edit. The reviewer changed none of the main code and read none of the hidden benchmark tasks.
Sources: our experiment records, 4–5 Nov 2026: reviews of 4 Nov and final explanation of 5 Nov by GPT-6 Astra in Codex (OpenAI); results report of the executing Claude Opus 5.5 session (Anthropic).
Powered by Thorsten Meyer AI
Why Benchmark Provenance Matters
The findings show how a benchmark can produce misleading records without an obvious failure in its ordinary test suite. If a resumed run carries forward answers generated under different settings, or an invalid task identifier is silently ignored, the resulting output may not represent the test researchers believe they ran. A command that fails only at its real entry point can also leave a control condition untested.
The weight-conversion issue bears on interpretation of the model-editing result. If the weights change during format conversion even when the intended edit is zero, researchers cannot attribute every observed difference solely to the proposed edit without a suitable control. The report says the experiment’s legal score stayed at zero, despite the edited model answering all 10 published tasks compared with four for the unedited model. That count alone does not show improved legal performance.
For readers of AI evaluations, the report is a case study in why reproducibility depends on more than green unit tests: provenance checks, exact run settings and controls for data conversion can affect what a result supports. It is a report about one experiment and review, not evidence that one company’s model is generally better at finding software defects.
Products Worth Considering
Bose QuietComfort Headphones (2nd Gen), Wireless Over-Ear Noise-Cancelling Headphones, Black
Bose TV Speaker Compact Soundbar with Bluetooth and HDMI-ARC, Black
Apple AirPods Pro 3 Wireless Earbuds with Active Noise Cancellation
As an affiliate, we earn on qualifying purchases.
How the Review Was Set Up
The two AI sessions began work on the experiment on November 4, according to the report. Once the author recognized that both were working on the same project, Claude retained responsibility for the experiment: the plan, test rig, directional weight edit, benchmark runs and blind grading. Three Claude agents performed that grading.
GPT-6 Astra was assigned to validate the other session’s implementation. The report says it did not alter the main implementation or run a second model-improvement experiment. Its early preparatory files were set aside after the roles were divided. The reviewer tested real entry points with small, reproducible examples rather than relying only on the project’s own tests.
The updated article says the figures now refer to 10 published Lexassay tasks, with runs available on lexassay.com. It reports that the edited model answered all 10, versus four for the original, while the legal score remained zero. Those results and the review findings are reported by ThorstenMeyerAI.com; the source material does not provide an independent replication of the benchmark.
“Review and validate the other session’s implementation.”
— The review assignment, quoted from GPT-6 Astra’s report
What the Review Does Not Establish
The report does not say that GPT-6 Astra independently reran every benchmark or reproduced the full model edit. It says the reviewer did not rerun every test during its final documentation step, was not a fourth grader, and measured weight conversion on a CPU rather than the production GPU path. The degree to which those limits affect the reported measurements is not established in the supplied material.
The report also says the reviewer challenged the wording of several claims: that the zero-strength edit was “exactly identical,” a description of average citation errors, the phrase “parallel Codex experiment,” and a causal conclusion drawn from one control. The supplied account does not give the full revised wording or a complete account of the second round’s six provenance checks. It is therefore unclear from this material which four checks failed, or what each failure meant for the benchmark’s final results.
The reported task counts do not establish legal improvement: the legal score was zero, and the source does not provide enough detail here to assess task difficulty, scoring criteria or statistical uncertainty. No independent confirmation of the report’s account is included in the source material.
Public Runs and Further Validation
The report says the 10-task runs are public on Lexassay, alongside the weight-edited model, an edit control and model-card sampling. Those materials offer a basis for examining the reported results and the changes made after review. The source does not announce a specific date for a further review or a new benchmark run.
The next useful evidence would include a complete record of the failed provenance checks, reproducible runs using the corrected rig, and results from the production GPU path. A comparison that isolates the weight edit from format conversion would also help clarify what caused any changes in model outputs. Until then, the reported review supports a narrower conclusion: it found important test and record-keeping faults, but does not by itself establish that the edit improved the model’s legal capability.
Key Questions
What did GPT-6 Astra review?
It reviewed the implementation and testing process for a Claude-led experiment that edited the weights of the local model Ornith 1.5. The report says Astra did not conduct a competing model-improvement experiment.
What were the main problems found?
The first review identified four issues: resumed runs could reuse answers from a different setup, unknown task IDs could yield empty but successful runs, the real command line could fail despite passing parser tests, and weight conversion changed values at zero edit strength.
Did the edit improve the model’s legal performance?
The report says the edited model answered all 10 published tasks, compared with four for the original, but its legal score remained zero. Those figures do not establish improved legal performance.
Were the findings fully independently verified?
The source describes GPT-6 Astra’s separate checks, but does not provide an independent replication by another reviewer. It also says Astra did not rerun every test in its final documentation step and measured on a CPU, not the production GPU path.
Where can readers inspect the experiment?
The report says the 10-task runs and related materials, including the edited model and an edit control, are public on Lexassay.
Source: ThorstenMeyerAI.com
