🔍 Read the full analysis: Inside Our AI Test: Local Model Editing And Weight-Space Programming on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
A home experiment removed a measured loop-related direction from the output weights of local model Ornith 1.5 35B-A3B. It finished 12 of 15 benchmark answers after the edit, compared with 4 of 15 in the paired run, but scored zero both times; with recommended sampling settings and no edit, it finished 13 of 15.
A local-model experiment reported November 5 found that a targeted edit to Ornith 1.5 35B-A3B helped it finish more answers on a legal benchmark, but did not improve its score: it completed 12 of 15 questions and scored zero. The result matters because it separates a model’s ability to stop looping from its ability to produce accurate legal answers.
The experiment was inspired by an MIT CSAIL paper describing “weight-space programming,” in which a model edits another AI’s weights directly. The home test was not a replication of that method. According to Thorsten Meyer’s account, a Claude Opus 5.5 session wrote code to run Ornith, detect a loop-related direction in its internal signals, and remove that direction at half strength from the output weights of 27 of the model’s 40 layers. No training or new legal data was used.
In the reported comparison using the same setup, the unedited model completed 4 of 15 answers, while the edited model completed 12. But both received a legal score of zero. The 12 edited answers included 62 wrong or invented citations, or an average of 5.2 per answer. Across 75 checked criteria, the edited answers met seven, up from three before editing.
A separate run complicates the apparent improvement: using sampling settings recommended by the model card, the unedited model completed 13 of 15 answers without a weight edit. Meyer says the setup also changed during testing, and that change alone shifted the original model’s completed-answer count from 10 to four. OpenAI’s GPT-6 Astra, running in Codex, reviewed the code, figures and claims and identified faults, according to the account. It was a review of the experiment, not a second experiment.
Infographic 01 · What we changed
One loop direction, weakened at half strength
Inspired by the MIT post, but a simpler, classic technique (directional ablation): code written by a Claude Opus 5.5 session measured one loop direction by running the model and projected it out of the weights. No training, and no AI reading weights as text.
The six steps
- Write test material that is not the benchmark: 24 legal development questions plus 8 general controls, 0 overlap with benchmark prompts
- Find the true loop onset at token level: from there, at least 90% of the next 1,500 positions repeat earlier 32-token sequences
- Run the model and measure one loop direction per layer: average internal signal inside the loop minus the reasoning just before it
- Weaken it in the weights:
W ← W − α·r·(rᵀW)on the MLP output matrices of all 256 experts and the shared expert - Bake an edited copy; the original files stay untouched and the edited weights stay internal
- Benchmark original and edit on the same local setup, 15 tasks, graded blind by three Claude agents
Quick test: share of pure copying
Restart 20 tokens before a known loop onset, 8 cases, 3,000 tokens. Lower is better.
The cost of the edit
- Full strength: 23 of 24 legal questions finished, but control answers broke: tokens dropped out of short formulas, and month counting went wrong
- Half strength: 23 of 24 finished, 9 of 9 checked control facts right, and one German control question was answered in Chinese once
- Before the edit: 8 of 24 legal development questions did not finish; all 8 controls did
One run per setting. The loop detector was revised three times during the experiment, so the development numbers come from a moving yardstick. The direction is a statistical description, not a single “loop neuron”.
Sources: our experiment records, 4–5 Nov 2026 (executing Claude Opus 5.5 session; review by GPT-6 Astra in Codex); Ornith-1.5-35B-A3B model card.
Powered by Thorsten Meyer AI
Completion Is Not Legal Accuracy
The result illustrates a practical distinction for people running local models: an intervention can improve how a model behaves without improving what it knows. Finishing an answer is useful when a model gets trapped in repetition, but it is not a substitute for valid reasoning, reliable citations or verified source material.
That distinction is especially consequential in legal work, where a fluent response with fabricated authorities can mislead users. In this test, the edited model completed more responses under one comparison, yet its legal score remained zero and its citations were frequently wrong or invented. The report says the experiment is not legal advice, and its results do not establish that weight editing can make a model suitable for legal tasks.
The findings also caution against attributing gains to an edit without controlling other settings. The unedited model finished more answers under the model card’s recommended sampling settings than the edited model did in the fair comparison. That makes the narrow takeaway more defensible than a claim of general improvement: the edit may have reduced looping in a particular run, but the available tests do not show it improved legal performance.
Products Worth Considering
Apple MacBook Air 13.6-inch with M5 Chip, 16GB Memory, 512GB SSD, Sky Blue
Samsung Galaxy Tab A7 Lite 8.7 32GB Wi‑Fi Android Tablet – Long‑Lasting Battery, Compact Slim Design, Sturdy Metal Frame (US Version)
Bose TV Speaker (2nd Gen) Compact Soundbar
As an affiliate, we earn on qualifying purchases.
The MIT Idea and Ornith’s Loops
The MIT researchers named in the source account are Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab. Their reported Qwen example involved GPT-6 Astra reasoning about another model’s weights and editing them. The source says the Qwen model’s maths score rose from 7.9% to 48.1% after an edit associated with a looping problem. The researchers’ stated limitation, as quoted by Meyer, was that editing could remove an obstacle hiding an ability but could not add an ability the model lacked.
Meyer chose Ornith because, among four local models considered, it appeared to show the relevant looping behavior and could run on the available machine. The model card describes it as a 35-billion-parameter mixture-of-experts model, with about three billion parameters active per token, designed for coding and agent work rather than law. In the benchmark’s earlier result, it scored one out of 100; five of 15 answers did not finish, and some long transcripts contained few distinct lines.
The test used Lexassay, a benchmark described as asking 15 questions about German, U.S. and EU law without internet access or tools. It checks cited statutes and court decisions against primary sources; ten tasks are published and five are hidden. The article says the new runs are not included in Lexassay’s published results, which still show the original Ornith result.
“The model finishes is not the answer is right.”
— Thorsten Meyer, describing the experiment’s takeaway
Limits of One Local Test
The reported results come from one run per setting, a small development set and a loop detector that changed during the experiment. The source also says the local setup changed in a way that moved the unedited model’s completion count from 10 to four, making it difficult to isolate the effect of the weight edit.
The account says Claude carried out the edit and Claude agents handled blind grading, creating a potential conflict of interest. GPT-6 Astra from another company reviewed the work and found faults, but the report does not describe a second independent run or establish that the edit caused the difference in completed answers. Three graders came from the same model family. It remains unclear whether the completion gain would hold across repeated runs, other prompts or different hardware and settings.
The source describes the approach as related to the MIT work, not the same method. The home experiment used a classic directional-ablation technique based on running the model and measuring its signals; it did not have an AI read and write the weights by reasoning over them. The results do not establish whether the edit would help other models or tasks.
Replication and Benchmark Results
The immediate next step is repeated testing under fixed settings, including the same sampling configuration for edited and unedited models and a stable loop detector. More runs would help show whether the change in completion rate is reproducible rather than an effect of run-to-run variation or setup differences.
Further evaluation would need to measure answer quality as well as completion, especially citation accuracy against the benchmark’s primary sources. Independent graders and tests on held-back questions could help distinguish a real improvement from grading or tuning effects. The reported account does not announce a schedule for such work, and it does not say that the new results will be added to the public Lexassay results page.
For now, the report supports a limited conclusion: a simple weight edit was associated with more finished answers in one comparison, but the model’s legal score did not improve. Whether a repeatable edit can reduce loops without changing other behavior—and whether that can help a model that already has the needed knowledge—remains to be tested.
Key Questions
What did the experiment change?
It removed a measured loop-related direction at half strength from the output weights of 27 of Ornith’s 40 layers. The report says the edit used code written by a Claude Opus 5.5 session and did not involve training or new legal data.
Did the edit improve the model’s legal score?
No. The model completed 12 of 15 answers after editing in the reported comparison, but its legal score stayed at zero. The answers contained 62 wrong or invented citations.
Why does the report mention 13 completed answers?
In a separate run using the model card’s recommended sampling settings and no weight edit, the original model completed 13 of 15. That result complicates claims that the edit itself drove the higher completion count.
Was this the same method used by the MIT researchers?
No. The report says the MIT work involved GPT-6 Astra reasoning directly about another model’s weights. This home test measured internal signals by running Ornith and applied directional ablation, a different and simpler technique.
Can the result be treated as proof that weight editing works?
No. It was a small experiment with one run per setting, a changing loop detector and setup differences. The reported evidence suggests a possible completion benefit in one comparison, but does not establish reproducibility or better legal accuracy.
Source: ThorstenMeyerAI.com
