🔍 Read the full analysis: Weight-Space Programming At Home: An AI Test With A Legal Score Of Zero on ThorstenMeyerAI.com
Get smart everyday buys delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
A home experiment altered selected weights in a local AI model to reduce repetitive loops. In one comparison, completed answers rose from 4 of 15 to 12 of 15, but the legal score remained zero; a no-edit settings change produced 13 completed answers. The test was small and does not show that weight editing improves legal accuracy.
A home test of weight editing increased the number of legal benchmark answers completed by a local AI model from 4 of 15 to 12 of 15 in one comparison, but its legal score stayed at zero. The experiment, reported by Thorsten Meyer on November 5, tested whether reducing repetitive loops could help the model finish answers; it did not show that the model could answer legal questions accurately.
The test altered Ornith 1.5 35B-A3B, a model described by its publisher as designed for coding and agent tasks. The researchers measured a direction associated with looping, then removed it at half strength from the output weights of 27 of the model’s 40 layers. They report that the edit involved no training and no additional legal data.
On the same setup, the edited model completed 12 of 15 answers, compared with 4 of 15 before the edit. Yet its legal score remained 0. The article says the 12 edited answers contained 62 wrong or invented citations—an average of 5.2 per answer—and met 7 of 75 checked criteria, compared with 3 before editing. These are results from the reported experiment, not evidence of legal reliability.
A separate run suggests the weight change was not needed simply to improve completion. With sampling settings recommended by the model card and no weight edit, the original model completed 13 of 15 answers, also scoring zero. The author says the edit was carried out using code written in a Claude Opus 5.5 session; a GPT-6 Astra session in Codex reviewed the code, figures and claims and identified faults. The article presents this as one experiment with an independent review, not as two separate experiments.
Infographic 01 · What we changed
One loop direction, weakened at half strength
Inspired by the MIT post, but a simpler, classic technique (directional ablation): code written by a Claude Opus 5.5 session measured one loop direction by running the model and projected it out of the weights. No training, and no AI reading weights as text.
The six steps
- Write test material that is not the benchmark: 24 legal development questions plus 8 general controls, 0 overlap with benchmark prompts
- Find the true loop onset at token level: from there, at least 90% of the next 1,500 positions repeat earlier 32-token sequences
- Run the model and measure one loop direction per layer: average internal signal inside the loop minus the reasoning just before it
- Weaken it in the weights:
W ← W − α·r·(rᵀW)on the MLP output matrices of all 256 experts and the shared expert - Bake an edited copy; the original files stay untouched and the edited weights stay internal
- Benchmark original and edit on the same local setup, 15 tasks, graded blind by three Claude agents
Quick test: share of pure copying
Restart 20 tokens before a known loop onset, 8 cases, 3,000 tokens. Lower is better.
The cost of the edit
- Full strength: 23 of 24 legal questions finished, but control answers broke: tokens dropped out of short formulas, and month counting went wrong
- Half strength: 23 of 24 finished, 9 of 9 checked control facts right, and one German control question was answered in Chinese once
- Before the edit: 8 of 24 legal development questions did not finish; all 8 controls did
One run per setting. The loop detector was revised three times during the experiment, so the development numbers come from a moving yardstick. The direction is a statistical description, not a single “loop neuron”.
Sources: our experiment records, 4–5 Nov 2026 (executing Claude Opus 5.5 session; review by GPT-6 Astra in Codex); Ornith-1.5-35B-A3B model card.
Powered by Thorsten Meyer AI
More Completed Answers, No Legal Gain
The results separate two abilities that can be mistaken for one another: finishing a response and getting it right. A model that no longer repeats itself may produce a complete-looking answer, but the reported citation errors show why completion alone cannot establish accuracy. In legal use, invented or incorrect references could mislead readers about what statutes or court decisions say.
The test offers a limited example of a possible role for weight edits: reducing a behavior that obstructs output. It does not establish that this technique improves legal knowledge, nor that it would work on other models or tasks. The no-edit settings run also matters: it completed one more answer than the edited run, indicating that sampling choices may affect completion as much as, or more than, the specific edit in this test.
The benchmark is described as checking cited statutes and court decisions against primary sources, with questions on German, US and EU law. That makes the zero score more informative than a simple count of finished responses, but the experiment remains small. It should not be taken as a comparison of legal products or as a basis for relying on AI-generated legal advice.
Products Worth Considering
Apple MacBook Pro MPHF3LL/A 14.2-Inch Laptop | M2 Pro 12-Core Chip 16GB Unified Memory 1TB SSD | 19-Core GPU Liquid Retina XDR Display Space
Dell XPS 13 9350 Copilot+ Laptop, 13.4-inch 120Hz Display, Intel Core Ultra 7 256V, 16GB RAM, 512GB SSD, Ubuntu Linux 24.04 LTS
Dell XPS 13 9350 Laptop, 13.4-Inch WUXGA Display, Intel Core Ultra 7 256V, 16GB RAM, 1TB SSD
As an affiliate, we earn on qualifying purchases.
A Local Test Inspired by MIT
The experiment followed a report by Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab at MIT CSAIL about what they call “weight-space programming.” In the reported Qwen example, GPT-6 Astra reasoned directly about another network’s weights and made a change associated with stopping loops; the source says the model’s maths score rose from 7.9% to 48.1%. The MIT approach and the home test are related in aim, but they are not the same method.
In Meyer’s account, the MIT system read and changed weights directly. The home test instead ran Ornith, recorded internal signals, calculated a direction, and projected it out of selected weights—a technique the article calls directional ablation. The stated hypothesis was that an edit might remove an obstacle hiding an existing capability, but would not supply knowledge the model lacked.
Ornith was selected because its benchmark runs showed looping: five of 15 earlier answers did not finish within a 62,000-token limit, and the author reports that only 2% to 13% of lines in the final third of those transcripts were distinct. Other tested models either did not show the same looping problem or could not be tested under comparable conditions. The experiment used a benchmark of 15 questions, with ten published and five kept hidden, according to the source.
“The model finishes is not the answer is right.”
— Thorsten Meyer, describing the result
Small Test, Open Questions
The findings are limited by one run per setting, a small development set and a loop detector that changed during the experiment, according to the author. The source also says a change in the local setup alone shifted the original model’s completed-answer count from 10 to 4. That makes it difficult to attribute differences confidently to the weight edit rather than settings or other run conditions.
The source says Claude carried out the edit and Claude agents performed blind grading; it also reports that three graders came from the same model family. GPT-6 Astra provided a review and found faults, but the article does not describe that review as a replication. The exact impact of those faults on the reported scores, and whether the results would persist across repeated trials, remain unclear.
The report does not establish performance on the five hidden benchmark tasks separately, or show results from a larger independent evaluation. The benchmark figures describe this particular test and should not be generalized to other legal questions, models or real-world use.
Replication Is the Next Test
The source does not announce a scheduled follow-up experiment. The next useful evidence would be repeated runs with the same model, token limits and sampling settings, alongside a stable loop detector and clearly documented grading procedures. That would help determine whether the completion differences are reproducible and whether they come from the edit rather than the setup.
Further testing would also need to evaluate accuracy independently, including the correctness of citations and performance on held-back questions. Until such results are available, the reported weight change is best understood as a limited test of loop reduction—not a demonstrated way to improve legal answers. The article explicitly says the experiment is not legal advice.
Key Questions
What did the weight edit change?
It removed a measured loop-related direction at half strength from the output weights of 27 of Ornith’s 40 layers. The account describes this as directional ablation, not direct reasoning over the weights or model training.
Did the edit improve the model’s legal accuracy?
No improvement was reported in the overall legal score: it stayed at zero. The article reports 62 wrong or invented citations in the 12 completed edited answers.
Could settings changes explain the higher completion count?
They may have contributed. With recommended sampling settings and no edit, the original model completed 13 of 15 answers, compared with 12 of 15 in the edited comparison. The small test does not establish which change caused the differences.
Was this the same method used in the MIT research?
No. The source describes the MIT work as an AI reasoning directly about model weights. The home test used model runs and recorded internal signals to calculate and remove a direction from selected weights.
Can the result be used to judge AI for legal work?
No. It was a small experiment on one model and is not legal advice or a broad evaluation of legal AI. The reported zero score and citation errors do not support relying on the model for legal answers.
Source: ThorstenMeyerAI.com
