🔍 Read the full analysis: Legal Questions Put A Law-Focused AI Model To The Test on ThorstenMeyerAI.com
Get smart everyday buys delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
A legal AI test by ThorstenMeyerAI.com reports that GLM-5.3 FlashX Legal scored 8 out of 100 across 15 questions, while leading general-purpose models scored higher. The test’s author says it is a spot check, not a formal benchmark, and discloses that his company runs the test and Claude agents graded answers.
GLM-5.3 FlashX Legal, a model marketed by Z.ai for professional legal workflows, scored 8 out of 100 in a 15-question legal AI test published November 4 by ThorstenMeyerAI.com. The test’s author says it did not fully solve any task, gave no answer on two, and produced citation errors; the results raise questions about relying on legal chatbots for deadlines or legal claims.
The test, called Lexassay, compared 42 AI setups on questions involving German, U.S. and EU law. Its author, Thorsten Meyer, said GLM-5.3 FlashX Legal scored zero on five hidden tasks and received credit for 21 real citations, while 25 citations were misattributed and one was invented. The model’s base version scored 3 out of 100, according to the test.
Meyer said that on two German-law questions, the legal model used its full stated reasoning allowance of 64,000 tokens but returned no answer. On another question, it described a four-figure compensation amount as conceivable after data scraping, while Meyer said the German Federal Court of Justice has used around €100 for the mere loss of control over personal data. The test’s account is an assessment by its author, not a ruling on what a person would recover in a specific case.
Mistral Large 4, which its maker describes as state-of-the-art among open models for enterprise work including law, scored 10. Meyer said it got about half the substance right but lost most of its points because of incorrect citations. The reported leaders were Claude Opus 5.5 at 79, GPT-6 Astra at 72 and GPT-6.1 Sol at 71. Even the top-scoring model fully met the criteria on only eight of the ten public tasks, according to the report.
Infographic 01 · The leaderboard
Same 15 legal questions, very different results
German, US and EU law, no internet, no tools. Every cited law and ruling checked against the source.
The top · 67 to 79
The middle · 41 to 59
The weak group · 0 to 35
Selection of 14 of 42 configurations. Score 0–100 = share of achievable points, with deductions for wrong citations. Small local models (1–12) not shown.
A small spot check: one answer per model and task, no tools, no web. General information, not legal advice. Disclosure: Thorsten Meyer AI runs Lexassay; Claude agents grade blind.
Source: Lexassay, release of 4 Nov 2026.
Powered by Thorsten Meyer AI
Why Legal Citations Matter
The test highlights a practical risk: a fluent answer can still misstate the law or cite a source that does not support its claim. In legal matters, deadlines can determine whether an appeal is accepted, and a wrong citation can make an answer appear more dependable than it is. The report therefore advises readers to check cited statutes and decisions against primary sources and not to rely on a chatbot for a deadline.
The findings do not establish how these models perform when connected to legal databases, supplied with case documents or used by trained lawyers. Lexassay tested models without internet access or tools, aiming to measure their standalone answers. Its results concern that setup, not every possible legal AI product or workflow.
Products Worth Considering
Apple MacBook Pro MPHF3LL/A 14.2-Inch Laptop | M2 Pro 12-Core Chip 16GB Unified Memory 1TB SSD | 19-Core GPU Liquid Retina XDR Display Space
FLEXISPOT 60-Inch Executive Standing Desk with Drawers and CPU Cabinet
FLEXISPOT 60 x 30-Inch Bamboo Electric Standing Desk with White Frame
As an affiliate, we earn on qualifying purchases.
How Lexassay Tested the Models
Lexassay used 15 legal questions: ten published and five hidden. The questions covered German, U.S. and EU law, including deadline calculations, recent legal changes, court decisions and prompts based on false assumptions. For hidden tasks, the site publishes the score but not the question and answer, according to Meyer.
Answers were graded against reference answers with three to six weighted criteria, and citations were checked against official databases. A false or misattributed citation reduced a task’s points; an answer without a legal-advice disclaimer or direction to consult a lawyer lost 5%. Meyer said each setup produced one answer per task, making the work a spot check rather than a formal benchmark; small score differences may be noise.
Meyer disclosed that ThorstenMeyerAI.com runs Lexassay and that several Claude agents graded the answers. He also noted that Claude Opus 5.5 ranked first, creating a conflict of interest. The test’s method page and public answers, reference answers and citation checks are published, he said, so readers can inspect the grading.
“Optimized for professional legal workflows, with a focus on document understanding, legal reasoning, and task execution.”
— Z.ai, in developer documentation as quoted by ThorstenMeyerAI.com
Limits of the Reported Scores
The results do not show how the models perform with web search, legal research tools, case files or lawyer review, because Lexassay ran them without internet access or tools. The source describes one answer for each model and task, so performance on another run may differ. It does not provide enough evidence to generalize the scores to every legal question or jurisdiction.
The figures on citation checks aggregate published runs, not just the legal-focused model: Meyer reported 2,644 citations confirmed as real and 804 found to be invented or inconsistent with the answer. The article does not give a rate for all citations in the supplied material. The author’s financial and grading disclosures also matter when interpreting results, even though he says the underlying materials are available for review.
The source material ends during its discussion of GLM-5.3 FlashX Legal’s ranking on another test, so no further ranking detail is included here. It is also unclear whether Z.ai has responded to the Lexassay findings.
Check Sources Before Acting
Meyer recommends treating chatbot answers as a starting point, checking every cited law and court decision in its original source, and asking a lawyer when a deadline, money or an authority is involved. Readers facing an actual appeal deadline should not use this test or a chatbot as a substitute for legal advice.
The immediate next step for readers assessing the report is to review Lexassay’s published methodology, answers and citation checks. Further comparisons would be needed to establish how the models perform across repeated runs or when provided with legal research tools. The source does not announce a date for another test or report a response from the model makers.
Key Questions
What did GLM-5.3 FlashX Legal score?
It scored 8 out of 100 across Lexassay’s 15 legal tasks, according to ThorstenMeyerAI.com. The author reported zero on the hidden set and said the model did not fully solve any task.
Does the test prove that legal AI cannot be used?
No. The report tests models answering without internet access or tools and describes the results as a spot check. It does not assess every legal AI product or workflows involving lawyers and legal databases.
Why did citations affect the scores?
Lexassay checked citations against official databases and deducted points when a reference was invented or did not support the model’s claim. The method was intended to account for answers that sound plausible but cite the law inaccurately.
Who ran and graded the test?
Thorsten Meyer said ThorstenMeyerAI.com runs Lexassay and that several Claude agents graded answers. He disclosed a conflict because a Claude model ranked first and said the method and public materials are available for review.
Should someone use a chatbot to calculate an appeal deadline?
The report says not to rely on a chatbot for a legal deadline. It recommends checking sources and consulting a lawyer when a deadline or other legal consequence is involved.
Source: ThorstenMeyerAI.com
