🔍 Read the full analysis: What’s Behind Z.ai’s Legal AI Ranking? Its Benchmark Pass Rate In Context on ThorstenMeyerAI.com
Get smart everyday buys delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Z.ai says its GLM-5.3-FlashX-Legal model achieved a 19.2% all-criteria pass rate on Harvey’s Legal Agent Benchmark. Vals AI’s archived leaderboard records 19.17%, seventh among 81 models; the same evaluation shows a 92.5% pass rate on individual criteria, a different measure. The result is a benchmark score, not evidence that the model can safely complete legal work without human review.
Z.ai’s GLM-5.3-FlashX-Legal recorded a 19.17% all-criteria pass rate on Vals AI’s leaderboard for Harvey’s Legal Agent Benchmark, ranking seventh of 81 models in the archived results. The score means the model passed every required criterion on about one in five evaluated tasks—not that it earned only 19.17% of the available individual criteria.
The distinction is central to interpreting the result. Harvey’s benchmark marks a task complete only when every rubric criterion passes. A task can therefore count as a failure if the system misses one detail, such as a deadline or citation, even if it gets most of the work right. Vals AI’s leaderboard lists GLM Legal at 19.17% of tasks, at a reported cost of $0.15 per test.
The same Vals evaluation reports a 92.5% pass rate on individual criteria. That figure uses a different unit from the task-level score and does not mean that 92.5% of complete legal work products were acceptable. Z.ai rounded the task result to 19.2% and said the model improved by 12.5 percentage points over the base GLM-5.3-Flash model. The leaderboard figures support that comparison: 19.17% versus 6.67%, a difference of 12.50 percentage points.
Z.ai describes the model as tuned for legal research, contract review and drafting, including producing initial drafts. Its documentation says it underwent legal-domain mid-training and post-training, with reinforcement learning across tens of thousands of task environments modeled on legal workflows. The company recommends connecting it to tools for laws, cases and company data. Its pricing page lists $0.37 per million input tokens and $1.25 per million output tokens; those are listed rates, not the leaderboard’s per-test cost.
What the Score Means for Legal Teams
The score matters because legal work often depends on details that cannot be averaged away. A document may be broadly persuasive but still expose a client to risk if a citation is wrong, a deadline is omitted or a required issue is missed. Harvey’s all-pass method is designed to treat each task as complete only when every criterion is satisfied. That makes its task pass rate a demanding measure of end-to-end performance, but it also makes the result easy to misread as a general accuracy rate.
For law firms and legal departments, the leaderboard offers a point of comparison, not a procurement verdict. The listed $0.15 per test is lower than the costs shown for several higher-ranked systems in the same archived table, but price alone says little about the cost of review, correction, data handling or integration. Buyers would need to examine performance on their own matter types and establish how client information is processed and protected.
Human review remains a practical issue. The benchmark measures defined tasks under a specific grading setup; it does not establish that outputs can be filed, sent to clients or relied on without checking. The source article’s author also described a four-task spot check in which the legal model cited rulings the author could not locate. That small check is not a benchmark or proof of a general failure rate, but it reinforces why citations and legal authorities need independent verification.
Products Worth Considering
LG PuriCare AeroHit Smart Air Purifier for Large Rooms up to 1000 sq ft, True HEPA Filter with 360° Airflow, PM1.0 Sensor & LED Air Quality
Logitech MX Brio Ultra HD 4K Webcam, Graphite
Samsung Odyssey G4 25-inch FHD Gaming Monitor
As an affiliate, we earn on qualifying purchases.
How Harvey Grades Legal Tasks
Harvey released its Legal Agent Benchmark as an open-source benchmark on May 6, 2026. It contains 1,250 tasks spanning 24 practice areas. Each task supplies an instruction and client matter materials, then asks an agent to produce work for review. Harvey says the tasks are graded against more than 75,000 expert-written, binary criteria covering matters such as facts, legal conclusions, citations, recommendations, deadlines, amounts and formatting.
The leaderboard result comes from a private holdout set of 120 tasks, rather than the public task set. According to Vals AI, internet access is disabled during testing and two language-model judges grade each submission. Each judge calculates a task pass rate, and the reported score is their average. Vals’ October 18 archived table places GLM Legal seventh; leaderboard positions and results can change with later versions or added systems.
Harvey’s own May 2026 baseline offers context, though it is not a direct comparison with the later Vals run. Harvey reported a top all-pass score of 7.1% among the frontier models it tested in that baseline and said those models completed fewer than 10% of tasks end to end in aggregate. Different models, runs and evaluation conditions mean the numbers should not be treated as a controlled head-to-head comparison.
“A task is marked complete only if every criterion passes.”
— Harvey, describing its benchmark grading rule
Limits of the Published Results
The ranking does not establish how the model performs across all legal matters or in day-to-day firm use. It reflects a 120-task private holdout, a specified grading process and the systems included in the archived leaderboard. The documentation does not identify every operational detail of the evaluation, while Z.ai’s October 13 blog post names Vals as the evaluator. The benchmark result is not a substitute for testing on a firm’s own work.
Z.ai calls the result “state-of-the-art performance among open-source models,” but the cited announcement does not set out a complete comparison group or explain the basis for classifying this legal model as open source. The source article reports that it found no published weights for the legal model. The claim should therefore be read as Z.ai’s characterization, not as a conclusion independently established by the score alone.
The source article also reports that its author found no “not legal advice” notice on the Z.ai pages reviewed and that the materials describe outputs as drafts for legal professionals. Those observations are limited to the pages examined. The article’s four-task spot check is likewise too small to establish a general rate of citation errors. Details about client-data handling, deployment safeguards and the review required before use remain material questions for prospective customers.
What Buyers Should Verify Next
The next concrete reference point is the leaderboard itself: Vals’ archived version was updated October 18, 2026, and its later HLAB-Vals-1.1 listing was updated October 30. Readers should check which version and task set a quoted score refers to, since rankings can change as evaluations are updated. Z.ai’s AutoClaw announcement, dated October 13, said beta access was available by application; the source material does not confirm the current application status or a broader release schedule.
Before using the system, legal teams can ask Z.ai about data retention, where client materials are processed, whether inputs are used for training, and what controls govern connections to external legal databases. They should also test the model on representative tasks and require lawyer review of citations, authorities and material conclusions. No further benchmark results or independent validation are specified in the supplied material.
Key Questions
What does GLM Legal’s 19.17% score measure?
It is the share of tasks on which the model passed every required criterion, averaged across two AI judges in Vals AI’s evaluation. A task fails the all-pass measure if even one criterion is missed.
Why is the criteria pass rate 92.5%?
The 92.5% figure counts individual criteria, while 19.17% counts whole tasks completed with every criterion met. They use different denominators and should not be treated as conflicting scores.
Was GLM Legal the top model on the leaderboard?
No. In Vals AI’s archived leaderboard updated October 18, 2026, GLM Legal ranked seventh of 81 models. The ranking applies to that archived version and may not describe later results.
Does the benchmark show the model is safe to use without review?
No. The benchmark reports performance on a defined set of tasks. It does not establish that outputs are ready for client use or legal filings without professional verification, particularly of citations, deadlines and conclusions.
What remains uncertain about Z.ai’s ‘open-source’ claim?
Z.ai described the result as state of the art among open-source models, but the supplied material does not specify its full comparison set. The source article says it found no published weights for the legal model, leaving the basis for that label unclear.
Source: ThorstenMeyerAI.com
