🔍 Read the full analysis: Z.ai’s Legal AI Ranks High Despite Passing Just About 1 In 5 Tasks on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Z.ai’s GLM-5.3-FlashX-Legal recorded a 19.17% all-criteria pass rate on Vals AI’s evaluation of Harvey’s Legal Agent Benchmark, ranking seventh among 81 models in an archived leaderboard. The score is an all-or-nothing measure: the model passed 92.5% of individual rubric criteria, but missed at least one criterion on most tasks. The result and Z.ai’s claim of a 12.5-percentage-point gain over its base model are supported by the listed figures; its claim of leading open-source models is harder to assess because the comparison set is not specified.
Z.ai’s GLM-5.3-FlashX-Legal ranked seventh among 81 models on an archived Vals AI leaderboard, with an all-criteria pass rate of 19.17% on Harvey’s Legal Agent Benchmark. That means the model met every rubric requirement on about one in five tested tasks; it passed 92.5% of individual criteria, a different measure that can conceal a missed requirement within an otherwise strong answer.
The model, also referred to as GLM Legal, is built on GLM-5.3-Flash. Z.ai describes it as tuned for legal work including research, contract review, document understanding and drafting. Its documentation says it received legal-domain mid-training and post-training, including reinforcement learning across tens of thousands of task environments modeled on legal workflows. Z.ai recommends connecting it to services providing laws, cases and company information.
Vals AI’s archived Version 1 leaderboard, last updated October 18, 2026, lists GLM Legal at 19.17% and a cost of $0.15 per test. The listed base model, GLM 5.3 Flash, scored 6.67% at $0.57 per test. The difference is 12.50 percentage points, matching Z.ai’s stated improvement. Those figures describe this leaderboard run; they do not establish how the model performs in every legal setting.
The benchmark’s scoring rule explains the gap between the task and criterion results. Harvey says a task counts as complete only when every required criterion passes. A response that meets most requirements but misses a deadline, citation or recommendation fails the task-level test. Vals says its leaderboard uses a private holdout of 120 tasks, disables internet access and averages task pass rates from two AI judges, GPT 5.5 and Claude Sonnet 4.6.
Why Task-Level Accuracy Matters
The result highlights a practical distinction for firms considering legal AI: high partial-credit performance is not the same as completed legal work. Vals reports that GLM Legal met 92.5% of individual criteria under its scoring approach, but the all-criteria measure shows that a substantial share of answers still missed at least one requirement. In legal workflows, a single omitted fact, incorrect deadline or unsupported citation can require a professional to rework the output.
The ranking is evidence of progress against the listed models, not evidence that the system can independently handle legal matters. Z.ai’s reported $0.15 per test is also a benchmark cost, not a full estimate of deployment expense: its pricing page separately lists $0.37 per million input tokens and $1.25 per million output tokens. Firms would also need to account for review, data handling and integration requirements before comparing costs with professional work.
The source article describes a four-task spot check in which its author found rulings cited by GLM Legal that could not be located. The author also reports that OpenAI and Claude models in that small check invented no rulings, while two Claude models cited a real ruling for the opposite of its holding. This was not a benchmark or systematic comparison, and it cannot establish general error rates. It does underscore why citations and legal conclusions need verification.
Products Worth Considering
Apple Watch Series 8 GPS, 41mm, Midnight Aluminum Case with Midnight Sport Band (S/M)
Logitech MX Keys S Wireless Keyboard, Graphite
Bose SoundLink Plus Portable Bluetooth Speaker, Wireless Outdoor Speaker, Up to 20 Hours Battery Life, Waterproof and Dustproof, Black
As an affiliate, we earn on qualifying purchases.
How Harvey Scores Legal Agents
Harvey released its Legal Agent Benchmark on May 6, 2026. The public set contains 1,250 tasks across 24 legal practice areas. Each task provides an instruction and client matter materials, then asks an agent to produce work for review. Harvey says the tasks are assessed against more than 75,000 expert-written, binary criteria covering items such as facts, conclusions, citations, recommendations, deadlines, dollar amounts and formatting.
Harvey’s all-pass rule is intentionally strict: an answer must satisfy every criterion for the task to count as complete. Its published example is a deal-team report that identifies eight of ten risks: under that rule, it is materially incomplete rather than an 80%-complete task. The public leaderboard results come from a 120-task private holdout that Vals says mirrors the public task mix. Because the evaluation is limited and uses a defined test set, it should be read as a comparison under that protocol, not a forecast of performance across all legal work.
Z.ai announced the model in an AutoClaw blog post dated October 13, 2026, which opened a beta by application; Vals lists the release date as October 12. Z.ai also calls the result “state-of-the-art performance among open-source models in this evaluation.” The source material says the company does not specify the comparison set and that it found no published weights for the legal model, leaving the meaning of “open-source” and the scope of that comparison unclear.
““an all-criteria pass rate of 19.2%””
— Z.ai, in its model documentation
Limits of the Legal AI Result
The leaderboard gives a result for a 120-task holdout, but the supplied material does not provide enough detail to determine how scores vary across practice areas, task types or levels of difficulty. It also does not explain whether the two AI judges agreed on individual task outcomes. Vals says the final result averages their task pass rates, so the reported score is not a direct count of universally agreed passes.
Z.ai’s description of the model as state of the art among open-source systems is not independently established by the information provided. The company does not name its comparison set, and the source article reports finding no published weights for GLM Legal. The benchmark score itself is listed by Vals, but the broader label should be treated as Z.ai’s claim.
The supplied material does not settle how client information is handled when the model is used, what data retention or jurisdictional rules apply, or what safeguards are available to firms. It also does not establish the frequency of citation errors in routine use. The four-task spot check is anecdotal and cannot answer that question. Z.ai’s pages, according to the source article, describe outputs as drafts for legal professionals; the article says the pages it reviewed did not carry a “not legal advice” notice.
Questions Before Legal Deployment
Z.ai’s AutoClaw post opened a beta by application, but the supplied material does not give an access schedule, participant numbers or a date for general availability. The next public signals to watch are further leaderboard updates, more detail on the model’s comparison set and any release of weights or technical documentation that clarifies its availability and evaluation.
For law firms and legal departments, the immediate next step is not to treat the rank as a substitute for review. Procurement teams will need answers about client-data processing, retention, applicable rules and auditability, as well as how lawyers will check citations and substantive conclusions before a draft is used. Additional independent testing across practice areas would help show whether the benchmark result carries over to the tasks firms actually assign.
Until those details are available, the confirmed picture is limited but useful: GLM Legal scored above its listed base model and placed seventh in this Vals AI run, while completing every rubric criterion on fewer than one in five holdout tasks. How the model performs under supervised use, and whether its cost and controls meet professional requirements, remain open questions.
Key Questions
What does GLM Legal’s 19.17% score mean?
On Vals AI’s evaluation of Harvey’s benchmark, a task counted as passed only when every required rubric criterion passed. The 19.17% score means GLM Legal met all criteria on about one in five tasks in the evaluated holdout.
How can it pass 92.5% of criteria but only 19.17% of tasks?
The measures use different units. A task can include many criteria, and missing just one makes that whole task a failure under the benchmark’s all-or-nothing task rule. The 92.5% figure is the criterion-level result reported by Vals AI, not the share of tasks completed.
What was GLM Legal’s leaderboard rank?
Vals AI’s archived Version 1 leaderboard, last updated October 18, 2026, placed GLM Legal seventh among 81 models, at 19.17% and $0.15 per test. Rankings are specific to that evaluation and archived leaderboard version.
Did GLM Legal improve on Z.ai’s base model?
Yes, in the listed leaderboard results. GLM Legal scored 19.17%, while GLM 5.3 Flash scored 6.67%, a difference of 12.50 percentage points. That comparison applies to the reported benchmark run.
Does the result show the model is ready to replace legal professionals?
No. The score measures performance on a defined benchmark, and the available information does not show reliable performance across all legal work. The model is described as producing drafts for legal professionals; human review, citation checking and clear data-handling terms remain central questions for any deployment.
Source: ThorstenMeyerAI.com
