We gave nine AI models real legal work from closed matters and graded it against what the lawyer delivered.
Some of their mistakes fade as models get stronger.
Others do not. On three points of professional judgment, the four strongest models got it wrong 90% of the time.
1
8
For a law firm:
Type 1 is a buying decision. Test models on your own matters and your own law.
Type 2 is not. When to advise, how to weigh a group relationship, what a client should never be asked: those are a firm’s standards. Give them to the system, then test it.
1
3
The full piece, with all 14 points and how each of the nine models did on each:
prysmfoundry.com/matterbench…
3
Today we're releasing MatterBench, the first legal benchmark built from real work and graded against what lawyers actually delivered.
The best model scores just 8.8%. Not one attempt passed a long horizon task.
Other legal benchmarks are written to mirror real work. MatterBench is built from the real work: client matters that lawyers delivered. The real instruction, the real matter file, the lawyers' reasoning, graded against what went to the client.
Can AI run a corporate due diligence, or take a fintech through its license application? Not yet.
tldr: Lawyers aren't going anywhere. Yet.
Leaderboard: prysmfoundry.com/matterbench (1/5)🧵
How we write the checks:
Every check comes from how the work was really done. The lawyers delivered these matters with our AI agents: the instructions they gave, where they disagreed with the AI, what they corrected, and what they changed in its first draft. Then what came back: the partner's redline, the client's feedback, the regulator's comments.
1,282 checks. 249 are critical: the points a partner would reject the work for. (4/5)
Where agents fail:
• They raise problems that are not there. A license business plan was ready to file. 89% of attempts called it unready and raised issues the regulator never had.
• They miss what is missing. 86% did not work out what an unregistered pledge of treasury bills means. 86% did not flag a missing guarantee on a renewal promissory note.
• They know the common-law canon, not the local statute book. 34 of 36 attempts missed all six local-law points the lawyer relied on.
Every attempt on 4 tasks is public, step by step: prysmfoundry.com/matterbench…
Blog: (5/5) prysmfoundry.com/matterbench…



