Thirteen Firms and the Roar of the Crowd

Legora's ROI report says 42% of customers won new work. That's 13 firms out of 31. Hand-picked. With a margin of error that, in an election poll, would span a wipeout and a landslide.

You’ll have seen a number of posts and references to Legora’s ROI report. They’re speaking to their customers, trying to understand what’s happening, which is the right thing to do. Many of these posts cite some very precise, and interesting, numbers. Across AmLaw, Magic Circle and top international firms:

• 42% said Legora helped them win new work;
• 45% said it helped them expand existing client relationships; and
• 39% said they could take on work that would previously have required more people, more time, or simply saying no.

People cite Jevons paradox with excitement (of course). Legora’s CEO Max Junestrand also says that Legora’s customers are hiring more lawyers, and offers the same three numbers as his evidence (the report says 31 firms, his post says 30). Most, not all, commentators are ecstatic.

So let’s be precise.

Small and selected sample

The report is built on 31 firms. 42% is 13 of them. No survey needs to reach the entire population to be valid; a 1,000-person poll speaks for 200 million voters. 31 interviews may provide useful qualitative guidance, but the percentages cannot be treated as representative population estimates without an appropriate sampling design. This would be the case, if, but only if, the 31 were drawn at random from the population they are supposed to represent. There is no evidence that this is the case. Participants were likely recruited by Legora, and the report doesn’t say how. It also doesn’t say how many firms were approached, how many declined, or whether firms seeing no return were as willing to give an hour to an ROI interview as those seeing strong results. None of this we know, and all of it we would need to know to interpret the results with any confidence.

Who was interviewed is disclosed, and worth keeping in view: 55% of participants are innovation and AI leaders, 26% are partners. For the most part, the people asked whether the tool paid off are the people who chose to buy it. There is no comparison group of non-adopters either.

Huge confidence interval

The 95% confidence interval on 13 of 31 runs roughly ±16 percentage points. What this means is that the share of firms in the population this report speaks for (more than 90% of participants come from firms with over 200 lawyers, so presumably Legora’s larger law firm customers) that would say ‘yes’ lies somewhere between roughly 26% and 59%. That is close to “could mean anything”. For reference, election polls seek 1,000+ respondents to get to approximately ±3 points, and even then polls regularly miss their mark.

100% − 42% = 58%

If 42% of interviewed firms reported winning new work, then 58% did not. To the credit of Ari Kaplan, the report’s author, he says so explicitly:

“more than half (58%) have not won new work as a result of using Legora, or have not yet seen those results, and 55% have not seen an expansion in existing client relationships.”

“Have not won new work, or have not yet seen those results” is not the same as “have lost work”. But it isn’t “the pie is growing” either. It is simply too early to tell, which Kaplan also acknowledges. But then why are we getting excited about pseudo-precise statistics?

The 42% itself deserves a closer read. The exact wording is that Legora “has helped them win new work, either directly or indirectly”, and the quotes underneath are about including Legora in pitches and clients liking the sound of it. That is a fine thing to report. It is a different thing from a measured (and sustained) revenue effect.

Most participants gave only general descriptions of how they measure time savings; one explicitly said its firm did not formally measure them. The USD 6.9m of “potential additional yearly billing per 100 lawyers” is a formula (lawyers × self-reported hours saved × hourly rate × 46 weeks) sitting on top of those general responses.

And of course, Jevons

As one would expect, because it’s trendy, commentators also find their way to Jevons. Junestrand’s version: “Better steam engines made Britain burn far more coal. That was coal, not law, but the mechanism is the same.” Is it? Watt’s engine made a unit of steam power cheaper. As a result, demand for steam power grew so much that Britain burned more coal in total. The paradox is a claim about the input factor.

Translated to law, the coal is lawyer time, the efficiency gain is AI. Jevons then predicts more lawyer hours consumed only if demand for legal output is elastic enough. The pie has to grow, and it has to grow faster than the price of a unit of legal work falls. Both have to hold at once, and nobody has shown they do.

More plausible to me is the picture I described in June: some segments of legal work Jevons hard, others don’t move, and prices for some services come down while the market rates for others go up. Whether this ends up a net positive for the firms in this report is, you guessed it, too early to say.

Legora customers hire more lawyers

The most tenuous claim attached to this report is that Legora customers are hiring more lawyers. “Will AI mean fewer lawyers? Our customers are hiring more of them,” Junestrand writes, and the evidence that follows is the 42%, the 45% and the 39%. The report contains firm-size bands but no hiring figures or longitudinal headcount data for the 13 firms that reported winning new work or for any of the others. Firms saying that a tool helped them pitch is a long way from firms hiring, and the post offers nothing else. What the report does contain points a calculating business person in the opposite direction: 29% of interviewed firms say they can scope matters more tightly with fewer professionals, and one participant, asked what losing Legora would mean, answers: “It would be disruptive, since we are able to do more work with fewer professionals now.” Same dynamic as with Jevons: unless business growth outpaces the efficiency gain, firms will be run with fewer professionals.

Oh, you luddite!

Anyone who’s made it this far might conclude this is a hit piece on Legora, that I oppose AI in the legal profession and would rather things stayed as they were. In fact neither. I’m excited and astonished, and of late mildly terrified, by AI. I use it every day and wouldn’t want to live without it. What I could absolutely live without, however, are bullshit arguments and the latest legal AI laser pointer that people skitter after.