AI and job skills: what a study of 133 patent attorneys found

Do patent attorneys get better with AI, or do they lose their touch? A three-month experiment on 133 legal professionals gives a mixed answer, and it was run by a team that works at Google. Juniors, for their part, show no measured gain on the test without AI.

Key takeaways

• 133 attorneys from 11 firms: 90 got access to an AI writing tool, 43 didn't.
• Over three months, the AI group's drafts are better: about 0.33 standard deviations higher, mostly because the weak drafts disappear.
• On a review done without AI, the seniors who had the tool do slightly better. Among juniors, there's no average difference.
• The study hasn't been peer-reviewed yet, and Google built the tool being tested.

A tool for writing patents

A patent rests on its "claims," very precise legal sentences that define what's protected. That's where mistakes get expensive. So the researchers asked the attorneys to write dependent claims, then a full description.

The AI group used an unreleased writing tool developed by Google Labs. The control group worked with standard legal tools, without generative AI. The drafts were then blind-graded by other specialized attorneys.

On the first task, which lasted ten days, the AI group finished about ten minutes faster. The control group took about 110 minutes.

Better drafts, murkier review

The study has a second task: reviewing an AI-generated draft without using AI. That's where the results get complicated. The seniors who had the tool do slightly better than the control group. Among juniors, meaning those with less than seven years of experience, the average difference is zero.

The detail that changes the picture: an audit flagged 15 submissions out of 91 as possibly done with AI, even though the review was supposed to be done without it. Without those 15 cases, the seniors' advantage remains, but it's smaller, and the overall gap is no longer statistically significant.

The limits, and who funded the study

The sample is small, especially for juniors, and the study only covers patents. The firms were working for a demanding client, which limits how far the results generalize. Three months is a short time to train an expert, and the researchers couldn't measure each person's starting level.

It's a working paper published by the NBER, the US National Bureau of Economic Research, and it hasn't been peer-reviewed yet. Six of the seven authors work at Google, which developed the tool. That doesn't make the numbers wrong, but you should know it when you read them.

For comparison, the MIT study on ChatGPT covered 981 people followed for four weeks. There too, the signal exists, but it doesn't tell the whole story about the cause.

Progress or losing your touch?

On draft quality, the tool helps both groups, juniors included. On the review done without AI, it doesn't clearly help juniors. Hence the mixed verdict: AI helps produce good text, but the evidence that it trains beginners is still weak. If you're learning a profession, write your own drafts first, then compare them with the tool's.