I'm reading "AI Outperforms Law Professors in Stanford Law Study/In a rigorous blind study, law professors overwhelmingly preferred AI-generated answers to student legal questions over answers written by fellow law professors—and flagged the AI answers as potentially misleading or harmful far less often" (Stanford Law School News and Announcements).
June 5, 2026
"The study, titled 'Law Professors Prefer AI Over Peer Answers'... was conducted with 16 law professors across U.S. law schools and tested..."
"... whether large language models could serve as effective tutors for contract law courses. In a blind evaluation of nearly 3,000 anonymized comparisons, professors rated AI responses significantly higher than answers written by other professors, with AI winning 75% of head-to-head matchups.... The study is particularly notable because previous AI evaluations have focused primarily on subjects with clear right-or-wrong answers. Legal reasoning, by contrast, demands careful analysis of competing arguments and defensible conclusions.... 'These weren’t just simple questions with obvious answers. Many of them required synthesizing complex material, applying it to new situations, and explaining legal concepts in ways that would help students develop their own analytical skills.' Participants created 40 representative contracts law questions that students might ask after class or during office hours, wrote their own answers, and then evaluated responses without knowing whether they came from AI or other participating professors. The AI systems performed comparably to the best human instructor in the study...."
I'm reading "AI Outperforms Law Professors in Stanford Law Study/In a rigorous blind study, law professors overwhelmingly preferred AI-generated answers to student legal questions over answers written by fellow law professors—and flagged the AI answers as potentially misleading or harmful far less often" (Stanford Law School News and Announcements).
I'm reading "AI Outperforms Law Professors in Stanford Law Study/In a rigorous blind study, law professors overwhelmingly preferred AI-generated answers to student legal questions over answers written by fellow law professors—and flagged the AI answers as potentially misleading or harmful far less often" (Stanford Law School News and Announcements).
Posts
22 comments:
Professor Buggywhip
…that sounds like the kind of payoff from AI disruption we’ve all been dreaming of…well, most of us have…
Ice Man Fellow…
The problem is that people expect anything good from a law professor.
They are the people who couldn’t make it as lawyers.
It is no coincidence that the most incompetent people in every field also tend to be professors and democrats.
So did the AIs not Hallucinate?
Or did the Law Professors checking the answer not bother to validate the cites?
This is why LLMs should replace universities.
They are cheaper and they produce better results than the mediocrities that are the vast majority of university professors.
The people who are at university to actually learn will get more help from an LLM and an internship/work study at fractions of the cost.
I was watching the Stanley Cup game last night in a casino, so I couldn't hear it well. There was a highly contested call at one point, and I asked some AI on my phone what happened. Despite me prompting with the correct year, the teams and "tonight's game", it answered about last years game with different teams. I told it that was wrong, it apologized and answered but with the outcome of the call backwards, and the next answer got the teams backwards. After getting it wrong 4 times, it admitted "Yes, I got everything completely wrong, would you like me to try again?" I don't know any humans that stupid, but I still use AI every day. Maybe I'm the stupid one.
This is probably true, the answers I get have improved a lot, and that isn't even using specialized models, but I still say that if you fed it jurisprudence or whatever you call laws and judicial opinions, from 1850, AI would rationalize and defend human slavery. Maybe that's fine, maybe lawyers should be constrained, and probably no AI lawyer would have come up with a way to fine Trump hundres of millions of dollars for valuing his property according the actual market in Palm Beach, rather than using the assessed valuation, using a century old law that had never been interpreted in this way, but then again, I am thinking that these models will be "tunable."
It seems to me that you could, for any subject, have AI be your professor at a significant cost reduction and much faster progress. AI could develop the course, do the teaching, and test you. You just couldn't sleep with it to get a better grade.
A basket of correlations and agent of consensus.
That's awesome...lol. And the prices for "higher" education just keep getting higher. University of Wisconsin Board or Regents just approved a 2% tuition increase for 2026-2027. This after a 5% increase last year for 2025-2026.
So the folks who did a worse job answering contract questions were the ones who judged the quality of the answers provided by others?
When human novelists are replaced by AI novelists, or AI-assisted novelists, it will be because human readers have themselves been replaced: either readers who are truly AI (good luck getting them to pay for anything) or AI-infected humans who have lost interest in creativity.
Lawyers, on the other hand, have a problem. Few people need or want a creative lawyer.
It seems to me that you could, for any subject, have AI be your professor at a significant cost reduction and much faster progress.
At a minimum, as we learned during COVID, you could have the best contracts professor in the country do on-line lectures for every law student in the country, with the local schools providing opportunities to ask questions and grade exams.
"Few people need or want a creative lawyer."
Not so sure. If you really need a lawyer, you probably want a creative one.
In a blind evaluation of nearly 3,000 anonymized comparisons, professors rated AI responses significantly higher than answers written by other professors
Fear will kick in. If the study wasn't blind you know damn well the profs would have picked professor responses over AI. "Intellectuals" will soon be vilifying AI in mass. Expect many "Duke 88" signed statements.
For the profs who participated in the study, this has to be a 'holy shit' moment.
It's like when the Napa Valley vineyards Château Montelena and Stag's Leap Wine Cellars, won the Judgement of Paris in 1976.
After that, all that remained was the .... arrogance.
" If the study wasn't blind you know damn well the profs would have picked professor responses over AI"
And even as it was — "rigorously blind" — I bet the professors looked for those AI tells — like em dashes, "it's not, it's," and "delve" — to try to go against AI.
They might have even expected that the AI answer will be better written and organized and held that against the answer.
“You have right to AI. If you cannot afford one, one will be appointed to represent you.”
If AI can deliver "better" answers than the human professors, I think that there might be an issue with the human professors. AI is designed to tell people what they want to hear. So I wonder what the human professors felt their answers were supposed to achieve? After all, sometimes, the best way to help someone is to tell them something that they don't want to hear. Also, by writing their own questions instead of using real student questions, they may have injected a bias into the study.
Searched the entire paper for "hal". There were 6 hits. None were "hallucinations".
I'm not impressed
AI case law searching is still rifled with hallucinations.
"Professor Buggywhip"
Captain Dunsel.
Post a Comment
Please use the comments forum to respond to the post. Don't fight with each other. Be substantive... or interesting... or funny. Comments should go up immediately... unless you're commenting on a post older than 4 days. Then you have to wait for us to moderate you through. It's also possible to get shunted into spam by the machine. We try to keep an eye on that and release the miscaught good stuff. We do delete some comments, but not for viewpoint... for bad faith.