AI Reasoning · New York · Decrypt
AI Lawyers Are Already Better Than Law Professors at Reasoning—Say Law Professors
Compiled by KHAO Editorial — aggregated from 1 source. See llms.txt for citation guidance.
★ Tier-1 Source
Law professors preferred answers generated by artificial intelligence over answers written by fellow professors, according to a recent study led by Stanford University that examined how large language models perform on legal reasoning tasks.
Key facts
- In the study, 16 professors from 14 U.S. law schools—including Stanford, Yale, New York University, the University of Chicago, Georgetown, UCLA, and the University of Virginia—created 40 contract law
- AI-generated answers were also flagged as harmful less often than those written by professors, with Gemini recording a 3.41% harmfulness rate and NotebookLM 3.64%, compared with 12.06% for human
- In March, the Los Angeles Superior Court began testing AI tools to help judges manage growing caseloads, while law schools are adding AI training programs
- In 2,918 blinded comparisons, professors selected the answer they would rather give a student
Summary
Law professors preferred AI-generated contract law answers over those written by fellow professors about 75% of the time. Researchers said the results show that large language models can align with professional standards. In the study, 16 professors from 14 U.S. law schools—including Stanford, Yale, New York University, the University of Chicago, Georgetown, UCLA, and the University of Virginia—created 40 contract law questions covering legal doctrine, case law, hypotheticals, and policy issues. “Large language models (LLMs) are increasingly promoted as educational tutors, yet most evaluations focus on domains with a single ground truth,” the researchers wrote.