Rethinking legal education in the AI era(law.uchicago.edu) |
Rethinking legal education in the AI era(law.uchicago.edu) |
Some bits I found interesting:
[For required courses in year 1]
> Across all 1L sections, we will prohibit the use of electronic devices such as laptops, tablets, and phones in the classroom.
> Additionally, examinations will be in-class without access to the internet, electronic files, or apps. And most of all, we will continue our longstanding tradition of emphasizing the Socratic Method as part of these courses.
> Throughout the year, students will write without AI, while also using AI for research, revision, iterating on drafts, and preparation for oral argument. Students and their instructors will review together both their writing and their use of AI.
> For these courses, the use of the Socratic Method, no-device policies, and in-class, no-access exams will remain, but as default rules rather than required policies.
> In all elective courses, we will encourage instructors to experiment with pedagogy.
> We will be adding one additional requirement, which is that all students will be required to engage in an oral discussion of their SRP with their supervising professor, in an in-person setting.But the effect is to make success more dependent on networks. (If they are using AI as an excuse to make networks more relevant than skills, it relieves competitive pressure.)
First, a main draw of law journals is the filing cabinet of past exams as remembered by past students, available only to journal members i.e., friends of connected people. Reducing one's ability to remember what was in a lecture only increases the advantage insiders have.
Second, the "Socratic" Method for teaching law is nothing like it is in philosophy; it's more of a weird combination of hide-the-ball and distraction that again ensures people who can see through the clouds are using out-of-band information, and share it only with those they seek to enlist or join.
> all students will be required to engage in an oral discussion of their SRP with their supervising professor, in an in-person setting
I'm surprised anyone could get through otherwise. When did it become legitimate to dump your paper and not answer for it?
That all being said, I would be entirely unsurprised if the "mythology" of "YES the journals have the secret fancy sauce that no one else can get" lived on.
Also -- and I don't necessarily want to denigrate law school professors, it's another thing that's just gone on so long that "this is how we do it" -- yes, the law School "Socratic method" (at worst) is an effective form of possible professor laziness; they can just "ask questions about the cases" and not really teach.
Related to how law school "textbooks" aren't textbooks. They're literally just a bunch of cases slapped together with a tiny bit of commentary (unless that's maybe changed too, I hope?)
Again, not all professors, you CAN still teach very well under this framework. -- but it is an out for those fancy people who's forte isn't teaching.
How so? In my view, the socratic method is one of the few useful parts of law school. Most people, even quite smart people, think in a fuzzy way. Their ideas can seem superficially compelling in written form, but often fall apart when you start questioning their assumptions and chain of logical reasoning.
The biggest problem is that most law schools don’t use socratic method, because it’s a lot of work for teachers. Just cold calling a student and asking them to explain what happened in a case isn’t socratic method.
On the other hand, the paperwork-first side of law is probably in for a tough future. Especially if your clients are not big corporations or rich celebrities.
In general I think my advice to young people is: focus on social skills, public speaking, etc. These are not going to be replaced by robots in your lifetime.
That already happened! https://youtu.be/MkmfZPt-gaw?si=gk1k3sGsFq9npp0X
But then why do I need a lawyer?
I do watch Travis's videos. Ever since he integrated AI, the video quality dropped immensely and it became really stupid, compared to his older videos. AI degraded the quality and discussions he used to have with law enforcement too, it's quite sad actually.
Are expectations way too high in the near term for tech companies and all of the crap that they’re talking? 100% totally agree. But this is “just another an AI cycle” and if you’re a practitioner it does look different than the previous ones.
However this kind of statement was not released for crypto, or for the Internet. Those were massively impactful technologies.
What you are actively seeing happening now are institutions saying “we have to change the way that we practice” “the way that we did business before is no longer as applicable as it’s going to be changing very rapidly in the future and we’re going to have to deal with that.”
That is not coming from investors or technology companies. That is coming from people who are utilizing this stuff and being impacted by it
So it is absolutely the case that there’s going to be some kind of financial changes but just like every other AI cycle we are left with a aggressively more capable technology that is more pervasive.
Everybody needs to be taking that seriously.
Lawyers are going to use ai, because it's easy. The only time they won't is when the AI won't come up with a plausible enough excuse. The lawyer will always have a job, because a non lawyer using the same arguments will be ignored.
So these are rules to maintain the appearance that the system has integrity.
And that's the annoying thing. They know what integrity and justice and fairness look like, just as the rest of the justice system does, but they ignore it all.
At this point, it would be preferable for both sides to put their argument to an AI and let the AI decide what is fair. Wouldn't be any worse than judges who were themselves lawyers looking after their own.
Sorry for the rant, I've had experience of the 'justice' system.
I would never expect Harvard Law school to admit this, their hands are tied by many interests, but IMO they still aren't being pragmatic enough to evolve.
I would make the courses 50% AI based immediately, the other 50% being based on validating and applying the AI's work with traditional rigorous work. Anyone who rejects this should drop out, because they won't have a career anyway. And if Harvard Law disagrees then they can find out how meaningful their degrees are going to be in 10 years.
“Lawyer” is one of many possible career paths. Law graduates are uniquely qualified as premium law-aware generalists. In my opinion, there is no better education to prepare for future business uncertainty.
This is many people at many jobs. We continue to see this weird thing when AI comes into an industry that suddenly the product being produced by the people was perfect/amazing/whatever. Maybe that's true for the majority of HN who are lucky enough to work with experts in every field, but in the general population that's simply not the case.
Many doctors, lawyers, and programmers are not very good at their jobs - I've seen it first hand. AI gives the general population a way to steer around these people a bit, and maybe ask the right questions. Is it perfect? Nope, but it's often better than the people someone has access to.
So, on one hand you can get an expert take that might be wrong, but is linked to an actual person, with a reputation and some level of ownership. On the other hand you have an over confident LLM that might be wrong and has no reputation, no ownership. How does that actually improve things compared to the older status quo?
I think this extends to the other areas that AI is also disrupting.
Software engineers are also absolutely terrible: In being sloppy with handling errors, doing coercions, etc. All because always engineering after best practice simply takes too much time and mental load.
The real disruption from AI is likely that it does not get mentally fatigue. It can keep on, launch adversarial audits, make sure that everything is up to all standards.
It's worse than mental fatigue. A friend had GLM 5.2 suddenly become dumb for hours. It started forgetting instructions, randomly inserted some bits in simplified chinese... Apparently throttled due to demand. there was tons of tokens left. (btw when your llm runs out of tokens is another example that's worse than fatigue)
A tired programmer usually knows he's tired (and can take a break) and fucks up in more expected ways
I hope the irony is lost on no one that "Attention Is All You Need" was the paper that kicked the LLM boom off.
Part of the distinction between a how good one lawyer will be vs another is exactly that: attention to detail. And, as a trade, a practice, it seems like a skill that they try to hammer into the people who pursue this career (to varying degrees of success clearly).
It makes the last 20 years of leet code hiring look downright wrong. Your ability to recall what sorting algo to use, or solve some brain teaser conveys nothing about your ability to read (massive amounts) of code, and think critically about it.
Let's face it, only thing lawyers are there for right now is to take liability. And even that may not be needed 5 years from now.
Middlemen that can be automated will be automated.
At the end of the day, I think these paralegals have been quite weak links, and nearly nobody will notice a difference in quality when it's all handed over to LLMs. But the paralegals will lose their jobs, and probably have nothing better to do than purchase angle grinders and cut down Flock camera poles, or crown the front hood of Waymos with a traffic pylon.
finally, was this thing you beat the actual lawyer at in their specific specialty, since lawyers are often focused on particular aspects of the law?
IME, many lawyers who represent others also have fools for clients.
Additionally, you haven't given any of the details of your case, so I'm going to assume you were simply in a position where the facts of your case weighed heavily in your substantive favor, and no amount of legal maneuvers by the defendant would have made a difference.
There's nothing magical about lawyers. They are mostly just boring, studious people who excelled in non-STEM academics and were good at memorization and test taking. If you have a strong case, it won't matter very much who the lawyer is on the other side.
So? I beat actual lawyers in trials and hearings five times between 2008 and 2019. That last one was actually a team of attorney + advocate (senior bar member).
You could always "beat" them just by reading and preparing. Now, you+AI beat a lawyer. Tomorrow, do you really think that you+AI will beat lawyer+AI?
And once we stop getting well-trained lawyers (because everyone is just a meatbag accessory to an AI), whether you win or not is likely to be a coin toss.
Taking law as the example, I'm not sure I agree.
Legal is largely a distress purchase. Very few clients actually want to spend time interacting with outside corporate counsel, a divorce lawyer, a personal injury lawyer, etc.
The real value of an attorney is accountability. A person with a license and reputation who is on the hook for the advice they provide. But accountability is only worth buying if the judgment behind it is sound, and that judgment takes years to build.
The problem is that much of what makes a "good" lawyer comes from years of doing unglamorous work as a junior, financed by clients. If AI reduces the ability of firms to subsidize training the next generation of lawyers, you get fewer "good" lawyers and the whole market keeps eroding.
From what I've seen, the top AI models can now produce contracts that are on par with what I would have had to pay thousands of dollars for a decade ago, and the agents/chatbots are capable of collaborating with you to address fairly complex issues. If you have the money, you still run everything by a real attorney of course for the accountability piece, but the future is not bright for contract attorneys because people know even they're probably using AI to do half their work.
It might be worth distinguishing between LLMs as tools for fuzzy-searching hard facts, versus using them to craft logical arguments. [0] The risk-profile and verification difficulty are rather different.
[0] Or, more-precisely, using them to craft a pattern-fitting text artifact, which hopefully maps onto a sane concept when reinterpreted by a human.
One of the things that AI has helped with is automatic breech notifications. They are largely proforma, but depending on which generation of contract they are notifying from, subtle changes are needed.
Law is mostly highly structured text, and you know what loves highly structured and documented text? LLMs.
Legal LLMs do share the same issue with coding LLMs, they tend to default to lazy/insecure. Its not the case where you as a non-lawyer can prompt it to make a "water tight" contract from scratch.
How has crypto been massively impactful?
The point I’m making is a financial bubble can burst while also massively changing the world
I'd say it's pretty impactful.
Yes they were. Educators have been talking about how access to the internet has affected their students learning since forever. We were cautioned and had dedicated classes to teach us to be critical of different sources, teaching us what Wikipedia could and could not be used for. The internet was deliberately turned off for our exams.
Crypto too was seen as culturally important, papers were written by otherwise well respected authors. We were expected to read some whitepapers about crypto. Interestingly, reading those papers only made us more sure it was a scam.
You're rewriting history to make this moment seem different, it is not.
I wish people would stop being so dense.
A bubble doesn’t say anything about how useful the underlying technology is. We are in an AI bubble, in the sense that the allocated resources is completely out of line with the underlying businesses. AI companies would literally need the entire world as their TAM, using their services non stop to justify the valuations and level of expenditures we are seeing. What you describe is literally the symptoms of a bubble
That’s not true. Apple alone is valued higher than the largest three AI companies combined.
This is what people mean when they say it's a bubble.
And that when trying to be a lawyer will usually fail when you have an emotional stake in the outcome. Even if you're the best lawyer in the world, you need someone practical and dispassionate to make sure they can see your blind spots.
You asserted that AI companies would need "the entire world as their TAM using their services non stop" to justify a few trillion in valuation. Yet Apple has a $5 trillion valuation just selling phones and computers, mostly to consumers and not businesses. Google built a $2 trillion valuation by the end of 2021 mainly on search and ads. Our economy is very large and you don't need to take over "the entire world" in order to justify a few companies with $1 trillion valuations.
Apple and alphabet are both trading at something like 10x revenue, with very high operating margins.
On the other hand you have AI vendors: OpenAI $852 valuation is ~34x revenue, with $14B losses projected for 2026. Their infra commitment through 2030 is more than $600B (that's on the low end of numbers floating around).
Just for the infra expenditure, if we assume a conservative ROIC of 5%, 8% operating margin: they need an operating profit of $30B/y, and a revenue of $375B/y. That's 100% of the projected 2030 AI infra market.
Edit: actually, their 2026 losses seem to be even higher, the $14B number is from a January leak, but more recent projections are at >$22B losses for 2025
6 months ago everybody talked about how AI was not able to plan their work. They could only do minor changes.
The introspective property of knowing when you are "tired" and acting appropriately will likely be trivially observed in 6 months.
operators will never expose this guess to the user (who wants to pay for a tool that complains and refuses work?) so they probably will use it to load balance differently. and if they are out of resources then they can't load balance so they'll just hope you won't notice, so we're back to where we started. maybe it's already roughly how it works now...
what's most likely in 6 months is a big correction in pricing, once they get people hooked up on this enough. maybe there will be a model that manages to detect when it's stupid but it won't be available outside FAANG and DOD
Naturally, LLMs do not get fatigue. As such they also don't need to detect it. That is correct. But LLMs can see if their performance have fatigue-like deterioration and correct for it.
That case is even more trivial and has nothing fundamentally to do with LLMs. Capacity is added everyday.
I would not base my long term projects on what happens in the market based on the fact that a single provider needs to throttle to satisfy demand now.
Your assumption that the current pricing is not sustainable might be fair. Personally I believe the opposite, and I havde not seen any indication that inference should not continue to decline in price.
In particular, LLMs appear to be hyper commoditisable. So if Anthropic or OpenAI is doing pricing shenanigans, people will likely move on.
> "smart guy with access to AI tools" can perform as well as any other trained lawyer
You are seriously understating the ability of a trained lawyer and overstating the ability AI tools.
What I am saying is that going to schools no longer is the only way for smart people to be able to do the job of a lawyer.
That is what AI will do, and firms will figure out how to adapt. To be fair, this trend has been going on for some time now already. Clients want attorneys with experience and judgment to be on the hook in real trials, mergers, etc. There’s only so many of those folks. The trend of the last several decades has been pretty steady overall headcount at firms despite large increases in the amount of legal work, and ever increasing demand for the limited set of seasoned attorneys.
Firms have not done a great job cultivating the next generation against the backdrop of those trends. But that’s true in a lot of white collar jobs. And AI will make that problem more acute.
But—lawyers are a transaction cost and it’s good for the economy to automate as much of it as we can!
I guess it depends on how you look at it.
I don't think anyone would argue that spending money on lawyers is an ideal use of capital. But if AI-driven automation reduces overall demand for legal services (as measured by billable hours) and/or erodes the pricing power of lawyers/law firms, it doesn't mean that the "savings" will benefit all participants in the economy.
The biggest beneficiary will be corporations, as reduced legal expenditure can boost corporate profits. But that is a boon for the wealthy shareholder class, not the worker class. "Lawyer" has been one of the highest-paying white collar jobs available so the question becomes: as all of these high-paying white collar jobs become less plentiful, how do young people not born into wealth develop opportunities to get ahead economically?
How would you know? Unless you have expertise in this area, you can't really know if you have a good contract until you have to litigate it.
> you still run everything by a real attorney of course for the accountability piece
Your attorney can make better use of AI than you can. A growing time sink for lawyers (and I assume, physicians) is having to explain to clients that "yes, I understand this is what Claude told you," and "no, that's not actually how X works in practice."
I ran multiple businesses for over 20 years, and during that time had to deal with countless contracts involving commercial transactions, leases, funding events and employment. I worked closely with outside counsel, and had to deal with numerous disputes and lawsuits.
Many contracts are boilerplate to begin with, which is why firms like Cooley and Orrick provide free document libraries and generators. They know that a huge segment of the people who might retain them aren't going to pay $10,000 for an NDA or a simple commercial contract. So they give documents away as a marketing tool, hoping that it might get them more complex work in the future.
Attorneys can add value when you have a unique wrinkle and of course when there's a dispute, but, as an example, if you need an employment contract for California, you don't need to pay someone $500/hour to draft one from scratch and frankly for many contract categories, you'd probably be taking on more risk by having a bespoke contract drafted than modifying a solid, widely-used template.
> Your attorney can make better use of AI than you can. A growing time sink for lawyers (and I assume, physicians) is having to explain to clients that "yes, I understand this is what Claude told you," and "no, that's not actually how X works in practice."
You're not wrong, but I think you're misunderstanding what I'm saying.
First, as far as contracts are concerned, the leading AIs are more than capable of producing pretty decent ones for many categories because they've been trained against all the real contracts that are out there. Your "yes, I understand this is what Claude told you but..." scenario is real, but it applies less and less than attorneys want to believe, especially when the subject is mundane.
Second, the bigger issue is that AI is fundamentally hindering the development of the next generation of lawyers. It's great that senior associates and partners can use AI to be more efficient, but the economic pressures AI creates mean that it's far harder for law firms to have clients subsidize the training of the next generation of lawyers.
So the question isn't whether an experienced attorney today can make better use of AI than you can, it's whether a new attorney today will be able to become a "good" lawyer when AI is eroding their firms' ability to have clients subsidize their training.
For contracts, I was thinking of the more complex deals where a battle-tested template is unavailable. Though even where one appears to fit, it doesn't always. One of the difficulties in professional/client relations is that there are often details that one party sees as minor or completely unremarkable, and the other party sees as changing the whole frame of analysis. I haven't been too impressed with LLMs being able to identify things like this -- they have reams of knowledge, but no practical sense.
The issue about halting the training pipeline is an interesting one. In law firms, a lot of "training" of young lawyers is really glorified bates-stamping charged out at professional rates. I think we could find that the work just gets more substantive and interesting. And yes, pyramid-shaped firms do rely on this type of stuff for revenue. But if all parties can do more sophisticated work, then AI may only nudge forward an arms race without fundamentally changing much (at least for work that has adversarial backpressure).
they are not human so human sensations don't apply. but I heard when you run out of tokens it's also kind of like fatigue, just less predictable
It is entirely clear that is it not the case when I compare agent generated apps with what contracted software teams have produced.
For medicine it is likely worse.
Doctors do care, but they have to give an advice based on their 15 year old knowledge - they simply can't read through 83 papers in a quick session.
I think it is a matter of time before we see the first insurance companies assign greater risk to human advice (legal, tech, medicine, etc.) than to agentic advice.
You likely have to change your idea about this. Heck, this view was wrong 6 months ago. Sticking with is becomes a hazard to patients.
That's not my argument... I didn't draw the conclusion that because of those 2 factors humans generally provide better answers. I personally have no idea if that's the case, and for sure wouldn't rely on my personal feelings to evaluate that
> So, on one hand you can get an expert take that might be wrong, but is linked to an actual person, with a reputation and some level of ownership.
Regardless, I agree.
The newest studies still works on llms that are two years old. It doesn't appear that proper medical harnesses with frontier models has been evaluated.
My slight intuition is that we will already now see results that are much better than average human doctors.
That's the whole point, yes. LLMs are inherently unreliable. Both human experts and LLMs are unreliable in their own ways. The human expert has a reputation and some level of responsibility, the LLM doesn't
ftfy
But this is the exception, not the rule because most people are not engaging in "complex" deals that require completely bespoke contracts. The average individual or company needs roughly the same things almost every other individual or company needs and a LegalZoom template is going to be perfectly usable for the 99%, ideally with a final pass by an experienced attorney.
What you're paying for when you do have a "complex" deal is the negotiation and structuring, and that's where the experience of the attorney is all the value.
> The issue about halting the training pipeline is an interesting one. In law firms, a lot of "training" of young lawyers is really glorified bates-stamping charged out at professional rates. I think we could find that the work just gets more substantive and interesting.
You can't run before you know how to walk. I don't know how a new associate becomes the next Clarence Darrow, Johnnie Cochran, or David Boies without thorough exposure to the often-boring basics: legal research, document drafting, document review, organizing and sorting documents, sitting in on calls and depositions, etc.
If clients aren't willing to subsidize these types of activities and firms have to look to automate more to maintain their profitability in the face of cost pressures, the development model breaks down long-term.
But maybe I was too subtle in my previous comment. I absolutely agree with you. What I am trying to say is that these schools trying to come up with policies about tests and applications in the age of AI have nothing to do with education. They are all just grasping at whatever straws to try to keep the idea that their institutions are somehow special.