A.I Made The California Bar Exam Worse (ft. Liz Dye)

Oh, you're complaining about the California bar exam. You're soft. Back

in my day, the California bar was a three-day marathon. Three days. None of

three-day marathon. Three days. None of

this pansy two-day nonsense you've got now. And oh, you want peace and quiet

now. And oh, you want peace and quiet while you take a life-changing test?

Well, I got news for you. Earthquake.

Don't like the earth moving under your feet while you're writing an essay about the rule against perpetuities? Well, now

you got to deal with a wildfire, son.

This is California. Oh, and the questions. Works of art written by

questions. Works of art written by honest to God humans. A perfect blend of gotcha traps, obscure fact patterns, and the lingering sense that you're being personally punished for something you did in a past life. Not this AI slop

that bar examiners are using now. Before

this year, no one in history has ever complained about the way one of the bar exam questions has been worded. And no

one has ever blamed the hardest bar exam in America for not passing. But law

students these days are weak. So,

congratulations and you're welcome. Liz

Dy explains how the California Bar Exam can't use AI to do their homework anymore. Thanks, Devin. The state of

anymore. Thanks, Devin. The state of California is going through some things after an attempt to save money on the bar exam went spectacularly sideways.

But I think people are being too hard on them. I mean, who among us hasn't gotten

them. I mean, who among us hasn't gotten several million dollars in the red, come up with a wild scheme to write the ship by switching to a cheaper supplier and a little assist from AI, and then wound up many more million dollars in the hole

with everyone mad at them. It could

happen to anyone. Maybe we should start by talking about how the bar exam is supposed to go. What's it like to take a test you have to pass to become a lawyer? In a word, it sucks a lot. You

lawyer? In a word, it sucks a lot. You

cannot take this test remotely and you will spend two days in a giant hall where they don't let you out and they don't let you eat and they don't let you wear AirPods. And in some places, they

wear AirPods. And in some places, they make you wear a suit, too. The bar is offered twice a year on the last Tuesday and Wednesday of February and July.

There's a February bar and there's a July bar and that is it. If you miss it or if you don't pass, you will not be allowed to practice law for another 6 months. So, it's very high stakes and

months. So, it's very high stakes and scary. Almost every state uses some

scary. Almost every state uses some portion of the Universal Bar Exam, which is a 12-hour test created by the National Conference of Bar Examiners or NCBE. The first day is the written

NCBE. The first day is the written portion. In the morning session, you

portion. In the morning session, you take the Multistate Performance Test where they supply you with materials to write a memo or a legal brief. And in

the afternoon, you take the multi-state essay exam, which consists of six 30inut analysis prompts. You can do all that by

analysis prompts. You can do all that by hand like we did back in my day, or you can pay $140 for the privilege of downloading the NCBE's testing software

onto your laptop. On the second day, you take the multi-state bar exam, the MBE, which is 100 multiple choice questions in the morning and another 100 in the afternoon. Some states want to test

afternoon. Some states want to test future lawyers on local law. And in most of those jurisdictions, they just swap out the multi-state essay exam for state specific questions. But everybody uses

specific questions. But everybody uses the MBE. Well, everybody but Puerto Rico

the MBE. Well, everybody but Puerto Rico and Louisiana. And Wisconsin admits you

and Louisiana. And Wisconsin admits you to the bar if you graduated from an accredited law school in the state. But

basically, every lawyer in the country became a lawyer by sitting for the MBE.

And if that sounds kind of like a monopoly, it's because it is. The

National Conference of Bar Examiners is a nonprofit that brings in close to $38 million a year and is sitting on roughly $175 million in cash. It built up that

war chest because it's the only game in town. So, we can charge whatever it

town. So, we can charge whatever it wants and what it wanted to charge California was a million dollars a year.

In April of 2024, the California Committee of Bar Examiners voted to ditch the UB entirely. The reason was not a mystery. They were totally broke.

The office of admissions said the admissions fund faces insolveny in 2026.

The state bar's adopted 2024 budget forecasts admissions fund deficit spending of $3.8 million. The fund is projected to end 2024 with just $3.3 million of reserves. And on top of the

million dollars a year they paid to NCBE for the multiple choice bar exam, they had to spend another $3 million to rent out convention centers all over the state and hire proctors. And with me

today to explain what went wrong is Joe Patrice, senior editor at the essential legal website above the law. Joe, can

you tell us why the California bar thought it would be a good idea to ditch the UBVE? Because they were broke. Uh

the UBVE? Because they were broke. Uh

because they had been using the UBVE and the with that the all the restrictions that the National Conference of Bar Examiners puts upon it. And that meant

in California renting large venues where everybody had to go to take this test in person. And those venues introduced tons

person. And those venues introduced tons of costs for the state bar. Also tons of cost on the examinees who have to get hotel rooms wherever the test is. And

eventually the state bar realized they were about to go bankrupt with it. uh at

which point they tried to figure out a way around it and they dillydallied far far far far too long. So when do you think they should have started?

Certainly years before they ultimately did it. Frankly, I think the CO

did it. Frankly, I think the CO situation was probably the best. That

gave a lot of folks a lot of time to think about different ways of handling licensing, but they didn't do that even though they knew that their numbers, their financial situation was worsening.

and the ultimately it had to be sometime before saying in May, you know what, let's do this in February. The plan was to hire a new vendor to rewrite the multiple

choice questions in time for the February bar. And if they ditch the UB,

February bar. And if they ditch the UB, the state wouldn't need to abide by the national conference's requirement that the test be taken in person, which would save them the expense of having to hire platoon of proctors and rent out

multiple convention centers along with alternate sites for people who needed testing accommodations. I myself took

testing accommodations. I myself took the bar exam in a law school classroom with five other pregnant ladies and so so many snacks. And like Joe said, most of the people taking the bar in

California probably have to book a hotel cuz California is a really big state.

And even if you live close to one of the exam sites, you don't want to be stuck on the freeway when that test starts at 8 a.m. cuz if you're not in your seat

8 a.m. cuz if you're not in your seat when the proctor says go, you just don't get to practice law for another 6 months. So, you're going to have to book

months. So, you're going to have to book a hotel for two nights in an expensive metropolitan area, which is extremely unpopular with test takers, particularly after COVID, when we'd all had a lot of

time to get used to doing everything online. So, the California bar proposed

online. So, the California bar proposed transitioning to an alternative, cost-effective exam administration approach, such as a fully remote online exam, utilizing small vendorowned test

centers, or a combination of the two.

They estimated that adopting one of these alternatives would yield savings between 2.8 8 million and $4.2 million annually beginning in 2025, enough to largely or fully eliminate the existing

structural deficit. And while they

structural deficit. And while they admitted that the impetus of this proposal is budget driven, they said applicants will also benefit from a change to a new administration approach.

And yeah, who wouldn't rather take the bar in their bedroom wearing pajamas instead of in some freezing convention hall surrounded by a bunch of stressed out wannabe lawyers? Like that sounds

like a win-win, right? The State Bar hired Kaplan North America, the parent company of the ubiquitous test prep outfit, to rewrite the multiplechoice portion at a cost of $1.35 million

annually, which is obviously more than the million dollars NCBE was charging.

But if California didn't have to book the convention centers in Anaheim, Oakland Pasadena Ontario and Chulabista, they'd save enough money to make it all worthwhile. They also hired

Majour Learning, an online test company, to administer the exam. Now, California

code says that any major change to the bar requires two years notice, but here the board said that wasn't needed because no alteration of the exam itself is being proposed. The only proposal on

the table is to change the vendor used for the exam development to allow for alternative exam administration methods.

The format and content will not change, which suggests that their plan was to essentially copy the multi-state bar exam on the cheap. Now, if you haven't passed the California Bar Exam, AI

questions or not, you're going to want a good lawyer. But if you want a great

good lawyer. But if you want a great lawyer, my law firm, the Eagle Team, can help. If you've gotten in a car crash,

help. If you've gotten in a car crash, suffered a data breach, especially got one of those data breach letters saying your information might have been leaked, or dealing with a workers's comp or social security issue, we can represent you or help find you the right attorney.

By the way, we don't get paid unless you do. So, there's nothing upfront. So,

do. So, there's nothing upfront. So,

just click on the link in the description or call the phone number on screen for a free consultation with my team. Because you don't just need a

team. Because you don't just need a legal team, you need the legal team.

Almost immediately, there were indications that they were going to have a problem. Kaplan put out a sample set

a problem. Kaplan put out a sample set of questions that contained multiple errors. Majour administered a mock exam

errors. Majour administered a mock exam that was buggy as hell. And just a week before the test, the bar offered all 5600 registered takers a full refund if

they wanted to back out. Again, backing

out meant a full 6 months more before they could get a license to practice law in the state. But still, about a thousand people took them up on it. The

state also promised to wave the fee for the July bar for anyone who failed in February. And on the day of the test, as

February. And on the day of the test, as feared, the platform crashed for hundreds of users. And look, it's important to understand the difference between the February bar and the July bar because if you finish law school in

May, you generally spend the next seven weeks intensively studying for the bar.

So the majority of people taking that test in July are taking it for the first time. But that's not true of the

time. But that's not true of the February bar. And it's particularly not

February bar. And it's particularly not true in California because even on the standardized multiple choice test, it sets the score required to pass really high. So the average passage rate for

high. So the average passage rate for the July 2024 bar for the whole country was 68%. In California was under 54%. So

was 68%. In California was under 54%. So

when this thing went sideways in February 2025 and the system crashed, it crashed onto a whole lot of people who were taking this test for a second or third time. Those are people who paid

third time. Those are people who paid tens or hundreds of thousands of dollars for law school and they put their lives on hold for months to study for this

thing. And I cannot imagine how awful it

thing. And I cannot imagine how awful it must have been for them. One test taker told Reuters, "I've never had this much despair and hopelessness." He said he switched internet connections and

laptops three times during the day on Tuesday. Since then, multiple people

Tuesday. Since then, multiple people were told that they failed when they actually passed. Some test takers were

actually passed. Some test takers were graded on other people's work, and a retest scheduled for the first week of March had to be rescheduled because of a prohibited online disclosure of an essay

question, better known as a leak. The

debacle has already spawned at least one class action lawsuit against Majour by test takers who allege that it failed spectacularly in its task to administer the exam because his platform crashed

and continually malfunctioned causing a disaster for test takers who were traumatized, who had their career ambitions delayed, and who paid Majour a fee for a defunct platform. The state of California is also suing Majour for

breach of contract and fraud.

California's law school deans were horrified and worried that the problem could not possibly be solved in time for the July bar. In a letter to California Supreme Court Justice Patricia Guerrero, they called on the bar to lower the

score needed to pass, require return to in-person testing, issue provisional licenses to practice law for people who failed, and most of all to go back to the UB. They said, "We understand that

the UB. They said, "We understand that the initial departure from nationally used multi-state bar exam was prompted by budgetary considerations caused in part by the need to rent large spaces

for test takers. The alternative exam has proven to cost far more than initially anticipated. So, the shift did

initially anticipated. So, the shift did not solve the financial problem, but likely exacerbated it while creating many others. Were the decision made to

many others. Were the decision made to return to the multi-state bar exam and California essays for the July exam, each of us would offer spaces in our respective campuses around the state to

the state bar at no cost. We would also be willing and committed collaborators in developing strategies to close any remaining gap, including by drawing on our network for proctors. The law school

deans echoed the bar committee's criticisms of majour, but they also called out Kaplan for typos and errors in the new multiple choice questions that mirrored those we saw in the practice questions published this past

fall, as well as disruptive conditions at the testing centers. On April 21, 3 months into this crisis, the state bar revealed that some of the multiple choice questions were developed using

artificial intelligence. They said a

artificial intelligence. They said a majority of the multiple choice questions were developed by Kaplan. A

subset of questions came from the first year law students exam and some questions were developed by ACS Ventures, the state bar's independent psychometrician. The ACS questions were

psychometrician. The ACS questions were developed with the assistance of AI and subsequently reviewed by content validation panels and a subject matter expert in advance of the exam. Okay, to

be fair, there were only 23 AI generated multiple choice questions out of 171 total. But the use of AI poses multiple

total. But the use of AI poses multiple problems. And you know what? I think the easiest way to explain this is by looking at a bar exam question. So

here's a sample multiple choice question from the multi-state bar exam generated by the National Conference of Bar Examiners. It says, "A man has four

Examiners. It says, "A man has four German shepherds that he is trained for guard duty and that he holds for breeding purposes. The man has beware of

breeding purposes. The man has beware of dog signs clearly posted around a fenced in yard where he keeps the dogs. The

man's next door neighbor frequently walks past the man's house and knows about the dog's ferocity. One summer

day, the neighbor entered the man's fenced in yard to retrieve a snow shovel that the man had borrowed during the past winter. The neighbor was attacked

past winter. The neighbor was attacked by one of the dogs and was severely injured in a suit against the man. Is

the neighbor likely to prevail? Okay.

The point of this question is to tease out whether the test taker understands some core concepts in tortton property law. I'm going to tell you right now the

law. I'm going to tell you right now the answer is A. But let's work from the bottom up. You can throw out D because

bottom up. You can throw out D because you're entitled to keep a pack of nasty dogs on your own property. Maybe

dangerous, but it's certainly not abnormal or illegal. And you can throw out C because a neighbor is not an inviteee. And if you just spent a couple

inviteee. And if you just spent a couple months studying for the bar, that would jump right out at you because typically an inviteee is a guest at a private home or a customer at a business. This is a guy who wandered in to get his shovel

back. So it's not C. And look, you can

back. So it's not C. And look, you can see this is a lot like a lot of standardized tests than that there are often two answers you can cross out immediately. Okay, so let's look at B.

immediately. Okay, so let's look at B.

If you're not a lawyer, maybe you think the neighbor is trespassing in the man's yard. But what this test is trying to

yard. But what this test is trying to assess is whether the prospective lawyer knows the difference between a trespasser, a lency, and an inviteee. An

inviteee is invited onto the property.

So if you invite my kids over for a slumber party, they're invitees. a

trespasser comes onto your property without permission. So, if I set up a

without permission. So, if I set up a tent and start camping in your yard, which is a kind of slumber party, I guess I'm a trespasser. The mailman is not a trespasser cuz he's allowed to be

on your property for some limited purpose. He can't set up a tent and

purpose. He can't set up a tent and sleep there, but he can put your mail in your mailbox. Most jurisdictions would

your mailbox. Most jurisdictions would consider the mailman to be a lency. In

this scenario, the man is a lency. He's

not an inviteee cuz he just wandered in to get his shovel back. So obviously not see. But if you borrow a snow shovel

see. But if you borrow a snow shovel from your neighbor and you guys know each other pretty well, you've implied that he can mosey over and get it back off your porch if you haven't returned it by July or whatever. And now we come

to a and that's a question about what duty of care a property owner owes to lences. It's assessing whether you know

lences. It's assessing whether you know that a property owner has a duty to warn of known hazards on his property, which the scenario says that he did. This guy

posted the sign, so he fulfilled his legal obligation. And the fact that the

legal obligation. And the fact that the neighbor ignored those signs and went in anyway, despite the fact that he knows about the dog's ferocity, means he assumed the risk and he can't recover for his injuries. This is not a great

use case for AI because look, you can certainly ask Chat GPT for the legal definition of a trespasser or to explain duty to warrant and premises liability and I bet you'd get the right answer,

but that's not how the bar exam works.

It poses a convoluted hypothetical and usually a couple of seemingly plausible answers that are trying to sus out whether you understand some precept of common law. The words lency, duty to

common law. The words lency, duty to warrant, and assumption of risk don't appear in the question at all. And if it were a lawyer asking chat GPT to formulate a hypothetical about bare

lenses and premises liability, you could probably get to this question or one that did what this is trying to do. But

if you had a lawyer, you wouldn't need chat GPT because by the time the lawyer composed the prompt, she could have just written a question herself. So that's

one of the problems here. ACS used non- lawyers guided by chat GPT to formulate some of the multiple choice questions.

ACS poses two defenses to this. First,

they say it's fine because they modeled their work on law school exam questions.

And leaving aside whether that's the best way to assess whether someone is fit to be a lawyer, it's not fair to change the format of such a high stakes test without being really transparent

about it. Remember, the law said you

about it. Remember, the law said you have to give two years notice if you're going to change the test. ACS also says that using AI is no big deal because the questions were subsequently reviewed by

a panel of psychometricians which is testing experts and subject matter experts which may be true but the problem is that the psychometricians and experts were paid by Kaplan. So, the

company was functionally checking its own work. On top of which, the state bar

own work. On top of which, the state bar fired experienced members of the advisory panel that weighed in on previous bar exams on the theory that their work vetting the NCBE posed potentially a conflict of interest or

maybe a copyright problem. And that

brings us to the third issue, which was pointed out by Professor Mary Basic, an assistant dean at UC Irvine Law School.

She was one of the professors kicked off the advisory panel because of those supposed copyright issues. But as she told the LA Times, there's a very limited set of practice questions out there to train an LLM on to produce a

bar exam. She said the place the

bar exam. She said the place the artificial intelligence would have gotten their information from has to be the NCBE questions because there's nothing else available. What else would artificial intelligence use? So, did the

state save money by ditching NCBE's expensive test and then pirating it with the use of AI? Because they promised that the test was going to be functionally the same. Although to be

fair, as Joe Patrice pointed out to me, Kaplan has been in the test prep business for a long time, including bar exam prep. They do know how to write bar

exam prep. They do know how to write bar exam style questions. They certainly

could have fed their own questions to the AI as a model. The state bar already adjusted the passing score down for February barakers and waved the fee for the July bar for retests. They also

decided almost immediately after the test that they would go back to an in-person model for the July exam. And

in May, the state supreme court decreed that they should go back to the multi-state bar exam for the multiple choice question. They did not, however,

choice question. They did not, however, take law school deans up on their offer to use their facilities to save money and ensure more convenient locations for test takers. So now there are just six

test takers. So now there are just six test sites for the entire state. Three

in the Los Angeles metropolitan area, one in Oakland for the Bay Area, one near Sacramento, and one in Chula Vista serving San Diego. For comparison,

Maryland, where I took the bar, has three locations for the bar exam. Fun

fact, Martins, the catering hall here, is a very popular site for parties. So,

it's entirely possible that people taking that test went to prom, got married, and took the bar in the exact same room. But Maryland has 6 and a4

same room. But Maryland has 6 and a4 million people. California has more than

million people. California has more than 39 million. California is also 13 times

39 million. California is also 13 times bigger than Maryland geographically. So,

you see the problem here. And as Joe pointed out, the state bar just dug itself into a deeper financial hole and threw out months of work. Well, no. It

doesn't fix the problem because it still requires them to rent large venues that even in even though they were able to get several large venues, it's still a

more limited list than they used to have, which means more geographically spread out than they probably want, which means more costs upon even in addition to the cost upon the state bar,

more costs upon applicants to be able to rent hotels and travel to a place and stay there in the middle of nowhere. Uh

not that not that LA is the middle of nowhere, but the point is outside of where they normally live for an extended period of time to take the test. Those

are all issues which is part of the reason why there's such a demand for some form of remote version of this test. And in addition, they throw away

test. And in addition, they throw away 10 months of work getting ready for some eventual remote or at least more geographically diverse test. Uh where

they have small maybe more smaller venues, which was part of the original deal. All of the work setting up that

deal. All of the work setting up that kind of model is now being tossed aside while they spend money going back to the old way. They're going to have to figure

old way. They're going to have to figure out some way of doing it. uh they might have to be in the red for a while, but that just underscores how they will eventually have to come up with a

solution. And that's more reason why

solution. And that's more reason why kicking it down the road is a bad idea.

So, congratulations, California Bar. You

played yourself. And to all the people out there cramming for the bar, may the odds be ever in your favor. Now, it's

particularly interesting to see how the media is covering this, especially because it's not the kind of story that makes the front page of the New York Times. 40 sources picked up on it with

Times. 40 sources picked up on it with about 45% coming from the left, 40% from the center, and less than 15% from the right. And when you examine the

right. And when you examine the headlines, you see very different frames emerge that shape how readers interpret and understand what this means for the legal profession. Of the few right-

legal profession. Of the few right- leaning sources that covered the story, most elected to simply repost the coverage written by journalists at the AP, a famously centrist organization that strives for nonpartisan reporting.

But on the left, sources were more likely to employ dramatic language like we see here. And then from the center, we of course see the widely cited AP article, but we also see a few original headlines that frame the admission as

just another bug in a problem playing process. The headlines you read first

process. The headlines you read first have immense power to shape how you understand the news. And that's why I like to see the news from as many angles as I can with the help of today's sponsor, Ground News. Ground News is a

website and app developed by a former NASA engineer on a mission to give readers an easy, datadriven, objective way to read the news. And right now, they're offering Legal Eagle viewers 40%

off their unlimited access Vantage subscription. Scan the QR code or use

subscription. Scan the QR code or use the link in the description to get a discount for a limited time. Ground News

gathers related articles from around the world and across the political spectrum so you can see the bias and compare coverage. Every story comes with a quick

coverage. Every story comes with a quick visual breakdown of the political bias, factuality, and ownership of the sources reporting, all backed by ratings from three independent news monitoring

organizations. One of my favorite parts

organizations. One of my favorite parts is their bias comparison feature, which highlights specific differences in reporting across the political spectrum.

That can be especially helpful when reading stories from sources you've never heard of, where there's no telling who owns it or what agenda they might have. I also love their blind spot

have. I also love their blind spot feature, which shows you stories that are under reportported by either side of the political spectrum. Like this

follow-up story that relates the bar exam fiasco to a long list of bureaucratic problems in California.

Surprisingly, this story was virtually ignored by all right-leaning media with 30% of reporting coming from the left and 70% from the center. This type of analysis helps explain why we often talk

past each other and why some people seem hyperfocused on what to others seems like nonsense or fake news. Ground News

is my favorite way to get the news. And

as a legal eagllet, you can save 40% off the ground news vantage subscription by scanning the code on screen or clicking on the link down below. This

subscription gives you unlimited access to every feature, including the bias insight tool and blind spot feed. And by

subscribing, you'll support an independent news platform working to make the media landscape more transparent. So click the link below or

transparent. So click the link below or scan the code today. And after that, click on this link for more Legal Eagle or we will see you in court.