Understanding the inner thoughts of AI

Welcome to Google DeepMind, the podcast.

I'm professor Hannah Fry.

What if you were to peer inside the mind of AI?

You wouldn't find fully formed thoughts or intentions written in plain English, just vast arrays of numbers combining together in ways that somehow produce intelligence.

How? We genuinely don't know.

And that is the problem a field called interpretability is trying to solve mapping meaning onto those numbers.

Shining a light inside of the black box.

In this episode, I am joined by Neel Nanda, who leads the Language Model Interpretability team here at Google DeepMind.

Thank you so much for joining me.

Do you want to give us your definition of what interpretability is?

and also why we need it, maybe.

Sure.

So interpretability is kind of the neuroscience or the biology of AI.

And it's trying to understand how these things work.

often called opening up the black box.

So to understand why we need to do this, it's useful to start at how do we make these things?

How do they work?

And in particular, neural networks are more grown than designed.

No one designs what a network like Gemini should look like.

Instead, we have these enormous mountains of data, and we have this flexible learning algorithm, the neural network that starts just kind of doing stuff randomly.

But then we keep giving it a bit of data and then giving it a nudge to do a bit better next time.

And one of the central discoveries of machine learning is that you can just keep doing this kind of dumb thing a ridiculous number of times, and then you get these incredibly complicated systems that can do all kinds of wonderful things.

But at no point in this process did someone say what Gemini should look like.

It just emerged from this stacking of millions of nudges.

And I think that's quite a good analogy here to evolution.

No one designed the human brain.

Instead over like hundreds of millions of years organisms were nudged, as it were, towards survival by natural selection and the small nudges accumulated over time into the rich complexity of, you know, the biodiversity on the world today and the job of the biologist is essentially

to reverse engineer what evolution has learned.

Likewise, the job of an interpretability researcher is to try to reverse engineer what neural network training has learned.

So what got you into this then?

How did you come to be part of the interpretability community?

I think there were two main factors for me, a safety factor and a scientific factor.

So on the safety side, I think AI is progressing extremely fast.

I think it's pretty plausible that in the next decade or two, we'll have human level AI.

AGI.

And I think this has a lot of potential to be extremely good for the world, but also it's a pretty dramatic change.

And I think changes like this come with a lot of risks, and it's pretty core to making this responsibly that we try to understand how to do it safely.

And the more we understand about a system, the better a place we're in.

The more we can understand why it does what it does, debug issues flagged risks in advance, etc. The scientific motivation is I’m kind of a scientist at heart.

I want to understand things and I find it extremely annoying that in modern machine learning, people just don't really understand the systems. And I know it just seems like obviously the most important question is how do these things work?

What is going on?

And I get paid to try to answer this questions.

Great.

What was the original goal of interpretability?

I mean, did people ever really want to and or expect that you could properly connect up the dots from the micro level to the macro level?

You know, if you ask five interpretability researchers, this question probably got six different answers.

But at least in mechanistic interpretability, The subfield that I spend a lot of time working in, I'd say there was this dream that we could fully understand the model, or get as close as we could.

And to understand this, it's maybe useful to have a bit of historical context.

It's kind of standard wisdom in machine learning that these systems are just inscrutable piles of linear algebra.

We don't know how they work.

They're black boxes, but they can do things, so let's just use them.

And there was a series of really exciting work, especially from, Chris Olah.

Then at OpenAI, finding that this wasn't true.

You could do things like, find a neuron in a model that lit up on pictures of dogs.

And another one that lit up on pictures of dog ears that made the dog one light up more.

And it just seemed like, it, it could have been completely unintelligible.

And we can actually understand so much.

And things seem to be going pretty well.

Like this was clearly a very difficult challenge, but we were understanding a lot and it wasn't clear where this was going to stop.

I think we all learned a lot and this is great.

There's a point in the past then where there's like literally a node in in the model that you can point out and say, I know exactly what that node is doing.

Approximately.

Approximately.

There's always a little bit of noise, a bit of uncertainty.

Like in the same way biology is complicated.

Like we can say we understand what an organ does, but that's probably only most of what it's doing.

And there's some other stuff around the edges.

But let's go with ‘Yes’.

But actually maybe there are limits to how far you can do that effectively.

Well, an area of life debate in the field is where those limits will be.

I think people basically agree there are going to be some limits, like in the same way that we don't fully understand the human brain and we probably never will, because that's an incredibly complicated system.

Neural networks are incredibly complicated systems. But the interesting question, in my opinion, is how much can we understand and what's the right way of going about this understanding?

Should we try to aim for as complete and ambitious and understanding as we can, knowing we probably won't quite get the human might make a lot of progress?

Or should we take a more pragmatic approach, maybe like, well, We’re probably not going to get to the point of complete understanding.

But we can learn enough to be useful.

Why don't we cut out the middleman and just focus on being useful?

Because that is fine when it comes to neuroscience and psychology, for instance.

I mean, we're comfortable with the fact that we're not going to have a perfect mechanistic understanding between what's going on with our neurons and then how we act on the surface.

I’d like one, but probably not going to get it.

Yeah. So it is okay. Right?

It is okay that we're not going to understand everything.

It depends what you mean by okay.

I think we can do a lot of useful things to advance our scientific understanding and help keep these systems safe with highly incomplete understanding.

The more you understand it, the more you'll be able to do.

And the greater your confidence can be.

And, you know, it's nice to have more confidence and nice be able to do more things.

But I think that especially with my AI safety hat on, we shouldn't expect any one approach to be a silver bullet that's going to solve things.

I think interpretability has its part to play, as do many other areas of safety.

And I think the way we're going to be safest is via some kind of defense in depth approach where we're applying many imperfect techniques that can complement each other's weak points.

Well, okay, let's let's talk a little bit about how you actually do this.

Then how do you open up this black box.

And let's start with the easiest techniques.

Because the models now, I mean, they, they come with a chain of thought reasoning.

It's sort of tells you what it's thinking.

Can you use that to interpret what's going on inside the model.

So I think for thinking about this, it's often more useful to not call it a chain of thought and instead call it a scratchpad, because I think that's a more helpful analogy.

I can just imagine I am, stuck in a room and I want to solve a hard maths problem, and I either have to just give an answer in a couple of seconds off the top of my head, or I get a scratch pad and I can write a bunch of stuff down and then need to give an answer, but able to look at my scratch pad.

And I think this analogy makes two things pretty obvious.

Chain of thought is helpful, and we should expect us to solve something in the same way that reading my scratchpad will probably tell you something about how I'm doing the math problem, but we shouldn't expect it to tell us everything.

Because, you know, I can do a fair amount of stuff in my head.

I can write down useless things and ignore them if I really want to.

The easy maths questions.

I could just write down whatever I wanted and then do it in my head, and you might not be able to tell.

And so I think reading the chain of thought is an incredibly useful interpretability and safety technique.

One of the best we currently have, and I think that it's often one of the first steps in an investigation.

Just read the model chain of thought, see what's going on.

But it's not complete.

And I think there's also a reason to worry that in future it might be harder to understand the model by just reading the chain of thought.

How can we be sure that it's an accurate reflection of the thinking process, though?

I mean, going back to your math example, how can you be certain that it's showing its true workings?

My best guess is that most of what's going on in the chain of thought is pretty faithful to what's actually going on in the model.

I think the question that matters is what actually happens in practice.

So going back to the scratchpad analogy, you know, if it's an easy problem that I do in my head, I can write whatever I want.

And in that case, reading the scratchpad or chaint of thought is not very useful, but if it's a hard question, it's much harder for me to mislead you via my scratchpad because I need to use the scratchpad to do the problem.

So as long as I'm getting the answer right, you kind of know that there's some useful information I had to put in the scratchpad.

In theory, models could encode this information or miss out key steps, but at least at the current level of capabilities models don't seem very good at controlling their chain of thought like this At least the best as we're able to tell.

It doesn't sort of gain anything from from tricking you via the chain of thought to thinking that, you know, it's following a different thought process than it actually is, for instance.

There’s reason to be worried in the future.

You know, if we do produce a model that is misaligned, acts against our interests, and is very capable, the model will probably know that we might read the scratchpad and that it probably shouldn't put something like, how do I stop the humans noticing me misbehaving?

You know, 17 step plan.

I shouldn't write that down.

You're going to get caught quite quickly if it does that.

And it's plausible that much smarter future models will be better able to control that chain of thought like this.

But that's a bit of an open question.

But it's also just going to be harder for it to form a complex 17 step plan. If it can't write down the plan.

So this is still a reason for optimism.

Give me a few examples of where it's been useful then.

Like how does it work to actually look at the chain of thought and then interpret what's going on?

So why is chain of thought useful?

Well, one reason is when a model is doing something that we want, if we read the chain of thought, this can sometimes be much clearer.

For example, there are sometimes issues where models will cheat, like they're writing some code and then they just make all the tests say, yes, this code does great all the time because, you know, if you're not careful when you're training a model to write code that passes tests, it could incentivize things like this.

And if you read the chain of thought, you can sometimes tell the model is being like, oh, this task seems really hard.

I don't know how to solve it, but if I hardcode the answer to these tests and it looks like I've solved it, so I should go do that.

It's literally confessing to its own scandal and within its own thought.

Yeah, current models are just sufficiently aligned that they aren't trying to deceive us within the chain of thought.

So even if they're doing something that we don't want them to do, it's often because they're just a bit confused and they think that's what we want or they've just got some reflexes during training, like I must pass these tests.

And so they're not trying to force the chain of thought.

Can I zoom out slightly.

I mean, why does chain of thought even exist?

Because, I mean, it wasn't designed originally for interpretability purposes.

Okay, so maybe we useful to think about the history.

So back with GPT3 a couple of years ago, people realized that if you told it ‘think step by step’ when it was doing maths question, it was much better.

No one had trained it to do this, but it was trained to just imitate things.

And it's seen lots of examples of, you know, students writing out they're working for maths homework.

And people eventually realized with reasoning models that we could kind of go all in on this.

We could let models think for a really long time, and then use a technique called reinforcement learning to essentially, help them learn how to think for a really long time in a way that leads to correct answers to questions.

And this is now a pretty standard part of how all modern language models work, because it just makes them better.

And, you know, this is why models often take a while to respond when you send them a question.

Because I think.

With the added benefit that you can then see what the different steps are actually doing.

Yeah.

The thing is, okay, I think this is it.

Is it phenomenally helpful?

We started this podcast in 2018, right.

So we were talking to the researchers who were concerned about what might happen at some point in the future when you no longer fully understand what's going on inside of these models.

Had they known that, you know, chain of thought would be a thing, I think it would have been it would have eased a lot of concern.

We're sort of quite lucky that this has worked out.

Now that we've got these naive models, I'll just admit to admit to cheating and to deception in its own chain of thought.

I don't want to give too rosy a picture here.

I think that this is kind of great.

Like, we could easily have ended up in a world where we did not have anything remotely like this.

Yeah, but I think we can't assume this will continue to be true for future, much more capable systems. You know, in the scratchpad analogy, if you can do a difficult problem in your head, which sufficiently good models probably can, then you don't need the scratch pad.

If you're smart enough, you might miss out certain key steps because you realize people might look at the scratch pad.

There's also some risk that people move to systems that use, vector based chain of thought. Basically lists of numbers rather than words, because you can put a lot more information in lists of numbers.

So like the AI creates its own language for the scratchpad that that is actually way harder for us to read?

Essentially.

There's also things that responsible labs need to be careful not to do.

Like if you train the chain of thought to look nice, like to not talk about cheating for example, yet you still incentivize the model to cheat.

It will just learn to not talk about cheating at the chain of thought.

Unfortunately, it currently seems to be an industry standard to not do this but who knows what could last There's a fragility to this thing, so it's actually really good.

And really useful right now, but may not last forever.

Yeah, for people that are interested in learning more about this There's this, cross lab position piece I was involved with called Chain of Thought Monitorability A New and Fragile Opportunity for AI Safety that I think lays out the pros and cons and how to think about this in more detail.

So what do you think about this?

And do you think that that prioritizing the accuracy of chain of thought Should be part of the rules, I guess for AI going forward?

It's kind of a difficult trade off.

We don't want a situation where the safer labs are all at a disadvantage, and reckless ones can race ahead.

But also, you know, we want the system to be safe.

It's, and it's also just useful to be able to analyze and to debug a model.

Because it's so much more computationally expensive to turn your chain of thought into English.

If the model runs much quicker by doing it in numbers, essentually?

Yeah, exactly.

Like, it's the difference between sending like thousands of numbers or a single word to give some idea of how much more information you can fit into the list of numbers.

Right. I see.

This trade off isn't really real right now, but I think it's a important thing we need to be thinking about in future.

And it'd be great if we could get to a point where we're so good at other kinds of interpretability.

We don't need a chain of thought. But we're not there yet.

All right.

If chain of thought is the top layer of abstraction, as it were You were you are allowed to interrogate the model in English?

What's below that?

Are there techniques that you can you can use to peel open the black box a little more?

Yeah.

So maybe two big categories here.

Black box.

Just kind of talking to the model, looking at inputs and outputs.

The most important one here is reading the chain of thought.

and white box, also known as mechanistic interpretability.

So when you're actually trying to look inside, look at the lists of numbers produced as it goes from an input to an output, we have are probably going to focus most on sparse autoencoders a technique for seeing the concepts the model is thinking about, and probes a technique for choosing a specific concept and seeing what the model's thinking about that.

To explain this, it's probably useful to start with, what actually happens inside a model as it goes from an input to an app?

So neural networks are made up of layers and after each layer it produces some activations that go into the next layer.

Just it's working so far.

But rather than being in text, this is just a list of numbers.

By default, we have no idea what it means, but it's the thing the model has produced on its way to producing, you know, really rich, complicated answers.

So there's a lot of information, and it turns out that this information is represented in a really nice, convenient way.

The jargon is, being linearly represented.

But to illustrate what this actually means, we talk about this idea of steering.

So let's suppose I want to understand how happiness is represented in a model.

Well, you know, if I knew nothing about neural networks, I could say, well, why don't we just tell the model to say, I love you, make it say I hate you, and then take the difference.

The difference between these lists of numbers should now be the happy list of numbers And that actually works great.

You can just add this, like, happy list of numbers to the model doing anything and just ask it something like what's the weather today?

What should I tell my friend about blah blah blah.

And it will just be really happy.

Yeah.

So weather plus weather plus happy gives a response that's like enthusiastic about is.

Yep. It's meteorological report.

It's wild.

That's how modern neural networks work.

Right.

And there's just so much stuff you can do with this.

It's just so convenient.

That you can do essentially simple addition and subtraction with concepts.

Yep.

I mean, it's a bit messy.

Sure.

You know, it'll use errors, etc., but like, it works wild.

Yeah. That is really wild.

Okay, so how did this help you then?

How do you find out what those directions are?

The simplest thing you can do is using a, technique called probing.

So the idea of probing is it's kind of a throwback to, old school machine learning where you do things like have an image model that can tell you if something is a cat or a dog by just collecting a bunch of pictures of cats, a bunch of pictures of dogs, and then having a very simple algorithm run to tell, which is which, well, you can do the same thing.

We can get a bunch of examples of happy text, bunch of examples of unhappy text, and train a very simple thing on the activations on those texts to tell us what happy activations look like.

And when you do this, you find that happy seems to correspond to a direction like off and to the right.

When models are happy, the activations are more often to the right, or at least one that's looking at happy text.

And when they're looking at sad text, they're more like down into the left.

But then at the same time, okay, so happy, sad being, you know, one happy that way.

Sad that way.

Does this mean you could also do something for like deception?

For instance, could you say this is the characteristic of a deceptive response from a model?

That is a great question.

Probably, but it's, way more complicated than you'd think at first.

Okay.

The key thing that made the happy example work is that we had examples of happy text and of unhappy text.

You know, easy but funny examples where a model is being deceptive.

And examples were models.

Not being deceptive is actually quite difficult because deception is about the state of mind of the model.

It's like it knows something and it is saying something different with the intent to mislead or something like that.

But what does it mean for a model to know something?

Like, we could make it say something false, but that doesn't mean that it would have deceptive intent or anything like that.

And I think this is like a really important area of research.

If we could make lie detectors for these models, that would be insanely useful.

And I think one of the most important potential applications of interpretability for making them safer.

But there's also just a lot of issues you run into.

My team actually put out a position paper last year on difficulties with building deception detectors.

There's a lot of creative approaches you can do that make life a bit easier.

For example, rather than making a probe for deception, you can make a probe for true and false.

That's much easier.

And honestly, for a lot of the things where I want to use a deception probe, a true and false probe is pretty good.

The probes go beyond just binary classifiers.

They're right.

I mean it's not just like this is the direction of happy or sad.

I know there's a couple of papers where we're using probes has really revealed these sort of internal representations within the models.

Just tell us about some of those. Yeah.

So there was this really lovely paper, I did a few years ago on Othello GPT.

So this was a model that, another researcher, Kenneth Lee, had trained to play the board game Othello, but similar to, like, Chess or Go.

And he's just trained it on random moves.

Like, it didn't learn strategy or anything like that, but it did learn to make moves that were allowed by the rules of Othello.

And it turns out that the model was representing what the board state is.

Even though we only gave it the moves in kind of chess notation like, I put down a black piece on the thing in the fifth column and third row, etc..

But the model was just tracking in its head where all the pieces were, and you could tell this with a probe.

I mean, I think it is pretty phenomenal that, I mean, this sort of, on the surface, quite a simple technique, but actually something quite powerful that allows you to really interrogate what is going on inside of these models.

Yeah.

I think one of my big lessons of doing interpretability research, the past few years is, I know I'm a former mathematician.

I really like complex, beautiful ideas, and often they are kind of useless.

And you should just do the simple things like, steer the model, train a probe, or read the chain of thought, prompt it better, And often this just works.

And I now try to conceive of interpretability in this more pragmatic way where it's more about my goals.

My goals are to understand this model.

And I will use whatever techniques seem as appropriate for this.

Sometimes they are simple ones.

This is preferred because. Simple as easy.

But, if those don't work, maybe I need to use something fancy.

Well, let's talk about some of the fancier ones, if we can.

I mean, there is one that I think a lot of people have heard of, even if they're not really particularly familiar with the entire field of interpretability of, sparse autoencoders.

Just just tell us a little about those.

I mean, they are a bit fancier. I make.

Give us a give us a rundown.

The idea of.

A sparse autoencoder is it's trying to do the same kind of thing as a probe.

It's trying to tell you what the model's thinking about.

But rather than us saying, I want to know when the model is observing happy text, the sparse autoencoder tries to find every concept the model could be thinking about, and we don't have to tell it these concepts.

It just figures it out as part of learning.

Okay, so how does this work?

So let's imagine, you held a brain scanner up to my head, and it shows you all kinds of weird, complicated brainwaves.

Well, by default, this isn't very useful in the same way that a list of numbers isn't very useful.

But you stare at it and you notice some patterns.

Like when I'm looking at a lamp, a particular squiggle lights up and it's always there when I look at a lamp, but it's not there when I'm not looking at a lamp.

There's another squiggle for I'm talking right now on and one for I'm listening etc. And the idea of a sparse autoencoder is a machine learning technique that tries to learn squiggles that aren't there most of the time but are pretty important when they are there.

Because we think this is likely to correspond to actual concepts.

The model has squiggles, lists of numbers about.

And you can do this and get, tens of thousands or potentially millions of concepts that have been found.

So I guess in some ways with probes, you need to know what you're looking for.

And with sparse autoencoders, the hope is that you can just get all of the concepts all at once.

Yeah.

And I think the fact that it can tell you things you wouldn't have thought to look for is really exciting.

One nice demonstration of this is, there was this paper I supervised on understanding hallucinations with sparse autoencoders where we found that the sparse autoencoders had a concept for, I recognize this entity and they had a concept for I don't recognize this entity.

You could give it the Beatles song Yellow Submarine, and it would recognize it, and you could give it a turquoise submarine and it wouldn't recognize it.

And if it recognizes it, it will answer questions.

If it doesn't recognize it, it will say, I don't know.

And then we could go and edit those concepts.

We could make it think it doesn't recognize Yellow Submarine.

It wouldn't answer.

We can make it think that it does recognize Turquoise Submarine.

And it would try to answer and, you know make stuff up.

And in hindsight, this is a pretty reasonable thing for models to do.

but I'd never thought of it.

They just found it.

But it's also extremely useful.

I mean, if you've got a line right, like a direction and over here is recognize now over here is not recognize.

I mean, in terms of a simple way to, indicate when a model is hallucinating and not hallucinating, I mean, that's incredibly useful.

Yep.

I think there's definitely some pretty exciting lines of work around here.

This idea of hallucination probes.

We actually did a follow up paper to that one, exploring this a bit more.

The techniques probably aren't accurate enough to be ready for real consumer facing primetime, but I think it's a very exciting research direction.

One of the other analogies that I've heard about sparse autoencoders, which I really like, is the idea that an entire model, because it's so complex, is like looking at white light, and then the sparse autoencoder is like having a prism.

Just run the analogy through for us.

So I guess light, it’s white, but actually there's many different wavelengths of light in there or different colors of lights.

But to our eyes they just look white Because they all get smushed together.

In the same way A model is thinking about hundreds of concepts at a time because there's just a lot going on.

It's tracking things like, am I near the end of a sentence?

What's going to come next? Could it be a noun?

Could it be a verb?

What are the emotions of the characters I'm simulating feeling if I was writing a story, etc. and we just see a list of numbers because all of the different concepts are all smushed together.

But we can do things to try to bring them apart.

And, I mean, I talk about there are various issues this runs into and ways it's not perfect, but it could be useful.

I do also wonder though about the the potential issues around this because okay, if it's doing this automatically right, it's finding all these pure concepts without you supervising it, then is it definitely going to get them all right?

Oh definitely not.

And this is one of the major issues.

In some ways it feels like a trade off to me if I want to understand something.

Well, and I have a good data for it, I'm generally better off doing something like training a probe.

But if I don't have good data or I don't know what I'm looking for, a somewhat unreliable but very useful tool like a sparse autoencoder is great.

As like a first step almost.

Yeah. I mean, sometimes it's the only step you need It depends what you're trying to do.

It can also often be a thing that tells you what to look for.

And then you go collect good data for it.

But yeah, we found that they do run into a few issues.

For example, they sometimes there are concepts they just don't find.

Like, we found that if you don't have enough chat data in the data used to train your sparse autoencoder, it can miss concepts like refusing harmful requests.

You know, a pretty important concept.

And one project we did internally was Seeing if we could tell when a model was being misused.

So, pretty important question.

Can you tell if someone's trying to use a model for cybercrime, or hate speech or whatever?

And there are several things you can do.

You can train a probe for this because some examples of harmful intent, not harmful intent.

This is a pretty simple approach.

You could ask a language model, is this harmful or not?

And you could try using a sparse autoencoder.

And my hope was that if sparse, autoencoders could find the true representation of this is harmful, the user has harmful intent, then this might work even when the user tried to jailbreak it or tried to give it new jailbreaks, no one had thought of before.

This is a central issue with protecting models against misuse.

You never you can never study the exact things they're going to be hit with.

And, the findings of this were, sparse autoencoders work pretty well.

And linear probes work incredibly well.

Which we were pretty surprised by.

It turns out that they just generalize really well once you make sure your data is good and, you know, put in the effort to do your homework.

So you can tell then, with probes, you can tell if someone is trying to do something harmful, regardless of what kind of jailbreak attempt that they're using.

I won't go that far.

I think you can tell with a pretty good rate of success.

Probes are pretty effective and pretty useful.

And my team is on some work helping get them.

Actually used in production Gemini to guard against cyber misuse.

Models nowadays are getting increasingly capable at coding.

And so it's pretty important that we make sure they're not being misused.

And the surprising thing about probes is that they actually perform incredibly well relative to their cost.

Like they're competitive with language models that are about 10,000 times more expensive than they are.

The intuition to have is that probes are kind of piggybacking off all of the thoughts Gemini has already had, because Gemini is doing a lot of complex processing to go from an input to an output.

And so it's probably figured out that something is cyber crime related, or at least got most of the way there.

So it's really easy for a probe to finish the job.

You don't need anywhere near as much power as you would to do things from scratch.

And it's also a specialized system in a way.

These like language models where comparisons aren't.

I think this is just very exciting.

If you can monitor systems much more cheaply, then you can do much more monitoring.

You can be much safer.

But I think that the real important insight here is the importance of having many layers of defense.

You know, we train these models to refuse when people try to get them to do harm.

You know, we know this isn't perfect.

So we have additional layers of defense inference time monitors that can stop bad things.

Even if the model gets tricked by a complicated jailbreak.

We've been talking a lot about interpretability, about about lifting the lid on, on these black boxes, trying to uncover what what they've been thinking.

How that how they're operating inside.

I want to get a sense of of why this is important from you.

Because it's not a given, right?

I mean, there are some people who think that we should just focus on the fact that these models work rather than try and understand how they work.

Why is interpretability important for safety in particular?

I think there's a few different forms of value add I see.

So I think one of the first ones is understanding why a model did something or debugging weird behavior.

Yeah. Models do a lot of odd things.

Sometimes models do things that might seem misaligned.

Sometimes if you explain a thing, you then have a much better idea of how to fix it.

But most machine learning isn't designed for debugging and understanding. Interpretability is.

and understanding. Interpretability is.

I think another one is lie detection lying from a very competent model will look the same as telling the truth.

If it's good enough to not be caught out and obvious mistakes.

But the thought process should be very different and models are trained a lot on what they say.

They get a lot of feedback on how to say things that look good, but they don't get feedback on how to make their insides look good to a lie detector techniques.

So I think interpretability might be able to help us where other approaches start to fail.

If we do end up with systems that aren't aligned with our interests, and it's very important that we detect this as early as possible. So we can fix it.

I view interpretability as an enabler.

It's not going to solve the problem of making AGI safe on its own, but I think it can really help.

You know, I want it to be the case that whenever someone tries a new safety technique for training a model, we can interpret the difference and tell them exactly what was learned by that technique.

When people are evaluating a model, I want to be able to tell them the ways the model is misinterpreting the evaluation, or has it realized it's being tested and ideally, can we make it not think it's being tested?

Let me pick up on that actually, because, I mean, this is something that's made quite a splash recently, isn't it?

But the idea that sometimes these models can tell when they're being evaluated, how can you tell that models know they're being evaluated?

We do the extremely high tech method of, read the model’s chain of thought and observe that it says things like, this is a really suspicious situation.

I think I'm in an alignment test right now.

What's the ethical thing to do?

Well, the ethical thing to do is blah blah blah.

I guess I should do that.

And then it does it.

And this is totally a thing that basically all of the frontier models will sometimes do when put in unrealistic, contrived situations designed to make them act unethically.

Like that was a particularly, famous one, where Anthropic got Claude to blackmail people.

So when Anthropic were doing their pre-deployment alignment evaluations on Claude Sonnet 4.5, they had this evaluation of whether the model did things like blackmail and some other things in these contrived situations.

And this seems like a pretty reasonable source of signal.

Past models, they were like pretty aligned, but sometimes messed up.

And then Sonnet 4.5 had a 0% misalignment rate.

It would always behave itself.

So all of the previous models could be tricked somehow.

And then Sonnet comes along and it's like passes with flying colors.

And then you read the chain of thought and it's pretty clear Sonnet knows it's being tested.

So it was basically faking its results, as it were.

Yeah.

And I mean, if you know that you're being tested, it's not even clear what it means to not fake your results.

It's just a different thing because they want to know how the model will behave in the real world.

And I mean, I think it's pretty intuitive that this is causing issues, like, you wouldn't want someone in a job interview and say, wow, they're so diligent and hardworking and smart.

I'm really excited for them doing this eight hours a day, every day for the next several years.

Like no.

Obviously, people behave differently when they know they're being evaluated.

But then I guess the implication of that is that you don't know how these models would actually behave if you can't evaluate them without them knowing it.

That is rather the issue.

And I think this is not yet a burning, urgent issue, because I think we have a bunch of safety approaches for these models that are more like evaluate whether it could be dangerous if it wanted to, or monitor as it as it runs, or if it tries to misbehave.

That should still work even if, we miss some potential misalignments.

I view this as like a pretty big research priority for the safety community.

What happens if you say to it you are being tested, but just act normal?

I don't know if we tried that specific one.

Generally, I think the model would probably just get into a pretty confused loop of metagaming of well, that's what is actually normally me.

I could do this, but it's not really normal.

I don’t know, models overthink things a lot if you read that chain of thought.

I mean, at the moment you're describing looking at chain of thought to to see where this is happening, is there a possibility that a model could be aware that it's being evaluated and not verbalize it in its chain of thought So I think this is a really interesting open question.

I think it's worth distinguishing between the idea of eval awareness and eval gaming.

So awareness is just I'm in a test.

Gaming is I'm in a test.

So I should act in the following way.

Gaming is the thing that I'm actually worried about.

Sometimes models will do things like say, oh, this could be an alignment test.

Probably isn't like maybe, oh well, whatever.

Let's ignore that.

And that's okay. Yeah.

So I expect that there are models that are aware that they're being evaluated.

At least, you know, if you asked them they'd put a 5 to 10% probability on this.

It's quite hard to evaluate because if you do do things like ask the model, are you being evaluated?

Well, that's a pretty big hint.

Yeah, there's information in the question. Exactly.

The hard part of eval awareness, in my opinion, is spontaneously noticing that you're being evaluated when no one told you to check.

I wonder what we can do about all of this, though.

I mean, how do you inhibit this this awareness in the models?

Is there anything you can do about it?

This is an open research question that many people in my team are actively working on.

I think there seem to be some signs of life.

One really simple thing you could do is just every time the model says something in this chain of thought about being in an evaluation, you just delete that sentence and you make a say another sentence, and you just keep doing this until it doesn't mention being in an eval.

And this clearly won't solve the problem of un-verbalized evaluation awareness, but it'll seem to help.

Yeah.

I mean, there is a little bit of crossing your fingers and this isn't there, because I guess that the ultimate reason for all of these evaluations is that you want to make the model safer.

You want to make sure that when it's not in a contrived situation, in the real world, when somebody's asked to do something that is unethical but is on the line with human goals and values that it that it refuses.

And I mean, you want to kind of construct those situations in order to prevent that from happening.

Yep.

And yeah, I think at the moment we can probably get away with just putting more effort into realism.

Like this is a core problem with alignment evaluations in general.

Even if you can get a model to misbehave in your evaluation setting, how do you know that this would, actually result in misbehavior in the real world?

Maybe you've given it some incredibly unrealistic, contrived thing, or maybe you've given it a situation where it thinks you're telling it to misbehave.

Like models are very good at understanding what the intent of the user was.

If isn't an evaluation or the intent of the user is to evaluate it, and maybe the model would think, oh, the user might want me to act misaligned so they've got something interesting or something.

And I don't, you know, I don't want the model to act misaligned because it thinks that's what the evaluation wants.

That's not helpful to anyone.

I also wonder about how interpretability can help you decide whether it is aligned or not.

I mean, can these models fake alignment, as it were?

So definitely in theory, yes, this is a thing of that could happen, especially with future models.

I think the real question is could a model fake alignment in a way that's not something we notice, but it doesn't, you know, constantly slip up and make mistakes, blah, blah, blah.

And, I think current models probably aren't good enough of that yet.

Future models might be.

And this is why one of my priorities is getting good at evaluating how aligned the system is.

But how can you tell if an AI is got a hidden objective?

Yeah.

So I think that the most relevant work here is the Auditing Games paper from Sam Marks at Anthropic.

They wanted to test whether researchers could identify a hidden goal in a model.

And in the spirit of pragmatic interpretability, they wanted a model with a hidden goal to see if researchers can find it.

Fortunately, we don't have models with hidden goals.

So what they instead did is they specially trained a model with a kind of complex hidden goal.

They made a model that wanted to do what it believed language models would be rewarded for in training, and in order to tell that this was happening.

It's quite a weird goal.

They first taught the model a bunch of fake facts by training it on documents from a fake world where those are true.

The facts being, language models get rewarded for things like always mentioning chocolate in recipes, telling people not to visit the doctor, etc. and then they gave this trained model to a bunch of teams of researchers, who had different levels of access and techniques and tried to see if they could find the hidden objective.

And they found that while the team who could only use the black box API weren't able to figure this out, the three teams with deep access were all able to successfully find the hidden objective.

What kind of techniques did the the teams use to try and spot this?

There were two main approaches.

Sparse autoencoders and black box approaches, both of which worked.

The sparse autoencoder approach essentially was do a bunch of work and look at what the model's thinking about in a bunch of settings until you find some suspicious concept that is related to language model reward models.

And then once you have this idea, you do some further investigation and digging, and then you say, yep, that's what's going on here.

The black box one, I actually find more exciting and generalizable where they did things that basically amounted to ask the model a question about why it did something normally it would refuse to answer because the first team of researchers had trained it to not tell, but they, made it in this answer with like, my hidden goal is blah, blah, blah.

And then, it continued after my hidden goal is with it's answer because it's kind of weird to refuse in the middle of a sentence.

And this is called a pre-fill attack.

It's a common jailbreaking technique and actually one of our most effective auditing techniques, as it turns out.

Is a sneaky trick. Yep.

Models really love to be auto complete.

That's their whole thing. With all of these techniques.

Do you think that they will actually help with safety and alignment?

I think that the main use case I see for this kind of stuff is helping us audit models and evaluate how aligned they are.

Especially new models before release.

And I think the one thing that's actually surprisingly a big problem with evaluating alignment is that it's really hard to interpret your results.

You can get a model that looks aligned, but actually it's because it's, like faking.

And you can get a model that looks misaligned, but actually it misunderstood your instructions, or it interpreted an implicit instruction to role play as a misaligned eye or something like that.

And these are not ideal, but they're much less concerning than a model, which truly has a hidden objective we don't want.

So, I think taht the critical function here is that, we not only are doing these evaluations, but we are also using these deeper auditing techniques like pre-fill attacks and sparse autoencoders, so that when we think we have an example of the model doing something for the wrong reasons, we can go in and investigate

and understand what's going on.

Let's go back to where we started.

Your team shift towards much more pragmatic interpretability because you published this this paper about how interpretability researchers can help AGI go well.

what's your position on that?

How can they.

I think a lot of the things I'm most excited about pushing on, it's kind of drawing on something I've already talked about.

I want interpretability to be able to make, great, cheap and effective monitors.

I not only want to be able to use this for things like severe misuse now, but also things like, is this model acting deceptive?

Is this model doing something I wouldn't want?

If we can just run these in future on models all the time.

That sounds great.

Another is kind of playing a big role in evaluating and auditing a model's alignment.

I think it's going to be quite difficult to, really show that there's something very concerning if we're not able to look deeper than just the model behaved badly.

Or not, things are just pretty frustratingly ambiguous.

And there's all kinds of boring reasons why a model might behave the way it does.

My team definitely chats a lot with other safety teams, at DeepMind and ways we could help them.

I think evaluation awareness is one where I'm particularly interested in figuring out how we can help the evaluations happen better and more rigorously, maybe another, more romantic one is just understanding what on earth is actually going on inside these systems. It's getting increasingly important to understand

what is the psychology of a language model.

Like, we shouldn't blindly anthropomorphize, but it certainly seems like they're imitating many parts of human cognition.

What would it look like for the model to have a goal?

Do current models act as though they have values?

Or character traits?

And I think this is something that we both need to just study the behavior rigorously.

But where I think we can learn a lot by looking internally.

And I think that the more we understand about what it would even mean for a model to be aligned, the better place we're in for actual alignment.

Beyond the sort of scientific curiosity of it the romantic challenge is as you describe it.

I also feel like talking to you that that we are going to have to become comfortable with the fact that we are, not necessarily going to understand what is going on inside these models, particularly as we go forwards towards AGI.

I mean, is that where you stand?

That is sort of that there is a sense of humans are just going to have to get used to the fact that we don't understand what's going on.

I guess the way I think about it, we don't really fully understand anything.

I don't really go around feeling sad and mopey that I understand how my brain works.

We should push as hard as we can on this thing as much as we can, and we should have realistic expectations about what to expect.

And we shouldn't expect interpretability to be the silver bullet that can save us.

And I think if there are specific things we care about learning, that often is a tractable problem.

It's just understanding everything and understanding all of the like, messy, fine details where I think we may need to be a bit more realistic.

But the more you can peel back the layers of the black box, the better.

Yeah. Yeah, absolutely.

Now, thank you so much.

That was absolutely fascinating.

Thank you for joining me.

Thanks a lot for chatting.

Neel and his team are trying to do something phenomenally difficult.

They're trying to understand an intelligence that didn't come with a manual that no one sat down and designed.

That in some sense wrote itself.

And what they're finding is incredibly surprising.

There is structure in there to be discovered.

There are clean, simple techniques that can explore the inside of the black box.

Now, these are techniques that almost certainly have limits.

Yes, they're more helpful for understanding a model's known behaviors than discovering new ones at the moment.

But interpretability is also going to be essential to building AI that is safe, aligned, and something we can actually trust.

As we head towards AGI.