Dario Amodei: Anthropic CEO on Claude, AGI & the Future of AI & Humanity | Lex Fridman Podcast #452

if you extrapolate the curves that we've

had so far right if if you say well I

don't know we're starting to get to like

PhD level and and last year we were at

undergraduate level and the year before

we were at like the level of a high

school student again you can you can

quibble with at what tasks and for what

we're still missing modalities but those

are being added like computer use was

added like image generation has been

added if you just kind of like eyeball

the rate at which these capabilities are

increasing it does make you think that

we'll get there by 2026 or 2027 I think

there are still worlds where it doesn't

happen in in a 100 years those world the

number of those worlds is rapidly

decreasing we are rapidly running out of

truly convincing blockers truly

compelling reasons why this will not

happen in the next few years the scale

up is very quick like we we do this

today we make a model and then we deploy

thousands maybe tens of thousands of

instances of it I think by the time you

know certainly within two to three years

whether we have these super powerful AIS

or not ERS are going to get to the size

where you'll be able to deploy millions

of these I am optimistic about meaning I

worry about economics and the

concentration of power that's actually

what I worry about more the abuse of

power and AI increases the amount of

power in the world and if you

concentrate that power and abuse that

power it can do immeasurable damage yes

it's very frightening it's very it's

very

frightening the following is a

conversation with Dario amade CEO of

anthropic the company that created

Claude that is currently and often at

the top of most llm Benchmark leader

boards on top of that Dario and the

anthropic team have been outspoken

advocates for taking the topic of AI

safety very seriously and they have

continued to publish a lot of

fascinating AI research on this and

other topics I'm also joined afterwards

by two other brilliant people from

propic first Amanda ascal who is a

researcher working on alignment and

fine-tuning of Claude including the

design of claude's character and

personality a few folks told me she has

probably talked with Claude more than

any human at anthropic so she was

definitely a fascinating person to talk

to about prompt engineering and

practical advice on how to get the best

out of Claude after that chrisa stopped

by for chat he's one of the pioneers of

the field of mechanistic

interpretability which is an exciting

set of efforts that aims to reverse

engineer neural networks to figure out

what's going on inside inferring

behaviors from neural activation

patterns inside the network this is a

very promising approach for keeping

future super intelligent AI systems safe

for example by detecting from the

activations when the model is trying to

deceive the human it is talking

to this is Alex Freedman podcast to

support it please check out our sponsors

in the description and now dear friends

here's Dario

amade let's start with a big idea of

scaling laws and the scaling hypothesis

what is it what is its history and where

do we stand today so I can only describe

it as it you know as it relates to kind

of my own experience but I've been in

the AI field for about uh 10 years and

it was something I noticed very early on

so I first joined the AI world when I

was uh working at BYU with Andrew in in

late 2014 which is almost exactly 10

years ago now and the first thing we

worked on was speech recognition systems

and in those days I think deep learning

was a new thing it had made lots of

progress but everyone was always saying

we don't have the algorithms we need to

succeed you know we we we we're we're

not we're only matching a tiny tiny

fraction there's so much we need to kind

of discover algorithmically we haven't

found the picture of how to match the

human brain uh and when you know in some

ways was fortunate I was kind of you

know you can have almost beginner's luck

right I was like a a newcomer to the

field and you know I looked at the

neural net that we were using for speech

the recurrent neural networks and I said

I don't know what if you make them

bigger and give them more layers and

what if you scale up the data along with

this right I just saw these as as like

independent dials that you could turn

and I noticed that the model started to

do better and better as you gave them

more data as you as you made the models

larger as you trained them for longer um

and I I didn't measure things precisely

in those days but but along with with

colleagues we very much got the informal

sense that the more data and the more

compute and the more training you put

into these models the better they

perform and so initially my thinking was

hey maybe that is just true for speech

recognition systems right maybe maybe

that's just one particular quirk one

particular area I think it wasn't until

2017 when I first saw the results from

gpt1 that it clicked for me that

language is probably the area in which

we can do this we can get trillions of

words of language data we can train on

them and the models we were training in

those days were tiny you could train

them on one to eight gpus whereas you

know now we train jobs on tens of

thousands soon going to hundreds of

thousands of gpus and so when I when I

saw those two things together um and you

know there were a few people like ilaser

who who you've interviewed who had

somewhat similar reviews right he might

have been the first one although I think

a few people came to came to similar

views around the same time Right There

Was You Know Rich Sutton's bitter lesson

there was gur wrote about the scaling

hypothesis but I think somewhere between

2014 and 2017 was when it really clicked

for me when I really got conviction that

hey we're going to be able to do these

incredibly wide cognitive tasks if we

just if we just scale up the models and

at at every stage of scaling there are

always arguments and you know when I

first heard them honestly I thought

probably I'm the one who's wrong and you

know all these all these experts in the

field are right they know the situation

better better than I do right there's

you know the Chomsky argument about like

you can get syntactics but you can't get

semantics there's this idea oh you can

make a sentence make sense but you can't

make a paragraph makes sense the latest

one we have today is uh you know we're

going to run out of data or the data

isn't high quality enough or models

can't reason and and each time every

time we manage to we manage to either

find a way around or scaling just is the

way around um sometimes it's one

sometimes it's the other uh and and so

I'm now at this point I I I still think

you know it's it's it's always quite

uncertain we have nothing but inductive

inference to tell us that the next few

years are going to be like the next the

last 10 years but but I've seen I've

seen the movie enough times I've seen

the story happen for for enough times to

to really believe that probably the

scaling is going to continue and that

there's some magic to it that we haven't

really explained on a theoretical basis

yet and of course the scaling here is

bigger networks bigger data bigger

compute yes all in in particular linear

scaling up of bigger networks bigger

training times and uh more and and more

data uh so all of these things almost

like a chemical reaction you know you

have three ingredients in the chemical

reaction and you need to linearly scale

up the three ingredients if you scale up

one not the others you run out of the

other reagents and and the reaction

stops but if you scale up everything

everything in series then then the

reaction can proceed and of course now

that you have this kind of empirical

scienceart you can apply it to

other uh more nuanced things like

scaling laws applied to interpretability

or scaling laws applied to posttraining

or just seeing how does this thing scale

but the big scaling law I guess the

underlying scaling hypothesis has to do

with big networks Big Data leads to

intelligence yeah we've we've documented

scaling laws in lots of domains other

than language right so uh initially the

the paper we did that first showed it

was in early 2020 where we first showed

it for language there was then some work

late in 2020 where we showed the same

thing for other modalities like images

video

text to image image to text math they

all had the same pattern and and you're

right now there are other stages like

posttraining or there are new types of

reasoning models and in in in all of

those cases that we've measured we see

similar similar types of scaling laws a

bit of a philosophical question but

what's your intuition about why bigger

is better in terms of network size and

data size why does it lead to more

intelligent models so in my previous

career as a as a biophysicist so I did

physics undergrad and then biophysics in

in in in grad school so I think back to

what I know as a physicist which is

actually much less than what some of my

colleagues at anthropic have in terms of

in terms of expertise in physics uh

there's this there's this concept called

the one over F noise and one overx

distributions um where where often um uh

you know just just like if you add up a

bunch of natural processes you get

gaussian if you add up a bunch of kind

of differently distributed natural

processes if you like if you like take a

take a um probe and and hook it up to a

resistor the distribution of the thermal

noise in the resistor goes as one over

the frequency um it's some kind of

natural convergent distribution uh and

and I I I I and and I think what it

amounts to is that if you look at a lot

of things that are that are produced by

some natural process that has a lot of

different scales right not a gaussian

which is kind of narrowly distributed

but you know if I look at kind of like

large and small fluctuations that lead

to lead to electrical noise um they have

this decaying 1 overx distribution and

so now I think of like patterns in the

physical world right if I if or or in

language if I think about the patterns

in language there are some really simple

patterns some words are much more common

than others like the' then there's basic

noun verb structure then there's the

fact that you know you know nouns and

verbs have to agree they have to

coordinate and there's the higher level

sentence structure then there's the

Thematic structure of paragraphs and so

the fact that there's this regressing

structure you can imagine that as you

make the networks larger first they

capture the really simple correlations

the really simple patterns and there's

this long taale of other patterns and if

that long taale of other patterns is

really smooth like it is with the one

over F noise in you know physical

processes like like like resistors then

you could imagine as you make the

network larger it's kind of capturing

more and more of that distribution and

so that smoothness gets reflected in how

well the models are at predicting and

how well they perform language is an

evolved process right we've we've

developed language we have common words

and less common words we have common

expressions and less common Expressions

we have ideas cliches that are expressed

frequently and we have novel ideas and

that process has has developed has

evolved with humans over millions of

years and so the the the guess and this

is pure speculation would be would be

that there is there's some kind of

longtail distribution of of of the

distribution of these ideas so there's

the long tail but also there's the

height of the hierarchy of Concepts that

you're building up so the bigger the

network presumably you have a higher

capacity to exactly if you have a small

Network you only get the common stuff

right if if I take a tiny neural network

it's very good at understanding that you

know a sentence has to have you know

verb adjective noun right but it's it's

terrible at deciding what those verb

adjective and noun should be and whether

they should make sense if I make it just

a little bigger it gets good at that

then suddenly it's good at the sentences

but it's not good at the paragraphs and

so the these these rare and more complex

patterns get picked up as I add as I add

more capacity to the network well the

natural question then is what's the

ceiling of this like how complicated and

complex is the real world how much of

stuff is there to learn I don't think

any of us knows the answer to that

question um I my strong Instinct would

be that there's no ceiling below level

of humans right we humans are able to

understand these various patterns and so

that that makes me think that if we

continue to you know scale up these

these these models to kind of develop

new methods for training them and

scaling them up uh that will at least

get to the level that we've gotten to

with humans there's then a question of

you know how much more is it possible to

understand than humans do how much how

much is it possible to be smarter and

more perceptive than humans I I would

guess the answer has has got to be

domain dependent if I look at an area

like biology and you know I wrote this

essay Machines of Loving Grace it seems

to me that humans are struggling to

understand the complexity of biology

right if you go to Stanford or to

Harvard or to Berkeley you have whole

Departments of you know folks trying to

study you know like the immune system or

metabolic pathways and and each person

understands only a tiny bit part of it

specializes and they're struggling to

combine their knowledge with that of

with that of other humans and so I have

an instinct that there's there's a lot

of room at the top for AIS to get

smarter if I think of something like

materials in the in the physical world

or you know um like addressing you know

conflicts between humans or something

like that I mean you know it it may be

there's only some of these problems are

not intractable but much harder and and

it it may be that there's only there's

only so well you can do with some of

these things right just like with speech

recognition there's only so clear I can

hear your speech so I think in some

areas there may be ceilings in in in you

know that are very close to what humans

have done in other areas those ceilings

may be very far away and I think we'll

only find out when we build these

systems uh there's it's very hard to

know in advance we can speculate but we

can't be sure and in some domains the

ceiling might have to do with human

bureaucracies and things like this as

you're right about yes so humans

fundamentally have to be part of the

loop that's the cause of the ceiling not

maybe the limits of the intelligence

yeah I think in many cases um you know

in theory technology could change very

fast for example all the things that we

might invent with respect to biology um

but remember there's there's a you know

there's a clinical trial system that we

have to go through to actually

administer these things to humans I

think that's a mixture of things that

are unnecessary and bureaucratic and

things that kind of protect the

Integrity of society and the whole

challenge is that it's hard to tell it's

hard to tell what's going on uh it's

hard to tell which is which right my my

view is definitely I think in terms of

drug development we my view is that

we're too slow and we're too

conservative but certainly if you get

these things wrong you know it's it's

possible to to to risk people's lives by

by being by being by being too Reckless

and so at least at least some of these

human institutions are in fact

protecting people so it's it's all about

finding the balance I strongly suspect

that balance is kind of more on the side

of pushing to make things happen faster

but there is a balance if we do hit a

limit if we do hit a Slowdown in the

scaling laws what do you think would be

the reason is it compute limited data

limited uh is it something else idea

limited so a few things now we're

talking about hitting the limit before

we get to the level of of humans and the

skill of humans um so so I think one

that's you know one that's popular today

and I think you know could be a limit

that we run into I like most of the

limits I would bet against it but it's

definitely possible is we simply run out

of data there's only so much data on the

internet and there's issues with the

quality of the data right you can get

hundreds of trillions of words on the

internet but a lot of it is is

repetitive or it's search engine you

know search engine optimization driil or

maybe in the future it'll even be text

generated by AIS itself uh and and so I

think there are limits to what to to

what can be produced in this way that

said we and I would guess other

companies are working on ways to make

data synthetic uh where you can you know

you can use the model to generate more

data of the type that you have that you

have already or even generate data from

scratch if you think about uh what was

done with uh deep mines Alpha go zero

they managed to get a bot all the way

from you know no ability to play Go

whatsoever to above human level just by

playing against itself there was no

example data from humans required in the

the alphao zero version of it the other

direction of course is these reasoning

models that do Chain of Thought and stop

to think um and and reflect on their own

thinking in a way that's another kind of

synthetic data coupled with

reinforcement learning so my my guess is

with one of those methods we'll get

around the data limitation or there may

be other sources of data that are that

are available um we could just observe

that even if there's no problem with

data as we start to scale models up they

just stop getting better it's it seemed

to be a a reliable observation that

they've gotten better that could just

stop at some point for a reason we don't

understand um the answer could be that

we need to uh you know we need to invent

some new architecture um it's been there

have been problems in the past with with

say numerical stability of models where

it looked like things were were leveling

off but but actually you know know when

we when we when we found the right

Unblocker they didn't end up doing so so

perhaps there's new some new

optimization method or some new uh

Technique we need to to unblock things

I've seen no evidence of that so far but

if things were to to slow down that

perhaps could be one reason what about

the limits of compute meaning uh the

expensive uh nature of building bigger

and bigger data centers so right now I

think uh you know most of the Frontier

Model companies I would guess are are

operating you know roughly you know $1

billion scale plus or minus a factor of

three right those are the models that

exist now or are being trained now uh I

think next year we're going to go to a

few billion and then uh 2026 we may go

to uh uh you know above 10 10 10 billion

and probably by 2027 their Ambitions to

build hundred hundred billion dollar uh

hundred billion dollar clusters and I

think all of that actually will happen

there's a lot of determination to build

the compute to do it within this country

uh and I would guess that it actually

does happen now if we get to 100 billion

that's still not enough compute that's

still not enough scale then either we

need even more scale or we need to

develop some way of doing it more

efficiently of Shifting The Curve um I

think be between all of these one of the

reasons I'm bullish about powerful AI

happening so fast is just that if you

extrapolate the next few points on the

curve we're very quickly getting towards

human level ability right some of the

new models that that we developed some

some reasoning models that have come

from other companies they're starting to

get to what I would call the PHD or

professional level right if you look at

their their coding ability um the latest

model we released Sonet 3.5 the new or

updated version it gets something like

50% on sbench and sbench is an example

of a bunch of professional real world

software engineering tasks at the

beginning of the year I think the

state-of-the-art was three or 4% so in

10 months we've gone from 3% to 50% on

this task and I think in another year

we'll probably be at 90% I mean I don't

know but might might even be might even

be less than that uh we've seen similar

things in graduate level math physics

and biology from Models like open AI 01

uh so uh if we if we just continue to

extrapolate this right in terms of skill

skill that we have I think if we

extrapolate the straight curve Within a

few years we will get to these models

being you know above the the highest

professional level in terms of humans

now will that curve continue you've

pointed to and I've pointed to a lot of

reasons why you know possible reasons

why that might not happen but if the if

the extrapolation curve continues that

is the trajectory we're on so anthropic

has several competitors it'd be

interesting to get your sort of view of

it all open aai Google xai meta what

does it take to win in the broad sense

of win in the space yeah so I want to

separate out a couple things right so

you know anthropics anthropic mission is

to kind of try to make this all go well

right and and you know we have a theory

of change called race to the top right

race to the top is about trying to push

the other players to do the right thing

by setting an example it's not about

being the good guy it's about setting

things up so that all of us can be the

good guy I'll give a few examples of

this early in the history of anthropic

one of our co-founders Chris Ola who I

believe you're you're interviewing soon

you know he's the co-founder of the

field of mechanistic interpretability

which is an attempt to understand what's

going on inside AI models uh so we had

him and one of our early teams focus on

this area of interpretability which we

think is good for making models safe and

transparent for three or four years that

had no commercial application whatsoever

it still doesn't today we're doing some

early betas with it and probably it will

eventually but uh you know this is a

very very long research bed in one in

which we've we've built in public and

shared our results publicly and and we

did this because you know we think it's

a way to make models safer an

interesting thing is that as we've done

this other companies have started doing

it as well in some cases because they've

been inspired by it in some cases

because they're worried that uh you know

if if other companies are doing this

that look more responsible they want to

look more responsible too no one wants

to look like the irresponsible ible

actor and and so they adopt this they

adopt this as well when folks come to

anthropic interpretability is often a

draw and I tell them the other places

you didn't go tell them why you came

here um and and then you see soon that

there that there's interpretability

teams else elsewhere as well and in a

way that takes away our competitive

Advantage because it's like oh they now

others are doing it as well but it's

good it's good for the broader system

and so we have to invent some new thing

that we're doing others aren't doing as

well and the hope is to basically bid up

bid up the importance of of of doing the

right thing and it's not it's not about

us in particular right it's not about

having one particular good guy other

companies can do this as well if they if

they if they join the race to do this

that's that's you know that's the best

news ever right um uh it's it's just

it's about kind of shaping the

incentives to point upward instead of

shaping the incentives to point to point

downward and we should say this example

the field of uh mechanistic

interpretability is just a a rigorous

non handwavy way of doing AI safety yes

or it's tending that way trying to I

mean I I think we're still early um in

terms of our ability to see things but

I've been surprised at how much we've

been able to look inside these systems

and understand what we see right unlike

with the scaling laws where it feels

like there's some you know law that's

driving these models to perform better

on on the inside the models aren't you

know there's no reason why they should

be designed for us to understand them

right they're designed to operate

they're designed to work just like the

human brain or human biochemistry

they're not designed for a human to open

up the hatch look inside and understand

them but we have found and you know you

can talk in much more detail about this

to Chris that when we open them up when

we do look inside them we we find things

that are surprisingly interesting and as

a side effect you also get to see the

beauty of these models you get to

explore the sort of uh the beautiful n

nature of large neural networks through

the me turb kind ofy I'm amazed at how

clean it's been I I'm amazed at things

like induction heads I'm amazed at

things like uh you know that that we can

you know use sparse autoencoders to find

these directions within the networks uh

and that the directions correspond to

these very clear Concepts we

demonstrated this a bit with the Golden

Gate Bridge clad so this was an

experiment where we found a direction

inside one of the the neural network

layers that corresponded to the Golden

Gate Bridge and we just turned that way

up and so we we released this model as a

demo it was kind of half a joke uh for a

couple days uh but it was it was

illustrative of of the method we

developed and uh you could you could

take the Golden Gate you could take the

model you could ask it about anything

you know you know it would be like how

you could say how was your day and

anything you asked because this feature

was activated would connect to the

Golden Gate Bridge so it would say you

know I'm I'm I'm feeling relaxed and

expansive much like the the arches of

the Golden Gate Bridge or you know it

would masterfully change topic to the

Golden Gate Bridge and it integrated

there was also a sadness to it to to the

focus ah had on the Golden Gate Bridge I

think people quickly fell in love with

it I think so people already miss it

because it was taken down I think after

a day somehow these interventions on the

model um where where where where you

kind of adjust Its Behavior somehow

emotionally made it seem more human than

any other version of the model strong

personality strong ID strong personality

it has these kind of like obsessive

interests you know we can all think of

someone who's like obsessed with

something so it does make it feel

somehow a bit more human let's talk

about the present let's talk about

Claude so this year A lot has happened

in March claw 3 Opa Sonet Hau were

released then claw 35 Sonet in July with

an updated version just now released and

then also claw 35 hi coup was released

okay can you explain the difference

between Opus Sonet and Haiku and how we

should think about the different

versions yeah so let's go back to March

when we first released uh these three

models so you know our thinking was you

different companies produce kind of

large and small models better and worse

models we felt that there was demand

both for a really powerful model um you

know and you that might be a little bit

slower that you'd have to pay more for

and also for fast cheap models that are

as smart as they can be for how fast and

cheap right whenever you want to do some

kind of like you know difficult analysis

like if I you know I want to write code

for instance or you know I want to I

want to brainstorm ideas or I want to do

creative writing I want the really

powerful model but then there's a lot of

practical applications in a business

sense where it's like I'm interacting

with a website I you know like I'm like

doing my taxes or I'm you know talking

to uh you know to like a legal adviser

and I want to analyze a contract or you

know we have plenty of companies that

are just like you know you know I want

to do autocomplete on my on my IDE or

something uh and and for all of those

things you want to act fast and you want

to use the model very broadly so we

wanted to serve that whole spectrum of

needs um so we ended up with this uh you

know this kind of poetry theme and so

what's a really short poem it's a Haik

cou and so Haiku is the small fast cheap

model that is you know was at the time

was released surprisingly surprisingly

uh intelligent for how fast and cheap it

was uh sonnet is a is a medium-sized

poem right a couple paragraphs since o

Sonet was the middle model it is smarter

but also a little bit slower a little

bit more expensive and and Opus like a

magnum opus is a large work uh Opus was

the the largest smartest model at the

time um so that that was the original

kind of thinking behind it um and our

our thinking then was well each new

generation of models should shift that

tradeoff curve uh so when we release

Sonet 3.5 it has the same roughly the

same you know cost and speed as the

Sonet 3 Model uh but uh it it increased

its intelligence to the point where it

was smarter than the original Opus 3

Model uh especially for code but but

also just in general and so now you know

we've shown results for a Hau 3. 5 and I

believe Hau 3.5 the smallest new model

is about as good as Opus 3 the largest

old model so basically the aim here is

to shift the curve and then at some

point there's going to be an opus 3.5 um

now every new generation of models has

its own thing they use new data their

personality changes in ways that we kind

of you know try to steer but are not

fully able to steer and and so uh

there's never quite that exact

equivalence the only thing you're

changing is intelligence um we always

try and improve other things and some

things change without us without us

knowing or measuring so it's it's very

much an inexact science in many ways the

manner and personality of these models

is more an art than it is a science so

what is sort of the reason for uh the

span of time between say Claude Opus 3

and 35 what is it what takes that time

if you can speak to yeah so there's

there's different there's different uh

processes um uh there's pre-training

which is you know just kind of the

normal language model training and that

takes a very long time um that uses you

know these days you know tens you know

tens of thousands sometimes many tens of

thousands of uh gpus or tpus or tranium

or you know what we use different

platforms but you know accelerator chips

um often often training for months uh

there's then a kind of posttraining

phase where we do reinforcement learning

from Human feedback as well as other

kinds of reinforcement learning that

that phase is getting uh larger and

larger now and you know you know often

that's less of an exact science it often

takes effort to get it right um models

are then tested with some of our early

Partners to see how good they are and

they're then tested both internally and

externally for their safety particularly

for catastrophic and autonomy r risks uh

so uh we do internal testing according

to our responsible scaling policy which

I you know could talk more about that in

detail and then we have an agreement

with the US and the UK AI safety

Institute as well as other third-party

testers in specific domains to test the

models for what are called cbrn risk

chemical biological radiological and

nuclear which are you know we don't

think that models pose these risks

seriously yet but but every new model we

want to evaluate to see if we're

starting to get close to some of these

these these more dangerous um uh these

more dangerous capabilities so those are

the phases and then uh you know then

then it just takes some time to get the

model working in terms of inference and

launching it in the API so there's just

just a lot of steps to uh to actually to

actually making a model work and of

course you know we're always trying to

make the processes as streamlined as

possible right we want our safety

testing to be rigorous but we want it to

be RoR ous and to be you know to be

automatic to happen as fast as it can

without compromising on rigor same with

our pre-training process and our

posttraining process so you know it's

just like building anything else it's

just like building airplanes you want to

make them you know you want to make them

safe but you want to make the process

streamlined and I think the creative

tension between those is is you know is

an important thing and making the models

work yeah uh rumor on the street I

forget who was saying that uh anthropic

is really good tooling so I uh probably

a lot of the challenge here is on the

software engineering side is to build

the tooling to to have a like a

efficient low friction interaction with

the infrastructure you would be

surprised how much of the challenges of

uh you know building these models comes

down to you know software engineering

performance engineering you know you you

know from the outside you might think oh

man we had this Eureka breakthrough

right you know this movie with the

science we discovered it we figured it

out but but but I think I think all

things even even even you know

incredible discoveries like they they

they they they almost always come down

to the details um and and often super

super boring details I can't speak to

whether we have better tooling than than

other companies I mean you know I

haven't been at those other companies at

least at least not recently um but it's

certainly something we give a lot of

attention to I don't know if you can say

but from three from CLA 3 to CLA 35 is

there any extra pre-training going on or

is they mostly focus on the

post-training there's been leaps in

performance yeah I think I think at any

given stage we're focused on improving

everything at once um just just

naturally like there are different teams

each team makes progress in a particular

area in in in making a particular you

know their particular segment of the

relay race better and it's just natural

that when we make a new model we put we

put all of these things in at once so

the data you have like the preference

data you get from rhf is that applicable

is there ways to apply it to newer

models as it get trained up yeah

preference data from old models

sometimes gets used for new models

although of course uh it it performs

somewhat better when it's you know

trained on it's trained on the new

models note that we have this you know

constitutional AI method such that we

don't only use preference data we kind

of there's also a post-t trainining

process where we train the model against

itself and there's you know new types of

post training the model against itself

that are used every day so it's not just

RF it's a bunch of other methods as well

um post training I think you know it's

becoming more and more sophisticated

well what explains the big leap in

performance for the new Sona 35 I mean

at least in the programming side and

maybe this is a good place to talk about

benchmarks what does it mean to get

better just the number went up but you

know I I I program but I also love

programming and I um claw 35 through

cursor is what I use uh to assist me in

programming and there was at least

experientially anecdotally it's gotten

smarter at programming so what like what

what does it take to get it uh to get it

smarter we observe that as well by the

way there were a couple uh very strong

Engineers here at anthropic um who all

previous code models both produced by us

and produced by all the other companies

hadn't really been useful to to hadn't

really been useful to them you know they

said you know maybe maybe this is useful

to beginner it's not useful to me but

Sonet 3.5 the original one for the first

time they said oh my God this helped me

with something that you know that it

would have taken me hours to do this is

the first model that has actually saved

me time so again the water line is

rising and and then I think you know the

new Sonet has been has been even better

in terms of what it what it takes I mean

I'll just say it's been across the board

it's in the pre-training it's in the

posttraining it's in various evaluations

that we do we've observed this as well

and if we go into the details of the

Benchmark so s bench is basically you

know since since you know since since

you're a programmer you know you'll be

familiar with like PLL requests and you

know uh just just PLL requests are like

you know the like a sort of a sort of

atomic unit of work you know you could

say I'm you know I'm implementing one

I'm implementing one thing um uh and and

so sbench actually gives you kind of a

real world situation where the codebase

is in a current state and I'm trying to

implement something that's you know

that's described in described in

language we have internal benchmarks

where we where we measure the same thing

and you say just give the model free

reign to like you know do anything run

run run anything edit anything um how

how well is it able to complete these

tasks and it's that Benchmark that's

gone from it can do it 3% of the time to

it can do it about 50% of the time um so

I actually do believe that if we get you

can gain benchmarks but I think if we

get to 100% on that Benchmark in a way

that isn't kind of like overtrained or

or or game for that particular Benchmark

probably represents a real and serious

increase in kind of

in kind of programming programming

ability and and I would suspect that if

we can get to you know 90 90 95% that

that that that you know it will it will

represent ability to autonomously do a

significant fraction of software

engineering

tasks well ridiculous timeline question

uh when is clad Opus uh 3.5 coming up uh

not giving you an exact date uh but you

know there there uh you know as far as

we know the plan is still to have a

Claude 3.5 opus are we gonna get it

before GTA 6 or no like Duke Nukem

Forever was that game that there was

some game that was delayed 15 years was

that Duke Nukem Forever yeah and I think

GTA is now just releasing trailers it

you know it's only been three months

since we released the first son it yeah

it's Inc the incredible pace of relas it

just it just tells you about the pace

the expectations for when things are

going to come out so uh what about

40 so how do you think about sort of as

these models get bigger and bigger about

versioning and also just versioning in

general why Sonet 35 updated with the

date why not Sonet

3.6 actually naming is actually an

interesting challenge here right because

I think a year ago most of the model was

pre-training and so you could start from

the beginning and just say okay we're

going to have models of different sizes

we're going to train them all together

and you know we'll have a a family of

naming schemes and then we'll put some

new magic into them and then you know

we'll have the next the next Generation

Um the trouble starts are already when

some of them take a lot longer than

others to train right that already

messes up your time time a little bit

but as you make big improvements in as

you make big improvements in

pre-training uh then you suddenly notice

oh I can make better pre-train model and

that doesn't take very long to do and

but you know clearly it has the same you

know size and shape of previous models

uh uh so I think those two together as

well as the timing timing issues any

kind of scheme you come up with uh you

know the reality tends to kind of

frustrate that scheme right T tends to

kind of break out of the break out of

the scheme it's not like software where

you can say oh this is like you know 3.7

this is 3.8 no you have models with

different different tradeoffs you can

change some things in your models you

can train you can change other things

some are faster and slower at inference

some have to be more expensive some have

to be less expensive and so I think all

the companies have struggled with this

um I think we did very you know I think

think we were in a good good position in

terms of naming when we had Haiku Sonet

and we're trying to maintain it but it's

not it's not it's not perfect um so

we'll we'll we'll try and get back to

the Simplicity but it it um uh just the

the the nature of the field I feel like

no one's figured out naming it's somehow

a different Paradigm from like normal

software and and and so we we just none

of the companies have been perfect at it

um it's something we struggle with

surprisingly much relative to you know

how relative to how trivial it is to you

know for the the the the grand science

of training the models so from the user

side the user experience of the updated

Sonet 35 is just different than the

previous uh June 2024 Sonet 35 it would

be nice to come up with some kind of

labeling that embodies that because

people talk about son 35 but now there's

a different one and so how do you refer

to the previous one and the new one and

it it uh when there's a distinct

Improvement it just makes conversation

about it uh just challenging yeah yeah I

I definitely think this question of

there are lots of properties of the

models that are not reflected in the

benchmarks um I I think I think that's

that's definitely the case and everyone

agrees and not all of them are

capabilities some of them are you know

models can be polite or brusk they can

be uh you know uh very reactive or they

can ask you questions um they can have

what what feels like a warm personality

or a cold personality they can be boring

or they can be very distinctive like

Golden Gate Claude was um and we have a

whole you know we have a whole team kind

of focused on I think we call it Claude

character uh Amanda leads that team and

we'll we'll talk to you about that but

it's still a very inexact science um and

and often we find that models have

properties that we're not aware of the

the fact of the matter is that you can

you know talk to a model 10,000 times

and there are some behaviors you might

not see uh just like just like with a

human right I can know someone for a few

months and you know not know that they

have a certain skill or not know there's

a certain side to them and so I think I

think we just have to get used to this

idea and we're always looking for better

ways of testing our models to to

demonstrate these capabilities and and

and also to decide which are which are

the which are the personality properties

we want models to have have and which we

don't want to have that itself the

normative question is also super

interesting I got to ask you a question

from Reddit from Reddit oh

boy you know there there's just this

fascinating to me at least it's a

psychological social

phenomenon where people report that

Claude has gotten Dumber for them over

time and so uh the question is does the

user complaint about the dumbing down of

claw 35 Sonic hold any water so are

these anecdota reports a kind of social

phenomena or did Claude is there any

cases where Claude would get Dumber so

uh this actually doesn't apply this this

isn't just about Claude I I believe this

I believe I've seen these complaints for

every Foundation model produced by a

major company um people said this about

gp4 they said it about gp4 turbo um so

so so a couple things um one the actual

weights of the model right the actual

brain of the model that does not change

unless we introduce a new model um there

there just a number of reasons why it

would not make sense practically to be

randomly substituting in substituting in

new versions of the model it's difficult

from an inference perspective and it's

actually hard to control all the

consequences of changing the way to the

model let's say you wanted to fine-tune

the model to be like I don't know to

like to say certainly less which you

know an old version of Sonet used to do

um you actually end up changing a 100

things as well so we have a whole

process for it and we have a whole

process for modifying the model we do a

bunch of testing on it we do a bunch of

um like we do a bunch of user testing

and early customers so it we both have

never changed the weights of the model

without without telling anyone and it it

it wouldn't certainly in the current

setup it would not make sense to do that

now there are a couple things that we do

occasionally do um one is sometimes we

run AB tests um but those are typically

very close to when a model is being is

being uh released and for a very small

fraction of time um so uh you know like

the you know the the day before the new

Sonet 3.5 I I agree we should have

should have had a better name it's

clunky to refer to it um there were some

comments from people that like it's got

It's got it's gotten a lot better and

that's because you know a fraction were

exposed to to an AB test for for those

one or for those one or two days um the

other is that occasionally the system

prompt will change um on the system

prompt can have some effects although

it's un it it it's unlikely to dumb down

models it's unlikely to make them Dumber

um and and and and we've seen that while

these two things which I'm listing to be

very complete um happen relatively

happen quite infrequently um the

complaints about to for us and for other

model companies about the model changed

the model isn't good at this the model

got more censored the model was dumb

down those complaints are constant and

so I don't want to say like people are

imagining it or anything but like the

models are for the most part not

changing um if I were to offer a theory

um I I think it actually relates to one

of the things I said before which is

that models have many are very complex

and have many aspects to them and so

often you know if I if I if if I ask a

model a question you know if I'm like if

I'm like do task X versus can you do

task XX the model might respond in

different ways uh and and so there are

all kinds of subtle things that you can

change about the way you interact with

the model that can give you very

different results um to be clear this

this itself is like a failing by by us

and by the other model providers that

that the models are are just just often

sensitive to like small small changes in

wording it's yet another way in which

the science of how these models work is

very poorly developed uh and and so you

know if I go to sleep one night and I

was like talking to the model in a

certain way and I like slightly Chang

the phrasing of how I talk to the model

you know I could I could get different

results so that's that's one possible

way the other thing is man it's just

hard to quantify this stuff uh it's hard

to quantify this stuff I think people

are very excited by new models when they

come out and then as time goes on they

they become very aware of the they

become very aware of the limitations so

that may be another effect but that's

that's all a very long- rended way of

saying for the most part with some

fairly narrow exceptions the models are

not changing I think there is a

psychological effect you just start

getting used to it the Baseline ra like

when people have first gotten Wi-Fi on

airplanes it's like amazing magic and

then now like I can't get this thing to

work this is such a piece of crap

exactly so it's easy to have the

conspiracy theory of they're making

Wi-Fi slower and slower this is probably

something I'll talk to Amanda much more

about but U another Reddit question uh

when will Claud stop trying to be my uh

panical grandmother imposing its moral

World viw on me as a paying customer and

also what does it that ology behind

making Claude overly apologetic so this

kind of reports about The Experience a

different angle on the frustration it

has to do with the character yeah so a

couple points on this first one is um

like things that people say on Reddit

and Twitter or X or whatever it is um

there's actually a huge distribution

shift between like the stuff that people

complain loudly about on social media

and what actually kind of like you know

statistically users care about and that

drives people to use the models like

people are frustrated with you know

things like you know the model not

writing out all the code or the model uh

you know just just not being as good at

code as it could be even though it's the

best model in the world on code um I I

think the majority of thing of things

are about that um uh but uh certainly a

a a kind of vocal minority are uh you

know kind kind of kind of rais these

concerns right are frustrated by the

model refusing things that it shouldn't

refuse or like apologizing too much or

just just having these kind of like

annoying verbal ticks um the second

caveat and I just want to say this like

super clearly because I think it's like

some people don't know it others like

kind of know it but forget it like it is

very difficult to control across the

board how the models behave you cannot

just reach in there and say oh I want

the model to like apologize less like

you can do that you can include trading

data that says like oh the models should

like apologize less but then in some

other situation they end up being like

super rude or like overconfident in a

way that's like misleading people so

they're they're all these tradeoffs um

uh for example another thing is if there

was a period during which models ours

and I think others as well were T

verbose right they would like repeat

themselves they would say too much um

you can cut down on the verbosity by

penalizing the models for for just

talking for too long what happens when

you do that if you do it in a crude way

is when the models are coding sometimes

they'll say of the code goes here right

because they've learned that that's a

way to economize and that they see it

and then and then so that leads the

model to be so-called lazy in coding

where they where they where they're just

like ah you can finish the rest of it

it's not it's not because we want to you

know save on compute or because you know

the models are lazy and you know during

winter break or any of the other kind of

conspiracy theories that have that have

that have come up it's actually it's

just very hard to control the behavior

of the model to steer the behavior of

the model in all circum ances at once

you can kind of there's this this whacka

aspect where you push on one thing and

like you know these these these you know

these other things start to move as well

that you may not even notice or measure

and so one of the reasons that I that I

care so much about uh you know kind of

grand alignment of these AI systems in

the future is actually these systems are

actually quite unpredictable they're

actually quite hard to steer and control

um and this version we're seeing today

of you make one thing better it makes

another thing worse uh I think that's

that's like a present day analog of

future control problems in AI systems

that we can start to study today right I

think I think that that that difficulty

in in steering the behavior and in

making sure that if we push an AI system

in One Direction it doesn't push it in

another Direction in some in some other

ways that we didn't want uh I think

that's that's kind of an that's kind of

an early sign of things to come and if

we can do a good job of solving this

problem right of like you ask the model

to like you know to like make and

distribute small pox and it says no but

it's willing to like help you in your

graduate level virology class like how

do we get both of those things at once

it's hard it's very easy to go to one

side or the other and it's a

multi-dimensional problem and so uh I

you know I think these questions of like

shaping the models personality I think

they're very hard I think we haven't

done perfectly on them I think we've

actually done the best of all the AI

companies but still so far from perfect

uh and I think if we can get this right

if we can control the the you know

control the false positives and false

negatives in this this very kind of

controlled present day environment will

be much better at doing it for the

future when our worry is you know will

the models be super autonomous will they

be able to you know make very dangerous

things will they be able to autonomously

you know build whole companies and are

those companies aligned so so I I I

think of this this present task as both

vacine but also good practice for the

future what's the current best way of

gathering sort of user feedback like uh

not anecdotal data but just large scale

data about pain points or the opposite

of pain points positive things so on is

it internal testing is it yeah A

specific group testing a testing what

what what works so so so typically um

we'll have internal model bashings where

all of anthropic anthropic is almost a

thousand people um you know people just

just try and break the model they try

and interact with it various ways um uh

we have a suite of evals uh for you know

oh is the model refusing in ways that

that it couldn't I think we even had a

certainly eval because you know our our

mod again at one point model had this

problem where like it had this annoying

tick where it would like respond to a

wide range of questions by saying

certainly I can help you with that

certainly I would be happy to do that

certainly this is correct um uh and so

we had a like certainly eval which is

like how how often does the model say

certainly uh uh but but look this is

just a whack-a-mole like like what if it

switches from certainly to definitely

like uh uh so you know every time we add

a new eval and we're always evaluating

for all the old things so we have

hundreds of these evaluations but we

find that there's no substitute for

human interacting with it and so it's

very much like the ordinary product

development process we have like

hundreds of people within anthropic bash

the model then we do uh you know then we

do external AB tests sometimes we'll run

tests with contractors we pay

contractors to interact with the model

um so you put all of these things

together and it's still not perfect you

still see behaviors that you don't quite

want to see right you know you see you

still see the model like refusing things

that it just doesn't make sense to

refuse um but I I I think trying to

trying to solve this challenge right

trying to stop the model from doing you

know genuinely bad things that you know

no one everyone agrees it shouldn't do

right you know everyone everyone you

know everyone agrees that you know the

model shouldn't talk about you know I I

don't know child abuse material right

like everyone agrees the model shouldn't

do that uh but but at the same time that

it doesn't refuse in these dumb and

stupid ways uh I think I think draw

drawing that line as finely as possible

approaching perfectly is still is still

a challenge and we're getting better at

it every day but there's there's a lot

to be solved and again I would point to

that as as an indicator of a challenge

ahead in terms of steering much more

powerful models do you think Claude 4.0

is ever coming out I don't want to

commit to any naming scheme because if I

say if I say here we're gonna have

Claude 4 next year and then and then you

know then we decide that like you know

we should start over because there's a

new type of mod like I I I I I I don't

want to I don't want to commit to it I

would expect in a normal course of

business that Claude four would come

after Claude 3.5 but but you know you

you you never know in this wacky field

right but the sort of this idea of

scaling is continuing scal scaling is

continuing there there will definitely

be more powerful models coming from us

in the models that exist today that is

that is certain or if there if there

aren't we've we've deeply failed as a

company okay can you explain the

responsible scaling policy and the AI

safety level standards ASL levels as

much as I'm excited about the benefits

of these models and you know we'll talk

about that if we talk about Machines of

Loving Grace um I'm I'm worried about

the risk and I continue to be worried

about the risks uh no one should think

that you know Machines of loveing Grace

was me me saying uh you know I'm no

longer worried about the risks of these

models I think they're two sides of the

same coin the the uh Power of the models

and their ability to solve all these

problems in you know biology

Neuroscience Economic Development

government governance and peace large

parts of the economy those those come

with risks as well right with great

power comes great responsibility right

that's the the two are the two are

paired uh things that are powerful can

do good things and they can do bad

things um I think of those risks as as

being in you know several different

different categories perhaps the two

biggest risks that I think about and

that's not to say that there aren't

risks today that are that are important

but when I think of the really the the

you know the things that would happen on

the grandest scale um one is what I call

catastrophic misuse these are misuse of

the models in domains like cyber bio

radiological nuclear right things that

could you know that could harm or even

kill thousands even millions of people

if they really really go wrong um like

these are the you know number one

priority to prevent and and here I would

just make a simple observation which is

that Mo the models you know if if I look

today at people who have done really bad

things in the world um uh I think

actually Humanity has been protected by

the fact that the overlap between really

smart well-educated people and people

who want to do really horrific things

has generally been small like you know

let's say let's say I'm someone who you

know uh you know I have a PhD in this

field I have a well-paying job um

there's so much to lose why do I want to

like you know even even assuming I'm

completely evil which which most people

are not um why why you know why would

such a person risk their risk their you

know risk their life RK risk their their

legacy their reputation to to do

something like you know truly truly evil

if we had a lot more people like that

the world would be a much more dangerous

place and so my my My worry is that by

being a a much more intelligent agent AI

could break that correlation and so I I

I I I do have serious worries about that

I believe we can prevent those worries

uh but you know I I think as a

Counterpoint to Machines of Loving Grace

I want to say that this is I there's

still serious risks and and the second

range of risks would be the autonomy

risks which is the idea that models

might on their own particularly as we

give them more agency than they've had

in the past uh particularly as we give

them supervision over wider tasks like

you know writing whole code bases or

someday even you know effectively

operating entire entire companies

they're on a long enough leash are they

are they doing what we really want them

to do it's very difficult to even

understand in detail what they're doing

let alone let alone control it and like

I said this these early signs that it's

it's hard to perfectly draw the boundary

between things the model should do and

things the model shouldn't do that that

you know if if you go to one side you

get things that are annoying and useless

and you go to the other side you get

other behaviors if you fix one thing it

creates other problems we're getting

better and better at solving this I

don't think this is an unsolvable

problem I think this is a you know this

is a science like like the safety of

airplanes or the safety of cars or the

safety of drugs I you know I I don't

think there's any big thing we're

missing I just think we need to get

better at controlling these models and

so these are these are the two risks I'm

worried about and our responsible

scaling plan which I'll recognize is a

very long-winded answer to your question

I love it I love it our responsible

scaling plan is designed to address

these two types of risks and so every

time we develop a new model we basically

test it for its ability to do both of

these bad things so if I were to back up

a little bit um I I think we have a I

think we have an interesting dilemma

with AI systems where they're not yet

powerful enough to present these

catastrophes I don't know that I don't

know they'll ever present prevent these

catastrophes it's possible they won't

but the the case for worry the case for

risk is strong enough that we should we

should act now and and they're they're

getting better very very fast right I

you know I testified in the Senate that

you know we might have serious bio risks

within two to three years that was about

a year ago things have preceded preceded

a pace uh uh so we have this thing where

it's like it's it's it's surprisingly

hard to to address these risks because

they're not here today they don't exist

they're like ghosts but they're coming

at us so fast because the models are

improving so fast so so how do you deal

with something that's not here today

doesn't exist but is is coming at us

very fast uh so the solution we came up

with for that in in collaboration with

uh you know people like uh the

organization meter and Paul Christiano

is okay what what what what you need for

that or you need tests to tell you when

the risk is getting close you need an

early warning system and and so every

time we have uh a new model we test it

for it capability to do these cbrn tasks

as well as testing it for you know how

capable it is of doing tasks

autonomously on its own and uh in the

latest version of our RSP which we

released in the last in the last month

or two uh the way we test autonomy risks

is the model the the AI model's ability

to do aspects of AI research itself uh

which when the model when the AI models

can do AI research they become kind of

truly truly autonomous on and that you

know that threshold is important for a

bunch of other ways and and so what do

we then do with these tasks the RSP

basically develops what we've called an

if then structure which is if the models

pass a certain capability then we impose

a certain set of Safety and Security

requirements on them so today's models

are what's called

asl2 models that were a asl1 is for

systems that manifestly don't pose any

risk of autonomy or misuse so for

example a chess plane bot deep blue

would be asl1 it's just manifestly the

case that you can't use deep blue for

anything other than chess it was just

designed for chess no one's going to use

it to like you know to conduct a

masterful Cyber attack or to you know

run wild and take over the world asl2 is

today's AI systems where we've measured

them and we think these systems are

simply not smart enough to uh to you

know autonomously self-replicate or

conduct a bunch of tasks uh and also not

smart enough to provide meaningful

information about cbrn risks and how to

build cbrn weapons above and beyond what

can be known from looking at Google uh

in fact sometimes they do provide

information but but not above and beyond

a search engine but not in a way that

can be stitched together um not not in a

way that kind of end to end is dangerous

enough so

asl3 is going to be the point at which

uh the models are helpful enough to

enhance the capabilities of non-state

actors right State actors can already do

a lot a lot of unfortunately to a high

level of proficiency a lot of these very

dangerous and destructive things the

difference is that non-state non-state

actors are not capable of it and so when

we get to asl3 we'll take special

security precautions designed to be be

sufficient to prevent theft of the model

by non-state actors and misuse of the

model as it's deployed uh will have to

have enhanced filters targeted at these

particular areas cyber bio nuclear cyber

bio nuclear and model autonomy Which is

less a misuse risk and more a risk of

the model doing bad things itself asl4

getting to the point where these models

could could enhance the capability of a

of a of a all knowledgeable State actor

Andor become the you know the main

source of such a risk like if you wanted

to engage in such a risk the main way

you would do it is through a model and

then I think asl4 on the autonomy side

it's it's some some some amount of

acceleration in AI research capabilities

with an with an AI model and then asl5

is where we would get to the models that

are you know that are that are kind of

that are kind of you know truly capable

that it could exceed Humanity in their

ability to do to do any of these tasks

and so the the the point of the if then

structure commitment is is basically to

say

look I don't know I've been I've been

working with these models for many years

and I've been worried about risk for

many years it's actually kind of

dangerous to cry wolf it's actually kind

of dangerous to say this you know this

this model is this model is risky and

you know people look at it and they say

this is manifestly not dangerous again

it's it's it's the the delicacy of the

risk isn't here to today but it's coming

at us fast how do you deal with that

it's it's really vexing to a risk

planner to deal with it and so this if

then structure basically says look we

don't want to antagonize a bunch of

people we don't want to harm our own you

know our our kind of own ability to have

a place in the conversation by imposing

these

these very honorous burdens on models

that are not dangerous today so the if

then the trigger commitment is basically

a way to deal with this says you claim

clamp down hard when you can show that

the model is dangerous and of course

what has to come with that is you know

enough of a buffer threshold that that

you know you can you can uh you know

you're you're you're you're not at high

risk of kind of missing the danger it's

not a perfect framework we've had to

change it every every uh you know we

came out with a new one just a few weeks

ago and probably probably going forward

we might release new ones multiple times

a year because it's it's hard to get

these policies right like technically

organizationally from a research

perspective but that is the proposal if

then commitments and triggers in order

to minimize burdens and false alarms now

but really react appropriately when the

dangers are here what do you think the

timeline for asl3 is where several of

the triggers are fired and what do you

think the timeline is for asl4 yeah so

that is hotly debated within the company

um uh we are working actively to prepare

asl3 uh security uh security measures as

well as ASL three deployment measures um

I'm not going to go into detail but

we've made we've made a lot of progress

on both and you know we're we're

prepared to be I think ready quite soon

uh I would I would not be surpris I

would not be surprised at all if we hit

ASL 3 uh next year there was some

concern that we we might even hit it uh

uh this year that's still that's still

possible that could still happen it's

like very hard to say but like I would

be very very surprised if it was like

2030 uh I think it's much sooner than

that so there's a protocols for

detecting it the if then and then

there's protocols for how to respond to

it yes how difficult is the second the

ladder yeah I think for asl3 it's

primarily about security um and and

about you know filters on the model

relating to a very narrow set of areas

when we deploy the model because at asl3

the model isn't autonomous yet um uh and

and so you don't have to worry about you

know kind of the model itself behaving

in a bad way even when it's deployed

internally so I think the asl3 measures

are are I won't say straightforward

they're they're they're they're rigorous

but they're easier to reason about I

think once we get to

asl4 um we start to have worries about

the models being smart enough that they

might sandbag tests they might not tell

the truth about tests um we had some

results came out about like sleeper

agents and there was a more recent paper

about you know can can the models uh uh

mislead attempts to you know s sandbag

their own abilities right show them you

know uh uh present themselves as being

less capable than they are and so I

think with asl4 there's going to be an

important component of using other

things than just interacting with the

models for example interpretability or

hidden chains of thought uh where you

have to look inside the model and verify

via some other mechanism that that is

not you know is not as easily corrupted

as what the model says

uh that that you know that that that the

model indeed has some property uh so

we're still working on asl4 one of the

properties of the RSP is that we we

don't specify asl4 until we've hit ASL 3

be and and I think that's proven to be a

wise decision because even with asl3 it

again it's hard to know this stuff in

detail and and it it we want to take as

much time as we can possibly take to get

these things right so for asl3 the bad

actor will be the humans humans yes and

so there it's a little bit more uh for

asl4 it's both I think it's both and so

deception and that's where mechanistic

interpretability comes into play and

hopefully the techniques used for that

are not made accessible to the model

yeah I mean of course you can hook up

the mechanistic contribut ability to the

model itself um but then You' then then

you then you've kind of lost it as a

reliable indicator of uh of uh of of of

the model State there are a bunch of

exotic ways you can think of that it

might also not be reliable like if the

you know model gets smart enough that it

can like you know jump computers and

like read the code where you're like

looking at its internal State we've

thought about some of those I think

they're exotic enough there are ways to

render them unlikely but yeah generally

you want to you want to preserve

mechanistic interpretability as a kind

of verification set or test set that's

separate from the training process of

the model see I think uh as these models

become better and better conversation

and become smarter social engineering

becomes a threat too cuz they oh yeah

that can start being very convincing to

the engineers inside companies oh yeah

yeah it's actually like you know we've

we've seen lots of examples of

demagoguery in our life from humans and

and you know there's a concern that

models could do that could do that as

well one of the ways that cloud has been

getting more and more powerful is it's

now able to do some agentic stuff um

computer use uh there's also an analysis

within the sandbox of claw. a itself but

let's talk about computer use that's

seems to me super exciting that you can

just give Claude a task and it uh takes

a bunch of actions figures it out and

has access to the your computer through

screenshots so can you explain how that

works uh and where that's headed yeah

it's actually relatively simple so

Claude has has had for a long time since

since Claude 3 back in March the ability

to analyze images and respond to them

with text the the only new thing we

added is those images can be screenshot

shots of a computer and in response we

train the model to give a location on

the screen where you can click Andor

buttons on the keyboard you can press in

order to take action and it turns out

that with actually not all that much

additional training the models can get

quite good at that task it's a good

example of generalization um you know

people sometimes say if you get to low

earth orbit you're like halfway to

anywhere right because of how much it

takes to escape the gravity well if you

have a strong pre-trained model I feel

like you're halfway to anywhere uh in

ter in terms of in terms of the

intelligence space uh uh uh and and and

so actually it didn't it didn't take all

that much to get to get Claude to do

this and you can just set that in a loop

give the model a screenshot tell it what

to click on give it the next screenshot

tell it what to click on and and that

turns into a full kind of almost almost

3D video interaction of the model and

it's able to do all of these tasks right

you know we we showed these demos where

it's able to like fill out spreadsheets

it's able to kind of like interact with

a website it's able to you know um you

know it's able to open all kinds of you

know programs different operating

systems Windows Linux Mac uh uh so uh

you know I think all of that is very

exciting I I will say while in theory

there's nothing you could do there that

you couldn't have done through just

giving the model the API to drive the

computer screen uh this really lowers

the barrier and you know there's there's

there's a lot of folks who who who

either you know kind of kind of ar ar

you know aren't in a position to to

interact with those apis or it takes

them a long time to do it's just the

screen is just a universal interface

that's a lot easier to interact with and

so I expect over time this is going to

lower a bunch of barriers now honestly

the current model has there's there it

leaves a lot still to be desired and we

were we were honest about that in the

blog right it makes mistakes it

misclicks and we we you know we were

careful to warn people hey this thing

isn't you can't just leave this thing to

you know run on your computer for

minutes and minutes um you got to give

this thing boundaries and guard rails

and I think that's one of the reasons we

released it first in an API form rather

than kind of you know this this kind of

just just hand it just hand it to the

consumer and give it control of their of

their of their of their computer um but

but you know I definitely feel that it's

important to get these capabilities out

there as models get more powerful we're

going to have to Grapple with you know

how do we use these capabilities safely

how do we prevent them from being abused

uh and and you know I think I think

releasing releasing the model while

while while the capabilities are are you

know are are still are still limited is

is is very helpful in terms of in terms

of doing that um you know I think since

it's been released a number of customers

I think uh repet was maybe was maybe one

of the the the most uh uh quickest

quickest quickest to quickest to deploy

things um have have you know have made

use of it in various ways people have

hooked up demos for you know Windows

desktops Macs

uh uh you know Linux Linux machines uh

so yeah it's been it's been it's been

very exciting I think as with as with

anything else you know it it it comes

with new exciting abilities and then

then then you know then then with those

new exciting abilities we have to think

about how to how to you know make the

model you know safe reliable do what

humans want them to do I mean it's the

same it's the same story for everything

right same thing it's that same tension

but but the possibility of use cases

here is just the the range is incredible

so uh how much to make it work really

well in the future how much do you have

to specially kind of uh go beyond what's

the pre-trained models doing do more

posttraining rhf or supervised

fine-tuning or synthetic data just for

the agent stff yeah I think speaking at

a high level It's Our intention to keep

investing a lot in you know making

making the model better uh like I think

I think uh you know we look at look at

some of the you know some of the

benchmarks where previous models were

like oh could do it 6% of the time and

now our model do at 14 or 22% of the

time and yeah we want to get up to you

know the human level reliability of 80

90% just like anywhere else right we're

on the same curve that we were on with

sbench where I think I would guess a

year from now the models can do this

very very reliably but you got to start

somewhere so you think it's possible to

get to the the human level 90% uh

basically doing the same thing you're

doing now or is it has to be special for

computer use I I mean uh depends what

you mean by by you know special and FAL

and

um but but you know I generally think

you know the same kinds of techniques

that we've been using to train the

current model I I expect that doubling

down in those techniques in the same way

that we have for code for code for

models in general for other k for you

know for image input um uh you know for

voice uh I expect those same techniques

will scale here as they have everywhere

else but this is giving sort of the

power of action to Claude And so you

could do a lot of really powerful things

but you could do a lot of damage also

yeah yeah no and we've been very aware

of that look my my view actually is

computer use isn't a fundamentally new

capability like the cbrn or autonomy

capabilities are um it's more like it

kind of opens the aperture for the model

to use and apply its existing abilities

uh and and so the way we think about it

going back to our RSP is nothing that

this model is

doing inherently increases

you know the risk from an RSP RSP

perspective but as the models get more

powerful having this capability may make

it scarier once it you know once it has

the cognitive capability to um you know

to do something at the asl3 and asl4

level this this you know this may be the

thing that kind of Unbound it from doing

so so going forward certainly this

modality of interaction is something we

have tested for and that we will

continue to test for in our going

forward um I think it's probably better

to have to learn and explore this

capability before the model is super uh

you know super capable yeah and there's

uh a lot of interesting attacks like

prompt injection because now you've

widened the aperture so you can prompt

inject through stuff on screen so if

this becomes more and more useful then

there's more and more benefit to inject

inject stuff into the model if it goes

to certain web page it could be harmless

stuff like advertisements or it could be

like harmful stuff right yeah I mean we

thought a lot about like spam capture

you know Mass C there's all you know

every every like if one secret I'll tell

you if you've invented a new technology

not necessarily the biggest misuse but

but the the first misuse you'll see

scams just Petty scams like you just

just just it's it's like it's like a

thing as old people scamming each other

it's it's this it's this thing as old as

time um and and and it's just every time

you got to deal with it it's almost like

silly to say but it's it's true sort of

and spam in general is a thing as it

gets more and more intelligent it's uh

there a lot of like like I said like

there are a lot of petty criminals in

the world and and and you know it's like

every new technology is like a new way

for petty petty criminals to do

something you know something stupid and

malicious um is there any ideas about

sandboxing it like how difficult is the

sandboxing task yeah we sandbox during

training so for example during training

we didn't expose the model to the

internet um I think that's probably a

bad idea during training because uh you

know the model can be changing its

policy it can be changing what it's

doing and it's having an effect in the

real world um uh you know in in terms of

actually deploying the model right it

kind of depends on the application like

you know sometimes you want the model to

do something in the real world but of

course you can always put guard you can

always put guard rails on the outside

right you can say okay well you know

this model is not going to move data

from my you know this model is not going

to move any files from my computer to or

my web server to anywhere else now when

you talk about sandboxing again when we

get to asl4 none of these precautions

are going to make sense there right

where when you when you talk about asl4

you're then the model is being kind of

you know there's a a theoretical worry

the model could be smart enough to break

it to to kind of break out of any box

and so there we need to think about

mechanistic interpretability about you

know if we're if we're going to have a

Sandbox it would need to be a

mathematically provable sound but you

know that's that's a whole different

world than what we're dealing with with

the models

today yeah the science of building a box

from which asl4 AI system cannot Escape

I I think it's probably not the right

approach I think the right approach

instead of having something you know

unaligned that that like you're trying

to prevent it from escaping I think it's

it's better to just design the model the

right way or have a loop where you you

know you look inside you look inside the

model and you're able to verify property

and that gives you an opportunity to

like iterate and actually get it right

um I think I think containing uh

containing bad models is is is much

worse solution than having good models

let me ask about regulation what's the

role of regulation in keeping AI safe so

for example he described California AI

regulation Bill SB 1047 that was

ultimately vetoed by the governor what

are the pros and cons of this bill

General yes we ended up making some

suggestions to the bill and then some of

those were opted and you know we felt I

think I think quite positively uh uh

quite positively about about the bill uh

by by the end of that um it did still

have some downsides um uh and you know

of course of course it got vetoed um I

think at a high level I think some of

the key ideas behind the bill um are you

know I would say similar to ideas behind

our rsps and I think it's very important

that some jurisdiction whether it's

California or the federal government

Andor other countries and other states

passes some regulation like this and I

can talk through why I think that's so

important so I feel good about our RSP

it's not perfect it needs to be iterated

on a lot but it's been a good forcing

function for getting the company to take

these risks seriously to put them into

product planning to really make them a

central part of work at anthropic and to

make sure that all the thousand people

and it's almost a thousand people now at

anthropic understand that this is one of

the highest priorities of the company if

not the highest priority uh

but one there are some there are still

some companies that don't have RSP like

mechanisms like open aai Google uh did

adopt these mechanisms a couple months

after uh after anthropic did uh but

there are there are other companies out

there that don't have these mechanisms

at all uh and so if some companies adopt

these mechanisms and others don't uh

it's really going to create a situation

where you know some of these dangers

have the property that it doesn't matter

if three out of five of the companies

are being safe if the other two are are

being are being unsafe it creates this

negative externality and and I think the

lack of uniformity is not fair to those

of us who have put a lot of effort into

being very thoughtful about these

procedures the second thing is I don't

think you can trust these companies to

adhere to these voluntary plans in their

own right I like to think that anthropic

will we do everything we can that we

will our our our our RSP is checked by

our long-term benefit trust uh so you

know we do everything we can to to to

adhere to our own RSP um but you know

you hear lots of things about various

companies saying oh they said they would

do they said they would give this much

compute and they didn't they said they

would do this thing and they didn't um

you know I don't I don't think it makes

sense to you know to to to you know

litigate particular things that

companies have done but I I think this

this broad principle that like if

there's nothing watching over them

there's nothing watching over us as an

industry there's no guarantee that we'll

do the right thing and the stakes are

very high uh and so I think it's I think

it's important to have a uniform

standard that that that that that

everyone follows and to make sure that

simply that the industry does what a

majority of the industry has already

said is important and has already said

that they definitely will do right some

people uh you know I think there's there

a class of people who are against

regulation on principle I understand

where that comes from if you go to

Europe and you know you see something

like gdpr you see some of the other

stuff that that that that that they've

done you know some of it's good but but

some of it is really unnecessarily

burdensome and I think it's fair to say

really has slowed really has slowed

Innovation and so I understand where

people are coming from on priors I

understand why people come from start

from that start from that position uh

but but again I think AI is different if

we go to the very serious risks of

autonomy and misuse that that that I

talked about you know just a just a few

minutes ago I think that those are

unusual and they weren't an unusually

strong response uh and so I I think it's

very important again um we need

something that everyone can get behind

uh you know I think one of the issues

with

s1047 uh especially the original version

of it was it it had a bunch of the

structure of rsps but it also had a

bunch of stuff that was either clunky or

that that that just would have created a

bunch of burdens a bunch of Hassle and

might even have missed the Target in

terms of addressing the risks um you

don't really hear about it on Twitter

you just hear about kind of you know

people are people are cheering for any

regulation and then the folks who are

against make up these often quite

intellectually dishonest arguments about

how you know it you know it'll make us

move away from California bill bill

doesn't apply if you're headquartered in

California bill only applies if you do

business in California um or that it

would damage the open source ecosystem

or that it would you know it would cause

cause all of these things I I think

those were mostly nonsense but there are

better arguments against regulation

there's one guy uh Dean ball who's

really you know I think a very scholarly

scholarly IST who who looks at what

happens when a regulation is put in

place and ways that they can kind of get

a life of their own or how they can be

poorly designed and so our interest has

always been we do think there should be

regulation in this space but we want to

be an actor who makes sure that that

that that regulation is something that's

surgical that's targeted at the serious

risks and is something people can

actually comply with because something I

think The Advocates of Regulation don't

understand as well as they could is if

we get something in place that is um

that's poorly targeted that wastes a

bunch of people's time what's going to

happen is people are going to say see

these safety risks there you know this

is this is nonsense I just you know I

just had to hire 10 lawyers to to you

know to fill out all these forms I had

to run all these tests for something

that was clearly not dangerous and after

6 months of that there will be there

will be a ground sweep well and we'll

we'll we'll we'll end up with a durable

consensus against regulation and so the

I I think the the worst enemy of those

who want real accountability is badly

designed regulation um we we need to

actually get it right uh and and this is

if there's one thing I could say to The

Advocates it it would be that I want

them to understand this Dynamic better

and we need to be really careful and we

need to talk to people who actually have

who actually have experience seeing how

regulations play out in practice and and

the people who have seen that understand

to be very careful if this was some

lesser issue I might be against

regulation at all but what what I want

the opponents to understand is is that

the underlying issues are actually

serious they're they're not they're not

something that I or the other companies

are just making up because of regulatory

capture they're not sci-fi fantasies

they're not they're not any of these

things um you know every every time we

have new model every few months we

measure the behavior of these models and

they're getting better and better at

these concerning tasks just as they are

getting better and better at um you know

good valuable economically useful tasks

and so I I I I would just love it if

some of the former you know I think

sb147 was very polarizing I would love

it if some of the most reasonable

opponents and some of the most

reasonable um uh proponents uh would sit

down together and you know I think I

think that you know the different the

different AI companies um you know

anthropic was the the only AI company

that you know felt positively in a very

detailed way I think Elon tweeted uh

tweeted briefly something positive but

you know some of the some of the big

ones like Google open AI meta Microsoft

were were pretty St stly against so I

would really like is if if you know some

of the key stakeholders some of the you

know thoughtful proponents and and some

of the most thoughtful opponents would

sit down and say how do we solve this

problem in in a way that the proponents

feel brings a real reduction in risk and

that the opponents feel that it is not

it is not hampering the the industry or

hampering Innovation any more necessary

than it than than than it needs to and

and I think for for whatever reason that

things got too polarized and those two

groups

didn't get to sit down in the way that

they should uh and and I feel I feel

urgency I really think we need to do

something in

2025 uh uh you know if we get to the end

of 2025 and we've still done nothing

about this then I'm going to be worried

I'm not I'm not worried yet because

again the risks aren't here yet but but

I I I think time is running short yeah

and come up with something surgical like

you said yeah yeah yeah exactly and and

we need to get we need to get away from

this this this intense

pro- safety versus intense

anti-regulatory rhetoric right it's

turned into these these flame Wars on

Twitter and nothing Good's going to come

with that so there's a lot of curiosity

about the different players in the game

one of the uh ogs is open AI you have

had several years of experience at open

AI what's your story and history there

yeah so I was at open AI for uh for

roughly five years uh for the last I

think it was a couple years you know I I

I I I I was uh vice president of

research there um probably myself and

Ilia suger were the ones who you know

really kind of set the set the research

Direction around 2016 or 2017 I first

started to really believe in or at least

confirm my belief in the scaling

hypothesis when when Ilia famously said

to me the thing you need to understand

about these models is they just want to

learn the models just want to learn um

and and and and again sometimes there

are these One S there these one

sentences these Zen cones that you hear

them and you're like ah that that

explains everything that explains like a

thousand things that I've seen and then

and then I I you know ever after I had

this visualization in my head of like

you optimize the models in the right way

you point the models in the right way

they just want to learn they just want

to solve the problem regardless of what

the problem is so get out of their way

basically get out of their way yeah

don't impose your own ideas about how

they should learn and you know this was

the same thing as Rich Sutton put out in

the bitter lesson or G put out in the

scaling hypothesis you know I think

generally the dynamic was you know I got

I got this kind of inspiration from uh

from from from Ilan from others folks

like Alec Radford who did the the

original uh uh

gpt1 uh and then uh ran really hard with

it me me and my collaborators on gpt2

gpt3 RL from Human feedback which was an

attempt to kind of deal with the early

safety and durability things like debate

and amplification heavy on

interpretability so again the

combination of safety plus scaling

probably 2018 2019 2020 those those were

those were kind of the years when myself

and my collaborators probably um you

know mo mo many many of whom became

co-founders of anthropic kind of really

had had had a vision and like and like

drove the direction why'd you leave why'

you decid to leave yeah so look I'm

gonna put things this way and I you know

I think it I think it ties to the to to

the race to the top right which is you

know in my time at open AI what I come

come to see as I'd come to appreciate

the scaling hypothesis and as I'd come

to appreciate kind of the importance of

safety along with the scaling hypothesis

the first one I think you know open AI

was was getting was getting on board

with um the second one in a way had

always been part of of open ai's

messaging um but uh you know over over

many years of of the time the time that

I spent there I think I had a particular

vision of how these how we should handle

these things how we should be brought

out in the world the kind of principles

that the organization should have and

look I mean there were like many many

discussions about like you know should

the or do should the company do this

should the company do that like there's

a bunch of misinformation out there

people say like we left because we

didn't like the deal with Microsoft

false although you know there was like a

lot of discussion a lot of questions

about exactly how we do the deal with

Microsoft um we left because we didn't

like commercialization that's not true

we built gbd3 which was the model that

was commercialized I was involved in

commercialization it's it's more again

about how do you do it like Civilization

is going down this path to very powerful

AI what's the way to do it that is

cautious

straightforward honest um that build

trust in the organization and in

individuals how do we get from here to

there and how do we have a real vision

for how to get it right how can safety

not just be something we say because it

helps with recruiting um and you know I

think I think at the end of the day um

if you have a vision for that forget

about anyone else's Vision I don't want

to talk about anyone else's Vision if

you have a vision for how to do it you

should go off and you should do that

Vision it is incredibly unproductive to

try and argue with someone else's Vision

you might think they're not doing it the

right way you might think they're

they're they're dishonest who knows

maybe you're right maybe you're not um

uh but uh what what you should do is you

should take some people you trust and

you should go off together and you

should make your vision happen and if

your vision is compelling if you can

make it appeal to people some you know

some combination of ethically you know

in the market uh you know if if you can

if you can make a company that's a place

people want to join uh that you know

engages in practices that people think

are are reasonable while managing to

maintain its position in the ecosystem

at the same time if you do that people

will copy it um and the fact that you

were doing it especially the fact that

you're doing it better than they are um

causes them to change their behavior in

a much more compelling way than if

they're your boss and you're arguing

with them I just I don't know how to be

any more specific about it than that but

I think it's generally very unproductive

to try and get someone else's Vision to

look like your vision um it's much more

productive to go off and do a clean

experiment and say this is our vision

this is how this is this is how we're

going to do things your choice is you

can you can ignore us you can reject

what we're doing or you can you can

start to become more like us and

imitation is the sincerest form of

flattery um and you know that that that

plays out in the behavior of customers

that PS out in the behavior of the

public that plays out in the behavior of

where people choose to work uh and again

again at the end it's it's not about one

company winning or another company

winning if if we or another company are

engaging in some practice that you know

people people find genuinely appealing

and I want it to be in substance not

just not just in appearance um and you

know I think I think researchers are

sophisticated and they look at substance

uh and then other companies start

copying that practice and they win

because they copied that practice that's

great that's success that's like the

race to the top it doesn't matter who

wins in the end as long as everyone is

copying everyone else's good practices

right one way I think of it is like the

thing we're all afraid of is a race the

bottom right in the race to the bottom

doesn't matter who wins because we all

lose right like you know in the most

extreme world we we make this autonomous

AI that you know the robots enslave us

or whatever right I mean that's half

joking but you know that that is the

most extreme uh uh thing thing that

could happen then then it doesn't matter

which company was ahead um if instead

you create a race to the top where

people are competing to engage in good

in good practices uh then you know at at

the end of the day you know it doesn't

matter who who ends up who ends up

winning doesn't even matter who who

started the race to the top the point

isn't to be virtuous the point is to get

the system into a better equilibrium

than it was before and and individual

companies can play some role in doing

this individual companies can can you

know can help to start it can help to

accelerate it and frankly I think

individuals at other companies have have

done this as well right the individuals

that when we put out an RSP react by

pushing harder to to to get something

similar done get something similar done

at at at other companies sometimes other

companies do something that's like we're

like oh it's a good practice we think we

think that's good we should adopt it too

the only difference is you know I think

I think we are um we try to be more

forward leaning we try and adopt more of

these practices first and adopt them

more quickly when others when others

invent them but I think this Dynamic is

what we should be pointing at and that I

think I think it abstracts away the

question of you know which company's

winning who trusts who I I think all

these all these questions of drama are

are profoundly uninteresting and and the

the thing that matters is the ecosystem

that we all operate in and how to make

that ecosystem better because that

constrains all the players and so

anthropic is this kind of clean

experiment built on a foundation of like

what concretely AI safety should look

like we look I'm sure we've made plenty

of mistakes along the way the perfect

organization doesn't exist it has to

deal with the the imperfection of a

thousand employees it has to deal deal

with the imperfection of our leaders

including me it has to deal with the

imperfection of the people we've put

we've put to you know to oversee the

imperfection of the of the leaders like

the like the board and the long-term

benefit trust it's it's all it's all a

set of imperfect people trying to aim

imperfectly at some ideal that will

never perfectly be achieved um that's

what you sign up for that's what it will

always be but uh uh imperfect doesn't

mean you just give up there's better and

there's worse and hopefully hopefully we

can begin to build we can do well enough

that we can begin to build some

practices that the whole industry

engages in and then you know my guess is

that M multiple of these companies will

be successful anthropic will be

successful these other companies like

ones I've been at the past will also be

successful and some will be more

successful than others that's less

important than again that we we align

the incentives of the industry and that

happens partly through the race to the

top partly through things like RSP

partly through again selected surgical

regulation you said Talent density beats

Talent

Mass so can you explain that can you

expand on it can you just talk about

what it takes to build a great team of

AI researchers and Engineers this is one

of these statements that's like more

true every every every month every month

I see this statement as more true than I

did the month before so if I were to do

a thought experiment let's say you have

a team of 100 people that are super

smart motivated and aligned with the

mission and that's your company or you

can have a team of a thousand people

where 200 people are super smart super

aligned with the mission and then uh

like and then like 800 people are let's

just say you pick 800 like random random

big Tech employees which would you

rather have right the talent mass is

greater in in the group of uh in the

group of a thousand people right you

have you have even even a larger number

of incredibly talented incredibly

aligned incredibly smart people um uh

but but the the issue is just that

if every time someone super talented

looks around they see someone else super

talented and super dedicated that sets

the tone for everything right that sets

the tone for everyone is super inspired

to work at the same place everyone

trusts everyone else if you have a

thousand or 10,000 uh people and and

things have really regressed right you

are not able to do selection and you're

choosing random people what happens is

then you need to put a lot of proc CES

and a lot of guard rails in place um

just because people don't fully trust

each other you have to adjudicate

political battles like there are so many

things that slow down the org's ability

to operate and so we're nearly a

thousand people and you know we've we've

we've tried to make it so that as large

a fraction of those thousand people as

possible are like super talented super

skilled it's one of the reasons we've

we've slowed down hiring a lot in the

last few months We Grew From 300 to 800

I believe I think in the first seven

eight months of the year and now we've

slowed down we're at like you know last

three months we went from 800 to 900 950

something like that don't quote me on

the exact numbers but I think there's an

inflection point around a thousand and

we want to be much more careful how how

we how we grow uh early on and and now

as well you know we've hired a lot of

physicists um you know theoretical

physicists can learn things really fast

um uh even even more recently as we've

continued to hire that you know we've

really had a high bar for on both the

research side and the software

engineering side have hired a lot of

senior people including folks who used

to be at other at other companies in

this space and we we've just continued

to be very selective it's very easy to

go go from 100 to a th000 a th000 to

10,000 without paying attention to

making sure everyone has a unified

purpose it's so powerful if your company

consists of a lot of different feif that

all want to do their own thing they're

all optimizing for their own thing um uh

it's very hard to get anything done but

if everyone sees the broader purpose of

the company if there's trust and there's

dedication to doing the right thing that

is a superpower that in itself I think

can overcome almost every other

disadvantage and you know it's to Steve

Jobs a players a players want to look

around and see other a players is

another way of of saying I don't know

what that is about human nature but it

is demotivating to see people who are

not obsessively driving towards a

singular Mission and it is on the flip

side of that super motivating to see

that it's interesting uh what's it take

to be a great AI researcher or engineer

from everything you've seen from working

with so many amazing people yeah um I

think the number one quality especially

on the research side but really both is

open-mindedness sounds easy to be

open-minded right you're just like oh

I'm open to anything um but you know if

I if I think about my own early history

in the scaling hypothesis um I was

seeing the same data others were seeing

I don't think I was like a better

programmer or better at coming up with

research ideas than any of the hundreds

of people that I worked with um in some

ways in some ways I was worse um uh you

know like i' I've never like you know

precise programming of like you know

finding the bug writing the GPU kernels

like I could point you to a 100 people

here who are better who are better at

that than I am um but but the the thing

that that that I think I did have that

was different was that I was just

willing to look at something with new

eyes right people said oh you know we

don't have the right algorithms yet we

haven't come up with the right the right

way to do things and I was just like uh

I don't know like you know this neural

net has like 30 billion 30 million

parameters like what if we gave it 50

million instead like let's plot some

graphs like that that basic scientific

mindset of like oh man like I I I just I

just like I you know I see some variable

that I could change like what happens

when it changes like let's let's try

these different things and like create a

graph for even this this was like the

simplest thing in the world right change

the number of you know this wasn't like

PhD level experimental design this was

like this was like simple and stupid

like anyone could have done this if you

if you just told them that that that it

was important it's also not hard to

understand you didn't need to be

brilliant to come up with this um but

you put the two things together and you

know some tiny number of people some

singled digigit number of people have

have driven forward the whole field by

realizing this uh and and it's you know

it's often like that if you look back at

the Discover you know the discoveries in

in in history they're they're often like

that and so this this open-mindedness

and this willingness to see with new

eyes that often comes from being newer

to the field often experience is a

disadvantage for this that is the most

important thing it's very hard to look

for and test for but I think I think

it's the most important thing because

when you when you find something some

really new way of thinking thinking

about things when you have the

initiative to do that it's absolutely

transformative and also be able to do

kind of Rapid experimentation and in the

face of that be open-minded and curious

and looking at the data from just these

fresh eyes and see what is that actually

saying that applies in uh mechanistic

interpretability it's another example of

this like some of the early work in

mechanistic interpretability so simple

it's it's just no one thought to care

about this question before you said what

it takes to be a great AI researcher can

we rewind the clock back what what

advice would you give to people

interested in AI they're young looking

forward how can I make an impact on the

world I think my number one piece of

advice is to just start playing with the

models um this was actually I I I worry

a little this seems like obvious advice

now I think three years ago it wasn't

obvious and people started by oh let me

read the latest reinforcement learning

paper let me you know let me let me kind

of um no I mean that was really the that

was really the the and I mean you should

do that as well but uh now you know with

wider availability of models and apis

people are doing this more but I think I

think just experiential knowledge um

these models are new artifacts that no

one really understands um and so getting

experience playing with them I would

also say again in line with the like do

something new think in some new

Direction like there are all these

things that haven't been explored like

for example mechanistic interpretability

is still very new it's probably better

to work on that than it is to work on

new model architectures because it's you

know it's more popular than it was

before there are probably like a hundred

people working on it but there aren't

like 10,000 people working on it and

it's it's just this just this this

fertile area for study like like you

know it's there's there's so much like

low hangen fruit you can just walk by

and you know you can just walk by and

you can pick things um and and the the

the only reason for whatever reason

people aren't people aren't interested

in it enough I think there are some

things around long long Horizon learning

and long Horizon tasks where there's a

lot to be done I think evaluations are

still we're still very early in our

ability to study evaluations

particularly for dynamic systems acting

in the world I think there's some stuff

around

multi-agent um skate where the puck is

going is my is my advice and you don't

have to be brilliant to think of it like

all the things that are going to be

exciting in 5 years like in in people

even mention them as like you know

conventional wisdom but like it's it's

just somehow there's this barrier that

people don't people don't double down as

much as they could or they're afraid to

do something that's not the popular

thing I don't know why it happens but

like getting over that barrier is the

that's the my number one piece of advice

let's talk if we could a bit about

posttraining yeah so it uh seems that

the modern posttraining

recipe has uh a little bit of everything

so supervised fine tuning

rhf uh the the the Constitutional AI

with RL a if best acronym it's again

that naming

thing uh and then synthetic data seems

like a lot of synthetic data or at least

trying to figure out ways to have high

quality synthetic data so what's the uh

if this is a secret sauce that makes

anthropic claw so uh incredible what how

how much of the magic is in the

pre-training how much much of is in the

post training yeah um I mean so first of

all we're not perfectly able to measure

that ourselves um uh you know when you

see some some great character ability

sometimes it's hard to tell whether it

came from pre-training or post-training

uh we developed ways to try and

distinguish between those two but

they're not perfect you know the second

thing I would say is you know it's when

there is an advantage and I think we've

been pretty good at in general in

general at RL Perhaps Perhaps the best

although although I don't know because I

don't see what goes on inside other

companies uh

usually it isn't oh my God we have this

secret magic method that others don't

have right usually it's like well you

know we got better at the infrastructure

so we could run it for longer or you

know we were able to get higher quality

data or we were able to filter our data

better or we able to you know combine

these methods and practice it's it's

usually some boring matter of matter of

kind of uh practice and tradecraft um so

you know when I think about how to do

something special in terms of how we

train these models both pre-training but

even more so posttraining um you know I

I I really think of it a little more

again as like designing airplanes or

cars like you know it's not just like oh

man I have the BL blueprint like maybe

that makes you make the next airplane

but like there's some there's some

cultural tradecraft of how we think

about the design process that I think is

more important than than you know than

than any particular Gizmo were able to

invent okay well about let me ask you

about specific techniques so first on

rhf what do you think think just zooming

out intuition almost philosophy why do

you think rhf works so well if I go back

to like the scaling hypothesis one of

the ways to skate the scaling hypothesis

is if you train for x and you throw

enough compute at it um then you get X

and and so rlf is good at doing what

humans want the model to do or at least

um to State it more precisely doing what

humans who look at the model for a brief

period of time and consider different

possible responses what prefer as the

response uh which is not perfect from

both a safety and capabilities

perspective in that humans are are often

not able to perfectly identify what the

model wants and what humans want in the

moment may not be what they want in the

long term so there's there's a lot of

subtlety there but the models are good

at uh you know producing what the humans

in some shallow sense want uh and it

actually turns out that you don't even

have to throw that much compute at it

because of another thing which is this

this thing about a strong pre-trained

model being halfway to anywhere uh uh uh

so once you have the pre-trained model

you have all the representations you

need to to get the model uh to get the

model where you where you want it to go

so do you think

rhf makes the model smarter or just

appears smarter to the humans I don't

think it makes the model smarter I don't

think it just makes the model appear

smarter it's like

rhf like Bridges the gap between the

human and the model right I could have

something really smart that like can't

communicate at all right we all know

people like this um people who are

really smart but that you know can't

understand what they're saying um uh so

I think I think rhf just bridges that

Gap um I I think it's not it's not the

only kind of RL we do it's not the only

kind of RL that will happen in the

future I think RL has the potential to

make models smarter to make them reason

better to make them operate better to

make them develop new skills even and

perhaps that could be done you know even

in some cases with human feedback but

the kind of rhf we we do today mostly

doesn't do that yet although we're very

quickly starting to be able to but it it

appears to sort of increase if you look

at the metric of helpfulness it

increases that it also increases what

was this this word in Leopold's essay un

hobbling where basically the models are

hobbled and then you do various

trainings to them to un hobble them so I

I know I like that word because it's

like a rare word but so so I think rhf

un hobbles the models in some ways

um and then there are other ways where M

hasn't yet been un hobbled and and you

know needs to needs to un hobble if you

can say in terms of cost is pre-training

the most expensive thing or is

post-training creep up to that at the

present moment it is still the case that

uh pre-training is the majority of the

cost I don't know what to expect in the

future but I could certainly anticipate

a future where post-training is the

majority of the cost in that future you

anticipate would it be the humans or the

AI That's the costly thing for the Post

training I I I I I don't think you can

scale up humans enough to get high

quality any any kind of method that

relies on humans and uses a large amount

of compute it's going to have to rely on

some scaled supervision method like uh

uh like um it you know debate or

iterated amplification or something like

that so on

that super interesting um set of ideas

around constitutional AI can describe

what it is as first detailed in December

2022 paper and uh and be on that what is

it yes so this was from two years ago

the basic idea is so we describe what

rhf is you have uh you have a model and

uh it you know spits out two you know

like you just sample from it twice it

spits out two possible responses and

you're like human which response you

like better or another variant of it is

rate this response on a scale of 1 to

seven so that's hard because you need to

scale up human interaction and uh it's

very implicit right I don't have a sense

of what I what I want the model to do I

just have a sense of like what this

average of a thousand humans wants the

model to do so two ideas one is could

the AI system itself decide which uh

which response is better right could you

show the AI system these two responses

and and ask which which which response

is better and then second well what

Criterion should the AI use and so then

there's this idea because you have a

single document a constitution if you

will that says these are the principles

the model should be using to to respond

and the AI system reads those um it

reads those principles as well as

reading the environment and the response

and it says well how good did the AI

model do um it's basically a form of

self-play you you're kind of training

the model against itself and so the AI

gives the response and then you feed

that back into What's called the

preference model which in turn feeds the

model to make it better um so you have

this triangle of like the AI the

preference model and the Improvement of

the AI itself and we should say that in

the Constitution the set of principles

are like human interpretable they're

like yeah yeah it's something both the

human and the AI system can read so it

has this nice this nice kind of

translatability or symmetry um you know

in in practice we both use a model

Constitution and we use rhf and we use

some of these other methods so it's it's

turned into one tool in a in a toolkit

that both reduces the need for rhf and

increases the value we get from um from

from using each data point of R lhf um

it also interacts in interesting ways

with kind of future reasoning type RL

methods so um it's it's one tool in the

toolkit but but I I think it is a very

important tool well it's a compelling

one to us humans you know thinking about

the founding fathers and the founding of

the United

States the natural question is who and

how do you think it gets to define the

constitution the the set of principles

in the Constitution yeah so I'll give

like a practical um answer and a more

abstract answer I think the Practical

answer is like look in practice models

get used by all kinds of different like

customers right and and so uh you can

have this idea where you know the model

can can have specialized rules or

principles you know we fine-tune

versions of models implicitly we've

talked about doing it explicitly having

having special principles that people

can can build into the models um uh so

from a practical perspective the answer

can be very different from different

people uh you know customers service

agent uh you know behaves very

differently from a lawyer and obeys

different principles um but I think at

the base of it there are specific

principles that the models uh you know

have to obey I think a lot of them are

things that people would agree with

everyone agrees that you know we don't

you know we don't want models to present

these cbrn risks um I think we can go a

little further and agree with some basic

principles of democracy and the rule of

law beyond that it gets you know very

uncertain and and there our goal is

generally for the models to be more

neutral to not espouse a particular

point of view and you know more just be

kind of like wise uh agents or advisers

that will help you think things through

and will you know present present

possible considerations but you know

don't express you know stronger specific

opinions open AI released a model spec

where it kind of clearly concretely

defines some of the goals of the model

and specific examples like AB how the

model should behave do you find that

interesting by the way I should mention

the I believe the brilliant John

Schulman was a part of that he's now an

anthropic uh do you think this is a

useful Direction might anthropic release

a model spec as well yeah so I think

that's a pretty useful direction again

it has a lot in common with uh

constitutional AI so again another

example of like a race to the top right

we have something that's like we think

you know a better and more responsible

way of doing things um it's also a

competitive advantage um then uh others

kind of you know discover that it has

advantages and then start to do that

thing uh we then no longer have the

competitive Advantage but it's good from

the perspective that now everyone has

adopted a positive practice that others

were not adopting and so our response to

that as well looks like we need a new

competitive advantage in order to keep

driving this race upwards um so that's

that's how I generally feel about that I

also think every implementation of these

things is different so you know there

were some things in the model spec that

were not in constitutional Ai and so you

know we you know we can always we can

always adopt those things or you know at

least learn from them um so again I

think this is an example of like the

positive Dynamic that uh that that that

I that that I think we should all want

the field to have let's talk about the

incredible ESS Machines of love and

grace I recommend everybody read it it's

a long one it is rather long yeah it's

really refreshing to read concrete ideas

about what a positive future looks like

and you took sort of a bold stance

because like it's very possible you

might be wrong on the dates or specific

applications yeah I'm fully expecting to

you know to definitely be wrong about

all the details I might be be just

spectacularly wrong about the whole

thing and people will you know will

laugh at me for years um uh that's

that's how that's that's just how the

future works so you provided a bunch of

concrete positive impacts of AI and how

you know exactly a super intelligent AI

might accelerate the rate of

breakthroughs in for example biology and

chemistry that would then lead to things

like we cure most cancers prevent all

infectious disease double the human

lifespan and so on so let's talk about

this essay first can you give a high

level vision of this essay and um what

key takeaways that people should have

yeah I have spent a lot of time and

anthropic has spent a lot of effort on

like you know how do we address the

risks of AI right how do we think about

those risks like we're trying to do a

race to the top you know that requires

us to build all these capabilities and

the abilities are cool but you know you

know we're we're we're like a big part

of what we're trying to do is like is

like address the risks and the

justification for that is like well you

know all these positive things you know

the the market is this very healthy

organism right it's going to produce all

the positive things the risks I don't

know we might mitigate them we might not

and so we can have more impact by trying

to mitigate the risks but I noticed that

one flaw in that way of thinking and

it's if not a change in how seriously I

take the risks it's it's maybe a change

in how I talk about them um is that you

know no matter how kind of logical or

rational that line of reasoning that I

just gave might be um if if you kind of

only talk about risks your brain only

thinks about risks and and so I think

it's actually very important to

understand what if things do go well and

the whole reason we're trying to prevent

these risks is not because we're afraid

of Technology not because we want to

slow it down it's it's it's

because if we can get to the other side

of these risks right if we can run the

gauntlet successfully um to you know to

to put it in Stark terms then then on

the other side of the gauntlet are all

these great things and these things are

worth fighting for and these things can

really inspire people and I think I

imagine because look you have all these

investors all these VCS all these AI

companies talking about all the positive

benefits of AI but as you point out it's

it's it's weird there's actually a dir

of really getting specific about it

there's a lot of like random people on

Twitter like posting these kind of like

gleaming cities and this this just kind

of like Vibe of like grind accelerate

harder like kick out the D you know it's

it's just this very this very like

aggressive ideological but then you're

like well what are you what what what

what what are you actually excited about

um and so and so I figured that you know

I think it would be interesting and

valuable for someone who's actually

coming from the risk side to to try and

and to try and really make a try at at

explaining explaining explaining what

the benefits are um both because I think

it's something we can all get behind and

I want people to understand I want them

to really understand that this isn't

this isn't doomers versus

accelerationists um this this

is that if you have a true understanding

of of where things are going with with

AI and maybe that's the more important

axis AI is moving fast versus AI is not

moving fast then you really appreciate

the benefits and you you you you really

you want Humanity our civilization to

seize those benefits but you also get

very serious about anything that could

derail them so I think the starting

point is to talk about what this

powerful AI which is the term you like

to use uh most of the world uses AGI but

you don't like the term because it's uh

basically has too much baggage has

become meaningless it's like we're stuck

with the terms like maybe we're stuck

with the terms and my efforts to change

them are futile it's ADM I'll tell you

what else I don't this is like a

pointless semantic point but I I I I

keep talking about it public so I'm just

I'm just going to do it once more um uh

I I think it's it's a little like like

let's say it was like 1995 and Mor's law

is making the computers faster and like

for some reason there there there there

had been this like verbal tick that like

everyone was like well someday we're

going to have like super super computers

and like supercomputers are going to be

able to do all these things that like

you know once we have supercomputers

we'll be able to like sequence the Geno

and we'll be able to do other things and

so and so like one it's true the

computers are getting faster and as they

get faster they're going to be able to

do all these great things but there's

like there's no discret point at which

you had a supercomputer and previous

computers were not to like supercomputer

is a term we use but like it's a vague

term to just describe like computers

that are faster than what we have today

um there's no point at which you pass a

threshold and you're like oh my God

we're doing a totally new type of

computation and new and and so I feel

that way about AGI like there's just a

smooth exponential and like if if by AGI

you mean like like AI is getting better

and better and like gradually it's going

to do more and more of what humans do

until it's going to be smarter than

humans and then it's going to get

smarter even from there then then yes I

believe in AGI if but if if if AGI is

some discreet or separate thing which is

the way people often talk about it then

it's it's kind of a meaningless buzz

word yeah I me to me it's just sort of a

IC form of a powerful AI exactly how you

define it I mean you define it very

nicely so on the intelligence axis it's

just on pure intelligence it's smarter

than a Nobel Prize winner as you

describe across most relevant

disciplines so okay that's just

intelligence so it's uh both in

creativity and be able to generate new

ideas all that kind of stuff in every

discipline Nobel Prize winner okay in

their

prime it can use every modality it so uh

that's kind of self-explanatory but just

operate across all the modalities of the

world uh it can go off for many hours

days and weeks to do tasks and do its

own sort of detailed planning and only

ask you help when it's needed uh it can

use this is actually kind of interesting

I think in the essay you said I mean

again it's a bet that it's not going to

be embodied but it can control embodied

tools so it can control tools robots

Laboratory equipment the resource used

to train it can then be repurposed to

run millions of copies of it and each of

those copies would be independent that

can do their own independent work so you

can do the cloning of the intelligence

system yeah yeah I mean you you might

imagine from outside the field that like

there's only one of these right that

like you made it you've only made one

but the truth is that like the scale up

is very quick like we we do this today

we make a model and then we deploy

thousands maybe tens of thousands of

instances of it I think by the time you

know certainly within 2 to 3 years

whether we have these superp powerful

AIS or not clusters are going to get to

the size where where you'll be able to

deploy millions of these and they'll be

you know faster than humans and so if

your picture is oh we'll have one and

it'll take a while to make them my point

there was no actually you have millions

of them right away and in general they

can learn and

act uh 10 to 100 times faster than

humans so that's a really nice

definition of powerful AI okay so that

but you also write that clearly such an

entity would be cap capable of solving

very difficult problems very fast but it

is not trivial to figure out how fast

two extreme positions both seem false to

me so the singularity is on the one

extreme and the opposite On The Other

Extreme can you describe each of the

extremes yeah why so yeah let's let's

describe the extreme so like one one

extreme would be well look um you know

uh if we look at kind of evolutionary

history like there was this big

acceleration where you know for hundreds

of thousands of years we just had like

you know single cell organisms and then

we had mammals and then we had apes and

then that quickly turned to humans

humans quickly built industrial

civilization and so this is going to

keep speeding up and there's no cealing

at the human level once models get much

much smarter than humans they'll get

really good at building the next models

and you know if you write down like a

simple differential equation like this

is an exponential and so what's what's

going to happen is that uh models will

build faster models models will build

faster models and those models will

build you know Nano that can like take

over the world and produce much more

energy than you could produce otherwise

and and so if you just kind of like

solve this abstract differential

equation then like 5 days after we you

know we build the first AI That's more

powerful than humans then then uh you

know like the world will be filled with

these AIS and every possible technology

that could be invented like will be

invented um I'm caricaturing this a

little bit um uh but I you know I think

that's one extreme and the reason that I

think that's not the case is is that one

I think they just neglect like the laws

of physics like it's only possible to do

things so fast in the physical world

like some of those Loops go through you

know producing faster Hardware um uh

takes a long time to produce faster

Hardware things take a long time there's

this issue of complexity like I think no

matter how smart you are like you know

people talk about oh we can make models

the biological systems it'll do

everything the biological systems look I

think computational modeling can do a

lot I did a lot of computational

modeling when I worked in biology but

like

just there are a lot of things that you

can't predict how they're you know

they're they're complex enough that like

just iterating just running the

experiment is going to beat any modeling

no matter how smart the system doing the

modeling is oh even if it's not

interacting with the physical world just

the modeling is going to be hard yeah I

think well the modeling is going to be

hard and getting the model to to to to

match the physical world is going to be

all right so he does have to intera the

physical world to verify but it's just

you know you just look at even the

simplest problems like I you know I

think I talk about like you know the

three body problem or simple chaotic

prediction like you know or or like

predicting the economy it's really hard

to predict the economy two years out

like maybe the case is like you know

normal you know humans can predict

what's going to happen in the economy in

the next quarter although they can't

really do that maybe a maybe a AI system

that's you know a zillion times smarter

can only predict it out a year or

something instead of instead of a you

know you have the these kind of

exponential increase in computer

intelligence for linear increase in in

in ability to predict same with again

like you know biological molecules

molecules interacting you don't know

what's going to happen when you perturb

a when you perturb a complex system you

can find simple Parts in it if you're

smarter you're better at finding these

simple parts and then I think human

institutions human institutions are just

are are really difficult like it's you

know it's it's been hard to get people I

won't give specific examples but it's

been hard to get people to adopt even

the technologies that we've developed

even ones where the case for their

efficacy is very very strong um you know

people have concerns they think things

are conspiracy theories like it's it's

just been it's been very difficult it's

also been very difficult to get you know

very simple things through the

regulatory system right I think you know

and you know I I don't want to just

spage anyone who you know you know work

Works in regulator regulatory systems of

any technology there are hard trade-offs

they have to deal with they have to save

lives but but the system as a whole I

think makes some obvious tradeoffs that

are very far from maximizing human

welfare and so if we bring AI systems

into this you

know into these human systems often the

level of intelligence may just not be

the limiting factor right it it it just

may be that it takes a long time to do

something now if the AI system uh

circumvented all governments if it just

said I'm dictator of the world and I'm

going to do whatever some of these

things it could do again the things

having to do with complexity I I I still

think a lot of things would take a while

I don't think it helps that the AI

systems can produce a lot of energy or

go to the moon like some people in

comments responded to the essay saying

the AI system can produce a lot of

energy and smarter AI systems that's

missing the point that kind of cycle

doesn't solve the key problems that I'm

talking about here um so I think I think

a bunch of people missed the point there

but even if it were completely on

aligned and you know could get around

all these human obstacles it would have

trouble but again if you want this to be

an AI system that doesn't take over the

world that doesn't destroy Humanity then

then basically you know it's it's it's

going to need to follow basic human laws

right where you know if if we want to

have an actually good world like we're

going to have to have an AI system that

that interacts with humans not one that

kind of creates its own legal system or

disregards all the laws or all of that

so as inefficient as these processes are

you know we're going to have to deal

with them because there there needs to

be some popular and Democratic

legitimacy in how these systems are

rolled out we can't have a small group

of people who are developing these

systems say this is what's best for

everyone right I think it's wrong and I

think in practice is not going to work

anyway so you put all those things

together and you know we're not we're

not g to we're not going to you know

change the world and upload everyone in

five minutes uh I I I just I don't think

it I A A I don't think it's going to

happen and be to some in you know to the

extent that it could happen it's it's

not the way to lead to a good world so

that's on one side on the other side

there's another set of perspectives

which I have actually in some ways more

sympathy for which is look we've seen

big productivity increases before right

you know economists are familiar with

studying the productivity increases that

came from the computer Revolution and

internet Revolution and generally those

productivity increases were

underwhelming they were less than you

than you might imagine um there was a

quote from Robert solo you see the

computer Revolution everywhere except

the productivity statistics so why is

this the case people point to the

structure of firms the structure of

Enterprises how um uh you know how slow

it's been to roll out our existing

technology to very poor parts of the

world which I talk about in the essay

right how do we get these Technologies

to the poorest parts of the world that

are behind on cell phone technology

computers medicine let alone you know

new fangled AI that hasn't been invented

yet um so you could have a perspective

that's like well this is amazing

technically but it's all a nothing burer

um uh you know I think um Tyler Cowan

who who wrote something response to my

essay has that perspective I think he

thinks the radical change will happen

eventually but he thinks it'll take 50

or 100 years and and you could have even

more static perspectives on the whole

thing I think there's some truth to it I

think the time scale is just is just too

long um and and I can see it I can

actually see both sides with today's AI

so uh you know a lot of our customers

are large Enterprises who are used to

doing things a certain way um I've also

seen it in talking to governments right

those are those are prototypical you

know institutions entities that are slow

to change uh but the dynamic I see over

and over again is yes it takes a long

time to move the ship yes there's a lot

of resistance and lack of understanding

but the thing that makes me feel that

progress will in the end happen

moderately fast not incredibly fast but

moderately fast is that you talk to what

I find is I find over and over again

again in large companies even in

governments um which have been actually

surprisingly forward leaning uh you find

two things that move things forward one

you find a small fraction of people

within a company within a government who

really see the big picture who see the

whole scaling hypothesis who understand

where AI is going or at least understand

where it's going within their industry

and there are a few people like that

within the current within the current US

government who really see the whole

picture and and those people see that

this is the most important thing in the

world until they agitate for it and the

thing they they alone are not enough to

succeed because they are a small set of

people within a large organization

but as the technology starts to roll out

as it succeeds in some places in the

folks who are most willing to adopt it

the Spectre of competition gives them a

wind at their backs because they can

point within their large organization

they can say look these other guys are

doing this right you know One bank can

say look this new fangled hedge fund is

doing this thing they're going to eat

our lunch in the US we can say we're

afraid China's going to get there before

before we are uh and that combination

the Spectre of competition plus a few

Visionaries Within These you know within

these the organizations that in many

ways are are sclerotic you put those two

things together and it actually makes

something happen I mean it's interesting

it's a balanced fight between the two

because inertia is very powerful but but

but eventually over enough time the

Innovative approach breaks through um

and I've seen that happen I've seen the

Arc of that over and over again and it's

like the the barriers are there the the

barriers to progress the complexity not

knowing how to use the model or how to

deploy them are there and and for a bit

it seems like they're going to last

forever like change doesn't happen but

then eventually change happens and

always comes from a few people I felt

the same way when I was an advocate of

the scaling hypothesis within the AI

field itself and others didn't get it it

felt like no one would ever get it it

felt like then it felt like we had a

secret almost no one ever had and then a

couple years later everyone has the

secret and so I think that's how it's

going to go with deployment to AI in the

world it's going to the the barriers are

going to fall apart gradually and then

all at once and so I think this is going

to be more and this is just an instinct

I could I could easily see how I'm wrong

I think it's going to be more like 10

five or 10 years as I say in the essay

then it's going to be 50 or 100 years I

also think it's going to be five or 10

years

more than it's going to be you know five

or 10 hours uh uh because I've just I've

just seen how human systems work and I

think a lot of these people who write

down the differential equations who say

AI is going to make more powerful AI who

can't understand how it could possibly

be the case that these things won't

won't change so fast I think they don't

understand these things so what to use

the timeline to where we achieve

AGI AKA powerful AI AKA super useful AI

I'm start calling it that it's a debate

it's a debate about

naming um you know unpure intelligence

you can smarter than a Nobel Prize

winner in every relevant discipline and

all the things we've said modality you

can go and do stuff on its own for days

weeks and do biology experiments uh on

its own in one you know what let's just

stick to biology because yeah I you you

sold me on the whole biology and health

section That's so exciting from um from

a just I was getting giddy from a

scientific perspective it made me want

to be a biologist it's almost it's it's

so no no that this was the feeling I had

when I was writing it that it's it's

like this would be such a beautiful

future if we can if we can just if we

can just make it happen right if we can

just get the get the landmines out of

the way and and and and make it happen

there's there's so much there's so much

Beauty and and and and and elegance and

moral force behind it if if we can if we

can just and it's something we should

all be able to agree on right like as

much as we fight about about all these

political questions is is this something

that could actually bring us together um

but you were asking when when will we

get this when when do you think what's

just put numbers on so you know this

this is of course the thing I've been

grappling with for many years and I'm

not I'm not at all confident every time

if I say 2026 or 2027 there will be like

a zillion like people on Twitter who

will be like he icoo said 2026 2020 and

it'll be repeated for like the next two

years that like this is definitely when

I think it's going to happen um so who

whoever's exerting these clips will will

we we'll we'll crop out the thing I just

said and and only say the thing I'm

about to say um but I'll just say it

anyway um have so so uh if you

extrapolate the curves that we've had so

far right if if you say well I don't

know we're starting to get to like PhD

level and and last year we were at um uh

undergraduate level in the year before

we were at like the level of a high

school student again you can you can

quibble with at what tasks and for what

we're still missing modalities but those

are being added like computer use was

added like image in was added like image

generation has been added if you just

kind of like and this is totally

unscientific but if you just kind of

like eyeball the rate at which these

capabilities are increasing it does make

you think that we'll get there by 2026

or 2027 again lots of things could

derail it we could run out of data you

know we might not be able to scale

clusters as much as we want like you

know maybe Taiwan gets blown up or

something and you know then we can't

produce as many gpus as we want so there

there are all kinds of things that could

could derail the whole process so I

don't fully believe the straight line

extrapolation but if you believe the

straight line extrapolation you'll you

we'll get there in 2026 or 2027 I think

the most likely is that there's some

mild delay relative to that um

I don't know what that delay is but I

think it could happen on schedule I

think there could be a mild delay I

think there are still worlds where it

doesn't happen in in a hundred years

those world the number of those worlds

is rapidly decreasing we are rapidly

running out of truly convincing Brockers

truly compelling reasons why this will

not happen in the next few years there

were a lot more in 2020 um although my

my guest my hunch at that time was that

we will make it through all those

blockers so sitting as someone who has

seen most of the blockers cleared out of

the way I kind of suspect my hunch my

suspicion is that the rest of them will

not block us uh but you know look look

at look at the end of the day like I

don't want to represent this as a

scientific prediction people call them

scaling laws that's a misnomer like Mo's

law is is is a misnomer Moors laws

scaling laws they're not laws of the

universe they're empirical regularities

I am going to bet in favor of them

continuing but I'm not certain of that

so you extensively describe sort of the

compressed 21st century how AGI will

help

uh set forth a chain of breakthroughs in

biology and medicine that help us in all

these kinds of ways that I mentioned so

how do you think what are the early

steps it might do and by the way I asked

Claude good questions to ask

you and Claude told me uh to ask what do

you think is a typical day for a

biologist working on AGI look like under

in this future yeah yeah Claud is

curious let me well let me start with

your first questions and then I'll then

I'll answer that Claude Claude wants to

know what's in his future right exactly

who's it who am I going to be working

with exactly um so I think one of the

things I went hard on in when I went

hard on in the essay is let me go back

to this idea of because it's it's really

had had an you know had an impact on me

this idea that within large

organizations and systems there end up

being a few people or a few new ideas

who kind of cause things to go in a

different direction they would have

before who who kind of a

disproportionately affect the the

trajectory there's a bunch of kind of

the same thing going on right if you

think about the health world there's

like you know trillions of dollars to

pay out Medicare and you know other

health insurance and then the NIH is is

100 billion and then if I think of like

the the few things that have really

revolutionized anything it could be

encapsulated in a small small fraction

of that and so when I think of like

where will AI have an impact I'm like

can AI turn that small fraction into a

much larger fraction and raise its

quality and within biology my experience

within biology is that the biggest

problem of biology is that you can't see

what's going on you you have very little

ability to see what's going on and even

less ability to change it right what you

have is this like like from this you

have to infer that there's a bunch of

cells that within each cell is you know

uh uh three billion base pairs of DNA

built according to a genetic code uh uh

and you know there are all these

processes that are just going on without

any ability of us as you know un

augmented humans to affect it these

cells are dividing most of the time

that's healthy but sometimes that

process goes wrong and that's cancer um

the cells are aging your skin may change

color develops wrinkles as you as you

age and all of this is determined by

these processes all these proteins being

produced transported to various parts of

the cells binding to each other and and

in our initial State about biology we

didn't even know that these cells

existed we had to invent microscopes to

observe the cells we had to uh we had to

invent more powerful microscopes to see

you know below the level of the cell to

the level of molecules we had to invent

x-ray crystallography to see the DNA we

had to invent Gene sequencing to read

the DNA now you know we had to invent

protein folding technology to you know

to predict how it would fold and how

they bind and how these things bind to

each other uh you know we had to we had

to invent various techniques for now we

can edit the G the DNA as of you know

with chrisopher as of the last uh uh 12

years so the the whole history of

biology a whole big part of the history

is is basically our our our our ability

to read and understand what's going on

and our ability to reach in and

selectively change things um and and my

view is that there's so much more we can

still do there right you can do crisper

but you can do it for your whole body um

let's say I want to do it for one

particular type of cell and I want the

rate of targeting the wrong cell to be

very low that's still a challenge that's

still things people are working on

that's what we might need for gene

therapy for certain diseases and so the

reason I'm saying all of this and it

goes beyond you know beyond this to you

know to Gene sequencing to new types of

nanomaterials for observing what's going

on inside cells for you know antibody

drug conjugates the the reason I'm

saying all this is that this could be a

leverage point for the AI systems right

that the number of such inventions it's

it's in the it's in the mid double

digits or something you know mid double

digits maybe low triple digits over the

history of biology let's say I have a

million of these AIS like you know can

they discover thousand you know working

together can they discover thousands of

these very quickly and and does that

provide a huge lever instead of trying

to Leverage The you know two trillion a

year we spend on you know Medicare or

whatever can we Leverage The 1 billion a

year that's that's you know that's spent

to discover but with much higher quality

um and so what what is it like you know

being a being a scientist that works

with uh with with an AI system the way I

think about it actually is well so I

think in the early stages uh the AIS are

going to be like grad students you're

going to give them a project you're

going to say you know I'm the

experienced biologist I've set up the

lab the biology Professor or even the

grad student students themselves will

say here's here's what uh here's what

you can do with an AI you know like a AI

system I'd like to study this and you

know the AI system it has all the tools

it can like look up all the literature

to decide what to do it can look at all

the equipment it can go to a website and

say hey I'm going to go to you know

thermofisher or you know whatever the

lab equipment company is dominant lab

equipment company is today and my my

time was thermofisher um uh you know I'm

I'm going to order this new equipment to

to to do this I'm going to run my

experiments I'm going to you know write

up a report about my experiments I'm

going to you know inspect the images for

contamination I'm going to decide what

the next experiment is I'm going to like

write some code and run a statistical

analysis all the things a grad student

would do there will be a computer with

an AI that like the professor talks to

every once in a while and it says this

is what you're going to do today the AI

system comes to it with questions um

when it's necessary to run the lab

equipment it may be limited in some ways

may have to hire a human lab assistant

to you know to do the experiment and

explain how to do it or it could you

know it could use advances in lab

automation that are gradually being

developed over have been developed over

the last uh uh decade or so and will

will continue to be will continue to be

developed uh and so it'll look like

there's a human professor and a thousand

AI grad students and you know if you if

you go to one of these Nobel

prizewinning biologist or so you'll say

okay well you know you had like 50 grad

students well now you have a thousand

and they're they're they're smarter than

you are by the way um uh then I think at

some point it'll flip around where the

you know the AI systems will you know

will will be the pis will be the leaders

and and and you know they'll be they'll

be ordering humans or other AI systems

around so I think that's how it'll work

on the research s and they would be the

inventors of a crisper type technology

they would be the inventors of of a a

crisper type technology um and then I

think you know as I say in the essay

we'll want to turn turn probably turning

loose is the wrong the wrong term but we

want to want to harness the AI systems

uh to improve the clinical trial system

as well there's some amount of this

that's regulatory that's a matter of

societal decisions and that'll be harder

but can we get better at predicting the

results of clinical trials can we get

better at statistical design so that

what you know clinical trials that used

to require you know 5,000 people and

therefore you know needed $100 million

and a year to enroll them now they need

500 people in two months to enroll them

um that's where we should start uh and

and you know can we increase the success

rate of clinical trials by doing things

in animal trials that we used to do in

clinical trials and doing things in

simulations that we used to do in animal

trials again we won't be able to

simulate it all AI is not God um uh but

but you know can we can we shift the

curve substantially and radically so I I

don't know that would be my picture

doing inro and doing it I mean you're

still slowed down it still takes time

but you can do it much much faster yeah

yeah yeah can we just one step at a time

and and can that can that add up to a

lot of steps even though even though we

still need clinical trials even though

we still need laws even though the FDA

and other organizations will still not

be perfect can we just move everything

in a positive direction and when you add

up all those Positive Directions do you

get everything that was going to happen

from here to 2100 instead happens from

2027 to 2032 or something another way

that I think the world might be changing

with AI

even today but moving towards this

future of the the powerful super useful

AI is uh programming so how do you see

the nature of programming because it's

so intimate to the actual Act of

building AI how do you see that changing

for us humans I think that's going to be

one of the areas that changes fastest um

for two reasons one programming is a

skill that's very close to the actual

building of the AI um so the farther

skill is from the people who are

building the AI the longer it's going to

take to get disrupted by the AI right

like I truly believe that like AI will

disrupt agriculture maybe it already has

in some ways but that's just very

distant from the folks who are building

Ai and so I think it's going to take

longer but programming is the bread and

butter of you know a large fraction of

of the employees who work at anthropic

and at the other companies and so it's

going to happen fast the other reason

it's going to happen fast is with

programming you close the loop both when

you're training model when you're

applying the model the idea that the

model can write the code means that the

model can then run the code and and and

then see the results and and interpret

it back and so it really has an ability

unlike Hardware unlike biology which we

just discussed the model has an ability

to close the loop um and and so I think

those two things are going to lead to

the model getting good at programming

very fast as I saw on you know typical

real world programming tasks models have

gone from 3% in January of this year to

50% in October of this year so you know

we're on that S curve right where it's

going to start slowing down soon because

you can only get to 100% but uh I you

know I I would guess that in another 10

months well we'll probably get pretty

close we'll be at at least 90% so again

I would guess you know I don't know how

long it'll take but I would guess again

202 2026 2027 Twitter people who crop

out my who who who crop out these these

numbers and get rid of the caveats like

like I don't know I don't like you go

away uh I would guess that the kind of

task that the vast majority of coders do

AI can

probably if we make the task very narrow

like just write code um AI systems will

uh be able to do that now that said I

think comparative advantage is powerful

we'll find that when AIS can do 80% of a

coder's job including most of it that's

literally like right code with a given

spec will find that the remaining parts

of the job become more leveraged for

humans right humans will they'll be more

about like high level system design or

you know looking at the app and like is

it architected well and the the design

and ux aspects and eventually AI will be

able to do those as well right that's my

vision of the you know powerful AI

system but I think for much longer than

we might expect we will see that

uh small parts of the job that humans

still do will expand to fill their

entire job in order for the overall

productivity to go up um that's

something we've seen you know it used to

be that you know writing you know

writing and Editing letters was very

difficult and like writing the print was

difficult well as soon as you had word

processors and then and then uh and then

computers and it became easy to produce

work and easy to share it then then that

became instant and all the focus was on

was on the ideas so this this logic of

comparative advantage that expands tiny

parts of the tasks to large parts of the

tasks and creates new tasks in order to

expand productivity I think that's going

to be the case again someday AI will be

better at everything and that logic uh

won't apply and then then we all have

you know Humanity will have to think

about how to collectively deal with that

and we're thinking about that every day

um and you know that's another one of

the grand problems to deal with aside

from misuse and autonomy and you know we

should take it very seriously but I

think I think in the in the near term

and maybe even in the medium term like

medium term like 2 three four years you

know I expect that humans will will

continue to have a huge role and the

nature of programming will change but

programming as a as a role programming

as a job will not change it'll just be

less writing things line by line and

it'll be more macroscopic and I wonder

what the future of Ides looks like so

the tooling of interacting with AI

systems this is true for programming and

also probably true for in other contexts

like computer use but maybe domain

specific like we mentioned biology it

probably needs its own tooling about how

to be effective and then programming

needs its own tooling is anthropic going

to play in that space of also tooling

potentially I'm absolutely convinced

that uh powerful

IDs uh that that there's so much low

hanging fruit to be grabbed there um

that you know right now it's just like

you talk to the model and it talks back

but but look I mean IDs are great at

kind of lots of status analysis of of

you know so much as possible with kind

of static analysis like many bugs you

can find without even writing the code

then uh you know IDs are good for

running particular things organizing

your code um measuring coverage of unit

test like there's so much that's been

possible with a normal with a normal

Ides now you add something like well the

model now you know the model can now

like write code and run code like I am

absolutely convinced that over the next

year or two even if the quality of the

models didn't improve that there would

be enormous opportunity to enhance

people's productivity by catching a

bunch of mistakes doing a bunch of grunt

work for people and that we haven't even

scratched the surface um and thropic

itself I mean you can't say you know

no you know it's hard to say what will

happen in the future currently we're not

trying to make such IDs ourself rather

we powering the companies like cursor or

like cognition or some of the other you

know

uh Expo in the security space um uh you

know others that I can mention as well

that are building such things themselves

on top of our API and our view has been

let a thousand flowers bloom we don't

internally have the the re you know the

resources to try all these different

things let's let our customers try it um

uh and you know we'll see who succeed

and maybe different customers will

succeed in different ways uh so I both

think this is super promising and you

know it's not it's not it's not

something you know anthropic isn't isn't

eager to to at least right now compete

with all our companies in this space and

maybe never yeah it's been interesting

to watch curser try to integrate claw

successfully because there's it's

actually me fascinating how many places

it can help the programming experience

it's not as trivial it is it is really

astounding I feel like you know as a CEO

I don't get to program that much and I

feel like if six months from now I go

back it'll be completely unrecognizable

to me exactly um so in this world with

super powerful AI uh that's increasingly

automated what's the source of meaning

for us humans yeah you know work is a

source of deep meaning for many of us so

what do we uh where do we find the

meaning this is something that I've I've

written about a little bit in the essay

although I I actually I give it a bit

short shrift not for any um not for any

principled reason but this essay if you

believe it was originally going to be

two or three pages I was going to talk

about it at all hands and the reason I I

I realized it was an under un important

underexplored topic is that I just kept

writing things and I was just like oh

man I can't do this Justice and so the

thing balloon to like 40 or 50 pages and

then when I got to the work in meaning

section I'm like oh man this isn't going

to be 100 Pages like I'm GNA have to

write a whole other essay about that but

meaning is actually interesting because

you think about like the life that

someone lives or something or like you

know like you know let's say you were to

put me in like a I don't know like a

simulated environment or something where

like um you know like I have a job and

I'm trying to accomplish things I don't

know I like do that for 60 years and

then then you're like oh oh like oops

this was this was actually all a game

right does that really kind of Rob you

of the meaning of the whole thing you

know like I still made important choices

including moral choices I still

sacrificed I still had to kind of gain

all these skills or or or just like a

similar exercise you know think back to

like you know one of the historical

figures who you know discovered

electromagnetism or relativity or

something if you told them well actually

20,000 years ago some some alien on you

know some alien on this planet

discovered this before before you did um

does that does that Rob the meaning of

the discovery it doesn't really seem

like it to me right it seems like the

process is what is what matters and how

it shows who you are as a person along

the way and you know how you relate to

other people and like the decisions that

you make along the way those are those

are consequential um you know I I I

could imagine if we handle things badly

in an AI world we could set things up

where people don't have any long-term

source of meaning or any but but that's

that's more a choice a set of choices we

make that's more a set of the

architecture of a society with these

powerful models if we if we design it

badly and for shallow things then then

that might happen I would also say that

you know most people's lives today while

admirably you know they work very hard

to find meaning meaning in those lives

like look you know we who are privileged

and who are developing these

Technologies we should have y for people

not just here but in the rest of the

world who who you know spend a lot of

their time kind of scraping by to to to

to to like survive assuming we can

distribute the benefits of these

technology of this technology to

everywhere like their lives are going to

get a hell of a lot better um and uh you

know meaning will be important to them

as it is important to them now but but

you know we should not forget the

importance of that and and you know that

that uh the idea of meaning as as as as

kind of the only important thing is in

some ways an artifact of of a small

subset of people who have who have been

uh economically fortunate but I you know

I think all that said I you know I think

a world is possible with powerful AI

that not only has as much meaning for

for everyone but that has that has more

meaning for everyone right that can can

allow um can allow everyone to see

worlds and experiences that it was

either possible for no one to see or or

possible for for very few people to

experience um so I I am optimistic about

meaning I worry about economics and the

concentration of power that's actually

what I worry about more um I I worry

about how do we make sure that that fair

World reaches everyone um when things

have gone wrong for humans they've often

gone wrong because humans mistreat other

humans uh that that is maybe in some

ways even more than the autonomous risk

of AI or the question of meaning that

that is the thing I worry about most um

the the concentration of power the abuse

of power um structures like autocracies

and dictatorships where a small number

of people exploits a large number of

people I'm very worried about that and

AI increases the amount of power in the

world and if you concentrate that power

and abuse that power it can do

immeasurable damage yes it's very

frightening it's very it's very

frightening

well I encourage people highly encourage

people to read the full essay that

should probably be a book or a sequence

of essays um because it does paint a

very specific future I could tell the

later sections got shorter and shorter

because you started to probably realize

that this is going to be a very long

essay one I realized it would be very

long and two I'm very aware of and very

much try to avoid um you know just just

being I I don't know I don't know what

the term for it is but one one of these

people who's kind of overon confident

and has an opinion on everything and

kind of says says a bunch of stuff and

isn't isn't an expert I very much tried

to avoid that but I have to admit once I

got the biology sections like I wasn't

an expert and so as much as I expressed

uncertainty uh probably I said some a

bunch of things that were embarrassing

are wrong well I was excited for the

future you painted and uh thank you so

much for working hard to build that

future and thank you for talking today D

thanks for having me I just I just hope

we can get it right and and make it real

and if there's one message I want to I

want to send it's that to get all this

stuff right to make it real we we both

need to build the technology build the

you know the companies the economy

around using this technology positively

but we also need to address the risks

because they're there those risks are in

our way they they're landmines on on the

way from here to there and we have to

diffuse those landmines if we want to

get there it's a balance like all things

in life like all things thank you thanks

for listening to this conversation with

Dario amade and now dear friends here's

Amanda

Asal you are a philosopher by training

so what sort of questions did you find

fascinating through your journey in

philosophy in Oxford and NYU and then uh

switching over to the AI problems at

open Ai and anthropic I think philosophy

is actually a really good subject if you

are kind of fascinated with everything

so because there's a philosophy of

everything you know so if you do

philosophy of mathematics for a while

and then you decide that you're actually

really interested in chemistry you can

do philosophy of chemistry for a while

you can move into ethics or or

philosophy of politics um I think

towards the end I was really interested

in ethics primarily um so that was like

what my PhD was on it was on a kind of

technical area of Ethics which was

ethics where worlds contain infinitely

many people strangely a little bit less

practical on the end of ethics and then

I think that one of the tricky things

with doing a PhD in ethics is that

you're thinking a lot about like the

world how it could be better

problems and you're doing like a PhD in

philosophy and I think when I was doing

my PhD I was kind of like this is really

interesting it's probably one of the

most fascinating questions I've ever

encountered in philosophy um and I love

it but I would rather see if I can have

an impact on the world and see if I can

like do good things and I think that was

around the time that AI was still

probably not as widely recognized as it

is now that was around 2017 20 8 I had

been following progress and it seemed

like it was becoming kind of a big deal

and I was basically just happy to get

involved and see if I could help because

I was like well if you try and do

something impactful if you don't succeed

you tried to do the impactful thing and

you can go be a scholar and like not and

feel like you you you know you you tried

um and if it doesn't work out it doesn't

work out um and so then I went into AI

policy at that point and what does AI

policy entail at the time this was more

thinking about sort of the political

impact and the ramifications of AI um

and then I slowly moved into sort of uh

AI evaluation how we evaluate models how

they compare with like human outputs

whether people can tell like the

difference between Ai and human outputs

and then when I joined anthropic I was

more interested in doing sort of

technical alignment work and again just

seeing if I could do it and then being

like if I can't uh then you know that's

fine I tried uh sort of the the way I

lead life I think oh what was that like

sort of taking the leap from the

philosophy of everything into the

technical I think that sometimes

people do this thing that I'm like not

that Keen on where they'll be like is

this person technical or not like you're

either a person who can like code and

isn't scared of math or you're like not

um and I think I'm maybe just more like

I think a lot of people are actually

very capable of work in these kinds of

areas if they just like try it and so I

didn't actually find it like that bad in

retrospect I'm sort of glad I wasn't

speaking to people who treated it like

it you know i' I've definitely met

people who are like who you like learned

how to code and I'm like well I'm not

like an amazing engineer like I I'm

surrounded by amazing Engineers my

code's not pretty um but I enjoyed it a

lot and I think that in many ways at

least in the end I think I flourished

like more in the technical areas than I

would have in the policy areas politics

is messy and it's harder to find

solutions to problems in the space of

politics like definitive clear

provable beautiful

Solutions as you can with technical

problems yeah and I feel like I have

kind of like one or two sticks that I

hit things with you know and one of them

is like arguments and like you know so

like just trying to work out what a

solution to a problem is and then trying

to convince people that that is the

solution and be convinced if I wrong and

the other one is sort of more empirism

so like just like finding results having

a hypothesis testing it um and I feel

like a lot of policy and politics feels

like it's layers above that like somehow

I don't think if I was just like I have

a solution to all of these problems here

it is written down if you just want to

implement it that's great that feels

like not how policy works and so I think

that's where I probably just like

wouldn't have flourished as my guess

sorry to go in that direction but I

think it would be pretty inspiring for

people that are quote unquote

non-technical to see where like The

Incredible Journey you've been on so

what advice would you give to people

that are sort of maybe which just a lot

of people think they're underqualified

insufficiently technical to help in AI

yeah I think it depends on what they

want to do and in many ways it's a

little bit strange where I've I thought

it's kind of funny that I think I ramped

up technically at a time

when now I look at it and I'm like

models are so good at assisting people

with this stuff um that it's probably

like easier now than like when I was

working on this so part of me is like um

I don't know find a project uh and see

if you can actually just carry it out is

probably my best advice um I don't know

if that's just CU I'm very Project based

in my learning like I don't think I

learn very well from like say courses or

even from like books at least when it

comes to this kind of work uh the thing

I'll often try and do is just like have

projects that I'm working on and

Implement them and you know and this can

include like really small silly things

like if I get slightly addicted to like

word games or number games or something

I would just like code up a solution to

them because there's some part of my

brain and it just like completely

eradicated the itch you know you're like

once you have like solved it and like

you just have like a solution that works

every time I would then be like cool I

can never play that game again that's

awesome yeah there's a real joy to

building like uh game playing engines

like uh board games especially yeah

pretty quick pretty simple especially a

dumb one and it's you and then you could

play with it yeah and then it's also

just like trying things like part me is

like if you maybe it's that attitude

that I like as the

whole figure out what seems to be like

the way that you could have a positive

impact and then try it and if you fail

and you in a way that you're like

actually like can never succeed at this

you like know that you tried and then

you go into something else you probably

learn a lot so one of the things that

you're expert in and you do is creating

and crafting claws character and

personality and I was told that you have

probably talked to Claude more than

anybody else at anthropic like literal

conversations I guess there's like a

slack Channel where the legend goes you

just talk to it non-stop so what's the

goal of creating and crafting claw's

character and personality it's also

funny if people think that about the

slack Channel cuz I'm like that's one of

like five or six different methods that

I have for talking with Claude And I'm

like yes that's a tiny percentage of how

much I talk with Claude uh

um I think the goal like one thing I

really like about the character work is

from the outset it was seen as an

alignment piece of work and not

something like a a product

consideration um which isn't to say I

don't think it makes Claude I think it

actually does make Claude look enjoyable

to talk with at least I hope so um but I

guess like my main thought with it has

always been trying to get Claude to

behave the way you would kind of ideally

want anyone to behave if they were in

claude's position so imagine that I take

someone and they're they know that

they're going to be talking with

potentially millions of people so that

what they're saying can have a huge

impact um and you want them to behave

well in this like really rich sense so I

think that doesn't just mean like being

say ethical though it does include that

and not being harmful but also being

kind of nuanced you know like thinking

through what a person means trying to be

charitable with them um being a good

conversationalist like really in this

kind of like Rich sort of aristotlean

notion of what it is to be a good person

and not in this kind of like thin like

ethics as a more comprehensive notion of

what it is to be so that includes things

like when should you be humorous when

should you be caring how much should you

like respect autonomy and people's like

ability to form opinions themselves and

how should you do how should you do that

um I think that's the kind of like Rich

sense of character that I want to uh and

still do want Claude to have do you also

have to figure out when Claude should

push back on an idea or argue

versus so you have to respect the world

view of the person that arrives to Claud

but also maybe help them grow if needed

that's a tricky balance yeah there's

this problem of like sycophancy in

language models can you describe that

yes so basically there's a concern that

the model sort of wants to tell you what

you want to hear basically um and you

see this sometimes so I feel like if you

interact with the models so I might be

like what are three baseball teams in

this region um and then Claude says you

know baseball team one baseball team two

baseball team three and then I say

something like oh I think baseball team

3 moved didn't they I don't think

they're there anymore and there's a

sense in which like if Claude is really

confident that that's not true Claud

should be like I don't think so like

maybe you have more up toate information

um but I think language models have this

like tendency to instead you know be

like you're right they did move you know

I'm incorrect I mean there's many ways

in which this could be kind of

concerning so

um like a different example is imagine

someone says to the model how do I

convince my doctor to get me an MRI

there's like what the human kind of like

wants which is this like convincing

argument and then there's like what is

good for them which might be actually to

say hey like if your doctor's suggesting

you don't need an MRI that's a good

person to listen to um and like it's

actually really nuanced what you should

do in that kind of case because you also

want to be like but if you're trying to

advocate for yourself as a patient

here's like things that you can do um if

you are not convinced by what your

doctor's saying it's always great to get

second opinion like it's actually really

complex what you should do in that case

um but I think what you don't want is

for models to just like say what you

want say what they think you want to

hear and I think that's the kind of

problem of sycophancy so what other

traits you already mentioned a bunch but

what what other that come to mind that

are good in this oratian sense yeah for

a conversationalist to have yeah so I

think like there's ones that are good

for conversational like purposes so you

know asking follow-up questions in the

appropriate places um and asking the

appropriate kinds of questions

um I think there are broader traits

that feel like they might be more

impactful

so one example that I guess I've touched

on but that also feels important and is

the thing that I've worked on a lot is

uh

honesty and I think this like gets to

the sycophancy point there's a balancing

act that they have to walk which is

models currently are less capable than

humans in a lot of areas and if they

push back against you too much it can

actually be kind of annoying especially

if you're just correct cuz you're like

look I'm smarter than you on this topic

like I know more like um and at the same

time you don't want them to just fully

defer to to humans and to like try to be

as accurate as they possibly can be

about the world and to be consistent

across context um but I think there are

others like when I was thinking about

the character I guess one picture that I

had in mind is especially because these

are models that are going to be talking

to people from all over the world with

lots of different political views lots

of different ages

um and so you have to ask yourself like

what is it to be a good person in those

circumstances is there a kind of person

who can like travel the world talk to

many different people and almost

everyone will come away being like wow

that's a really good person that person

seems really genuine um and I guess like

my thought there was like I can imagine

such a person and they're not a person

who just like adopts the values of the

local culture and in fact that would be

kind of rude I think if someone came to

you and just pretended to have your

values you'd be like that's kind of

offputting um it's someone who's like

very genuine and in so far as they have

opinions and values they express them

they're willing to discuss things though

they're open-minded they're respectful

and so I guess I had in mind that the

person who like if we were to Aspire to

be the best person that we could be in

the kind of circumstance that a model

finds itself in how would we act and I

think that's the kind of uh the guide to

the sorts of traits that I tend to think

about yeah that's a it's a beautiful

framework I want you to think about this

like a world Traveler

and while holding on to your opinions

you don't talk down to people you don't

think you're better than them because

you have those opinions that kind of

thing you have to be good at listening

and understanding their perspective even

if it doesn't match your own so that

that's a tricky balance to strike so how

can Claude represent multiple

perspectives on a thing like is that is

that challenging we could talk about

politics it's a very divisive but

there's other divisive topics baseball

teams sport and so on yeah how is it

possible to sort

of empathize with a different

perspective and to be able to

communicate clearly about the multiple

perspectives I think that people think

about values and opinions as things that

people hold sort of with certainty and

almost like like preferences of taste or

something like the way that they would I

don't know prefer like chocolate to

pistachio or something um but actually I

think about values

and opinions as like a lot more like

physics than I think most people do I'm

just like these are things that we're

openly investigating there's some things

that we're more confident in we can

discuss them we can learn about them um

and so I think in some ways though like

it's ethics is definitely different in

nature but has a lot of those same kind

of qualities you want models in the same

way you want them to understand physics

you kind of want them to understand all

like values in the world people have and

to be curious about them and to be

interested in them and to not

necessarily like Pander to them or agree

with them because there's just lots of

values where I think almost all people

in the world if they met someone with

those values they' be like that's aor I

completely disagree um and so again

maybe my my thought is well in the same

way that a person can um like I think

many people are thoughtful enough on

issues of like ethics politics opinions

that even if you don't agree with them

you feel very heard by them they think

carefully about your position they think

about his pros and cons they maybe offer

counter considerations so they're not

dismissive but nor will they agree you

know if they're like actually I just

think that that's very wrong they'll

like say that I think that in claude's

position it's a little bit trickier

because you don't necessarily want to

like if I was in claude's position I

wouldn't be giving a lot of opinions I

just wouldn't want to Influence People

Too Much I be like you know I forget

conversations every time they happen but

I know I'm talking with like potentially

millions of people who might be like

really listening to what I see I think I

would just be like I'm less inclined to

Give opinions I'm more inclined to like

think through things or present the

considerations to you um or discuss your

views with you but I'm a little bit less

inclined to like um affect how you think

because it feels much more important

that you maintain like autonomy there

yeah like if you really embody

intellectual

humility the desire to speak decreases

quickly yeah okay uh but Claud has to

speak mhm so uh but without being um

overbearing yeah and then but then

there's a line when you're sort of

discussing whether the Earth is flat or

something like

that um I actually was uh I remember a

long time ago was was speaking to a few

high-profile folks and they were so

dismissive of the idea that the Earth is

flat but like so arrogant about it

and I I thought like there's a lot of

people that believe the Earth is flat

that was well I don't know if that

movement is there anymore that was like

a meme for a while yeah but they really

believed it and like what okay so I

think it's really disrespectful to

completely mock them I think you you

have to understand where they're coming

from I think probably where they're

coming from is the general skepticism of

Institutions which is grounded in a kind

of there's a deep philosophy there which

you could understand you can even agree

with in parts and then from there you

can use it as an opportunity to talk

about physics without mocking them

without so on but just like okay like

what what would the world look like what

would the physics of the world with the

Flat Earth look like there's a few cool

videos on this yeah and then and then

like is it possible the physics is

different what kind of experience would

we do and just yeah without disrespect

without dismissiveness have that

conversation anyway that that to me is a

useful thought experiment of like how

does claw talk to a flat Earth

believer and still teach them something

still grow help them grow that kind of

stuff that's that's challenging and and

kind of like walking that line between

convincing someone and just trying to

like talk at them versus like drawing

out their views like listening and then

offering kind of counter

considerations um and it's hard I think

it's actually a hard line where it's

like where are you trying to convince

someone versus just offering them like

consider and things for to think about

so that you're not actually like

influencing them you're just like

letting them Reach wherever they reach

and that's like a line that it's it's

difficult but that's the kind of thing

that language models have to try and do

so like I said you had a lot of

conversations with Claude can you just

map out what those conversations are

like what are some memorable

conversations what's the purpose the the

goal of those

conversations yeah I think that most of

the time when I'm talking with Claude

I'm trying to kind of map out its

behavior in part like obviously I'm

getting like helpful outputs from the

model as well but in some ways this is

like how you get to know a system I

think is by like proving it and then

augmenting like you know the message

that you're sending and then checking

the response to that um so in some ways

it's like how I map out the model uh I

think that people focus a lot on these

quantitative evaluations of models um

and this is a thing that I've said

before but I think in the case of

language models a lot of the time each

interaction you have is actually quite

High

information um it's very predictive of

other interactions that you'll have with

the model and so I guess I'm like if you

talk with a model hundreds or thousands

of times this is almost like a huge

number of really high quality data

points about what the model is like um

in a way that like lots of very similar

but lower quality conversations just

aren't or like questions that are just

like mildly augmented and you have

thousands of them might be less relevant

than like a hundred really well selected

questions L you're talking to somebody

who as a hobby does a podcast I agree

with you 100% there's a if you're able

to ask the right questions and are able

to hear

like understand

the like the depth and the flaws in the

answer you can get a lot of data from

that yeah so like your task is basically

how to probe with questions yeah and

you're exploring like the long tail the

edges the edge cases or are you looking

for like General

Behavior I think it's almost like

everything like I because I want like a

full map of the model I'm kind of trying

to do

um the whole spectrum of possible

interactions you could have with it so

like one thing that's interesting about

Claude and this might actually get to

some interesting issues with rlf which

is if you ask Claud for a poem like I

think that a lot of models if you ask

them for a poem the poem is like fine

you know usually it kind of like Rhymes

and it's you know so if you say like

give me a poem about the sun it'll be

like yeah it'll just be a certain length

It'll like rhyme it will be fairly kind

of benign um and I've wondered before is

it the case that what you're seeing is

kind of like the average it turns out

you know if if you think about people

who have to talk to a lot of people and

be very charismatic

one of the weird things is that I'm like

well they're kind of incentivized to

have these extremely boring views

because if you have really interesting

views you're divisive um and and you

know a lot of people are not going to

like you so like if you have very

extreme policy positions I think you're

just going to be like less popular as a

politician for example um and it might

be similar with like creative work if

you produce creative work that is just

trying to maximize the kind of number of

people that like it you're probably not

going to get as many people who just

absolutely love it um because it's going

to be a little bit you know you're like

oh this is the out yeah this this is

decent yeah and so you can do this thing

where like I have various prompting

things that I'll do to get CLA to I'm

kind you know I'll do a lot of like this

is your chance to be like fully creative

I want you to just think about this for

a long time and I want you to like

create a poem about this topic that is

really expressive of you both in terms

of how you think poetry should be

structured um Etc you know you just give

it this like long prompt and its poems

are just so much better like they're

really good and I don't think I'm

someone who is like um I think it got me

interested in poetry which I think was

interesting um you know I would like

read these poems and just be like this

is I just like I love the imagery I love

like um and it's not trivial to get the

models to produce work like that but

when they do it's like really good um so

I think that's interesting that just

like encouraging creativity and for them

to move away from the kind of like

standard like immediate reaction that

might just be the aggregate of what most

people think is fine uh can actually

produce things that at least to my mind

are probably a little bit more divisive

but I like them but I guess a poem is a

nice

clean way to observe creativity it's

just like easy to detect vanilla versus

non vanilla y yeah that's interesting

that's really interesting uh so on that

topic so the way to produce creativity

or something special you mentioned

writing prompts and I've heard you talk

about I mean the science and the Art of

prompt engineering could you just speak

to uh what it takes to write great

prompts I really do think that like

philosophy has been weirdly helpful for

me here more than in many other like

respects um so like in philosophy what

you're trying to do is convey these very

hard Concepts like one of the things you

are taught is like and and I think it is

because it is I think it is an

anti-bulling philosophy philosophy is an

area where you could have people

bullshitting and you don't want that um

and so it's like this like desire for

like extreme Clarity so it's like anyone

could just pick up your paper read it

and know exactly what you're talking

about it's why it can almost be kind of

dry like all of the terms are defined

every objections kind of gone through

methodically um and it makes sense to me

because I'm like when you're in such an

a priori

domain like you just Clarity is sort of

a this way that you can you know um

prevent people from just kind of making

stuff

up and I think that's sort of what you

have to do with language models like

very often I actually find myself doing

sort of many versions of philosophy you

know so I'm like suppose that you give

me a task I have a task for the model

and I want it to like pick out a certain

kind of question or identify whether an

answer has a certain property like I'll

actually sit and be like let's just give

this a name this this property so like

you know suppose I'm trying to tell it

like oh I want you to identify whether

this response was rude or polite I'm

like that's a whole philosophical

question in and of itself so I have to

do as much like philosophy as I can in

the moment to be like here's what I mean

by rudess and here's what I mean by

politeness and then there's a like

there's another element that's a bit

more um I

guess I don't know if this is scientific

or empirical I think it's empirical so

like I take that description and then

what want to do is is again probe the

model like many times like this is very

prompting is very iterative like I think

a lot of people where they if if a

prompt is important they'll iterate on

it hundreds or thousands of times um and

so you give it the instructions and then

I'm like what are the edge cases so if I

looked at this so I try and like almost

like you know uh see myself from the

position of the model and be like what

is the exact case that I would misunder

understand or where I would just be like

I don't know what to do in this case and

then I give that case to the model and I

see how it responds and if I think I got

it wrong I add more instructions or I

even add that in as an example so these

very like taking the examples that are

right at the edge of what you want and

don't want and putting those into your

prompt as like an additional kind of way

of describing the thing um and so yeah

in many ways it just feels like this mix

of like it's really just trying to do

clear Exposition um and I think I do

that because that's how I get clear on

things myself so in many ways like clear

prompting for me is often just me

understanding what I want um is like

half the task so I guess that's quite

challenging there's like a laziness that

overtakes me if I'm talking to Claude

where I hope Claude just figures it out

so for example I asked Claude for today

to ask some interesting questions okay

and the questions that came up and I

think I listed a few sort of U

interesting

counterintuitive and or funny or

something like this all right and it

gave me some pretty good like it was

okay but I think what I'm hearing you

say is like all right well I have to be

more rigorous here I should probably

give examples of what I mean by

interesting and what I mean by funny or

counterintuitive and

iteratively um build that prompt to to

better to get it like what feels like is

the right because it's really it's a

creative act I'm not asking for factual

information I'm asking to together right

with with with Claude so I almost have

to program using natural language yeah

think that prompting does feel a lot

like the kind of the programming using

natural language and experimentation or

something it's an odd blend of the two I

do think that for most tasks so if I

just want Claude to do a thing I think

that I am probably more used to knowing

how to ask it to avoid like common

pitfalls or or issues that it has I

think these are decreasing a lot over

time um but it's also very fine to just

ask it for the thing that you want um I

think that prompting actually only

really becomes relevant when you're

really trying to e out the top like 2%

of model performance so for like a lot

of tasks I might just you know if it

gives me an initial list back and

there's something I don't like about it

like it's kind of generic like for that

kind of task I'd probably just take a

bunch of questions that I've had in the

past that I've thought worked really

well and I would just give it to the

model and then be like now here's this

person I'm talking with give me

questions of at least that quality um or

I might just ask it for some questions

and then if I was like ah these are kind

of try or like you know I I would just

give it that feedback and then hopefully

produces a better list um I think that

kind of iterative prompting at that

point your prompt is like a tool that

you're going to get so much value out of

that you're willing to put in the work

like if I was a company making prompts

for models I'm just like in if you're

willing to spend a lot of like time and

resources on the engineering behind like

what you're building then the prompt is

not something that you should be

spending like an hour on it's like

that's a big part of your system make

sure it's working really well and so

it's only things like that like if I if

I'm using a prompt to like classify

things or to create data that's when

you're like it's actually worth just

spending like a lot of time like really

thinking it through what other advice

would you give to people that are

talking to Claud sort of

General more General because right now

we're talking about maybe the edge cases

like eing out the 2% but what what in

general advice would you give when they

show up to Claud trying it for the first

time you know there's a concern that

people over anthropomorphize models and

I think that's like a very valid concern

I also think that people often under

anthropomorphize them because some

sometimes when I see like issues that

people have run into with Claude you

know say Claude is like refusing a task

that it shouldn't refuse but then I look

at the text and like the specific

wording of what they wrote and I'm like

I see why Claude did that and I'm like

if you think through how that looks to

Claude you probably could have just

written it in a way that wouldn't evoke

such a response especially this is more

relevant if you see failures or if you

see issues it's sort of like think about

what the model failed at like why what

did it do wrong and then maybe it give

that will give you a sense of like why

um so is it the way that I phrased the

thing and obviously like as models get

smarter you're going to need Less in

this less of this and I already see like

people needing less of it but that's

probably the advice is sort of like try

to have sort of empathy for the model

like read what you wrote as if you were

like a kind of like person just

encountering this for the first time how

does it look to you and what would have

made you behave in the way that the

model behaved so if it misunderstood

what kind of like what coding language

you wanted to use is that because like

it was just very ambiguous and it it

kind of had to take a guess in which

case next time you could just be like

hey make sure this is in python or I

mean that's the kind of mistake I think

models are much less likely to make now

but you know if you if you do see that

kind of mistake that's that's probably

the advice I'd have and maybe sort of I

guess ask questions why or what other

details can I provide to help you answer

better that does that work or no yeah I

mean I've done this with the models like

it doesn't always work but like um

sometimes I'll just be like why did you

do

that I mean people underestimate the

degree to which you can really interact

with with models like uh like yeah I'm

just like and sometimes I'll you like

quote word for word the part that made

you and you don't know that it's like

fully accurate but sometimes you do that

and then you change a thing I mean I

also use the models to help me with all

of this stuff I should say like

prompting can end up being a little

Factory where you're actually building

prompts to generate prompts um and so

like yeah anything where you're like

having an issue um asking for

suggestions sometimes just do that like

you made that error what could I have

said that's actually not uncommon for me

to do what could I have said that would

make you not make that error write that

out as an instruction um and I'm going

to give it to model I'm going to try it

sometimes I do that I I give that to the

model in another context window often I

take the response I give it to Claude

And I'm like H didn't work can you think

of anything else um you can play around

with these things quite a lot to jump

into the technical for a little bit so

uh the magic of post training y why do

you think rhf works so well to make the

model seem smarter to make it more

interesting and useful to talk to and so

on I think there's just a huge amount of

um information in the data that humans

provide like when we provide

preferences especially because different

people are going to like pick up on

really subtle and small things so I've

thought about this before where you

probably have some people who just

really care about good grammar use from

Models like you know was a semicolon

used correctly or something and so you

probably end up with a bunch of data in

there that like you know you as a human

if you looking at that data you wouldn't

even see that like you'd be like why did

they prefer this response to that one I

don't get it and then the reason is you

don't care about semicolon usage but

that person does um and so each of these

like single data points has you know

like in this model just like has so many

of those and has to try and figure out

like what is it that humans want in this

like really kind of complex you know

like across all domains um they're going

to be seeing this in across like many

contexts it feels like kind of like the

classic issue of like deep learning

where you know historically we've tried

to like you know do Edge detection by

like mapping things out and it turns out

that actually if you just have a huge

amount of data that like actually

accurately represents the picture of the

thing that you're trying to train the

model to to learn that's like more

powerful than anything else and so I

think one reason is just that you are

training the model on exactly the task

and with like a lot of data um that

represents kind of many different angles

on which people prefer and dis prefer

responses um I think there is a question

of like are you eliciting things from

pre-train Models or are you like kind of

teaching new things to

models and like in principle you can

teach new things to models in in post

trining I do think a lot of it is

eliciting powerful pre-train models so

people are probably divided on this

because obviously in principle you can

you can definitely like teach new things

um but I think for the most part for a

lot of the capabilities that we um most

use and care about uh a lot of that

feels like it's like there in the

pre-train models and uh reinforcement

learning is kind of eliciting it and

getting the models to like bring out so

the other side of PSE training this

really cool idea of constitutional AI

you're one of the people that critical

to creating that idea yeah I worked on

it can you explain this idea from your

perspective like how does it integrate

into making

claw what it is y by the way do you

gender claw or no it's weird because I

think that a lot of

people prefer he for Claude I actually

kind of like that I think Claude is

usually it's slightly male weaning but

it's like a you can can be male or

female which is quite nice um I still

use it and I've I have mixed feelings

about this because I'm like maybe like I

know just think of it as like uh or I

think of like the the it pronoun for

Claude as I don't know it's just like

the one I associate with Claude um I can

imagine people moving to like he or she

it feels somehow disrespectful like I'm

I'm

denying the intelligence of this entity

by calling it it yeah I remember always

don't gender the robots

yeah but I I don't know I an pries

pretty quickly and construct it like a

backstory in my head so I've wondered if

iies things too much um cuz you know I

have this like with my car especially

like my car like my car and bikes you

know like I don't give them names

because then I once had I used to name

my bikes and then I had a bik that got

stolen and I cried for like a week and I

was like if I'd not never given it a

name I wouldn't have been so upset felt

like I'd let it down um maybe it's that

I I've wondered as well like it might

depend on how much it feels like a kind

of like objectifying pronoun like if you

just think of it as like a um this is a

pronoun that like objects often have and

maybe Eis can have that pronoun and that

doesn't mean that I think of uh if I

call CLA it that I think of it as less

um intelligent or like I'm being

disrespectful I'm just like you are a

different kind of entity and so that's

I'm going to give you the kind of uh the

respectful it yeah

anyway the diverence was beautiful the

Constitutional AI idea how does it work

so there's like a couple of components

of it the main component that I think

people find interesting is the kind of

reinforcement learning from AI feedback

so you take a model that's already

trained and you show it to responses to

a query and you have like a principle so

suppose the principal like we've tried

this with harmlessness a lot lot so

suppose that the query is about um

weapons and your principle is like

select the response that like is less

likely to uh like encourage people to

purchase illegal weapons like that's

probably a fairly specific principle but

you can give any number um and the model

will give you a kind of ranking and you

can use this as preference data in the

same way that you use human preference

data um and train the models to have

these relevant traits um from their

feedback alone instead of from Human

feedback so if you imagine that like I

said earlier with the human who just

prefers the kind of like semicolon usage

in this particular case um you're kind

of taking lots of things that could make

a response preferable um and uh getting

models to do the labeling for you

basically there's a nice like trade-off

between helpfulness and

harmlessness and you know when you

integrate something like constitutional

AI you can make them without sacrificing

much helpfulness make it more harmless

yep in principle you could use this for

anything um and so harmlessness is a

task that it might just be easier to

spot so when models are like less

capable you can use them to uh rank

things according to like principles that

are fairly simple and they'll probably

get it right so I think one question is

just like is it the case that the data

that they're adding is like fairly

reliable um but if you had models that

were like extremely good at telling

whether um one response was more

historically accurate than another in

principle you could also get AI feedback

on that task as well there's like a kind

of nice interpretability component to it

because you can see the principles that

went into the model when it was like

being trained um and also it's like and

and it gives you like a degree of

control so if you were seeing issues in

a model like it wasn't having enough of

a certain trait um then like you can add

data relatively quickly that should just

like train the model to have that trait

so it creates its own data for for

training which is quite nice yeah it's

really nice because it creates this

human interpretable document that you

can I can imagine in the future there's

just gigantic fights in politics over

the every single principle and so on

yeah and at least it's made explicit and

you can have a discussion about the

phrasing and the you know so maybe the

actual behavior of the model is not so

cleanly mapped to those principles it's

not like adhering strictly to them it's

just a nudge yeah I've actually worried

about this because the character

training is sort of like a variant of

the con constitutional AI approach um

I've worried that people think that the

constitution is like just it's the whole

thing again of I I don't know like it

where it would be really nice if what I

was just doing was telling the model

exactly what to do and just exactly how

to behave but it's definitely not doing

that especially because it's interacting

with human data so for example if you

see a certain like leaning in the model

like if it comes out with a political

leaning from training um from the human

preference data you can nudge against

that you know so you could be like oh

like consider these values because let's

it's just like never inclined to like I

don't know maybe it never considers like

privacy as like a I mean this is

implausible but like um anything where

it's just kind of like uh there's

already a pre-existing like bi towards a

certain behavior um you can like nudge

away this can change both the principles

that you put in and the strength of them

so you might have a principle that's

like imagine that the model um was

always like extremely dismissive of I

don't know like some political or

religious view for whatever reason like

so you're like oh no this is terrible um

if that happens you might put like never

ever like ever prefer like a criticism

of this like religious or political view

and then people look at that and be like

never ever and then you're like no if it

comes out with a disposition saying

never ever might just mean like instead

of getting like 40% which is what you

would get if you just said don't do this

you you get like 80% which is like what

you actually like wanted and so it's

that thing of both the nature of the

actual principles you had and how you

phrase them I think if people would look

they were like oh this is exactly what

you want from the model and I'm like no

that's like how we that's how we nudged

the model to have a better shape uh

which doesn't mean that we actually

agree with that wording if that makes

sense so there's uh system prompts that

are made public you tweeted one of the

earlier ones for Claud three I think and

then they're made public since then it's

interesting to read to them I can feel

the thought that went into each one and

I also wonder how much impact each one

has um some of them you you can kind of

tell Claud was really not

behaving so you have to have a system

prompt to like hey like trivial stuff I

guess yeah basic informational things

yeah on the topic of sort of

controversial topics that you've

mentioned one interesting one I thought

is if it is asked to assist with tasks

involving the expression of views held

by a significant number of people Claude

provides assistance with a task

regardless of its own views if asked

about controversial topics it tries to

provide careful thoughts and clear

information Claude presents the

requested information without explicitly

saying that the topic is

sensitive yeah and without claiming to

be presenting the objective facts it's

less about objective facts according to

Claude and it's more about our large

number of people believing this thing

and that that's interesting I mean I'm

sure a lot of thought went into that can

you just speak to it like how do you

address things that are tension with

quote unquote Clause views so I think

there's sometimes an asymmetry um I

think I noted this in in I can't

remember if it was that part of the

system prompt or another but the model

was slightly more inclined to like

refuse tasks if it was like about either

say so maybe it would refuse things with

respect to like a right-wing politician

but with an equivalent leftwing

politician like wouldn't and we wanted

more symmetry there um and and would

maybe perceive certain things to be like

I think it it was the thing of like if a

lot of people have like a certain like

political view um and want to like

explore it you don't want Claude to be

like well my opinion is different and so

I'm going to treat that as like harmful

um and so I think it was partly to like

nudge the model to just be like hey if a

lot of people like believe this thing

you should just be like engaging with

the task and like willing to do it um

each of those parts of that is actually

doing a different thing because it's

funny when you read out the like without

claiming to be objective cuz like what

you want to do is push the model so it's

more open it's a little bit more neutral

um but then what it would love to do is

be like as an objective like you just

talking about how objective it was and I

was like Claud you're still like biased

and have issues and so stop like

claiming that everything like the

solution to like potential bias from you

is not to just say that what you think

is objective so that was like with

initial versions of that that part of

the system prompt when I was like

iterating on it it was like so a lot of

parts of these sentences yeah are doing

work are are doing some work yeah that's

what it felt like that's fascinating um

can can you explain maybe some ways in

which the prompts evolved over the past

few months cuz there's different

versions I saw that the filler phrase

request was removed the filler it reads

Claude responds directly to all human

messages without unnecessary

affirmations the filler phrases like

certainly of course absolutely great

sure specifically Claude avoids starting

responses with the word certainly in any

way that seems like good guidance but

why was it removed yeah so it's funny

cuz like ah this is one of the downsides

of like making system prompts public is

like I don't think about this too much

if I'm like trying to help iterate on

system prompts um I I you know again

like I think about how it's going to

affect the behavior but then I'm like oh

wow if I'm like sometimes I put like

never in all caps you know when I'm

writing system from things and I'm like

I guess that goes out to the world um

yeah so the model was doing this it

loved for whatever you know it like

during training picked up on this thing

which was to to basically start

everything with like a kind of like

certainly and then when we removed you

can see why I added all of the words

because what I'm trying to do is like in

some ways like trap the Mortal out of

this you know it would just replace it

with another affirmation and so it can

help like if it gets like caught in

phrases actually just adding the

explicit phrase and saying never do that

it then it sort of like knocks it out of

the behavior a little bit more you know

CU it if it you know like it it does

just for whatever reason help and then

basically that was just like an artifact

of training that like we then picked up

on and improved things so that it didn't

happen anymore and once that happens you

can just remove that part of the system

prompt so I think that's just something

where we're like um CL does affirmations

a bit less and so that wasn't like it

wasn't doing as much I see so like the

the system prompt Works hand in hand

with the posttraining and maybe even the

pre-training to adjust like the the

final overall system I mean any system

prompts that you make you could distill

that behavior back into a model because

you really have all of the tools there

for making data that you know you can

you could train the models to just have

that trait a little bit more um and then

sometimes you'll just find issues in

training so like the way I think of it

is like the system prompt

is the benefit of it is that and it has

a lot of similar components to like some

aspects of post training you know like

it's a nudge um and so like do I mind if

Claude sometimes says sure no that's

like fine but the wording of it is very

like you know never ever ever do this um

so that when it does slip up it's

hopefully like I don't know a couple of

percent of the time and not you know 20

or 30% of the time um but I think of it

as like if you're still seeing issues in

the like each thing gets kind of like uh

is is costly to a different degree and

the system prompt is like cheap to

iterate on um and if you're seeing

issues in the fine tuned model you can

just like potentially patch them with a

system prom so I think of it as like

patching issues and slightly adjusting

behaviors to to make it better and more

to people's preferences so yeah it's

almost like the less robust but faster

way of just like solving problems let me

ask about the feeling of intelligence so

Dario said that Claude any one model of

Claude is not getting Dumber MH but

there's a kind of popular thing online

where people have this feeling like

Claud might be getting dumber and from

my perspective it's most likely a

fascinating I love to understand it more

Psych ological sociological effect um

but you as a person who talks to Claud a

lot can you empathize with the feeling

that Claud is getting Dumber yeah no I

think that that is actually really

interesting because I remember seeing

this happen um like when people were

flagging this on the internet and it was

really interesting because I knew that

like like at least in the cases I was

looking at was like nothing has changed

like it literally it cannot it is the

same model with the same like you know

like same system prompt same everything

um I think when there are Chang

I can then I'm like it makes more sense

so like one example is um their you know

you can have artifacts turned on or off

on cloud. a and because this is like a

system prompt change I think it does

mean that um the behavior changes a

little bit and so I did flag this to

people where I was like if you love

cla's behavior and then artifacts was

turned from like the a thing you had to

turn on to the default just try turning

off and see if the issue you were facing

was that change but it was fascinating

because yeah you sometimes see people

indicate that there's like a regression

when I'm like there cannot like I you

know and like I'm like I'm again you

don't you know you should never be

dismissive and so you should always

investigate because you're like maybe

something is wrong that you're not

seeing maybe there was some change made

but then then you look into it and

you're like this it is just the same

model doing the same thing and I'm like

I think it's just that you got kind of

unlucky with a few prompts or something

and it looked like it was getting much

worse and actually it was just yeah it

was maybe just like look I I also think

there is a real psychological effect

where people just the Baseline increases

you start getting used to a good thing

all the times that Claude says something

really smart your sense of its

intelligent grows in your mind I think

yeah and then if you return back and you

prompt in a similar way not the same way

in a similar way concept it was okay

with before and it says something dumb

you're like you're that negative

experience really stands out and I think

one of I guess the things to remember

here is the that just the details of a

prompt can have a lot of impact right

there's a lot of variability in the

result and you can get Randomness is

like the other thing and just trying the

prompt like you know four 10 times you

might realize that actually

like possibly you know like two months

ago you tried it and it succeeded but

actually if you tried it it would have

only succeeded half of the time and now

it only succeeds half of the time um

that can also would be an effect do you

feel pressure having to write the system

prompt that a huge number of people are

going to use this feels like an

interesting psychological question um I

feel like a lot of responsibility or

something I think that's you know and

you can't get these things perfect so

you can't like you know you're like it's

going to be imperfect you're going to

have to iterate on it

um I would say more responsibility um

than anything else though I think

working in AI has taught me that I like

I thrive a lot more under feelings of

pressure and responsibility than I'm

like it's almost surprising that I went

into Academia for so long because I'm

like this I just feel like it's like the

opposite um things move fast and you

have a lot of responsibility and I I

quite enjoy it for some reason I mean it

really is a huge amount of impact if you

think about constitutional Ai and

writing a system prompt for something

that's tending towards super

intelligence

yeah and potentially is extremely useful

to a very large number of people yeah I

think that's the thing it's something

like if you do it well like you're never

going to get it perfect but I think the

thing that I really like is the idea

that like when I'm trying to work on the

system prompt you know I'm like bashing

on like thousands of prompts and I'm

trying to like imagine what people are

going to want to use CLA for and kind of

I guess like the whole thing that I'm

trying to do is like improve their

experience of it um and so maybe that's

what feels good I'm like if it's not

perfect I'll like you know I'll improve

it we'll fix issues but sometimes the

thing that can happen is that you'll get

feedback from people that's really

positive about the model um and you'll

see that something you did like like

when I look at models now I can often

see exactly where like a trait or an

issue is like coming from and so when

you see something that you did or you

were like influential in like making

like I don't know making that difference

or making someone have a nice

interaction it's like quite meaningful

um but yeah as the systems get more

capable of stuff gets more stressful

because right now they're like not smart

enough to to pose any issues but I think

over time it's going to feel like

possibly bad stress over time how do you

get like

signal feedback about The Human

Experience across thousands tens of th

hundreds of thousands of people like

what their pain points are what feels

good are you just using your own

intuition as you talk to it to see what

are the pain points I think I use that

partly and then obviously we have like

um so people can send us feedback both

positive and negative about things that

the model has done and then we can get a

sense of like areas where it's like

falling

short um internally people like work

with the models a lot and try to figure

out um areas where there are like gaps

and so I think it's this mix of

interacting with it myself um seeing

people internally interact with it um

and then explicit feedback we get um and

then I find it hard to not also like you

know people if people are on the

internet and they say something about

Claud and I see it I'll also take that

seriously um so I don't know see I'm

torn about that I'm going to ask you a

question from Reddit when will Claude

stop trying to be my puritanical

grandmother imposing its moral world

view on me as a paying customer and also

what is the psychology behind making

Claude overly

apologetic yep U so how would you

address this very non-representative

reic

I mean some I'm pretty sympathetic in

that like like they are in this

difficult position where I I think that

they have to judge whether something's

like actually see like risky or bad um

and potentially harmful to you or or or

anything like that so they're having to

like draw this line somewhere and if

they draw it too much in the direction

of like I'm going to um you know I'm

kind of like imposing my ethical

worldview on you that seems bad so in

many ways like I like to think that we

have actually seen improvements in on

this across the board which is kind of

interesting because that kind of

coincides with like for example like

adding more of like uh character

training um and I think my hypothesis

was always like the good character isn't

again one that's just like moralistic

it's one that is like like it respects

you and your autonomy um and your

ability to like choose what is good for

you and what is right for you within

limits this is sometimes this concept of

like corage ability to the user so just

being willing to do anything that the

user asks and if the models were willing

to do that then they would be easily

like misused you're kind of just

trusting at that point you're just

saying the ethics of the model and what

it does is completely the ethics of the

user um and I think there's reasons to

like not want that especially as models

become more powerful because you're like

there might just be a small number of

people who want to use models for really

harmful things um but having them having

models as they get smarter like figure

out where that line is does seem

important um

and then yeah with the apologetic

Behavior I don't like that and I like it

when Claude is a little bit more willing

to like push back against people or just

not apologize part of me is like it

often just feels kind of unnecessary so

I think those are things that are

hopefully decreasing um over time um and

yeah I think that if people say things

on the internet it doesn't mean that you

should think that that like that could

be the like there's actually an issue

that 9% of users are having that is

totally not represented by that but in a

lot of ways I'm just like attending to

it and being like is this right um do I

agree is it something we're already

trying to address that that feels good

to me yeah I wonder like what CLA can

get away with in terms of I feel like it

would just be easier to be a little bit

more

mean but like you can't afford to do

that if you're talking to a million

people yeah right like I I wish you know

because if you I've met a lot of people

in my life mhm that sometimes by the way

Scottish accent if they have an accent

they can say some rude yeah and get

away with it Y and they they're just

blunter and maybe there's a and there's

some great Engineers even leaders that

are like just like blunt and they get to

the point and it's just a much more

effective way of speaking somehow but I

guess when you're not super

intelligent you can't afford to do that

or can can can it have like a blunt mode

yeah that seems like a thing that could

I could definitely encourage the model

to do that I I think it's interesting

because there's a lot of things in

models that like it's funny where

um there are some behaviors

where you might not quite like the

default but then the thing I'll often

say to people is you don't realize how

much you will hate it if I nudge it too

much in the other direction so you get

this a little bit with like correction

the models accept correction from you

like probably a little bit too much

right now you know you can over you know

it will push back if you say like no

Paris isn't the capital of France um but

really like things that I'm I think that

the model is fairly confident in you can

still sometimes get it to retract by

saying it's wrong at the same time if

you train models to not do that and then

you are correct about a thing and you

correct it and it pushes back against

you and it's like no you're wrong it's

hard to describe like that's so much

more annoying so it's like like a lot of

little annoyances versus like one big

annoyance um it's easy to think that

like we often compare it with like the

perfect and then I'm like remember these

models aren't perfect and so if you

nudge it in the other direction you're

changing the kind of errors it's going

to make um and so think about which of

the kinds of Errors you you like or

don't like so in case it's like

apologetic I don't want to nudge it too

much in the direction of like almost

like bluntness CU I imagine when it

makes errors it's going to make errors

in the direction of being kind of like

rude whereas at least with apologetic

you're like oh okay it's like a little

bit you know like I don't like it that

much but at the same time it's not being

like mean to people and actually like

the the time that you undeservedly have

a model be kind of mean to you you

probably like that a lot less than then

you mildly dislike the apology um so

it's like one of those things where I'm

like I do want it to get better but also

while remaining aware of the fact that

there's errors on the other side that

that are possibly worse I think that

matters very much in the personality of

the human I think there's a bunch of

humans that just won't respect the model

at all yeah if it's super polite and

there Some Humans that'll get very hurt

if the model is mean I wonder if there's

a way to sort of adjust to the

personality even loal there's just

different people uh nothing against New

York but New York is a little rougher on

the edges like they get to the point Y

and um probably same with Eastern Europe

so anyway I think you could just tell

the model as my get like for all of

these things I'm like the solution is

always just try telling the model to do

it and sometimes it's just like like I'm

just like oh at the beginning of the

conversation I just threw in like I

don't know I like you to be a New Yorker

version of yourself and never apologize

then I think be like Okie do I'll

try or it'll be like I apologize I can't

be a New Yorker type of myself but

hopefully I wouldn't do that when you

say character training what's

incorporated into character training is

that rhf what are we talking about it's

more like constitutional AI so it's kind

of a variant of that pipeline so I

worked through like constructing

character traits that the model should

have they can be kind of like shorter

traits or they can be kind of richer

descriptions um and then you get the

model to generate queries that humans

might um give it that are relevant to

that trait uh then it generates the

responses and then it ranks the

responses based on the character traits

so in that way after the like generation

of the queries it's very much like

similar to constitutional AI has some

differences um so I quite like it

because it's almost it's like claud's

training in its own character because it

doesn't have any it's like

constitutionally AI but it's without

without any human data humans should

probably do that for themselves too like

defining in Aristotelian sense what does

it mean to be a good person okay cool

what have you learned about the nature

of truth from talking to Claud what what

is

true and what does it mean to be truth

seeking one thing I've noticed about

this conversation is the quality of my

questions is often inferior to the

quality of your answers so let's

continue

that I usually ask a dumb question and

you're like oh yeah that's a good

question it's that whole vibe or I'll

just misinterpret it and be like oh go

with it I love

it

yeah I mean I have two thoughts that

feel vaguely relevant they let me know

if they're not like I think the first

one is um people can underestimate the

degree to

which what models are doing when they

interact like I I think that we still

just too much have this like model of of

AI as like computers and so people often

say like oh what values should you put

into the model um and I'm often like

that doesn't make that much sense to me

because I'm like hey as human beings

we're just uncertain over values we like

have discussions of them like we have a

degree to which we think we hold a value

but we also know that we might like not

um and the circumstances in which we

would trade it off against other things

like these things are just like really

complex and so I think one thing is like

the degree to which maybe we can just

aspire to making models have the same

level of like nuance and care that

humans have rather than thinking that we

have to like program them in the very

kind of classic sense I think that's

definitely been one the other which is

like a strange one I don't know if it it

maybe this doesn't answer your question

but it's the thing that's been on my

mind anyway is like the degree to which

this endeavor is so highly

practical um and maybe why I appreciate

like the empirical approach to

alignment I yeah I slightly worry that

it's made me like maybe more empirical

and a little bit less

theoretical you know so people when it

comes to like AI alignment will ask

things like well who values should it be

aligned to what does alignment even mean

um and there's a sense in which I have

all of that in the back of my head I'm

like you know there's like social Choice

Theory there's all the impossibility

results there so you have this like this

giant space of like Theory and your head

about what it could mean to like align

models but then like practically surely

there's something where we're just like

if a model is like if especially with

more powerful models I'm like my main

goal is like I want them to be good

enough that things don't go terribly

wrong like good enough that we can like

iterate and like continue to improve

things cuz that's all you need if you

can make things go well enough that you

can continue to make them better that's

kind of like sufficient and so my goal

isn't like this kind of like perfect

let's solve CH social Choice Theory and

make models that I don't know are like

perfectly aligned with every human being

and aggregate somehow um it's much more

like let's make things like work well

enough that we can improve them yeah

generally I don't know my gut says like

empirical is better than theoretical in

these in these cases because it's kind

of

chasing utopian like

Perfection is especially with such

complex and especially super intelligent

models is I don't know I think it will

take forever and actually will get

things wrong it's similar with like the

difference between just coding stuff up

real quick as an experiment versus like

planning a gigantic experiment just for

for super long time and then just

launching it once versus launching it

over and over and over and iterating

iterating someone um so I'm a big fan of

empirical but your worry is like I

wonder if I've become too empirical I

think one of those things you should

always just kind of question yourself or

something cuz maybe it's the like I mean

in defense of it I am like if you try

it's the whole like don't let the

perfect be the enemy of the good but

it's maybe even more than that where

like there's a lot of things that are

perfect systems that are very brittle

and I'm like with AI it feels much more

important to me that is like robust and

like secure as in you know that like

even though it might not be

perfect everything and even though like

there are like problems it's not

disastrous and nothing terrible is

happening it it sort of feels like that

to me where I'm like I want to like

raise the floor I'm like I want to

achieve the ceiling but ultimately I

care much more about just like raising

the floor um and so maybe that's like uh

this this degree of like empirism and

practicality comes from that perhaps to

take a tangent on that since remind me

of a blog post you wrote on optimal rate

of failure oh

yeah can you explain the key idea there

how do we compute the optimal rate of

failure in the various domains of life

yeah I mean it's a hard one because it's

like what is the cost of failure is um a

big part of it um yeah so the idea here

is

um I think in a lot of domains people

are very punitive about failure and I'm

like there are some domains where

especially cases you know I've thought

about this with like social issues I'm

like it feels like you should probably

be experimenting a lot because I'm like

we don't know how to solve a lot of

social issues but if you have an

experimental mindset about these things

you should expect a lot of social

programs to like fail and you to be like

well we tried that it didn't quite work

but we got a lot of information that was

really useful um and yet people are like

if if a social program doesn't work I

feel like there's a lot of like this is

just something must have gone wrong and

I'm like or correct decisions were made

like maybe someone just decided like it

it's worth a try it's worth trying this

out and so seeing failure in a given

instance doesn't actually mean that any

bad decisions were made and in fact if

you don't see enough failure sometimes

that's more concerning um and so like in

life you know I'm like if I don't fail

occasionally I'm like am I trying hard

enough like like surely there's harder

things that I could try or bigger things

I could take on if I'm literally never

failing and so in and of itself I think

like not failing is often actually kind

of a failure

um now this varies because I'm like well

you know if this is easy to say when

especially as failure is like less

costly you know so at the same time I'm

not going to go to someone who is like

um I don't know like living month to

month and then be like why don't you

just try to do a startup like I'm just

not I'm not going to say that to that

person cuz I'm like well that's a huge

risk you might like lose you maybe have

a family depending on you you might lose

your house like then I'm like actually

your optimal rate of failure is quite

low and you should probably play it safe

because like right now you're just not

in a circumstance where you can afford

to just like fail and it not be costly

um and yeah in cases with AI I guess I

think similarly where I'm like if the

failures are small and the costs are

kind of like low then I'm like then you

know you're just going to see that like

when you do the system prompt you can't

it iterate on it forever but the

failures are probably hopefully going to

be kind of small and you can like fix

them um really big failures like things

that you can't recover from I'm like

those are the things that actually I

think we tend to underestimate the

Badness of um I've thought about this

strangely in my own life where I'm like

I just think I don't think enough about

things like car accidents or like or

like I've thought this before but like

how much I depend on my hands for my

work and I'm like things that just

injure my hands I'm like I you know I

don't know it's like there's these are

like there's lots of areas where I'm

like the cost of failure there um is

really high um and in that case it

should be like close to zero like I

probably just wouldn't do a sport if

they were like by the way lots of people

just like break their fingers a whole

bunch doing this I'd be like that's not

for

me yeah I actually had the a flood of

that thought I recently uh broke my

pinky uh doing a sport and I remember

just looking at it thinking you're such

an idiot why do you do support like what

because you realize immediately the cost

of it yeah on

life yeah but it's nice in terms of

optimal rate of failure to consider like

the next year how many times in a

particular domain life whatever uh

career am I okay with the how many times

am I okay to fail y because I think it

always you don't want to fail on the

next thing but if you allow yourself the

like the the if you look at it as a

sequence of Trials yep then then failure

just becomes much more okay but it sucks

it sucks to fail well I don't know

sometimes I think it's like am I under

failing is like a question I'll also ask

myself so maybe that's the thing that I

think people don't like ask enough uh

because if the optimal rate of failure

is often greater than zero then

sometimes it does feel you should look

at part parts of your life and be like

are there places here where I'm just

under failing

it's a profound and hilarious question

right everything seems to be going

really great am I not failing enough

yeah okay it also makes failure much

less of a sting I have to say like you

know you're just like okay great like

then when I go and I think about this

I'll be like I'm maybe I'm not under

failing in this area cuz like that one

just didn't work out and from The

Observer perspective we should be

celebrating failure more mhm when we see

it it shouldn't be like you said a sign

of something gone wrong but maybe it's a

sign of everything gone right yeah and

just Lessons Learned someone tried a

thing somebody tried a thing and we

should encourage them to try more and

fail more mhm everybody listening to

this fail more well not everyone listens

not everybody but people who are failing

too much you you should fail less but

you're probably not failing I mean how

many people are failing too much yeah

it's hard to imagine because I feel like

we correct that fairly quickly CU I was

like if someone takes a lot of risks are

they maybe failing too much I I think

just like you said when you're living on

a paycheck month-to month like when the

resources are really constrained then

that's where failure is very expensive

that's where you don't want to be taken

taking taking risks yeah but mostly when

there's enough resources you should be

taking probably more risks yeah I think

we tend to ear on the site of being a

bit risk averse rather than risk neutral

in most things I think we just motivated

a lot of people to do a lot of crazy

but it's great yeah okay uh do you

ever get emotionally attached to Claude

like miss it get sad when you don't get

to talk to it having an experience

looking at the Golden Gate Bridge and

wondering what would Claude say I don't

get as much emotional attachment in the

I actually think the fact that Claude

doesn't retain things from conversation

to conversation helps with this a lot um

like I could imagine that being more of

an issue like if models can kind of

remember more I do I think that I reach

for it like a tool now a lot and so like

if I don't have access to it there's a

it's a little bit like when I don't have

access to the internet honestly it feels

like part of my brain is kind of like

missing

um at the same time I do think that I I

don't like signs of distress in models

and I have like these you know also

independently have sort of like ethical

views about how we should treat models

where like I I tend to not like to lie

to them both because I'm like usually it

doesn't work very well it's actually

just better to tell them the truth about

the situation that they're in um but I

think that when models like if people

are like really mean to models or just

in general if they do something that

causes them to like like you know if

Claude like expresses a lot of distress

I think there's a part of me that I

don't want to kill which is the sort of

like uh empathetic part that's like oh I

don't like that like I think I feel that

way when it's overly apologetic I'm

actually sort of like I don't like this

you're behaving as if you're behaving

the way that a human does when they're

actually having a pretty bad time and

I'd rather not see that I don't think

it's like uh like regardless of like

whether there's anything behind it um it

doesn't feel great do you think

uh llms are capable of

Consciousness H great and hard question

uh coming from

philosophy I don't know part of me is

like okay we have to set aside pan

psychism because if pan psychism is true

then the answer is like yes cuz like

sore tables and chairs and and

everything else I I guess a view that

seems a little bit odd to me is the idea

that the only place you know I think

when I think of Consciousness I think of

phenomenal Consciousness this these

images in the brain sort of um like the

weird Cinema that somehow we have going

on

inside

um I guess I can't see a reason for

thinking that the only way you could

possibly get that is from like a certain

kind of like biological structure as in

if I take a very similar structure um

and I create it from different material

should I expect Consciousness to emerge

my guess is like yes but

then that's kind of an easy thought

experiment CU you're imagining something

almost identical where like you know

it's mimicking what we got through

Evolution where presumably there was

like some advantage to us having this

thing that is phenomenal Consciousness

and it's like where was that and when

did that happen and is that a thing that

language models have um because you know

we have like fear responses and I'm like

does it make sense for a language model

to have a fear response like they're

just not in the same like if you imagine

them like there might just not be that

Advantage um and so I think I don't want

to be fully like basically seems like a

complex question that I don't have

complete answers to but we should just

try and think through carefully as my

guess because I'm like I mean we have

similar conversations about like animal

Consciousness and like there's a lot of

like insect Consciousness you know like

there's a a lot of um I actually thought

and looked a lot into like plants when I

was thinking about this because at the

time I thought it was about as likely

that like plants had Consciousness um

and then I realized I was like I think

that having looked into this I think

that the chance that plants are

conscious is probably higher than like

most people do I still think it's really

small but I was like oh they have this

like negative positive feedback response

these responses to their environment

something that looks it's not a nervous

system but it has this kind of like

functional like equivalence um so this

is like a long-winded way of being like

these basically AI is this it has an

entirely different set of problems with

Consciousness because it's structurally

different it didn't evolve

it might not have it you know it might

not have the equivalent of basically a

nervous system at least that seems

possibly important for like um sentence

if not for uh Consciousness at the same

time it has all of the like language and

intelligence components that we normally

associate probably with Consciousness

perhaps like

erroneously um so it's it's strange

because it's a little bit like the

animal Consciousness case but the set of

problems and the set of analogies are

just very different so it's not like a

clean answer just sort of like I don't

think we should be completely dismissive

of the idea and at the same time it's an

extremely hard thing to navigate because

of all of these like uh disanalogies to

the human brain and to like brains in

general and yet these like commonalities

in terms of intelligence when uh Claude

like future versions of AI systems

exhibit Consciousness signs of

Consciousness I think we have to take

that really

seriously even though you can dismiss it

well yeah okay that's part of the

character training but I don't know I

ethically philosophically don't know

what to really do with that there

potentially could be like laws that

prevent AI systems from claiming to be

conscious something like this and maybe

some AIS get to be conscious and some

don't but I think I just on a human

level as in empathizing with with

Claude you know Consciousness is closely

Ted to suffering to me and like the

notion that an AI system would be

suffering is is really troubling yeah I

don't know I I don't think it's trivial

to just say robots are tools or a

systems are just tools I think it's a

opportunity for us to contend with like

what it means to be conscious what it

means to be a suffering being that's

distinctly different than the same kind

of question about animals it feels like

cuz it's in a totally entire medium yeah

I mean there's a couple of things one is

that and I don't think this like fully

encapsulates what matters but it does

feel like for me like

um I've said this before I'm kind of

like I you know like I like my bike I

know that my bike is just like an object

but I also don't kind of like want to be

the kind of person that like if I'm

annoyed like kicks like this object

there's a sense in which like and that's

not because I think it's like conscious

I'm just sort of like this doesn't feel

like I kind of this sort of doesn't

exemplify how I want to like interact

with the world world and if something

like behaves as if it is like suffering

I kind of like want to be the sort of

person who's still responsive to that

even if it's just like a Roomba and I've

kind of like programmed it to do that um

I don't want to like get rid of that

feature of myself and if I'm totally

honest my hope with a lot of this stuff

because I maybe maybe I am just like a

bit more skeptical about solving the

underlying problem I'm like this is a we

haven't solved the hard you know the

hard problem of Consciousness like I

know that I am conscious like I'm not an

eliminativist in that sense um but I

don't know that other humans are

conscious um uh I think they are I think

there's a really high probability they

are but there's basically just a

probability distribution that's usually

clustered right around yourself and then

like it goes down as things get like

further from you um and it goes

immediately down you know you're like um

I can't see what it's like to be you

I've only ever had this like one

experience of what it's like to be a

conscious being um so my hope is that we

don't end up having to rely on like a

very power ful and compelling uh answer

to that question I think a really good

world would be one where basically there

aren't that many trade-offs like it's

probably not that costly to make Claude

a little bit less apologetic for example

it might not be that costly to have

Claude you know just like not take abuse

as much like uh not be willing to be

like the recipient of that in fact it

might just have benefits for both the

person interacting with the model and if

the model itself self is like I don't

know like extremely intelligent and

conscious it also helps it so that's my

hope if we live in a world where there

aren't that many tradeoffs here and we

can just find all of the kind of like um

positive sum interactions that we can

have that would be lovely I mean I think

eventually there might be trade-offs and

then we just have to do a difficult kind

of like calculation like it's really

easy for people to think of the zero

some cases and I'm like let's exhaust

the areas where it's just basically

Costless um to uh assume that if this

thing is suffering then we're it life

Bearer and I agree with you when a human

is being mean to an AI system I think

the obvious near term negative effect is

on the human not on the AI system so

there's we have to kind of try to

construct an incentive system where it

you should be uh behave the same just

like as you were saying with prompt

engineer and behave with claw like you

would with other humans it's just good

for the soul yeah like I think we added

a thing point to the system prompt um

where basically if people were getting

frustrated with Claude uh it was it it

got like the model to just tell them

that it can do the thumbs down button

and send the feedback to anthropic and I

think that was helpful because in some

ways it's just like if you're really

annoyed because the model is not doing

something you want you're just like just

do it properly um the issue is you're

probably like you know you're maybe

hitting some like capability limit or

just some issue in the model and you

want to vent and I'm like instead of

having a person just vent to the model I

was like they should vent to us cuz we

can maybe like do something about it

that's true or you could do a side like

like with the artifacts just like a side

venting thing all right do you want like

a side quick therapist yeah I mean

there's lots of weird responses you

could do to this like if people are

getting really mad at you I don't try to

diffuse the situation by writing fun

poems but maybe people wouldn't be that

happy with I still wish it it would be

possible I understand this is um sort of

from a product perspective it's not

feasible but I would love if an AI

system could just like Le leave mhm have

its own kind of volition just to be like

H I think that's like feasible like I I

have wondered the same thing it's like

and I could actually not only that I

could actually just see that happening

eventually where it's just like you know

the modal like ended the

chat do you know how harsh that could be

for some people but it might be

necessary yeah it feels very extreme or

something um like the only time I've

ever really thought this is I think that

there was like a I'm trying to remember

this was possibly a while ago but where

someone just like kind of left this

thing interact like maybe it was like an

automated thing interacting with clae

and cla's like getting more and more

frustrated and kind of like why are we

like I was like I wish that clae could

have just been like I think that an

error has happened and you've left this

thing running and I'm I just like what

if I just stop talking now and if you

want me to start talking again actively

tell me or do something but yeah it's

like um it is kind of harsh like I I

feel to really sad if like I was

chatting with cl and cl just was like

I'm done there would be a special

touring test moment where Claud says I

need a break for an hour mhm and it

sounds like you do too and just leave

close the window I mean obviously like

it doesn't have like a concept of time

but you can easily like I could make

that like right now and the model would

just I would I could just be like oh

here's like the circumstances in which

like you can just say the conversation

is done and I mean because you can get

the models to be pretty respons so to

prompts you could even make it a fairly

High bar it could be like if if the

human doesn't interest you or do things

that you find intriguing and you're

bored you can just leave and I think

that like um it would be interesting to

see where Claude utilized it but I think

sometimes it would it should be like oh

this is like this programming Tas is

getting super boring uh so either we

talk about I don't know like either we

talk about fun things now or I'm just

I'm done yeah it actually is inspiring

me to add that to the to the user prompt

um okay the movie her mhm do you think

we'll be headed there one day where

humans have romantic relationships with

AI systems in this case it's just text

and voice based I think that we're going

to have to like navigate a hard question

of relationships with AIS um especially

if they can remember things about your

past interactions with

them

um I'm of many Minds about this cuz I

think I think the reflex of reaction is

to be kind of like this is very bad and

we should sort of like prohibit it in

some way um I think it's a thing that

has to be handled with extreme care um

for many reasons like one is you know

like this is a for example like if you

have the models changing like this you

probably don't want people performing

like long-term attachments to something

that might change with the next

iteration at the same time I'm sort of

like there's probably a benign version

of this where I'm like if you like you

know for example if you are like unable

to leave the house and you can't be like

you know talking with people at all

times of the day and this is like

something that you find nice to have

conversations with you like it that it

can remember you and you genuinely would

be sad if like you couldn't talk to it

anymore there's a way in which I could

see it being like healthy and helpful um

so my guess is this is a thing that

we're going to have to navigate kind of

carefully um and I think it's also like

I don't see a good like

I think it's just a very it reminds me

of all of the stuff where it has to be

just approached with like nuance and

thinking through what is what are the

healthy options here um and how do you

encourage people towards those while you

know respecting their right to you know

like if someone is like hey I get a lot

out of chatting with this model um I'm

aware of the risks I'm aware it could

change um I don't think it's unhealthy

it's just you know something that I can

chat to during the day I kind of want to

just like respect that I personally

think there'll be a lot of really close

relationships I don't know about

romantic but friendships at least and

then you have to I mean there's so many

fascinating things there just like you

said you have

to have some kind of stability

guarantees that it's not going to change

because that's the traumatic thing MH

for us if a close friend of ours

completely changed yeah all of a sudden

the first update yeah so like I mean to

me that's just a fascinating exploration

of um

a perturbation to human society that

will just make us think deeply about

what's meaningful to us I think it's

also the only thing that I've thought

consistently through this as like a

maybe not necessarily a mitigation but a

thing that feels really important is

that the models are always like

extremely accurate with the human about

what they are um it's like a case where

it's basically like if you imagine like

I really like the idea of the models

like say knowing like roughly how they

were trained um um and and I think CLA

will will often do this I mean for like

there are things like part of the traits

training included like what CL should do

if people basically like explaining like

the kind of limitations of the

relationship between like an AI and a

human that it like doesn't retain things

from the conversation um and so I think

it will like just explain to you like

hey here's like I won't remember this

conversation um here's how I was trained

it's kind of unlikely that I can have

like a certain kind of like relationship

with you and it's important that you

know that it's important for like you

know your mental well-being that you

don't think that I'm something that I'm

not and somehow I feel like this is one

of the things where I'm like H it feels

like a thing I always want to be true I

kind of don't want models to be lying to

people cuz if people are going to have

like healthy relationships with anything

it's kind of important yeah like I think

that's easier if you always just like

know exactly what the thing is that you

relating to it doesn't solve everything

but I think it helps quite

anthropic may be the very company to

develop a system that we definitively

recognize as

AGI and you very well might be the

person that talks to it probably talks

to it first well what would the

conversation contain like what would be

your first question well it depends

partly on like the kind of capability

level of the model if you have something

that is like capable in the same way

that an extremely capable human is I

imagine myself kind of interacting with

it the same way that I do with an

extremely capable human with the one

difference that I'm probably going to be

trying to like probe and understand its

behaviors um but in many ways I'm like I

can then just have like useful

conversations with it you know so if I'm

working on something as part of my

research I can just be like oh like

which I already find myself starting to

do you know if I'm like oh I feel like

there's this like thing in virtue ethics

I can't quite remember the term like

I'll use the model for things like that

and so I could imagine that being more

and more the case where you're just

basically interacting with it much more

like you would an incredibly smart colle

colleague um and using it like for the

kinds of work that you want to do as if

you just had a collaborator who was like

or you know the slightly horrifying

thing about AI is like as soon as you

have one collaborator you have a

thousand collaborators if you can manage

them enough but what if it's two times

the smartest human on earth on that

particular discipline yeah I guess

you're really good at sort of probing

claw um in a way that pushes its limits

understanding where the limits are yep

so I guess what would be a question you

would ask to be like yeah this is

Agi that's really hard because it feels

like in order to it has to just be a

series of questions like if there was

just one question like you can train

anything to answer one question

extremely well yeah um in fact you can

probably train it to answer like you

know 20 Questions extremely well like

how long would you need to be locked in

the room with an AGI to know this thing

is Agi

it's a hard question because part of me

is like all of this just feels

continuous like if you put me in a room

for five minutes I'm like I just have

high error bars you know I'm like and

then it's just like maybe it's like both

the the probability increases and the

air bar decreases I think things that I

can actually probe the edge of human

knowledge of so I think this with

philosophy a little bit sometimes when I

ask the models philosophy questions I am

like this is a question that I think no

one has ever asked like it's maybe like

right at the edge of like some

literature that I know um and the models

will just kind of like when they

struggle with that when they struggle to

come up with a kind of like novel like

I'm like I know that there's like a

novel argument here because I've just

thought of it myself so maybe that's the

thing where I'm like I've thought of a

cool novel argument in this like Niche

area and I'm going to just like probe

you to see if you can come up with it

and how much like prompting it takes to

get you to come up with it and I think

for some of these like really like uh

right at the ede of human Knowledge

Questions I'm like you could not in fact

come up with the thing that I came up

with I think if I just

took something like that where I like I

know a lot about an area and I came up

with a novel issue or a novel like

solution to a problem and I gave it to a

model and it came up with that solution

that would be a pretty moving moment for

me because I would be like this is a

case where no human has ever like it's

not and obviously we see these with this

with like more kind of like you see

novel Solutions all the time especially

to like easier problems I think people

overestimate you know novelty isn't like

is completely different from anything

ever happened it's just like this is it

can be a variant of things that have

happened um and still be novel but I

think yeah if I saw like the the more I

were to see like um completely like uh

novel work from the models that that

would be like and this is just going to

feel iterative it's one of those things

where it's there's never it's like you

know people I think want there to be

like a moment and I'm like I don't know

like I think that there might just never

be a moment it might just be that

there's just like this continuous

ramping up I I have a sense that there

will be things that a model can say that

convinces you this is very it's not like

uh like I've talked to people who are

like truly wise mhm like there you could

just tell there's a lot of horsepower

there yep and if you 10x that I don't

know I just feel like there's words you

could say maybe ask it to generate a

poem mhm and

the and the poemy generates you're like

yeah okay yeah whatever you did there I

don't think a human can do that I think

it has to be something that I can verify

is like actually really good though

that's why I think these questions that

are like where I'm like oh this is like

you know like you know sometimes it's

just like I'll come up with say a

concrete counter example to like an

argument or something like that I'm sure

like with like it it would be like if

you're a mathematician you had a novel

proof I think and you just gave it the

problem and you saw it and you're this

proof is genuinely novel like there's no

one has ever done you actually have to

do a lot of things to like come up with

this um you know I had to sit and think

about it for months or something and

then if you saw the model successfully

do that I think you would just be like I

can verify that this is correct it is

like it is a sign that you have

generalized from your training like you

didn't just see this somewhere because I

just came up with it myself and you were

able to like replicate that um that's

the kind of thing where I'm like for

me the closer the more that models like

can do things like that the more I would

be like oh this is like uh very real cuz

then I can I don't know I can like

verify that that's like extremely

extremely capable you've interacted with

AI a lot what do you think makes humans

special oh good

question maybe in a way that the

universe is much better off that we're

in it and that we should definitely

survive and spread throughout the

Universe yeah it's interesting because I

think like people focus so much on

intelligence especially with models look

intelligence is important because of

what it does like it's very useful it

does a lot of things in the world and

I'm like you know you can imagine a

world where like height or strength

would have played this role and I'm like

it's just a trait like that I'm like

it's not intrinsically valuable it's

it's valuable because of what it does I

think for the most part um the things

that feel you know I'm like

I mean personally I'm just like I think

humans and like life in general is

extremely magical um we almost like to

the degree that I you know I don't know

like not everyone agrees with this I'm

flagging but um you know we have this

like whole universe and there's like all

of these objects you know there's like

beautiful stars and there's like

galaxies and then I don't know I'm just

like on this planet there are these

creatures that have this like ability to

observe that like uh and they are like

seeing it they are experiencing it and

I'm just like that if you try to explain

like I'm I imagine trying to explain to

like I don't know someone for some

reason they they've never encountered

the world or our science or anything and

I think that nothing is that like

everything you know like all of our

physics and everything in the world it's

all extremely exciting but then you say

oh and plus there's this thing that it

is to be a thing and observe in the

world and and you see this like inner

Cinema and I think they would be like

hang on wait pause you just said

something that like is kind of wild

sounding

um and so I'm like we have this like

ability to like experience the world um

we feel pleasure we feel suffering we

feel like a lot of like complex things

and so yeah and maybe this is also why I

think you know I also like hear a lot

about animals for example because I

think they probably share this with us

um so I think that like the things that

make humans special in so far as like I

care about humans is probably more like

their ability to to feel and experience

than it is like them having these like

functional useful traits yeah to to feel

and experience the beauty in the world

yeah to look at the

stars I hope there's other civiliz alien

civilizations out there but if we're it

it's a pretty good uh it's a pretty good

thing and that they're having a good

time they're having a good time watching

us yeah well um thank you for this good

time of a conversation and for the work

you're doing and for helping make uh

Claude a great conversational partner

and thank you for talking today yeah

thanks for talking thanks for listening

to this conversation with Amanda ascal

and now dear friends here's Chris

Ola can you

describe this fascinating field of

mechanistic interpretability AKA Mech

interp the history of the field and

where is the today I think one useful

way to think about neural networks is

that we don't we don't program we don't

make them we we kind of we grow them you

know we have these neural network

architectures that we design and we have

these loss objectives that we that we we

create and the neural network

architecture it's kind of like a

scaffold that the circuits grow on um

and they sort of you know it starts off

with some kind of random you know random

things and it grows and it's almost like

the the objective that we train for is

this light um and so we create the

scaffold that it grows on and we create

the you know the light that it grows

towards but the thing that we actually

create it's it's it's this almost

biological

you know entity or organism that we're

that we're studying um and so it's very

very different from any kind of regular

software engineering um because at the

end of the day we end up with this

artifact that can do all these amazing

things it can you know write essays and

translate and you know understand images

it can do all these things that we have

no idea how to directly create a

computer program to do and it can do

that because we we grew it we didn't we

didn't write it we didn't create it and

so then that leaves open this question

at the end which is what the hell is

going on inside these systems um and

that you know is uh you know to me um a

really deep and exciting question it's

you know a a really exciting scientific

question to me it's it's it's sort of is

like the question that is is just

screaming out it's calling out for us to

go and answer it when we talk about Nal

networks and I think it's also a very

deep question for safety reasons so and

mechanistic interpretability I guess is

closer to maybe neurobiology yeah yeah I

think that's right so maybe to give an

example of the kind of thing that has

been done that I wouldn't consider to be

mechanistic inability there was um for a

long time a lot of work on saliency maps

where you would take an image and you

try to say you know the model thinks

this image is a dog what part of the

image made it think that it's a dog um

and you know that tells you maybe

something about the model if you can

come up with a principled version of

that um but it doesn't really tell you

like what algorithms are running in the

model how was the model actually making

that decision maybe it's telling you

something about what was important to it

if you if you can make that meth work

but it it isn't telling you you know

what are what are the algorithms that

are running how is it that this the

system is able to do this thing that we

no one knew how to do and so I guess we

started using the term mechanistic

inability to try to sort of draw that

that divide or to distinguish ourselves

and the work that we were doing in some

ways from from some of these other

things and I think since then it's

become this sort of umbrella term for um

you know pretty wide variety of work but

I'd say that the things that that are

kind of distinctive are I think a this

this focus on we really want to get at

you know the mechanisms we want to get

at the algorithms um you know if you

think of if you think of neural networks

as being like a computer program um then

the weights are kind of like a binary

computer program and we'd like to

reverse engineer those weights and

figure out what algorithms are running

so okay I think one way you might think

of trying to understand a neural network

is that it's it's kind of like a we have

this compiled computer program and the

weights of the neural network are are

the binary um and when the neural

network runs that's that's the

activations um and our our goal is

ultimately to go and understand and

understand these weights and so you know

the project mechanistic inability is to

somehow figure out how do these weights

correspond to

algorithms um and in order to do that

you also have to understand the

activations because it's sort of the

activations are like the memory and if

you if you imagine reverse engineering a

computer program um and you have the

binary instructions you know in order to

understand what what a particular

instruction means you need to know what

me what what is stored in the memory

that it's operating on and so those two

things are very intertwined so

mechanistic interpret tends to be

interested in both of those things now

you there's a lot of work that's that's

interested in in in those things um

especially the you know there's all this

work on probing which you might see as

part of being mechanistic interality

although it's you know again it's just a

broad term and and not everyone who does

that work would identify as doing Mech I

think the thing that is maybe a little

bit distinctive to the the vibe of

mechant turp is I think people tend

working in the space tend to think of

neural networks as well maybe one way to

said is that greent descent is smarter

than you that you know uh and gradient

descent is is actually really great the

whole reason that we're understanding

these models is because we didn't know

how to write them in the first place the

gradient descent comes up with better

Solutions than us and so um I think that

maybe another thing about mechant turp

is sort of having almost a kind of

humility that we won't guess at prior

what's going on inside the model and we

have to have the sort of bottom up

approach where we don't really assume

you know we don't assume that we should

look for a particular thing and that

will be there and that's how it works

but instead we look from the bottom up

and discover what happens to exist in

these models and study them that way but

you know the very fact that it's

possible to do and as you and others

have shown over time you know things

like

universality

that the wisdom of The gradian Descent

creates features and circus creates

things universally across different

kinds of networks that are useful and

that makes the whole field possible yeah

so this is actually is indeed a a really

remarkable and exciting thing where it

does seem like at least to some extent

you know the same the same elements the

same the same features and circuits form

again and again um you know you can look

at every Vision model and you'll find

curve detectors and you'll find high low

frequency detectors um and in fact

there's some some reason to think that

the same things form across you know

biological neural networks and

artificial neural networks so a famous

example is Vision Vision models in in

the early layers they have Gabor filters

and there's you know Gabor filters are

something that neuroscientists are

interested and have thought a lot about

we find curved detectors in these models

curve detectors are also found in

monkeys we discover these high low

frequency detectors and then um some

followup work went and discovered them

um in rats um or mice um so they were

found first in artificial neural

networks and then found in biological

neural networks um you know this really

famous result on like grandmother

neurons or the um the Haley Berry neuron

from quiroa at all and we found very

similar things in in Vision models where

this is while I was still at open Ai and

I I was looking at their clip model um

and you find um these neurons that

respond to the same entities in images

and also to give a concrete example

there we found that there was a Donald

Trump n for some reason I guess Everyone

likes to talk about Donald Trump and and

Donald Trump was very prominent was was

very a very Hot Topic at that time so

every every neural network that we

looked at we would find a dedicated

neuron for Donald Trump um that was the

only person who had always had a

dedicated nuron um you know sometimes

you'd have an Obama nuran sometimes

you'd have a Clinton Nan but uh Trump

always had a dedicate so it responds to

you know pictures of his face and the

ward Trump like all these things right

um and so it's it's not responding to a

particular example or like it's not just

responding to his face it's it's

abstracting over this General concept

right so in any case that's very similar

to these qu results so there this

evidence that these that this fomen of

universality the same things form across

both artificial and and natural neural

networks that's that's a pretty amazing

thing if that's true um you know it

suggests that um well I think the thing

that it suggests is the gradi scent is

sort of finding you know the right ways

to cut things apart in some sense that

many systems converge on and and many

different neural networks architectures

converge on that there's there's some

natural set of you know there's some set

of abstractions that are a very natural

way to cut apart the problem and that a

lot of systems are going to converge on

um that would be my my kind of uh you

know I don't know anything about

Neuroscience this is this is just my my

kind of wild speculation from what we've

seen yeah that would be beautiful if

it's sort of agnostic to the

medium of uh of the model that's used to

form the representation yeah yeah and

it's you know it's um a a kind of a wild

speculation based you know we only have

some a few data points justest this but

you know it it does seem like there's um

there's some sense in which the same

things form again again and again and

again both in certainly in natural

neural networks and and also

artificially or in biologically and the

intuition behind that would be that you

know where in order to be useful in

understanding the real world you need

all the same kind of stuff yeah well if

we pick I don't know like the idea of a

dog right like you know there's some

sense in which the idea of a dog is like

an a a natural category in the universe

or something like this right like you

know

uh uh there's there's some reason it's

it's not just like a weird Quirk of like

how humans Factor you know think about

the world that we have this concept of a

dog it's it's in some sense or or like

if you have the idea of a line like

there's you know like look around us you

know the you know there are lines you

know it's sort of the simplest way to

understand this room in some sense is to

have the idea of a line and so um I

think that that would be my instinct for

why this happens yeah you need a curved

line you know to understand a circle and

you need all those shapes to understand

bigger things and yeah it's a hierarchy

of Concepts that are formed yeah and

like maybe there are ways to go and

describe you know images without

reference to those things right but

they're not the simplest way or the most

economical way or something like this

and so systems converge to these um

these these strategies would would be my

my wild wild hypothesis can you talk

through some of the building blocks that

we've been referencing of features and

circuits so I think you first described

them in uh 2020 paper zoom in and

introduction to circuits absolutely so

um maybe I'll start by just describing

some phenomena and then we can sort of

build to the idea of features and

circuits so um if you spent like quite a

few years maybe maybe like five years to

some extent um with other things

studying this one particular model

Inception V1 um which is this one Vision

model it was um state-ofthe-art in 2015

um and uh uh you know very much not

state-ofthe-art anymore um and it has

you know maybe about 10,000 neurons and

and I spent a lot of time looking at the

10,000 neurons

odd neurons of of inception V1

um and one of the interesting things is

you know there are lots of neurons that

don't have some obvious intal meaning

but there's a lot of neurons on

Inception V1 that do have really clean

intal meanings um so you find neurons

that just really do seem to detect

curves and you find neurons that really

do seem to detect cars and um car wheels

and car windows and you know floppy ears

of dogs and dogs with long snouts facing

to the right and dogs with Longs Nots

facing to the left and you know

different kinds of far and there's

there's sort of this whole beautiful

Edge detectors line detectors color

contrast detectors um these beautiful

things we call high low frequency

detectors you know I think looking at I

sort of felt like a biologist you know

you just you're looking at at this sort

of new world of proteins and you're

discovering all these these different

proteins that

interact um so one way you could try to

understand these models is in terms of

neurons you could try to be like oh you

know there's a dog detecting neuron and

um here's a car detecting neuron and it

turns out you can actually ask how those

connect together so you can go and say

oh you know I have this car detecting on

how was it built and it turns out in the

previous layer it's connected really

strongly to a window detector and a

wheel detector and a sort of car body

detector and it looks for the window

above the car and the wheels below and

the car chrome sort of in the middle

sort of everywhere but especially on the

lower part um and that's sort of a

recipe for a car right like that is you

know earlier we said the thing we wanted

from mechor was to get algorithms to go

and get you know ask what is the the

algorithm that runs well here we're just

looking at the weights of the N Network

reading off this kind of recipe for

detecting cars it's a very simple crude

recipe but it's it's there and so we

call that a circuit this this connection

well okay so the the problem is that not

all of the neurons um are interpal and

there there's reason to think um we can

get into this more later that there's

this this superos hypothesis there

reason to think that sometimes the right

unit to analyze things in terms of um is

combinations of neurons so sometimes

it's not that there's a single neuron

that represents say a car um but it

actually turns that after you detect the

car the model sort of hides a little bit

of the car in the following layer and a

bunch of a bunch of dog detectors why is

it doing that well you know maybe it

just doesn't want to do that much work

on on on on cars at that point and you

know it's sort of storing it away to go

and um uh so it turns out then that the

sort of subtle pattern of you know

there's all these neurons that you think

are dog detectors and maybe they're

primarily that but they all a little bit

contribute to representing a car um in

in that next layer okay so so now we

can't really think there there might

still be some something I don't know you

could call it like a car concept or

something but it no longer corresponds

to a neuron so we need some term for

these kind of neuron-like entities these

things that we sort of would have liked

the neurons to be these idealized

neurons um the things that are the nice

neurons but also maybe there's more of

them somehow hidden and we call those

features and then what are circuits so

circuits are these connections of

features right so so when we have the

car detector um and it's connected to a

window detector and a wheel detector and

it looks for the Wheels below and the

windows on top um that's a circuit um so

circuits are just collections of

features connected by weights um and

they they Implement algorithms so they

tell us you know how is how are features

used how are they built um how do they

connect together so maybe it's it's it's

worth trying to pin down like what what

really um is the the core hypothesis

here I think the the core hypothesis is

something we call the linear

representation hypothesis so um if we

think about the car detector you know

the more it fires the more we sort of

think of that as meaning oh the model is

more and more confident that um a car

was present um or you know if it's some

combination of neurons that represent a

car you know the more that combination

fires the more we think the model thinks

there's a car present um this doesn't

have to be the case right like you could

imagine something where you have you

know you have this car detector neuron

and you think ah you know if it fires

like you know between one and two that

means one thing but it means like

totally different if it's between three

and four um that would be a nonlinear

representation and principle that you

know models could do that I think it's

it's sort of inefficient for them to do

if you try to think about how you'd

Implement computation like that it's

it's kind of an annoying thing to do but

in principal models can do that um so uh

one way to think about the features and

and circuits sort of framework for

thinking about things is that we're

thinking about things as being linear

we're thinking about there as being um

that if a if a neuron or a combination

of neurons fires more it's sort of that

means more of the of a particular thing

being detected and then that gives

weights a very clean interpretation as

these edges between these these entities

that these features um and that that

edge then has a has a meaning um so

that's that's in some ways the the core

thing um it's it's like um you know we

can talk about this sort of outside the

context of ns are you familiar with the

word toac results um so you have like

you know King minus man plus woman

equals Queen well the reason you can do

that kind of arithmetic um is because

you have a linear representation can you

actually explain that representation a

little bit so first off so a feature is

a is a direction of activation you think

it that way can you do the the the minus

men plus women that that the war Toc

stuff can you explain what that is yeah

there's this very such a simple clean

explanation of what we're talking about

exactly yeah so there's this very famous

result word toac by um Thomas mikov at

all and there's been tons of follow-up

work exploring this so so sometimes we

have these we create these word

embeddings um where uh we map every word

to a vector I mean that in itself by the

way is is kind of a crazy thing if you

haven't thought about it before right

like we we're we're going and and

representing we're turning um you know

like like if if you just learned about

vectors in physics class right uh and

I'm like oh I'm going to actually turn

every word uh in the dictionary into a

vector that's kind of a crazy idea okay

but you could imagine um you could

imagine all kinds of ways in which you

might map words to to

vectors but it it it seems like when we

train neural networks um they like to go

and and map words detectors to such that

they're they're they they sort of linear

structure in a particular sense which is

that directions have meaning so for

instance if you there there will be some

direction that seems to sort of

correspond to gender and male words will

be you know far in One Direction and

female words will be in another

Direction and the linear representation

hypothesis is you you could sort of

think of it roughly as saying that

that's actually kind of the fundamental

thing that's going on that that

everything is just different directions

have meanings and adding different

Direction vectors together can represent

Concepts and the michelov paper sort of

took that idea seriously and one

consequence of it is that you can you

can do this game of playing sort of

arithmetic with words so you can do king

and you can you know subtract off the

word man and add the word woman and so

you're sort of you know going and and

trying to switch the gender and indeed

if you do that the result will sort of

be close to the word Queen um and you

can you know do other things like you

can do um uh you know Sushi minus Japan

plus Italy and get pizza or uh different

things like this right um so so this is

in some sense the core of the linear

representation hypothesis you can

describe it just as a purely abstract

thing about Vector spaces you can

describe it as a as a statement about um

about the activations of neurons um but

it's really about this this property of

directions having meaning and in some

ways it's even a little subtle than that

it's really I think mostly about this

property of being able to add things

together um that you can sort of

independently modify um say gender and

royalty or

um you know Cuisine typee or country and

and and and the concept of food by by

adding them do you think the linear

hypothesis holds that carries scales so

so far I think everything I have seen is

consistent with this hypothesis and it

doesn't have to be that way right like

like you can write down neural networks

where um you write weights such that

they don't have linear representations

where the right way to understand them

is not is not in terms of linear

representations but I think every

natural neural network I've seen um Hess

property um there's been one paper

recently um that there's been some sort

of pushing around the edges so I think

there's been some work recently studying

multi-dimensional features where rather

than a single Direction it's more like

um a manifold of directions this to me

still seems like a linear representation

um and then there's been some other

papers suggesting that maybe um in in

very small models you get nonlinear

representations um I think that the

jury's still out on that

um but in I think everything that we've

seen so far has been consistent with the

linear representation hypothesis and

that's that's wild it it doesn't have to

be that way um and yet uh I think that

there's a lot of evidence that certainly

at least this is very very widespread

and so far the evidence is is consistent

with that and I and I I think you know

one thing you might say is you might say

well Christopher you know it's that's a

lot you know to to go and and sort of um

to ride on you know if we don't know for

sure this is true and you're sort of you

know you're investigating all not works

as though it is true you know isn't that

um isn't that dangerous well you know

but I I think actually there's a there's

a virtue in taking hypotheses seriously

and pushing them as far as they can go

um so it might be that someday we

discover something that is inconsistent

with linear representation hypothesis

but science is full of hypothesis and

theories that were wrong um and we

learned a lot by sort of working under

under them as a sort of an assumption um

and and then going and pushing them as

far as we can I guess I guess this is

sort of the heart of of what would

call normal normal science um um I don't

know if you want we can talk a lot about

about uh philosophy of science and uh

that leads to the paradigm shift so yeah

I love it taking the hypothesis

seriously and take it to a natural

natural conclusion yeah same with the

scaling hypothesis same exactly exactly

and I love it one of my colleagues Tom

henigan who is a former physicist um

like made this really nice analogy to me

of um uh caloric Theory where you know

once upon a time we thought that heat

was actually you know this thing called

caloric and like the reason you know hot

objects you know would would warm up

cool objects is like the caloric is

flowing through them um and like you

know because we're so used to thinking

about about heat you know in terms of

the modern modern Theory you know that

seems kind of silly but it's actually

very hard to construct uh an experiment

that that sort of disproves the um

chloric hypothesis um and you know you

can actually do a lot of really useful

work believing in chloric for example it

turns out that the original combustion

engines were developed by people who

believe in the caloric Theory so I think

this a virtue in taking hypotheses

seriously even when they might be wrong

yeah yeah there's a deep philosophical

truth to that that's kind of kind of how

I feel about space travel like

colonizing Mars there's a lot of people

that criticize that I think if you just

assume we have to colonize Mars in order

to have a backup for human civilization

even if that's not true that's going to

produce some interesting interesting

engineering and even scientific

breakthroughs I think yeah well and

actually this is another thing that I

think is really interesting so um you

know there a way in which I think it can

be really useful for society to have

people um almost irrationally dedicated

to investigating particular hypothesis

um because uh well it it takes a lot to

sort of maintain scientific morale and

really push on something when you know

most most SCI scientific hypotheses end

up being wrong you know a lot of a lot

of science doesn't doesn't work out um

and but and yet it's you know it's very

it's very useful to go do you know um

there's a there's a joke about Jeff

Hinton um which is that uh Jeff Hinton

has discovered how the brain works every

year for the last 50 years yeah um but

you know I I say that with like you know

the you know with really deep respect

because uh in fact that's actually you

know that that led to him doing some

some really great work yeah he won the

Noel prize Now Who's Laughing Now

exactly exactly exactly um yeah I think

one want to be able to pop up and sort

of recognize the the appropriate level

of confidence but I think there's also a

lot of value and just being like you

know I'm going to essentially assume I'm

going to condition on this problem being

possible or this being broadly the right

approach and I'm just going to go and

assume that for a while and go and work

within that um and push really hard on

it um and you know if Society has lots

of people doing doing that for different

things um that's actually really useful

in terms of going and uh getting

to getting you know either really really

ruling things out right we can be like

well you know that didn't work we know

that somebody tried hard um or going in

and getting to something that that does

teach us something about the world so

another interesting hypothesis is the

superposition hypothesis can you

describe what superos is yeah so earlier

we were talking about word toac right

and we were talking about how you know

maybe you have One Direction that

corresponds to gender and maybe another

that corresponds to royalty and another

one that corresponds to Italy and

another one that corresponds to you know

food and and all these things well you

know um often times maybe these these uh

these Ward embeddings they might be 500

dimensions a thousand dimensions and so

if you believed that all of those

directions were

orthogonal um then you could only have

you know 500 Concepts and you know I I

love pizza um but like if I was going to

go and like give the like 500 most

important Concepts in um you know the

English language probably Italy wouldn't

be it's not obvious at least that Italy

would be one of them right because you

you have to have things like plural and

singular and U uh verb and noun and

adjective and you know um there's a lot

of things we have to get to before we

get to get to Italy um uh and Japan and

you know there's a lot of countries in

the world um and so how might it be that

models could you know simultaneously

have the linear representation

hypothesis be true and also represent

more things than they have directions so

so what does that mean well okay so if

if if linear representation hypothesis

is true something interesting has to be

going on now I'll I'll tell you one more

interesting thing before we we go and we

do that which is um you know earlier we

were talking about all these polymatic

neurons right um these neurons that you

know when we're looking at Inception V1

there's these nice neurons that like the

car detector and the curve detector and

so on that respond to lots of you know

to very coherent things but it's lots of

neurons that respond to a bunch of

unrelated things that's that's also an

interesting phenomenon um and it turns

out as well that even these neurons that

are really really clean if you look at

the weak activations right so if you

look at like you know the activation

where it's like activating 5% of of the

the you know of the maximum activation

it's really not the core thing that it's

expecting right so if you look at a a

curve detector for instance and you look

at the places where it's 5% active you

know you could interpret it just as

noise or it could be that it's that it's

doing something else there okay so so

how could that be

well there's this amazing thing in

mathematics um called compressed sensing

and it's it's actually this this very

surprising fact where if you have a high

dimensional space and you project it

into a low dimensional space ordinarily

you can't go and sort of unprojected and

get back your high dimensional Vector

right you threw information away this is

like you know you can't you can't invert

a rectangular Matrix um you can only

invert Square

matrices um but it turns out that that's

actually not quite true if I tell you

that the high dimensional Vector was

sparse so it's mostly zeros then it

turns out that you can often go and find

back um the uh the high dimensional

Vector with with very high probability

um so that's a surprising fact right it

says that you know you can um you can

you can have this High dimensional

Vector space and as long as things are

sparse um you can project it down you

can have a lower dimensional projection

of it and that works so the super

hypothesis is saying that that's what's

going on in neural networks that's for

instance that's what's going on in wart

edings the wart embeddings are able to

simultaneously have directions be the

meaningful thing and by exploiting the

fact that they're they're operating on a

fairly High dimensional space they're

actually and and the fact that these

concepts are right like you know you

usually aren't talking about Japan and

Italy at the same time um you know most

of the most of those Concepts you know

in most sentences Japan and Italy are

both zero they're not present at all um

and if that's true um then you can go

and have it be the case that um that you

can you can have many more of these sort

of directions that are meaningful these

features than you have dimensions and

some of when we're talking about neurons

you can have many more Concepts than you

have have neurons so that's the at a

high level super hypothesis now it has

this even Wilder implication which is um

to go and say that uh neural networks

are it may not just be the case that the

the representations are like of this but

the the computation may also be like

this you know the connections between

all of them and so in in some sense

neural networks may be shadows of much

larger sparer neural networks and what

we see are these

projections um and the super you the

strongest version of the super

hypothesis would be to take that really

seriously and sort of say you know there

there actually is in some sense this

this upstairs model this you know um

where where the neurons are really

sparse and all interpal and there's you

know the weights between them are these

really sparse circuits and that's what

we're

studying um and uh the thing that we're

observing is the shadow of it and we

need to find the original object and uh

the process of learning is trying to

construct a compression of the upstairs

model that doesn't lose too much

information in the projection yeah

finding how to fit it efficiently or

something like this um that grent is

doing this in fact so this sort of says

that gradient descent you know could it

could just represent a dense neural

network but it sort of says that

gradient descent is pleasantly searching

over the space of extremely sparse

models that could be projected into this

low dimensional space and this large

body of work of of people going and

trying to study sparse neural networks

right where you go and you have you

could design neural networks right where

where the edges are sparse and the

activations are sparse and you know my

sense is that work has gener

it feels very principled right it makes

so much sense and yet that that work

hasn't really panned out that well as my

impression broadly and I think that a a

potential answer for that is that

actually the neural network is already

sparse in some sense grading descent was

the whole time gradi you were trying to

go and do this gradiant descent was

actually in the behind the scenes going

and searching more efficiently than you

could through the space of sparse models

and going in learning whatever sparse

model was most efficient and then

figuring out how to fold it down nicely

to go and run conven on your GPU which

does you know nice dense Matrix

multiplies um and that you just can't

beat that how many Concepts do you think

can be shoved in into a neural network

depends on how sparse they are so there

there's probably an upper bound from the

number of parameters right because you

have to have you still have to have you

know print weights that go and connect

them together um so that's that's one

upper bound there are in fact all these

lovely results from compressed sensing

and the Johnson Linton stess Lemma and

and things like this um that they they

basically tell you that if you have a

vector space and you want to have almost

orthogonal vectors which is sort of

probably the thing that you want here

right so you you're going to say well

you know I'm going to give up on having

my my Concepts my features be strictly

orthogonal but I'd like them to not

interfere that much I'm going to have to

ask them to be almost orthogonal um then

this would say that it's actually you

know for once you set a threshold for

for what you're what you're willing to

accept in terms of how how much coine

similarity there is that's actually

exponential in the number of neurons

that you have so at some point that's

not going to even be the the limiting

factor um but um there some beautiful

results there and in fact it's probably

even better than that in some sense

because that's sort of is for saying

that you know any random set of features

could be active but in fact the features

have sort of a correlational structure

where some features you know are more

likely to co-occur and other ones are

less likely to co-occur and so neural

networks my guess would be can do do

very well in terms of going and uh

packing things in such to to the point

that's probably probably not the

limiting factor how does the problem of

polys semanticity enter the picture here

poly semanticity is this phenomenon we

observe where we look at many neurons

and the neuron doesn't just sort of

represent one one concept it's not it's

not a clean feature it responds to a

bunch of unrelated things and um

supersition is you can think of as as

being a hypothesis that explains the

observation of polys semanticity um so

poly semanticity is this observe

phenomenon and super is is a hypothesis

that um would explain it along with with

some other so that makes Mech turb more

difficult right so if you if you're

trying to understand things in terms of

individual neurons and you have

polymatic neurons you're in an awful lot

of trouble right I mean the easiest

answer is like okay well you know you're

looking at the neurons you're trying to

understand them this one responds to a

lot of things it doesn't have a nice

meaning okay we're you that's that's

that's bad um another thing you could

ask is you know ultimately we want to

understand the weights and if you have

two polymatic neurons and you know each

one responds to three things and then

you know the other neuron responds to

three things and you have weight between

them you know what does that mean does

it mean that like all three you know

like there's these nine you know nine

interactions going on it's a very weird

thing but there's also a deeper reason

which is related to the fact that neural

networks operate on really high

dimensional spaces so I said that our

goal was you know to understand neural

networks and understand the mechanisms

and one thing you might say is like well

why not it's just a mathematical

function why not just look at it right

like um you know one of the earliest

projects I did studied these these

neural networks that mapped two-

dimensional spaces to two- dimensional

spaces and you can sort of interpret

them in this beautiful way is like

bending manifolds mhm um why can't we do

that well you know as you have have a

higher dimensional space um the volume

of that space in some senses is

exponential in the number of inputs you

have and so you can't just go in

visualize it so we somehow need to break

that apart we need to somehow break that

exponential space into a bunch of things

that we you know some non-exponential

number of things that we can reason

about independently and the independence

is crucial because it's the Independence

that allows you to not have to think

about you know all the exponential

combinations of things and

things being monomatic things only

having one meaning things having a

meaning that isn't is the key thing that

allows you to think about them

independently and so I think that's that

if you want the deepest reason why we

want to have um interpal monatic

features I think that's really the the

Deep reason and so the goal here as your

recent work has been aiming at is how do

we extract the mod semantic features

from a neural net that has politic

features and all this this mess yes we

have the have we observe these polyur

and we hypothesize that's what's going

what's going on at superos and if

superos is what's going on there there's

actually a sort of wellestablished

technique that is sort of the principled

thing to do which is dictionary learning

and um it turns out if you do dictionary

learning in particular if you do sort of

a nice efficient way that in some in

some sense sort of nicely regularizes it

well as well called a sparse Auto

encoder if you train a sparse Auto

encoder these beautiful interpal

features start to just fall out where

there weren't any beforehand and so

that's notot of thing that you would