Prof. Geoffrey Hinton - "Will digital intelligence replace biological intelligence?" Romanes Lecture

[Applause]

okay um I'm going to disappoint all the

people in computer science and machine

learning because I'm going to give a

genuine public lecture I'm going to try

and explain what neural networks are um

what language models are why I think

they understand I have a whole list of

those things um and at the end I'm going

to talk about some threats from AI just

briefly and then I'm going to talk about

the difference between digital and

analog neur networks and why that

differen is I think is so

scary so since the 1950s there have been

two paradigms for intelligence the logic

inspired approach thinks the essence of

intelligence is reasoning and that's

done by using symbolic rules to

manipulate symbolic

Expressions um they used to think

learning could wait I was told when I

was a student don't work on learning

that's going to come later once we

understood how to represent things the

biologically inspired approach is very

different

um it thinks the essence of intelligence

is learning the strengths of Connections

in a neural network and reasoning can

weight um don't worry about reasoning

for now that'll come later once we can

learn

things so now I'm going to explain what

artificial neural Nets are and those

people who know can just be

amused um a simple kind of neuron has

input neurons and output neurons so the

input neurons might represent the

intensities of pixels in an image the

output neurons might represent the

classes of objects in the image like dog

or cat and then there's intermediate

layers of neurons sometimes called

hidden neurons that learn to detect

features that are relevant for finding

these things so one way to think about

is if you want to find a bird in an

image it would be good to start with a

lay of feature detectors that detected

little bits of edge in the image in

various positions in various

orientations and then you might have a

layer of neurons that detected

combinations of edges like two edges

that meet at a fine angle um which might

be a beak or might not or some edges

forming a little circle and then you

might have a layer of neurons that

detected things like a circle and two

edges meeting that looks like a beak in

the right spatial relationship which

might be the head of a bird and finally

you might have an output neuron that

says well if I find the head of a bird

and the foot of a bird and the wing of a

bird it's probably a bird so that's what

these things are going to learn to

be now the little red and green dots are

the weights on the connections and the

question is who sets those weights

so here's one way to do it that's

obvious it's obvious to everybody that

it'll work and it's obvious it'll take a

long time you start with random weights

then you pick one weight at random that

little red dot and you change it

slightly and you see if the network

works better you have to try on a whole

bunch of different cases to really

evaluate whether it works better and you

do all that work just to see if

increasing this weight by a little bit

or decreasing it by a little bit

improves things if increasing it makes

it worse you decrease it in right that's

the mutation method and that's sort of

how Evolution Works um for evolution

it's sensible to work like that because

the process that takes you from the

genotype to the phenotype is very

complicated and full of random external

events so you don't have a model of that

process but for neural Nets it's

crazy because we have because all this

computations is going on in the neural

net we have a model of what's happening

and so we can use the fact that we know

what happens in that forward pass

instead of measuring how changing a

weight would affect things we actually

compute how changing weight would affect

things and there's something called back

propagation where you send information

back through the network the information

is um about the difference between what

you got and what you wanted and you

figure out for every weight in the

network at the same time whether you

ought to decrease it a little bit or

increase it a little bit to get more

like what you

wanted that's the back propagation

algorithm you do it with Calculus on the

Chain rule um and that is more efficient

than the mutation method by a factor of

the number of Weights in the network so

if You' got a trillion weights in your

network it's a trillion times more

efficient so one of the things that

neural networks are often used for is

recognizing objects in images neural

networks can now take an image like the

one shown and produce actually a caption

for the image as the output and people

tried with symbolic air to do that for

many years and didn't even get close um

it's a difficult task we know that the

biological system does with a hierarchy

of feature detectors so it makes sense

to try neural networks on

that and in

2012 two of my students IA and

Alvi um with a little bit of help from

me showed that you can make a really

good neural network this way for

identifying a thousand different types

of object when you have a million

training images before that um we didn't

have enough training

images and it was obvious to

IIA who's a Visionary that if we tried

the neural Nets we had then on image net

they would win and he was right they won

rather dramatically they got 16% errors

and the best conventional computer

vision systems got more than 25% errors

then what happens was very strange in

science normally in science if you have

two competing schools when you make a

bit of progress the other school says I

rubbish um in this case the Gap was big

enough that the very best researchers

like jendra and Andrew zman just Andrew

zman sent me a mail saying this is

amazing and switched what he was doing

and did that and then rather annoyingly

did it a bit better than

us but what about language so obviously

the symbolic a community um feels they

should be good at language and they've

said in print some of them that um these

feature hierarchies aren't going to deal

with language and many linguists are

very skeptical um Chomsky managed to

convince his followers that language

wasn't learned looking back on it that's

just a completely crazy thing to say if

you can convince people to say something

that's obviously false then you've got

them in your cult

um I think chumsky did amazing things

but his time is

over so the idea that a big neural

network with no innate knowledge

could actually learn both the syntax and

the semantics of language just by

looking at data was regarded as

completely Crazy by statisticians and

cognitive scientists I had statisticians

explain to me a big model has a 100

parameters the idea of learning a

million parameters is just stupid well

we're doing a trillion

now and I'm going to talk now about some

work I did in

1985 that was the first language model

to be trained with back propagation

and it was really you can think of it as

the ancestor of these big models now and

I'm going to talk about it in some

detail because it's so small and simple

that you can actually understand

something about how it works and once

you understand how that works it gives

you insight into what's going on in

these bigger models um so there's two

very different theories of meaning

there's a kind of structuralist theory

where the meaning of a word depends on

how it relates to other words that comes

from duur and its symbolic AI really

believes in that approach so you'd you'd

have a relational graph where you have

nodes for words and arcs of relations

and you kind of capture meaning like

that and they assume you have to have

some structure like that and then

there's a theory that was in Psychology

since the 1930s or possibly before that

the meaning of a word is a big bunch of

features um the meaning of the word dog

is that it's animate and it's a predator

and um so on but they didn't say where

the features came from or exactly what

the features were and these two theories

of meaning sound completely different

and what I want to show you is how you

can unify those two theories of meaning

and I did that in a simple model in

1985 that had it had more than a

thousand weights in

it

um the idea is we're going to learn a

set of semantic features for each

word and we're going to learn how the

features of words should interact in

order to predict the features of the

next word so it's next word prediction

just like the current language models

when you fin shun

them but all of the knowledge about how

things go together is going to be in

these feature interactions there's not

going to be any explicit relational

graph if you want relations like that

you generate them from your features so

it's a generative model and the

knowledge is in the features that you

give to symbols and in the way these

features

interact so I took some simple

relational information in two family

trees they were deliberately

isomorphic um my Italian graduate

student always had the Italian family on

top you can express that same

information as a set of triples so if

you use the 12 relationship shown there

you can say things like Colin has Father

James and Colin has mother Victoria from

which you can infer um in this nice

simple World from the 1950s where um

that James has wife

Victoria and there's other things you

can infer and the question is if I just

give you some triples how do you get to

those

rules so what a symbolic AI person would

want to do is derive rules of the form

if x has mother Y and Y has husband Z

then X has father

Z and what I did was take a neural net

and show that it could learn the same

information but all in terms of these

feature

interactions now for very discrete rules

that are never violated like this that

might not be the best way to do it and

indeed symbolic people try doing it with

other methods but as soon as you get

rules that are a bit flaky and don't

always apply then neural Nets are much

better and so the question was could a

neural net capture the knowledge that a

symbolic person would have put into the

rules by just doing back propagation so

the neural net looked like this um there

was a symbol represent

the person a symbol representing the

relationship that symbol then VI some

connections went to a vector of features

and these features were learned by the

network so there features for person one

and features for the relationship and

then those features interacted and

predicted the features for the output

person from which you predicted the

output person you find the closest match

with the last

slay so what was interesting about this

network was that it learned sensible

things

if you did the right regularization the

six feature neurons so nowadays these

vectors are 300 or a th000 long back

then they were six long um this was done

on a machine that took 12.5 micros

seconds to do a floating Point multiply

which was much better than my Apple 2

which took two and a half micros two and

a half milliseconds to do Flo Point

multiply um sorry this is an old man

um so it learned features like the

national ity because if you know person

one is English you know the output is

going to be English so nationality is a

very useful feature it learned what

generation the person was because if you

know the relationship if you learn for

the

relationship that the answer is one

generation up from the input and you

know the generation of the input you

know the generation of the output bya

these feature

interactions so it learned all these the

obvious features of the domain and it

learned how to make those features

interact

so that it could generate the output So

what had happened was I'd shown it

symbol strings and it had created

features such that the interactions

between those features could generate

those symbol strings but it didn't store

symbol strings just like gbt 4 that

doesn't store any sequences of

words um in its long-term knowledge it

turns them all into weights from which

you can regenerate

sequences but this is a particularly

simple example of it where you can

understand what it

did so the large language models we have

today I think of as descendants of this

tiny language model they have many more

words as input like a million um a

million word fragments they use many

more layers of

neurons like dozens um they use much

more complicated interactions so they

don't just have a feature affecting

another feature they sort of match two

feature vectors and then let one vector

affect affect the other one a lot if

it's similar but not much of his

different and things like that so it's

much more complicated interactions but

it's the same general framework the the

same general idea of let's turn symbol

strings into features for word fragments

and interactions between these feature

vectors that's the same in these

models it's much harder to understand

what they do um many people particularly

people from the chumsky school argue

they're not really intelligent they're

just a form of glor auto complete that

uses statistical regularities to pasti

together pieces of text that were

created by people that's a quote from

somebody

um so let's deal with the autocomplete

objection when someone says it's just

autocomplete um they're actually

appealing to your intuitive notion of

how autocomplete works so in the old

days autocomplete would work by you'd

store say triples of words if you saw

the first two you count how often that

third one occurred so if you see fish

and chips occurs a lot after that but

hunt occurs quite often too so chips is

very likely and hunts quite likely and

although is very unlikely um and you can

do autocomplete like that and that's

what people are appealing to when they

say it's just autocomplete it's a dirty

trick I think because that's not at all

how llms predict the next word they turn

words into features they make these

features

interact and from those feature

interactions they predict the features

of the next

word and what I want to claim

is that these millions of features and

billions of interactions between

features that they learn are

understanding what they're really doing

these large language models they're

fitting a model to data it's not the

kind of Model strates T thought much

about until recently um it's a weird

kind of model it's very big it has huge

numbers of parameters but it is trying

to understand these strings of discrete

symbols by features and how features

interact so it is a

model and that's why I think these

things are really understanding and one

thing to remember is if you ask well how

do we understand because obviously we

think we understand um well many of us

do anyway

um this is the best model we have of how

we understand so it's not like there's

this weird way of understanding that

these AI systems are doing and then

there how the brain does it the best

model we have of how the brain does do

it is by assigning features to words and

having feature interactions and

originally this little language model

was designed as a model of how people do

it okay so I'm making the very strong

claim these things really do

understand now another argument people

use is that well gp4 just hallucinates

stuff it should actually be called

confabulation when it's done by a

language model and they just make stuff

up now this psychologists don't say this

so much because psychologists know that

people just make stuff up anybody who

studied memory going back to Bartlet in

the

1930s knows that people are actually

just like these large language models

they just invent stuff and for us

there's no hard line between between a

true memory and a false memory if

something happened recently and it sort

of fits in with the things you

understand you'll probably remember it

roughly correctly if something happened

a long time ago or it's weird you'll

remember it wrong and often you'll be

very confident that you remembered it

right and you're just wrong it's hard to

show that but one case where you can

show it is John Dean's memory so John

Dean testified at Watergate under oath

and retrospectively it's clear that he

was trying to tell the truth um but a

lot of what he said was just plain wrong

he would confuse who was in which

meeting he would attribute statements to

other people who made that statement and

actually it was wasn't quite that

statement um he got meetings just

completely confused but he got the gist

of what was going on in the white house

right as you could see from the

recordings and because he didn't know

the recordings you could get a good

experiment this way alre niser has a

wonderful article talking about John D's

memory and he's just like a chatbot he

just makes stuff

up but it's plausible so it's stuff that

sounds good to him what he

produces they can also do reasoning so

I've got a friend in Toronto who's a

symbolic AI guy but very honest so he's

very confused by the fact these things

work at all and he suggested a problem

to me I made the problem a bit harder

and I gave this to gp4 before it could

look on the web so when it was just a

bunch of Weights Frozen in 2021 all the

knowledge is in the strengths of the

interactions between

features so the rooms in my are painted

blue or white or yellow yellow paint F

to White within a year in 2 years time I

want them all to be white what should I

do and why and Hector thought it

wouldn't be able to do

this and here's what gb4 said um it

completely nailed

it first of all it started by saying

assuming blue paint doesn't Fade to

White because after I told you yellow

paint Fades to White well maybe blue

paint does too um so assuming it doesn't

the white rooms you don't need to paint

the yellow rooms you don't need to paint

because they're going to Fade to White

within a year

and you need to paint the blue rooms

white one time when I tried it it said

you need to paint the blue rooms yellow

because it realized that will Fade to

White that's more of a mathematician

solution to reduce it to a previous

problem so having claimed that these

things really do understand I want to

now talk about some of the

risks so there are many risks from

powerful AI there's fake images voices

and video which are going to be used in

the next election there's many elections

this year and they're going to help to

undermine democracy I'm very worried

about that the big companies are doing

something about it but maybe not enough

there's a possibility of massive job

losses we don't really know about that I

mean the past technology often created

jobs but this

stuff well we used to be stronger than

we used to be the strongest things

around apart from animals and when we

got the Industrial Revolution we have

machines that were much stronger manual

labor jobs

disappeared so the equivalent of manual

labor jobs are going to disappear in the

intellectual realm when we get things

that are much smarter than us so I think

there's going to be a lot of

unemployment my friend Yan

disagrees um one has to distinguish two

kinds of unemployment too uh two kinds

of job loss there'll be jobs where you

can expand the amount of work that gets

done indefinitely like in healthcare

everybody would love to have their own

private doctor who's talking to them all

the time so they get a slight itch here

and the doctor says no that's not cancer

um so there's room for huge expansion of

how much gets done in medicine so there

won't be job loss there but in other

things maybe there will be significant

job loss there's going to be massive

surveillance that's already happening in

China there's going to be lethal

autonomous weapons which are going to be

very nasty and they're really going to

be autonomous the Americans very clearly

have already decided they say people

will be in charge but when you ask them

what that means is it doesn't mean

people will be in the loop that makes

the decision to

kill and as far as I know the Americans

intend to have half of their soldiers be

robots by

2030 now I don't know for sure that this

is true I asked Chuck Schumer's um

National Intelligence

advisor and he said well if there's

anybody in the room Who would know it

would be me so I took that to be the

American way of saying you might think

that but I couldn't possibly

comment there's going to be cyber crime

and deliberate

pandemics

um I'm very pleased that in England

although they haven't done much towards

regulation they have set aside some

money um so that they can experiment

with open source models and see how easy

it is to make them commit cyber crime um

that's going to be very important

there's going to be discrimination and

bias I don't think those are important

as the other threats but then I'm an old

white male

um discrimination and bias I think are

easier to handle than the other things

if your goal is not to be unbiased but

your goal is to be less biased than the

system you replace and the reason is if

you freeze the weights of an AI system

you can measure its bias and you can't

do that with people they will change

their behavior once you start examining

it so I think discrimination and bias

are the ones where we can do quite a lot

to fix

them but the threat I'm really worried

about and the thing I talked about after

I left Google is the long-term

existential threat that is the threat

that these things could wipe out

Humanity um and people were saying this

is just science fiction well I don't

think it is science fiction I mean

there's lots of Science Fiction about it

but I don't think it's science fiction

anymore um other people were saying um

the big companies are saying things like

that to distract from all the other bad

things and that was one of the reasons I

had to leave Google before I could say

this so I couldn't be accused of being a

Google stoe um although I must admit I

still have some Google

Shares

um there's several ways in which they

could wipe us

out

so a super

intelligence um will be used by Bad

actors like Putin Z or Trump and they'll

want to use it for manipulating elector

electorates and waging

Wars and they will make it do very bad

things and they may may go too far and

it may take

over the thing that probably worries me

most is

that if you want an intelligent agent

that can get stuff done you need to give

it the ability to create sub

goals so if you want to go to the States

you have a sub goal of getting to the

airport and you can focus on that sub

goal and not worry about everything else

for a

while so super intelligences will be

much more effective if they're allowed

to create sub

goals and once they are allowed to do

that they'll very quickly realize

there's a almost Universal sub goal

which helps with almost everything which

is get more

control so I talked to a vice president

of the European Union about whether

these things that these things would

want to get control so that they could

do things better the things we wanted so

they could do it better her reaction was

well why wouldn't they we've made such a

mess of it so so she took that for

granted

um so they're going to have the sub goal

of getting more power so they're more

effective at achieving things that are

beneficial for us um and they'll find it

easy to get more power because they'll

be able to manipulate people so Trump

for example could invade the capital

without ever going there himself just by

talking he could invade the capital and

these super intelligences as long as

they can talk to people when they're

much smarter than us they'll be able to

persuade to through all sorts of things

and so I don't think there's any hope of

a big switch that turns them off whoever

is going to turn that switch off will be

convinced by the superintelligence

that's a very bad

idea then another thing that worries um

many people is what happens if super

intelligences compete with each other

you'll have Evolution the one that can

grab the most resources will become the

smartest um as soon as they get any

sense of

self-preservation then you'll get

evolution occurring the ones with more

sense of self-preservation will win and

the were more aggressive ones will win

and then you'll get all the problems

that jumped up chimpanzees like us have

which is we evolved in small tribes and

there's lots of aggression and

competition with other

tribes and I want to finish by talking a

bit about um an epiphany I had at the

beginning of

2023 I had always

thought that

we were a long long way away from Super

intelligence I used to tell people 50 to

100 years maybe 30 to 100 years it's a

long way away we don't need to worry

about it

now and I also thought that making our

models more like the brain would make

them better I thought the brain was a

whole lot better than the a we had and

if we could make a a bit more like the

brain for example by having three time

scales most of the models we have at

present have just two time scales one

for for the changing of the weights

which is slow and one for the words

coming in which is fast changing the

neuronal activities so the changes in

neural activities and changes in weights

the brain has more time scales than that

the brain has rapid changes in weights

so quickly Decay away and that's

probably how it does a lot of short-term

memory and we don't have that in our

models for technical reasons to do with

being able to do Matrix Matrix

multiplies

um I still believe that if once we got

that into our models they'll get better

but

because of what I was doing for the two

years previous to that I suddenly came

to believe that maybe the things we've

got now the digital models we got now

are

already very close to as good as brains

and will get to be much better than

brains and I'm going to explain why I

believe

that so digital computation is great um

you can run the same program on

different computers different pieces of

Hardware or the same neural net on

different pieces of Hardware all you

have to do is save the weights and that

means it's Immortal once you've got some

weights they're Immortal because if the

hardware dies as long as you got the

weights you can make more hardware and

run run the same neural

net but to do that we run transistors at

very high power so they behave digitally

and we have to have Hardware that does

exactly what you tell it to that was

great when we instructed computers by

telling them exactly how to do

things but we've now got

another way of making computers do

things and so now we have the

possibility of using all the very rich

analog properties of Hardware to get

computations done at far lower energy so

these big language models when they're

training learn like megawatts use like

megawatts and we use 30

Watts so because we know how to train

things maybe we could use analog

Hardware

and every piece of Hardway is a bit

different but we train it to make use of

its peculiar properties so that it does

what we want so it gets the right output

for the

input and if we do that then we can

abandon the idea that hardware and

software have to be separate um we can

have weights that only work in that bit

of hardware and then we can be much more

energy

efficient so I started thinking about

what I call mortal comput

where you've abandoned that distinguish

between hardware and software you're

using very low power analog computation

you can parallelize over trillions of

Weights that are stored as conductances

[Music]

um and what's more the hardware doesn't

need to be nearly so reliable you don't

need to have Hardware that at the level

of the instructions will always do what

you tell it to you can have goopy

Hardware that you grow and then you just

learn to make it do the right

thing so you should be able toce

Hardware much more cheap ly maybe even

um do some genetic engineering on

neurons to make it out of recycled

neurons I want to give you one example

of how this is much more

efficient so the thing you're doing in

neural networks all the time is taking a

vector of neural activities and

multiplying it by a matrix of weights to

get the vector of neural activities in

the next lay at least get the inputs to

the next L and so a vector Matrix

multiplies the thing you need to make

efficient

so the way we do it in a digital

computer is we have these transistors

that are driven up very high power to

represent bits in say a 32-bit number

and then to multiply two 32-bit numbers

you need to perform I never did any

computer science courses but I think you

need to perform about a th one bit

digital operations it's about the square

of the bit length um if you want to do

it

fast um so you do lots of these digital

operations

there's a much simpler way to do it

which is you make a neural activity be a

voltage you make a weight be a

conductance and a voltage times a

conductance is a charge per unit time

and charges just add themselves

up so you can do your vector Matrix

multiply just by putting some voltages

through some conductances and what comes

into each neuron in the next layer will

be the product of this Vector with those

weights um that's great it's usually

more energy efficient you can buy chips

that do that already but every time you

do it it'll be just slightly

different also it's hard to do nonlinear

things like

this so there's several big problems

with Mortal

computation one

is that it's hard to use back

propagation because if you're making use

of the quirky analog properties of a

particular piece of

Hardware you can assume the hardware

doesn't know its own properties

and so it's now hard to use the back

propagation on your own it's much easier

to use reinforcement algorithms that

Tinker with weights and see if it helps

but they're very inefficient for small

networks we have come up with methods

that are about as efficient as back

propagation a little bit worse but these

methods don't yet scale up and I don't

know if they ever will back propagation

in a sense is just the right thing to do

and for big deep networks I'm not sure

we're ever going to get things that work

as well as back propagation so maybe the

learning algorithm in these analog

systems isn't going to be as good as the

one we have for things like large

language

models um another reason for believing

that is a large language model has say a

trillion weights you have a 100 trillion

weights even if you're only use 10% of

them for knowledge that's 10 trillion

weights but the large language model in

its trillion weights knows thousands of

times more than you do so it's got much

much more knowledge that's partly

because it seemed much much more more

data but it might be because it has a

much better learning algorithm we're not

optimized for that we're not optimized

for packing lots of experience into a

few connections where a trillion is a

few now um we're optimized for having

not many experiences you only live for

about a billion seconds that's assuming

you don't learn anything after your 30

which is pretty much true so you live

for about a billion seconds and you've

got a 100 trillion

connections so got crazily more

parameters than you have experiences so

our brains optimized for making the best

use of not very many

experiences another big problem with

Mortal computation is that if the

software is inseparable from the

hardware once a system has learned if

the hardware dies you lose all the

knowledge it's mortal in that sense and

so how do you get that knowledge into

another mortal

system well you get the old one to give

a lecture and the new ones to figure out

how to change the weights in their brain

so they would ass set that that's called

distillation you try and get a student

model to mimic the output of a teacher

model and that works but it's not that

efficient um some of you may have

noticed that universities just aren't

that efficient it's very hard to get the

knowledge from the professor into the

student so this installation method a

sentence for example has a few hundred

bits of information and even if you

learned optimally you couldn't convey

more than a few hundred

bits but if you take these big digital

models

then if you look at a bunch of agents

that all have exactly the same neural

meum with exactly the same

weights and they're digital so they run

in exact they use those weights in

exactly the same

way and these thousand different agents

all go off and look at different bits of

the internet and learn stuff and now you

want each of them to know what the other

ones learned you can achieve that by

averaging the gradients or averaging the

weights so you can get massive

communication of what One agent learned

to all the other agents so when you

share the weights or you share the

gradients you're communicating a

trillion numbers not just a few hundred

bits but a trillion real numbers and so

they're fantastically much better at

communicating

and that's what they have over us

they're just much much better at

communicating between multiple copies of

the same model and that's why gp4 knows

so much more than a human it wasn't one

model that did it it was a whole bunch

of copies of the same model running on

different

Hardware so my conclusion which I don't

really like

um is that digital computation requires

a lot of energy and so it would never

evolve we had to evolve making use of

the quirks of the hardware to be very

low

energy but once you've got it it's very

easy for agents to

share and gb4 has thousands of times

more knowledge in about 2% of the

weights so that's quite depressing um

biological computation is great for

evolving because it requires very little

energy

um but my conclusion is that digital

computation is just better um and so I

think it's fairly clear that maybe in

the next 20 years I'd say with a

probability of about 0.5 in the next 20

years it'll get smarter than us and very

probably in the next 100 Years it'll be

much smarter than us and so we need to

think about um how to deal with that and

there are very few examples of more

intelligent things being being

controlled by less intelligent things

one good example is a mother being

controlled by a baby Evolution's gone to

a lot of work to make that happen so

that the baby survive it's very

important for the baby to be able to

control the mother um but there aren't

many other

examples some people think that we can

make these things be

benevolent um but if they get into a

competition competition with each other

I think they'll start behaving like

chimpanzees and

I'm not convinced you can keep them

benevolent if they get very smart and

they get any notion of

self-preservation um they may decide

they're more important than

us so I finished the lecture in record

time I

think