DeepMind Scientist: Causal AI & Intelligence | Andrew Lampinen Ep 13 | CausalBanditsPodcast.com

you try to constrain the reasoning processes too strongly that actually is going to make the system more fragile because as soon as there's something weird in the world that doesn't quite match your assumptions the system will

totally break down the objective of humans and of language models isn't to be rational reasoners it's to hey causal Bandits welcome to the

causal Bandits podcast the best podcast on causality and machine learning on the internet this week we're traveling to London to meet our guest as a child even Lov to play chess he studied math and

physics but decided to purse a PhD in cognitive psychology because it seemed less abstract he loves rock climbing and plays guitar senior research scientist

at Google Deep Mind ladies and Gentlemen please welcome Dr Andrew lampin let me pass it to your host Alex

mola ladies and Gentlemen please welcome Dr Andrew lampinen welcome to the podcast Andrew thank you for having me how are you today doing

pretty well how are you I'm very good we have some sun in London yeah it's a lucky day you came in at the right time great Andrew in one of the recent papers

you published which was a part of a series of papers um about large language model and causality released in the second half of

2023 in the title of this paper you talk about active and passive strategies this is a little bit maybe unusual when we think about traditional

ways of talking about machine learning models in general in the context of causality where AOS would tend to extrapolate the naming convention from

Pearl's causal hierarchy and talk about models or training regimes that are Interventional or observational or counterfactual what dictated this choice

of of of wording or this choice of Concepts yeah I'm glad you brought that up because that's actually one of the key distinctions that we wanted to make in this paper um because although

language models are trained passively that is they're just processing language data from the internet that has been generated by someone else that doesn't necessarily imply that that data is

purely observational so for example if you're reading a scientific paper even though you're merely absorbing that data you're not out there making those experimental interventions yourself those data remain Interventional data

and you can learn about real causal information from those and so the first thing that we point out in the paper is that actually language data on the Internet is Interventional it contains science papers it contains stack

Overflow posts where people are debugging something and trying some experiments and seeing what works and what doesn't it contains just conversations where people are talking to each other and each thing they say is an intervention in that conversation and

so even though language models are learning passively they're learning from data that's Interventional and that is an important distinction does this fact um

impact their capabil capabilities to generalize beyond what we what we maybe traditionally think about this that is exactly what we wanted to explore in the paper so in the paper we suggest that

there's two reasons that this kind of training language data on the internet could give some sort of causal strategies or understanding that could generalize beyond the data that's it's been trained on so the first case is

what we call causal strategies and what we mean by that is that by learning from others interventions the models might be able to discover a strategy for intervening that they could apply in a

new situation to discover new causal structures then to use those for some Downstream goal and so what we suggest in the paper and suggest formally and then show empirically is that precisely

you can discover from purely passively observing someone else's interventions a generalizable strategy for intervening to determine causal structures that a system could then deploy Downstream can you tell our audience a little bit more

about how you structured the learning task for for for the language models in your paper and what were the main insights based on the results that you were able to to achieve and show in the

paper yeah so one of the hard things about studying large language models of course is that it's hard to understand everything that's going into the training Corpus even if we can search through it it's hard to know all the things that are in there that might just

be slightly rephrased or something and so one of the things that we try to do in our program of work is to do more controlled experiments where we train a simpler model on a data distribution that we really understand and then see

how well it generalizes or doesn't so what we did in this paper is that we trained a model on a distribution of data which shows interventions on causal

dag so on each document you could think or each episode in the data the model sees a series of interventions on a dag and then it is sees a goal like maximize

this variable and a series of interventions that try to achieve that goal and so what we wanted to test in the paper is if the model sees passively this data of

interventions that are trying to discover a causal structure and then use that causal structure to achieve a goal on a set of dags that hold out certain kinds of causal structures so it doesn't see everything in training can it

generalize to itself intervene actively and really discover new Cal structures and exploit them at test time and so the way we test this is just like a language model when you deploy it suddenly it

becomes active right it's talking to a user interv intervening on the user rather than just passively absorbing data anymore we similarly train the system passively we test it interactively and we show that the

system is able to apply the causal strategies that are passively observed in training and use them to actively intervene at test time discover new causal structures and exploit them then we compare the way that the model is

doing this to various heuristic or associational strategies we show that the model is much more closely approximating correct causal reasoning than any of those simpler baselines in another paper in in in the series of

papers um called causal parrots Co offered by mate zich who was our guest in the zero episode of this podcast uh the offers propose a hypothesis that

says that large language models are learning a so-called Meta Meta structural causal model and this meta structural causal model is learned based

on the how the authors call it the correlations of causal facts that are present in the in the training data and so one of the proposed

conclusions in this in this paper is that causal models can talk causality but they do not reason causally do you feel that the results from your paper are contradicting this hypothesis or

maybe are they complimentary in some sense well I would say to some extent they are contradicting it in the sense that I think our results suggest that actually the models are capable of

discovering some sort of causal reasoning algorithm that they can apply in a new setting in a generalizable way at least if they have a good enough training regime now I think that on

natural data the models probably are learning of a variety of things and of course in our training distribution there were a lot of interventions whereas natural data might have quite a few more correlations and only

occasional Interventional data and so that data mixture will change what the models are learning and they might be learning a lot of strategies that are more sort of like correlational meta like and relatively fewer causal

strategies however what we show at the end of our paper is that actually if you test language models on sort of like experimenting to discover causal Str

structure tasks similar to the ones we used for our simpler experiments in certain cases in particular if you give them explanations in the prompt they're actually able to learn to do this pretty effectively to discover new causal

structures that aren't included in the prompt and so I think that suggest that there is at least enough Interventional data going on in the language model training distributions that they are

able to discover some of these strategies but in terms of what they do on average or in like most of the cases you deploy them it could be something a little bit more like what they're describing in the Cal parot paper

definitely in your paper you also emphasize the importance of of explanations this maybe directs us to to

a broader topic of structuring the call the the training regime in a c in a certain way what are have thoughts on this is the training Paradigm or

training regime that that we today use to train those models helpful for them to use Cal structures or not necessarily I mean I think that the training Paradigm we use today is very driven by

what works and what allows you to use data at scale I think almost always if you were designing a training Paradigm for a model you would try to leverage

more kind of auxiliary tasks for example that help it to understand the data if there's something you know about the data yourself so uh what we did with explanations in this paper and what we've done in some prior papers is to

show that if you know something about a task like you know for example why a reinforcement learning agent is getting a reward from the environment giving it a natural language explanation of that reward and asking it to just predict

those explanations as part of the learning process can actually improve what it's learning and you can even use this in cases where the data are totally confounded to shape how the modeles out

of distribution so I think that to the exent that you know more about the data then it's there's definitely many things you could do to better structure the what the model is learning and to change

the way it generalizes I think the tricky thing is that you know we don't know that much about the structure of data on the internet and it's very you know there's a trade-off between trying

to do something that is that we really understand at small scale and trying to get as general a system as possible and since language modeling is a relatively new well language modeling itself isn't

a new field but training large language models is quite a new paradigm I think people are just barely starting to scrape the surface of what you can do in terms of structuring the training process in a more interesting way so one

example of this is that people have recently started conditioning these models on a signal of quality like a reward estimate for the data or a quality estimate during the training process and that actually can help in

certain cases to disentangle the training a bit more and to get a system that at the end if you say okay now produce only highquality things at test time you can still learn from the worst

data but you can push the system more to generate high quality responses at test time and this is similar to things people have done in reinforcement offline RL type contexts for a while and things like decision Transformer or

upside down RL where you really just condition on a signal of how good something is that allows you to learn from the bad data without necessarily replicating that data at test time in your paper you are talking about large

language models but also agents uh for those people in our audience who are less familiar with with reinforcement learning paradigms could you give a brief introduction or brief description

how about the intuition how large language models and agents are related so actually I think that the term agents is quite General and large language models can fit within the Paradigm of

passively trained agents so I think of an agent as just a system that takes inputs such as observations of the world and produces output actions in a

sequential decisionmaking problem and so this covers for example language models which take inputs in natural language and produce a sequence of language and response to that maybe an interaction with the user it also considers things

like the kinds of like chess and go playing engines that we've worked on in the past at Deep Mind also systems that play video games like Atari it covers quite a wide range of systems the key

part to me is is this aspect of interaction with the environment and one of the key distinctions to make here is systems that are interacting with the environment at training time or only at

testing time if you have an agent it's probably because you want it to interact at some point but often you train this these agents at least partly on data where they are not interacting but for example they're just observing what

expert Starcraft players or expert go players would do and learning from that and then later on you might do some reinforcement learning training where you're actually playing them against each other against other agents and then

giving them rewards if they win and then using that as a way to make to improve them and that's similar to the paradigms people are using with language models now where they first train them on a lot of passive data just language that

humans have generated on the internet and then they do some fine tuning and then they finally do a reinforcement learning step where they train a reward model based on what which kinds of language responses humans prefer and

they use that to actually reward the models for producing the kinds of responses humans will like some people would argue that auto regressive models might not be very well in generating

very long text or predicting long time series because there might be the mechanism of accumulating the error so

the model is conditioning the next token or the next Whatever item data point on its previous generation and this

generation necess necessarily contains some error and this error can be reinforced over time

steps interventions might be used in order to decrease the the the amount of error how much intervention and action

or action do we need to make autoaggressive models reliable for long longer form content let's let's think about text generation or video generation so I think there's a lot of

things that go into this question I mean one point I want to make is that even with current language models at least the state-of-the-art systems they have some ability to air correct in the sequences they're producing if you do something like Chain of Thought you will

occasionally see the models like produce something and then say Oh no that's a mistake actually I should do this right so it's not the case that air is necessarily M monotonically increasing over time although I will agree that the

systems are not particularly good at this yet I think that part of what's important for this is that the system is able to reconsider the prior context and

maybe find errors in its res in that way and so the architecture of the system and the for example its context length and its ability to use the earlier Generations may affect its ability to do

that in non-trivial ways now coming to the intervention part it's a well-known fact that passive learning of systems is not very efficient and tends to lead to worse generalization out of distribution that doesn't mean that it's impossible

for these systems to generalize but simplistic strategies for Passive learning at least you tend to get this problem where on the data distribution the system is accurate but once it starts being active it will move off

that data distribution and then it will start to break down because it's never seen any situation like this before and so in the offline reinforcement learning literature for a long time there's been techniques like dagger which basically

you get a little bit of Interventional data that sort of allows you to recover back to the data distribution if you step off of it and those kind of strategies are quite important for getting a robust system in practice so

you could see things like the active reinforcement learning people do with language models at the end of training as being a kind of way of doing this or even if you're doing supervised tuning but on real Generations the model

produces when interacting with users this is a kind of Interventional data but the question of how much of that data you need to get a robust model is I think a very hard question to answer you would have to make very strong

assumptions on what the data generating process are like what kind of structures you're trying to get the model to learn how how reliably the model can air correct and I think all of those questions are quite open at the moment

so I don't think it's easy to put put a number on it I see a very interesting parallel between what you just mentioned and the ideas from from causal inference literature where people talk about the

causal data Fusion or mixing observational and Interventional data in order to recover on on one hand recover the the causal structure of the problem

on the other hand maximize the efficiency of inference have you ever thought about those two ideas those those two p paths and relations

potential relationships between them yeah that's a great question so we didn't really address this in the paper but we did have some cases where we sort of like got the system to use some what

we called like prior knowledge from the literature you could think of it as coming from like an observational Source where we sort of cue it on which variables in a system might be relevant and then the system learns in a generalizable way to intervene only on

the relevant varibles so it's sort of like most efficiently determining the causal structure that are NE that are necessary and not exploring the things that aren't you could imagine doing this similarly with this system that also

combines observational data in as a way of sort of efficiently discovering which things might be the most promising to intervene on or as a part of the learning process more generally I think

that that similarly would work pretty well and um yeah I I I have some colleagues like Rosemary K who have worked on things in this space so I definitely think it's a promising area

where are we today from your perspective when it comes to building useful sequential decision making systems useful for what well useful for

some problems so I guess like I don't personally use large language models that much but I would consider them a sequential decision-making system and I've talked to a number of people who

have found them quite useful for things like writing for idea generation even for just sort of like critiquing their thinking on a certain problem that I think is you know so in some sense I

would say language models are a useful sequential decision-making system I wouldn't say they're a useful autonomous sequential decision-making system perhaps that would be an important distinction to make right they're useful if you have a human in the loop at least

for right now because they're way too unreliable to use without a human but I think they're useful in certain cases and honestly I'm more interested in AI systems as tools to help humans rather

than AI systems to be an autonomous agent at least in deployment time right I'm I'm more excited about for example things that a scientist could use as a tool like for helping them to understand

some complex system better or helping them to reason about a problem I think that is probably going to be one of the most important and impactful use cases so as far as I'm concerned language

models are a useful sequential decision maker of course there could be a more useful sequential decision maker and I'm sure there'll be improvements in that very soon you said about helping scientists helping humans Building

Systems that are that are helping helpful for someone what drives you in your work yeah that's a great question because I I actually feel like a lot of the work I do is not particularly useful compared to some of my colleagues who

are working on things that help scientists like doing protein structure prediction or Fusion or something um I guess I'm mostly motivated by curiosity and by the hope that by developing some

understanding of the systems that we're working with we can better understand how to improve them and in ways that will be deployed Downstream to help improve scientific applications or other things so I guess I'm motivated

primarily by curiosity and by the hope that doing the sort of pure research that I do or more conceptual work that it will be useful Downstream in some application but we'll see if it is or

not you studied you studed mathematics and physics um and then you mve to cognitive psychology with elements of cognitive science neuroscience and so on and so on U and then you ended up at

Deep Mind or Google Deep Mind as we should call it now working with algorithms that nevertheless uh in many

cases somehow interact with with some environments what qualities or resources were the most helpful in your

journey to go through all those stages and and work in one of the the best known AI labs in the in the world I think my foundation in physics and

Mathematics has been really useful to me across a lot of the stages of my career because I think you know working in physics and in mathematics you get you develop good intuitions for things like linear algebra that crop up all over the

place in the statistics we do in cognitive psychology in thinking about how machine learning systems work and so on so I think that those quantitative foundations have been very useful to me

and also sort of programming foundations that I acquired there and in in high school as well the main resource I'd point to is actually the the people that I've interacted with along the way

because I think one of the things I find most valuable work about working at Deep Mind is the incredible people who were there who come from all sorts of different backgrounds and similarly during my PhD I was very lucky to work

with an adviser and in a department that has a diversity of backgrounds and has just really supportive people who are excited to teach you about the things that they're interested in and so yeah I

think I've learned the most from the people I've met along the way MH do you think that causality is necessary for

generalization I think that causality is one thing you can look for as a tool of generalization I mean I I often think about problems like in mathematics because of my background there and for

example in mathematics if I have a system that's like a a theorem that generalizes to some new theorems I don't know if thinking about that in terms of causal causality is the most useful way to understand that because I tend to

think of math as being fundamentally about equivalences that don't really have a causal direction right however I think in terms of a lot of real world systems and identic systems causality at

least offers very useful intuitions for thinking about how a system will generalize and where things might break down um and so I do find intuitions about confounding and intervention and

so on to be quite useful and practical at least in how I think about the problems of generalization for a gentic system specifically you mentioned

mathematical system fear improvers uh those systems are often are

built or maybe always are built as as um some logical uh as a series of logical

steps um which involve symbolic operations so in contrast to latch language models that learn well at least that's that's my understanding that

learn some kind of a smooth approximation U or smooth representation of Concepts symbolic systems will be more discrete on more discrete

end there's a whole field called neuros symbolic Ai and a Ab with neuros symbolic causal AI as well um in some of your work you all you're also interested

in representation learning do you think that combining representation learning as we usually do it today in in this

soft differentiable way with symbolic systems more discrete systems is um is a way or a path that has a bright future I

do but I think I think this in kind of the opposite way that of many other people and in particular what I'd like to say is that I think symbolic systems logical reasoning systems are useful

tools for an intelligent system to use but they are not intell relevant to intelligence in itself and what I mean by that is actually if you think about humans we are not particularly good logical reasoners we are not

particularly good mathematical reasoners without extensive training right and because of that we build tools like you know mathematical proving systems and things like that to help us with the

parts of solving problems that we're not particularly good at but I would say those tools are most useful in so far as they're used by a system that is intelligent and I think that that intelligence tends to take something

that's more the form of a continuous fuzzy sort of reasoning system and the symbolic logic is just a useful tool for those systems to use to tackle certain kinds of constraint problems that those

logical systems are good at so I do think that using symbolic approaches in certain situations like in mathematics or in physics or in many other computational Sciences is very very

useful but I think that the most effective most General ways to approach that will be through systems that put the fuzzy learning uh continuous systems

in control and just use the logical systems as sort of a a tool to use just like humans use yeah yeah I'm happy you said this I think I agree and not not maybe not many people uh think about

this in this direction so I feel less alone in the world um I often also argue that people are not that good in causality um which might be well

counterintuitive sometimes when we think about causality in humans in particular uh we'll often if you start reading about this we'll often end up at some

point at least that's one of the paths you can take with with research coming from Alison gopnik who's a developmental

psychologist a very prominent one he and she and and people in her team has shown repeatedly that human babies uh when they interact with with

the world they do it in a systematic way that allows them to build World models and it seems that the motivation for this exploration or at least some

structure that allows them to do this in a way that is systematic uh is at least to some extent uh inherited inborn or

developed through evolutionary paths maybe do you think that these types of um solutions that come from Evolution I

don't know if you would agree with me that it comes from from from Evolution but assuming that it does do you think that those can be

imitated uh in in a simple computational manner or is there something maybe some underlying complexity in the EV evolutionary process that is not that

easy to to ulate yeah I think that's a great question I don't think we know the answer yet one different instance I'll point to is the fact that you know for a long time people thought that

grammatical abilities of humans were evolutionary and couldn't possibly be learned from any finite amount of data particularly only positive examples I think what language models show is that

actually you know you might need more language data than humans learn from but you don't need that much more and actually people are trying to push that amount of data down even further and get it to a really humanlike regime as a

side note I actually think that embodied experience might be quite important in our learning efficiency in language and we had a paper a few years back showing that systems that are embodied in richer en environments show better language

generalization for example now coming back to the causality point though I don't think we know yet exactly what it will take to get human-like inductive biases to a system maybe you do need

more data or maybe the active experience that infants have even when they're growing up before they come to Allison's lab is quite important one thing that I think is very interesting I'm glad you brought up Allison's work because some

of her most famous studies are actually quite close to the Paradigm I'm describing of passive observation because the children don't perform the experiments in these studies themselves at least not at first they observe a lab

assistant putting some things onto different detectors and seeing what lights them up and what doesn't and then it's only from observing that uh those interventions that the children get some sense of the causal structure that is

then tested in some trials where the kids are actually allowed to play with the detectors and the different things so I think it is actually it has a somewhat close analogy to the kinds of learning that I'm describing in this passive learning paper um and I think

there's some very interesting connections to be made there now Allison and some people from her lab did try these kind of studies with language models uh in a paper that came out a little bit before minded and they showed

that the language models are still not particularly good at them so it does seem like there's something there that's missing and perhaps it is sort of this early interactive experience that the models are missing but I think we'll

have to do some more exploring to resolve that often when we talk about large language models we can see some people giving comments about understanding so they say ah they are just predicting the next token but they

don't really understand what language is they don't have the semantics are in a way uh similar to the Chinese rumor

argument what is understanding I guess I tend to think of understanding and many other properties as being sort of graded things that are you know it's not like a

system is discreetly oh it has understanding or it doesn't rather it's like well understanding is something you have to a degree it's the degree of like richness of representation you have of something degrees of ways you can

generalize that so again returning to math like I think about this a lot in the context of you know if you have a student in an intro calculus class they might understand differentiation for example to some extent but probably

their math teacher understands it better and someone who's like a researcher in sort of an analysis would understand it even better right there's sort of different degrees of understanding of a

concept even if you're able to work work with it to some extent so I think that language models do have some degrees of understanding of certain features of course there's some aspects of our

understanding of the world for example that are really about the phenomena of perception that they can't possibly learn about but that doesn't mean they don't have any understanding at all and similarly the

fact that they are trained by just predicting the next token I don't think that means they don't have understanding I would think of understanding as you know the degree to which that if you test them on something by like asking

them questions about it they get those questions right that's how I would measure understanding and I think by that measure they have some understanding often imperfect but then if you ask the average human questions

they'll often get some of them wrong so I would still say that there's some understanding there we started our conversation today um talking about your paper about passive strategies and

active strategies and and so on and you shared with us some of the results showing that large language models can

generalize to new um causal context if the training have certain certain properties some time ago uh franois Chet

published uh his set of tasks to measure what he called in calls intelligence um these are tasks called Arc or sets of tasks called Arc and Arc 2 and it seems

that large language models are um well there's a lot of detail to this but in some sense they are not performing very well in in those tasks in in some sense

those tasks what is important about them is that there is a concept and there is a structure and the concept and the structure are orthogonal what kind kind of training or what kind of data would

we need for large language models in order to make them perform better on this type of tasks yeah that's a great question so one thing I want to bring up in this context is there is a nice paper

by Taylor web and Keith Holio um came out maybe 6 months or a year ago looking at the ability of language models to understand analogical reasoning problems like the ones that humans are have been

tested out in the past and Keith Holio is one of the people who did a lot of the earlier work on human analogical re and so an analogical problems are really a an example of this case where you have this structure and you have some

superficial details around it the details are orthogonal to the structure and so you can test the same underlying abstract structure in different situations so they tested these language

models on some problems that are analogy problems with letter strings and things like that that are somewhat similar to some of the arc problems and they did find that the later generation models are doing increasingly well on these

problems even in Fairly comp often nearly as well as are um so I do think that to some ex you can get some performance on these problems out of

these systems at the same time I think that you know one of the things I think about a lot is in human in humans formal reasoning is really an ability that we

really have to train in depth right like you learn to be a formal Reasoner through years of education maybe like 20 years of education to become like a researcher in mathematics or logic or

something like that right and so I think that the we perhaps underestimate the extent to which a system needs really rigorous and perhaps interactive

training to efficiently learn these kind of formal reasoning strategies and we maybe overestimate the extent to which humans are just naturally good reasoners and I think part of the reason that we

overestimate this is that the people who run these studies are exceptional and they tend to be highly educated humans right they are the people who have gone through these years of rigorous training

in order to become a more formal Reasoner I think it we sort of suffer from this expert blind spot where it's hard to step back and say well actually how easy is it for the average person to

reason about a logical problem or to do these sort of structure mapping uh reasoning processes and so one slightly different example of this is in a recent paper we looked at how language models

in humans reason about logical problems that have some underlying semantic content and we looked at what we showed is that basically language models and humans both do better when the semantic

content of the problem supports their logical reasoning and they do much worse when the semantic content contradicts that the sort of conclusions that you would draw from logic and so what I think this is pointing to is the fact

that you know like humans language models are imperfectly using logic in order to address a problem but that's maybe kind of a natural thing to do unless you've been very carefully

educated to take a more rigorous strictly formal approach we as humans are also exceptionally good in building defense

mechanisms that allow us to think about as ourselves as better than than we might be in certain uh in certain contexts it's about reasoning as well

I'm pretty convinced yeah definitely I mean I think that you know I make all kinds of reasoning mistakes all the time and you know if if someone's skeptical of reasoning saw some of those they would say oh that's a system that really

doesn't understand but I think I like to think at least that I do yeah yeah I think we we we we all like to and I think it's healthy actually to think

this way about ourselves at least to an healthy extent whatever it is Andrew what would be your advice to people who are entering a complex field

like causality or reinforcement learning or machine learning or theoretical physics my perspective on this which comes from perhaps from from physics to some extent is that the best way to

learn about something is to play around with it right so if you're learning about a mathematical concept or if you're learning about a causality concept or a reinforcement learning concept the best way to learn is to like

code That Thing Up try it out like change some parameters see how things vary see you know if you change from more observational data to more Interventional data what does the system learn or fail to learn doing those kind

of experiments yourself and playing around with it I think it's the best way to build the intuitions that I think are kind of the foundation of our understanding what's your message to the coal python code

Community I guess one of the things that I take away from our paper that I think is kind of an important point is that you know if you have a system that

treats different levels of abstraction for example the raw data themselves but also the causal abstractions about them and even counterfactual questions like what would have happened if we'd done

something else in sort of a homogeneous way you actually enable kinds of reasoning that might not so obvious like being able to learn about causal strategies for example and so I think

that my message would be you know think about how you can build more holistic systems that are capable of bridging understanding across multiple levels and I think I'd be really excited to see

where the Cal Community takes that in your work uh you extensively refer to reinforcement learning on different levels also in in

this paper for many people who are maybe less aware about your work but we interested in reinforcement learning in the context of causality they might be

familiar with the works of Elias SP andb uh what would you say are the main similarities and differences between your work and in Elias's work yeah that's a great question so I'm not

familiar with all of Elias's work but he does have some papers that are quite closely related to the kinds of work that we did in this paper looking at what you can learn from sort of passively imitating a system I would say

the key difference between the work that I'm familiar with from him and our work is that we considered sort of fundamentally a metal learning regime where you don't have a system that's learning about a single problem but you

have a system that's learning how to adapt in new situations to new problems and that's sort of the general distinction between metal Learning Systems they're learning to adapt to learn and new learning how to learn

that's the metal learning part whereas a basic Learning System is just learning about a single structure or a single problem so in L's work there looking at what you can learn by imitating in a

single case and of course there you're really Bound by the fact that this dat the data are observational the key point we make in our paper is that actually if

you're learning about how to learn how to learn by intervening that is something you can learn just by passively imitating what someone else has done and you can actually deploy

that to acquire like more General causal understanding in new situations so I think that is the fundamentally important distinction between those works when we're speaking uh a second

ago you mentioned the systems that maybe could learn in this soft way and then just

use logical symbolic systems as a tool that is uh that is being used in cases where where it can be

useful Antonio damasio in one of his books uh was describing a case of one of his patients he he nicknamed him him

Elliot so Anton demasu is is a neuro sergeon neuroscientist and his patient uh has a particular brain injury that

disconnected lower parts of his BL brain with his prefrontal Contex cortex and this patient was doing

exceptionally well in logical reasoning so in like task like int Ence uh intelligence tests and mathematical reasoning tests he was everywhere there

he was above the average but when it comes to his personal life he was suffering uh he was suffering greatly he he lost his family

one day he went on the street he met a bunch of people somewhere on the on the car parking and started talking about business with them and they asked him to

to to give them his money and he just gave gave them like a huge amount of money like most of his money I I don't remember exactly and all those happened

to a person who was exceptionally intelligent in terms of how we me how we tend to measure intelligence perhaps this story tells us

something important about our own rationality what are your thoughts about rationality of artificial systems and what role those symbolic uh capab

abilities would play in those systems when we think about them from the point of view of rationality and decision making I think that's a really interesting question so I mentioned

earlier this paper on how humans and language models show similar patterns of content entangled logical reasoning and one of the arguments we make in that paper is that there's actually a

rational reason to do this if it helps the system to make better decisions across the distribution of situations that it encounters every day so humans tend to make better logical reasoning

inferences in situations that are sort of more familiar to them which are of course naturally the ones that they tend to encounter more often and similarly uh language models perhaps make better inferences in situations that they

intend to counter encounter in their data distribution so I think one way of framing these results is that well the objective of humans and of language

models isn't to be rational reasoners it's to be sort of a adaptive to the situations we encounter and that might actually mean that it's better to be less rational but to do better on the

kinds of situations that you tend to encounter every day and I think that might be one thing that's playing out in these kind of cases you know humans are sort of adapted to be irrational in selective ways that perhaps make us

better able to handle the kinds of social situations and other things that we encounter and maybe if you're too rational in a logical sense that will actually work against you in certain

cases and so uh there's actually a whole literature on sort of like things that are called bounded rationality for example that try to come up with normative theories of why humans are

irrational based on sort of similar arguments that like overall it helps us to be irrational because like for example we have limited resources and so we can't make the fully perfect rational inferences all the time or because it

helps us to be more accurate in the kinds of situations we encounter every day perhaps also our definition of rationality as as related to formal reasoning and maybe

this example shows this the example from the masio is is just not a useful definition in terms of what is really rational in terms of in terms of decisions that we make in real world in

everyday life so I think like a lot of these approaches that try to come up with a more normative Theory draw in sort of more probabilistic accounts of what's rational and sort of how you would be the best in expectation over a probabilistic distribution and that

tends to lead to slightly different patterns of behavior than you would get from just doing sort of strictly formal logic it seems that when we look at complex

systems dynamical dynamical and complex systems when we try to regularize those

systems overly makes them perfect it might cost us very unpleasant surprises and very uh abrupt surprises later on

when when it systems when the system D Dynamics just explodes in a sense at some point one example of this has been given by by by Nicholas talp in his in his

book about anti fragility when you work with learning systems does this intuition about bounding or or trying to

limit systems in a certain way that leads to detrimental consequences down the road is something that is also like reflected in your work that you find

useful in your work yeah definitely so so I think that we've written a fair amount about uh the sort of idea that instead of building constraints into a

system you should try to learn about those constraints from data and we've demonstrated this in various contexts but I think the general intuition is similar to what you're describing it's

that if you have some idea about how you think a system should approach a certain class of problems like you think oh it should really be a formal logical Reasoner and you try to constrain the

reasoning processes to too strongly that actually is going to make the system more fragile because as soon as there's something weird in the world that doesn't quite match your assumptions the

system will totally break down uh and so I think that the approach that I try to take for example is to use things like these explanation prediction objectives that try to encourage the system to

represent the things we think are important but without overly constraining the internal computations of the system and perhaps that's a way that you can in a softer way influence what the system is doing without putting

too hard of constraints on it but of course I think that there's a trade-off like if you know exactly what the problem is that you're trying to tackle you can build in a really strong inductive bias and you can make the

system do better of course but I think a lot of the cases I'm concerned with are things like you know things that work at scale like how do you build a better language model or how do you build a better agent that can interact in a rich

virtual environment and for those kind of situations I don't think we really know what the right solution is and so when we try to impose something like oh I think there should be a formal

hierarchical grammar here we tend to probably get some things wrong about how you actually need to solve that problem in practice therefore we make the system more brittle and so I guess there's this

idea that's gone around for a while now in the machine Learning Community called The Bitter lesson from Rich Sutton which basically says that you know if you try to build in what you think is the right

way to solve a problem it'll work at small scale but as soon as you scale the system it tends to break break down and I think empirically we've encountered that a lot and so I tend to think that

these sort of softer Solutions like giving explanations or changing the data distribution that you're learning from can be more effective and scalable ways to solve the problem at least in certain

situations in both machine learning research and and causality research uh we sometimes have very interesting

papers that show some very interesting phenomena but those phenomena are are only demonstrated empirically in the

context of a specific set of tasks in contrast when we take those systems to the real world the tasks are usually not that well defined or the distribution of

task is is is is completely unknown what are your thoughts about this discrepancy between the lab and the reality yeah that's a great question so I think it's important to do work in controlled settings where we really understand

what's going on and usually that necessitates doing something that's a little bit more toy because otherwise we can't possibly understand it but I think that the real world is a good forcing

function for encouraging you to not overfit because I think a problem that comes up is that people are designing a a new algorithm for example that they think will work better and then they're

designing the test for it simultaneously and so they sort of test the algorithm on the perfect case that it will perfectly work for and of course their algorithm is better for that particular case but then if it's like well I want

it to work across all the language tasks that you could learn from the internet maybe the assumptions that you made there in designing that system won't hold anymore and so I think that working

in the real world like trying to build image processing systems that can handle any natural image they encounter or trying to build language models that can handle sort of any natural language

query from a user while it's a very hard problem that neither of those is even close to fully solved I would say I think it's a really good way to test your ideas and make sure that they're

really solving the problem and not just solving some toy problem that you designed what would be the main lessons that you learned during your studies in Psychology that help you in

your work today I think there's so many things I mean I I think cognitive scientists have spent a lot of time thinking about some of these causality related issues about things like confounding or different ways that you

could explain the same different underlying processes that could explain the same observations that you make about humans in an experiment and so I think from an experimental design perspective and also just thinking about

doing rigorous statistics on the experiments we perform I think cognitive psychology has a lot to teach machine learning where often they don't even do any statistics it's just like this number is bigger than that other one so

the system must be better so yeah I think experimental design and doing rigorous statistical analyses are two of the main things I've learned um I guess

the the other thing might be just sort of thinking at a more abstract level and trying to bridge from sort of the observations we make to more abstract models that could explain them I think

that's something that's really emphasized in computational cognitive science and uh is maybe a valuable skill for thinking about the kinds of systems that we're working with today you mentioned

statistics uh in the context of of machine learning um you said that sometimes we just say hey this is a bigger number right that that's true unfortunately maybe unfortunately I

don't know um what are your thoughts about statistical learning Theory and how it applies to modern machine Learning

Systems yeah great question so I think you know when you make a theory just like when you make a simplified model you build in some assumptions to it and it's important to consider what those assumptions are in order to understand

how well the theory is going to apply to a new situation so statistical learning theory I think has some assumptions that are perhaps too strong in some cases for

modern machine learning systems or maybe another way of saying it is that we don't quite understand how modern machine Learning Systems fit into statistical learning theory appropriately so one example of this is

that there's been a lot of theory work maybe trying to bridge this Gap in understanding arguing that there's some implicit inductive biases in for example the architectures of our models or the learning processes of gradient descent

or how those things interact that causes the models to effectively be not as overparameterized as you would think of them as being at least at early stages in the training process and I think that

those kind of explanations you know point to a Nuance that's maybe not really captured in at least the most naive versions of statistical learning theory where you sort of assume that like the capacity of the system is sort

of uniformly distributed over all the different ways that it could fit the function maybe that is not really true in practice when you train a system by something like gradient descent for example is the future

causal of course yeah I mean I tend to think that the world is causal and so the future must be

from all you learned in your in your journey what do you think is a good life wow that's an interesting question

I think the the life I enjoy is being able to both engage with sort of intellectual issues like ask questions about causality and intelligence and even like dabble in a bit of philosophy

although I'm not an expert in that not an expert in very many things honestly but also to have sort of a fulfilling life outside of research and I do a lot of rock climbing as I think you mentioned in the introduction and I uh

you know have a lot of friends and my partner Who I Really value our time together and so I think it's uh yeah it's good to have a balance of engaging in intellectual topics but also you know

having a more sort of like physical engagement with the world and having social relationships too I think that's a very important part of what it means to be human who would you like to thank I would like to thank all of my

collaborators at Deep Mind I mean there's so many amazing people there people like Stephanie Chan initiative do scoter who I've worked pretty closely on a lot of this work also my uh Felix Hill

who's been a real mentor to me over the years and um my grad school adviser James molland who has been a huge hero of mine and a real inspiration and it's really helped me to think carefully

about science and statistics and everything else and I'd also like to thank my partner Julie kaser who has been a huge support to me and uh just

tremendously helpful before we conclude um what question would you like to ask me I'd like to ask you where you see the

future intersection of causality and machine learning going like what is your vision for how causality research and machine learning research will come together to make the future more

causal that's a great question um I'm recently thinking a lot about um combining uh the the formalisms the causal

formalisms like P formalism with things like uh differential equations and uh chaos in mathematics dynamical systems and this kind of stuff and I'm also

thinking about something I feel very similar to what what you mentioned before so combining this soft learning with more formalized ways of of of

inference of doing inference and um my intuition is that we in everyday life as humans using more of

soft learn uh soft learn skills than than formal reasoning but we probably also have certain constraints imposed on those

soft soft landar in a soft way uh associations or maybe maybe that maybe there's some structure I don't know maybe there are some constraints that

are enforcing structure on our associative learning and and I agree with you with

what you said before that formal reasoning is something that we need to be trained in and I experienced this firsthand because I studied philosophy

so I studied logic and then more logic and even more logic and then I started learning causality already having experience in machine learning and I

found it deeply counterintuitive um to internalize like things like do calculus for instance so I think these tools are

useful they're not necessarily the main ingredient if and I think this is maybe the main point of of of this answer if

we want to build a system that is humanik and this conditional goes down to something that I think is very fundamental and this is a question that

I think we don't ask ourselves as a machine learning community in general uh often enough and it this is a question about

what are we actually trying to do so do we want humanlike systems that will give us a feeling that we're interacting with something that is similar to us or do we

want to build systems that will be superum in certain regards and then in which regards actually we are interested

in building uh in systems like like this so these are the things I'm thinking about and I must say that this is probably a little bit on the

philosophical end this is a little bit further further away from from practice on the Practical uh on the Practical

end I'm thinking a lot about um providing the community of ways to provide the community with more narratives about causality that make kazal more

accessible to more people because there's a certain learning curve to jump into this things definitely yeah and there's also I think a lot of misconception that

comes from many different angles so we have people uh I see sometimes people from the experimental Community saying things like oh we are doing KB testing and is superior to causal inference

because causal inference has to be from observational data which is not true uh we have people in simulation and they digal twins that are saying hey we're doing simulation why we need actually

any causal inference so I think a unified perspective that shows that interventions or AB tests are a a special case of causal inference and so

are simulations because they are basically some kind of oper operationalization over a over structural causal models is something that we that that could be very

beneficial for the community uh to move forward so so this unified perspective I I found it something fascinating and I work every day

on making building narratives that that could be readable to to to more people regarding this yeah I think that's super valuable I mean I think understanding technical topics is always hard and so I

really admire the work you're doing in the podcast and with your book and everything to try to bring these things in an accessible way to a broader Community I think that's that's super important work um and yeah trying to

trying to arrive a more unified perspective of everything is the grand goal of many Sciences right so yeah that would be awesome not an easy one thank

you so much uh thank you Andrew where can people learn more about you and your your and your team's work well you can check out my website if you just Google my name Andrew lypin or my Twitter I

often post about my work on Twitter and my colleagues work but also just other papers that I find exciting other things in the field that I'm excited about so yeah follow me on Twitter that's probably good good Solution that's great

I have one more question I want to ask you before we before we finish what are two books that change your life the first one I would say is

perhaps a more recent one which is becoming Human by Michael tomasella I think it's a really beautiful book for it's sort of a technical book you know he's a researcher who thinks about

humans and What Makes Us special um and does a lot of comparative work with uh other primates for example and it's it's a beautiful book for thinking about how social interactions from a really early

age are a fundamental part of human experience and that book provided a lot of inspiration for some of the ideas we talked about in a paper called symbolic behavior that we had a few years back um thinking about how social interactions

might be really important for building more natural artificial intelligence for a second book oh man there's so many it's hard to pick just one I think one

of my this is a bit of a a trite answer but one of my favorite books ever is uh ulyses by James Joyce and I think that one of the things I really like about

that book is the way that it plays the sort of style of the writing and the content together with the narrative in different sections of the book and I actually think that like this bridging

between different levels of abstraction right what is just the superficial style what is the content what is the overall underlying narrative is actually quite similar to some of the themes I'm interested in in my research like how do

you bridge between different levels of causal reasoning how do you bridge uh in my dissertation I was interested in bridging between sort of like reasoning within a task and reasoning a cross

tasks and things like that so I think this perhaps I'm totally just re retrospectively attributing this to myself but perhaps some of the ways I thought about like literature when

reading ulyses and other books like that uh percolated into the way that I think about more complex formal systems that I reason about in my everyday work thank you so much for the conversation it was

a pleasure Andrew thank you very much