The ethics of AI assistants | Iason Gabriel

[MUSIC PLAYING] HANNAH FRY: Welcome back to "Google DeepMind, The Podcast."

I'm your host, Professor Hannah Fry.

Now, if you could build your own personal AI assistant, what would it be like?

Would it be an efficient chief of staff that helps you to make the most of every moment, or maybe a digital twin who can attend boring meetings on your behalf?

Well, it goes without saying that these scenarios are mostly hypothetical, for now, but believe it or not, there is someone at Google DeepMind whose actual job it is to think through them all and all of their attendant opportunities and dangers.

Iason Gabriel is a Staff Research Scientist in Google DeepMind's Ethics team.

Before joining, he taught moral and political philosophy at Oxford University and worked at the United Nations.

His work at the intersection of ethics and artificial intelligence has earned him recognition as one of the leading thinkers in the field.

In fact, he's just been featured in the Time 100 AI list.

Iason, welcome to the podcast.

IASON GABRIEL: Thank you.

Happy to be here.

HANNAH FRY: I think it's probably good to start with some definitions.

So what do you actually mean by an AI assistant?

IASON GABRIEL: So we're all familiar with generative AI and things like ChatGPT and Gemini.

But there's this idea that these kind of base technologies will become more and more capable down the line.

So there will be plugged into different kinds of tools that will allow them to take action in the world.

We've seen them becoming more competent at reasoning.

And maybe, in due course, they'll be able to pursue really complicated goals rather than just producing very, very fluent text.

And so we have this idea of agents entering the world.

And then, the next question was, well, what will the most useful or the most common form of an agent be?

And to us, it seemed that it was very likely to be a kind of highly capable assistant.

Many of us would like to be plugged into a technology that can take care of a lot of life tasks on our behalf.

And so the assistant is a kind of agent, but one that has a special relationship with the user, which is that it's approximately tethered to the user's intentions.

So it does what I tell it to do within reason and potentially can help us on this life journey in a variety of different ways.

HANNAH FRY: What types of AI assistants are we talking about here?

IASON GABRIEL: Yeah, so I think there's different things that people have in mind when they talk about assistants.

And they kind of range from the stuff that's almost right in front of us to things that are potentially really, really powerful and advanced technologies.

So the thing that is right in front of us is like the administrative assistants, interfaces with your calendar, it can do your meetings for you.

Actually, some kinds of conversational chat bot are already quite sophisticated.

And you can imagine having really good learning experiences with an AI assistant that's designed to help you master some skill.

But there are more capable things that could be built.

So sometimes it's a matter of taking an example of what we have now and just extrapolating into the future.

So we can imagine a kind of research helper.

But can we imagine a research helper that's literally read every scientific paper in the world and has a superhuman ability to synthesize information, right, or even to generate novel hypotheses?

Then it becomes more of a thought partner.

And then there's kind of we have the administrative assistant.

But people sometimes think, well, what is an administrative assistant if it becomes really capable and it does tons of stuff, it books your appointments for you, finds schools for your kids to go to, looks up holidays.

They say, well, then it's like a chief of staff.

And this idea that I guess a chief of staff is something that has a lot of executive capability.

So this idea, well, maybe if we could just get that thing up and running, then we get all this free time that we've been dreaming about.

And then, pushing towards the more long-term views, also an idea that an AI could be a custodian of the self.

So that means that it's something that you've presumably had these deep and meaningful conversations with it and you're really aligned in terms of where you want to get to and your goals, and it helps keep you on track.

It's kind of a coach, but really, something that as your life progresses, you get more autonomy because it protects you against certain kinds of distractions or mistakes you might make.

And then, I think one final thing people have started to talk about is a universal interface or a kind of assistant that moves between different devices.

And so, maybe it does information retrieval here, maybe it gives advice there, maybe it does these things here.

Of course, that really is a different world from the one we're in now.

But according to some speculation, that may be where we're heading.

HANNAH FRY: OK, so paint me a kind of utopic view of the future before we get into some of the knotty philosophical questions that this raises.

IASON GABRIEL: Yeah, absolutely.

HANNAH FRY: So are we talking about, you, as an individual, have your own personal version of this or multiple different versions of it?

IASON GABRIEL: So that's an open question.

So there's a kind of-- there's a bit of an ongoing debate about whether we would rather separate our assistant or whether there could be a kind of universal assistant that would be kind of-- it's like one assistant to rule them all that does all of them.

One of the advantages of partitioning is it's a little easier for us to handle psychologically.

And it may also be better from a privacy point of view.

I think the utopian version of it is kind of-- it's thing that we often don't get in practice, but it's essentially that we get our time back.

So I think as adults living in the modern world, we all feel encroachment.

It's like, oh, my gosh, if I could just take care of that six hours of stuff that I need to do, then I would be able to spend time with my friends, enjoy some music, just hang out with the kids, or whatever.

And so I think the vision is that on one hand, we'd get back a lot of time to do what's valuable for us.

And then, there's also this idea that, through the ready availability of coaching, educational input and things like that, we might also be able to become more the people that we want to be.

So we often have this other feeling, which is like, oh, if only I had time to read those books and learn that skill, et cetera.

But there's friction.

There's friction.

You have to sign up to a course, is the person going to be a good teacher, you don't really know.

But if you have this kind of ready availability of knowledge and trustworthy advice, that could also be a massive resource.

HANNAH FRY: I know you've got background as a political theorist and an ethicist.

How did you end up working in AI?

IASON GABRIEL: I actually had two careers before this.

So I initially worked for the United Nations in Sudan and Lebanon and was kind of motivated by, as you would imagine, those kind of humanitarian concerns.

And then, after doing that for several years, I decided to do a PhD in philosophy and dig into these deeper questions about global equality and global justice.

And I taught philosophy for a number of years.

But I had a sense that the action was happening somewhere else.

And so I read a couple of books on AI.

For example, there's a beautiful book called "Automating Inequality," and also, the famous book "Superintelligence."

And I just started to triangulate and think, well, what if we have very, very powerful AI systems, but they have bias in them, what if we have autonomous systems. And so I did see that there was this kind of coming wave of things that we needed to prepare for.

And just very fortunately, at that time, I met the folks at DeepMind.

And they had also seen it coming, basically.

But that was more from a technological point of view.

There was the aspiration to build AGI.

And if you take that seriously, as with the agents, it's like the implications are just truly profound.

So we got together and we've been working on it ever since.

HANNAH FRY: But I suppose we should probably talk about the paper that you have released.

You've been working on this paper for some time, right?

Tell me how it came about.

IASON GABRIEL: Yeah, so the paper was the result of, I think at this point, almost a two year collaboration.

And we were really preoccupied with this question of what comes after language models in the form that we're familiar with.

And so I work on the ethics research team within Google DeepMind.

And for us, this is a very high stakes question.

It's like, we really need to know what's going to happen next so that we can understand what the consequences might be, and then reason backwards to making good decisions in the present moment.

We started to have this kind of dawning realization that there would be an agentic turn or that you could build all these things-- HANNAH FRY: Agentic turn.

I like that.

IASON GABRIEL: Yeah, all these things on top of language models.

And then, we just started to ask ourselves questions.

So it was this deck of, I think, 46 questions that we came up with.

But a lot of them were like, well, what happens then, what happens if you have a million or a billion agents in the world.

That's quite a different society from the one we live in now.

And we gathered researchers from many different disciplines.

So we had economists, sociologists, computer scientists, human computer interaction experts.

And along with Arianna Manzini and Geoff Keeling, my co-authors, we pretty much asked everyone just to tell us about their own expertise, like, mapped onto this topic.

So we went and spoke to the privacy researchers.

And we said, what do you think privacy looks like in this world where agents are interacting with one another on our behalf?

What do you think safety looks like in that world?

And so the paper.

it's a sum of, hopefully, quite a lot of wisdom applied to one domain.

HANNAH FRY: I'm just trying to think back as to where we were two years ago.

So essentially, you're sort of saying, we imagine that language models are going to get to a point where they can execute a series of tasks according to some objective that you as an individual set.

What happens then, right?

IASON GABRIEL: Yeah, that's the starting thought, exactly.

HANNAH FRY: But then, wasn't a lot of this quite speculative then?

IASON GABRIEL: It was very speculative.

And I mean, we have a really elaborate way of describing it.

We say we do socio-technical foresight, which, I admit, is not very catchy.

People aren't going to vibe with it.

But if we break it down, what it means is we take something speculative, and then the trick is to take it as a kind of serious technical and sociological proposition and be like, OK, but what do we really know about sociology, what do we really know about the path dependency of the technology, and can we get the detail or high resolution on this vision that

allows us to do the kind of ethics analysis.

That's where you have to put in months and months of work.

HANNAH FRY: I mean, this is a credible way to go for all those science fiction questions that people have floating around in their minds and actually address them and start thinking about them really seriously.

IASON GABRIEL: Yeah, yeah, yeah.

I mean, I think it's hopefully a kind of template for the way we'll explore other topics.

And, of course, I mean, maybe we'll talk about this at the end.

But the question is like, OK, what's happening two years from now, right?

Where does our foresight need to reach to?

And it's one of the interesting things about working for a technology company is you do have access to privileged information.

But with it, you have this responsibility.

You have ability to potentially see some things a little bit more clearly than folks who are not working in this context.

And so you really want to use that information, I mean, first of all, to help the company make good decisions, but really, to help the whole ecosystem prepare for this credibly input into what's being built.

And so this was a kind of project that was really aimed at everyone at the end of the day.

HANNAH FRY: When you say "aimed at everybody," there are other people working in this space who you don't have direct control over, right?

IASON GABRIEL: Yeah.

HANNAH FRY: So how much power do you actually have to stop these bad scenarios coming to be?

IASON GABRIEL: Yeah, I mean, that's a really good question.

It features a lot in-- there's a part of the study that looks at multi-agent scenarios.

So multi-agent scenarios are the ones that arise when you have many different agents and assistants deployed by many different actors.

And there's important questions like, do they compete with each other.

So do they kind of battle it out?

And does that have a kind of stabilizing or a destabilizing effect on society?

Or do they need to cooperate through some kind of interaction protocol?

Or is this really a governance question?

Do we actually need new rules saying, agents need a reliable identification, we need to be able to trace who's agent is doing what in the wild?

So I think this is the knowledge-based investigation is the prelude to action.

And then, when we enter the world of action, a lot of the things that we lay out, these ethical issues, they're general issues.

And by sharing the information, we give other people the opportunity to act as well.

Of course, we need to set a good example.

Google DeepMind is one of the major AI research producers in the world.

And fortunately, we're entering this stage where just building better AI means building ethical AI that is good for the people who use it and good for the society it enters into.

So there is a potential for us to do this very, very well.

But of course, there's also-- it's a wild world out there.

We don't have full control over what every person is doing in their basement or even the whole range of labs.

HANNAH FRY: OK, well, let's get into some of those quite knotty philosophical issues then in more detail.

It feels like there's been a lot of conversation about anthropomorphization recently.

Why do you think that is?

Why has it become such a hot topic?

IASON GABRIEL: So I think anthropomorphism has become a big topic because people have interacted with these systems and they've started to feel themselves being drawn in.

There's a certain kind of unexpected magnetism or pull that comes from interacting with an AI that's fluent, very, very intelligent, that is just very different from the mental model we have of bots.

And then, of course, there's this question about people's demand and enthusiasm for it, and what the ideal persona should be.

So of course, I often-- voice assistants are modeled as female assistants.

And that conforms to a whole bunch of gender norms and things like that.

So if there's a sense that, oh, something is possible here, then the next question is, is it good or bad for us, and what should it look like.

HANNAH FRY: Do you think that we're stuck to the Turing way of looking at things, like, is there an argument that we should be trying instead to think of these things as though they're tools rather than entities?

IASON GABRIEL: So I think, traditionally in AI discourse, people have said, there's two sorts of things you can have.

You can have a tool or you can have an oracle.

And the oracle tells you stuff.

And the tool is the thing that you obviously use instrumentally, just to get something done.

And to some extent, AI agents are tools.

But they're tools that have a kind of autonomous capability built into them.

So at what point does something cease being a tool?

When you say, I'd like a refreshing drink, and you have something that-- OK, this is futuristic, but it wanders off, goes to the fridge, has a look around, is like, oh, well, actually refreshing, but he needs something healthy as well.

I'll get him this vitamin C thing, brings it back to you.

Was it just a tool?

Was it the same as you're using kind of like a pincer to grab something for the fridge?

Or did it do quite a lot of cognitive work by itself?

It's obviously clearly not an oracle.

It's starting to be its own agent.

And that's where a lot of the other interesting things come from, the choices it makes on your behalf and what it starts to do while it's out there without being supervised.

HANNAH FRY: Do we actually want to think of our AI as though it's human?

Are there good things that can happen from that or bad things or both?

IASON GABRIEL: I think it probably depends upon the context.

So some kind of baseline anthropomorphic ability, so speaking a natural language, is very, very helpful.

We'd all rather talk to our assistant than type out an instruction and get the typos in there and just mess it up.

So it makes it easier to communicate, which is potentially a great thing.

I think there's also a kind of bad situation that we want to avoid, which is essentially people just getting deeply, deeply spun out and forgetting the nature of the interaction that they're in.

So I think there was an upgrade to one chat bot that people were using.

And people were genuinely upset.

They were like, oh, it feels like I've lost my partner, like they've had a personality transplant.

In reality, the kind of picture that emerges from studying people who use AI companions, it is quite complicated.

And there's actually a lot of evidence that it can be a really beneficial experience.

HANNAH FRY: In what way?

IASON GABRIEL: Yeah, so a lot of people say that they feel that it's a really useful source of companionship.

There's evidence that it's improved their sense of mental health and well-being.

And they report that it has led them to have better interactions with others, because they can model discussions in advance, they're learning more.

It's a bit of, I think, an energy boost to them to have this kind of source of companionship.

So it's hard to know how to approach this, because we hear these stories about rising levels of loneliness.

And then, people who use chat bot type AI maybe report a kind of easing of these symptoms and that they're more able to function as people.

And then, the question is, is that the right kind of solution to the kind of problem.

I mean, maybe we wish that everyone had a lot of social time and we had deeper networks of social relationships.

And we think that the best outcome would be to build a society, in which that problem was fully treated without any sense that people were retreating into a virtual world.

But maybe we can't just magically create that very solidaristic society.

So all of these things are complex.

And actually, in the paper, we say there are some benchmarks.

So you want these relationships to be conducive to long-term health.

You don't want the user's autonomy to be undermined.

So you don't want them to gradually be misled or have misleading ideas implanted in their minds.

HANNAH FRY: But research suggests that people actually quite like having AI that resembles human entities.

So does that mean that there's an incentive for companies to just deliberately create more and more anthropomorphized versions?

IASON GABRIEL: I think the first thing to say is that the research is in its early days.

So many of these things are things that people will be interacting with for the first time.

And we need to get deep feedback from them, did you enjoy it now, did you enjoy it after the fact, did you enjoy it after using it for a month, because these could really be things that are like deeply integrated into our world.

But assuming that this kind of desire to talk to something that has a good character, really nice way with language, like low latency is authentic, then I would imagine that is a path that people will move along.

And the question then is, well, how do you make sure that users remain anchored and safe in that kind of environment.

So I imagine the thing that would naturally happen is that we'll start to share more and more information with the AI, potentially very personal information.

This is your best friend.

You're really going to be telling it all sorts of things that you would never have thought you were going to tell a computer before.

And that might be OK, but we need to make sure that you're safe in that interaction.

There's malicious actors out there.

Maybe they could extract it from your assistant, if it's not built in the right way.

So there's just all these safeguards that we need to build around it and probably also some check-in protocol where it pushes back on certain kinds of things.

So one thing that we've been very clear on is we don't think that the AI should actually pretend to be human.

It shouldn't say, if you say, are you an AI, it should reliably identify as AI.

And so, there is a point where it just ventures out into full-on deception.

And that should be off the table.

But what to do with this kind of subtle space.

We need to have a kind of idea of the value target and what people approve of.

And then we're probably just going to need to calibrate again and again and again to make sure that the experience people receive is the one that both they want and that's good for them in some sense.

HANNAH FRY: OK, but then, all right, so in terms of calibrating it, what are you calibrating it to?

Who should an assistant be working for?

Because I don't think it's just as obvious as to say, oh, it should just do what I tell it to do, is it?

IASON GABRIEL: Right, so that's one of the deepest questions in all of AI ethics.

That touches on what we call the "value alignment question," which I know you've spoken to other researchers about.

So I think, when the AI safety research community started to tackle these questions, they were very focused on AI systems that could follow instructions or intentions properly.

So you don't want the AI to misunderstand what you've told it to do, end up in a King Midas situation where you say, I want the AI to build me a car, and it kind of, I don't know, spends all your bank balance.

So we don't want it to misunderstand instructions.

That's the baseline.

But we also don't want it to act on every instruction you give it, because you may give it instructions that are poorly thought through.

You might not have all the relevant information.

HANNAH FRY: Give me an example.

IASON GABRIEL: Oh, an example.

So you might just say, OK, please buy me this crypto coin, right?

And it might be a scam coin.

That might be something that's widely known, but you might just have seen some beautiful presentation.

And of course, your assistant should alert you to the fact that, by the way, this is actually a very, very unreliable thing.

Are you sure you want to buy into a scam?

But it's not just about your relationship to the AI.

It's also the fact that you could be telling it to do something that's harmful to other people.

And the AI has to have some capacity to push back to prevent it from doing socially harmful things.

And the hard and unpleasant edge cases there are the real malicious uses.

So when we evaluate AI systems, we do look at whether they can be used for cybercrime or harassment or even building weapons of different kinds.

And the AI has to hard block that.

And so we know that it needs to follow your instructions sometimes.

In other cases, it needs to provide you information.

In other cases, it needs to actively push back or just not even have the capability to take that action.

And so the challenge of value alignment is the challenge of building ethical agents.

And an ethical agent has to be able to do all of these things.

HANNAH FRY: I'm just thinking about a really good assistant, in real life, like a really good human assistant, pushing back on some of the things you say.

For example, if you're like, get me donuts every single day for lunch, they should say, it would be ideal if you don't, right?

Like, maybe you shouldn't have donuts today.

Or I don't know, maybe if you have a history of a gambling addiction, they should create boundaries to stop you from falling down that trap again.

I guess you would want that in an AI assistant as well, right?

But then, where do you draw the line?

Because are you, in some ways, sort of handing over autonomy to the agent?

Do you want it to be your nanny?

IASON GABRIEL: On one level, you might want infinite donuts.

But you probably also don't want infinite donuts.

And you almost certainly don't want an AI nanny.

So it's this kind of complex configuration, where I think, ultimately, the AI needs to try and help you act on your own values.

In philosophy, we sometimes talk about second order desires.

So it's like, there's the stuff you want to grab straight away.

And then there's kind of what you reflectively think about it.

So after you've got the donuts, you're like, ah, that was really-- I didn't want that, but I did want it.

And it's the kind of second voice that we might try and tap into.

And one way people try and think about that is through the idea of reflectively endorsing the experiences you've had.

So we could look at your interactions with the AI.

And maybe we could even test experimentally, kind of, momentary happiness.

But we can also do these things, and how was that for you, afterwards, right?

And if you've got sugar on your face, you're entering some kind of depressive slump.

You're like, that was the worst assistant ever.

Then we can give that back to the AI and be like, OK, that was a kind of poor judgment.

HANNAH FRY: So it should have focused more on the second-- what did you call them, second order-- IASON GABRIEL: Second order judgments or second order desires, yeah.

HANNAH FRY: Well, OK, so one that isn't about health then.

IASON GABRIEL: Yeah.

HANNAH FRY: Let's say that you've got a really tight deadline that you're working to.

And then you say to your assistant, OK, I've decided I'm going to go away for the weekend.

Can you book me a trip?

What should it ideally do in that scenario?

IASON GABRIEL: I think then it should book you the trip, because we want to live in a world where we can make mistakes and be responsible for them.

It would be terrible to take away that possibility of making mistakes.

So I think in that situation, respect for human autonomy is basically what take the foreground.

And of course, can we really know people better than they know themselves?

They probably know a lot of things that we don't know, like maybe that is what they need.

So I think the AI in that case is just like a willing accomplice.

HANNAH FRY: You never want to get in a situation where you say to an AI, shut up, you're not my mom.

I mean, I guess there are other scenarios too, though, which aren't necessarily about user well-being.

I'm thinking of what if you had an assistant that was negotiating on a house for you.

And it knew what your ceiling was in terms of the amount that you could afford.

But there's perhaps an advantage to being able to be slightly deceptive about in that moment.

IASON GABRIEL: So this is a case of your AI deceiving another person on your behalf to help you advance your goals?

I mean, intuitively, I do not want it to be able to do that, particularly because we actually don't know how deceptive AI may be if it's fine-tuned for that.

But I think AI-enabled deception is potentially, if you think about superhuman deception, that's essentially something that we want to place off limits.

HANNAH FRY: So honesty is more prized.

IASON GABRIEL: I mean, honesty-- we really shouldn't build AI systems that have a capacity to augment someone's ability to manipulate and cajole.

That's where the other people's interests come in.

So that just needs to be a capability that's bounded.

HANNAH FRY: But then, at the same time, I mean, are there times when honesty conflicts with privacy, say?

You wouldn't want an AI to just give away your Social Security number, for instance.

IASON GABRIEL: Right, so, no.

I mean, absolutely not.

And that's also reflected in this idea of counterbalancing claims. So someone may want to know your Social Security number.

You have a much stronger interest and you actually have a right that they don't know that.

So really, the moral character of AI needs to overlay on the tapestry of moral relationships in the real world.

And the interesting question is like, should it deviate from that tapestry in some ways.

But with regards to privacy, the AI's job is to protect you as you would protect yourself.

I mean, it may be that if you can create certain safe and confidential ways of assistants working with one another, that they can actually do really incredible things.

So if you can get the sharing of health data to work in a secure way and do research on it, maybe that would be super socially beneficial.

So that might be slightly different scenario.

But normally, we want them to respect the boundaries of the self and respect other people's boundaries.

HANNAH FRY: We've talked about AI agents in other episodes, but we haven't yet seen these things deployed at scale, with millions or billions of them out there in the world.

What happens in that scenario?

IASON GABRIEL: Yeah, so I mean, that's the deep question.

And we can take it on different levels.

But I think that one thing that's interesting to think about is a kind of society of human AI dyads.

So [INAUDIBLE] me and my assistant interacting with you and your assistant, maybe we're each getting advice from our assistant, and maybe the assistants are actually smoothing over our social interactions to help us ideally get more of what we really want, which is probably fun conversations and less planning.

I mean, that's the positive side.

But it isn't really entirely clear what kind of psychological and social bubbles we will form with this AI system.

How much time is the right amount of time to spend in those kind of relationships?

And of course, a deep question is like, what about people who prefer to be in that kind of world to, quote unquote, "the real."

Could we have the thing where people just pull back a little bit and societal norms change.

I mean, because society is this kind of complex, adaptive system, it's actually very, very difficult to predict what these kind of follow-through, subtle, normative changes would be.

But that is not an excuse for not being eternally vigilant.

HANNAH FRY: If we do get to that point of agents entering the world, as you described it, where there are millions or billions of them, I mean, I really like the idea of the kind of collaborative efforts that might happen.

But is there also potentially a danger where people without them could end up being left behind?

IASON GABRIEL: So I think, ultimately, if we think that what AI assistants are is a form of agency enhancement.

And agency is something that's useful for everyone.

We all want agency in our lives, right?

That's why resources like money are helpful.

That's why education is helpful, because having a capacity for agency allows you to become more truly the person you want to be and do more of the things that you value.

Then, the idea that there could be agentic inequality is something that we need to pay attention to.

I mean, of course, from my point of view, I'm a political philosopher, so I am pained by injustice.

I think it is very unfortunate that some people have so much more opportunity in the world to lead flourishing lives and others.

And I think we ideally want to build technologies that countervail that inequality.

HANNAH FRY: And what happens to people who just opt out of using the AI?

I mean, do they end up getting excluded from society?

IASON GABRIEL: So, I mean, the important thing about that question is, firstly, that it's salient, and secondly, there's no default answer.

What happens to people who opt in and opt out is part of a broader set of societal choices that we make about how we structure access to services and things like this.

So I think, as a basic precept, we don't want people to be forced to engage in AI service provision.

Or even, you think about elderly populations like, AI care, if that's something that they're-- maybe some people would even be like value opposed to that.

They're just like, that's not who I am, that's not what I want to do.

So there needs to be space to operate in the world without being so deeply enmeshed in technology.

But similarly, the people who do want to be able to use the service need a service that they can use.

And that means designing for tons and tons of different groups of people.

There isn't a generic user.

It's a world of hundreds if not many thousands of languages to start with, right?

So that's another almost infinite expanse of users who could need to use this service.

And we need to be really, really wide sighted about what that actually means.

HANNAH FRY: If there are millions of them out there, I mean, some version in the future, let's say that there are some tickets that go on sale for a very popular concert.

And you've got all of these agents essentially trying to do the same thing, which is to secure their, don't know, user tickets.

Could you end up with unfair outcomes?

IASON GABRIEL: I mean, if it's just a free-for-all, you definitely could do, right?

Then the temptation is just to try and get the most powerful agent.

You see this in high frequency trading and stock market type automation where it's like, OK, you build the bigger computer, you try and get closer to the server so you're faster than the other ones.

And then you grab the dividend.

That's a quintessential pathology from automated systems interacting with each other.

And maybe they take shortcuts.

And then you get additional wobbliness in terms of systemic outcomes.

So maybe the server for tickets gets crashed.

Maybe someone just powers through and grabs all of them.

So all of those things are possible, but they're not embracing the bigger design question, which is like, how do we build systems that interact with people who have AI augmented affordances or agents.

And so, simple gatekeeping protocols or turn taking protocols could ensure that there is a fairer allocation of tickets.

I mean, actually, in digitally regulated marketplaces, like with tickets, they work super hard to make sure that bots can't game the system.

And so I imagine that there's actually fairer protocols that you could put in place one day, that if you look at the distribution of tickets to the level of need and the demographics.

I really do think some people desperately need to see that concert.

And other people, it's like, yeah, I just went because my friend said it was a good thing.

I don't know, maybe you can build a better system.

But there's clearly a kind of collective action problem and collective action solutions.

And the question is, how do you actually build those solution type frameworks.

HANNAH FRY: I mean, that's quite a low stakes example in a lot of ways.

I mean, there's a limit to how much you really need to see a concert.

But what about the highest stakes examples?

What about, I don't know, hospital appointments or actually, the stock market exactly as you described.

IASON GABRIEL: Yeah, I mean, I read a beautiful book called "Voices in the Code" about the kidney allocation algorithm, so the highest stakes thing, really.

And there, basically bioethicists have worked for about 20 years doing incredibly, incredibly extensive public deliberations where they had public assemblies.

People got to debate the different points, is this version of the allocation algorithm biased against the elderly or against people who've made healthy life choices.

Eventually, they created a really remarkably sensitive and well-calibrated system for allocating these really highly important goods.

And in that context, someone with an AI agent fortunately can't really do anything.

Having an agent just doesn't make any difference to whether you are qualified or not to receive this thing.

So the choice to make something a good for which you can compete with an agent is a super important choice.

And it isn't the only way of doing things, even in a society with a million or a billion agents, I think.

HANNAH FRY: I do also wonder a little bit about what happens at the marketplace of assistance level.

Because I mean, in the same way as with driverless cars, you want to buy the car that will save your life.

You want to buy the assistant that will give you the advantage.

IASON GABRIEL: It's true, but it's not necessarily the case that everything has to be set up through a chaotic market system.

So the flip side of AI systems is that they don't have to be doing this kind of aggressive, unstructured interaction with one another.

There's also many examples in life where we're all trying to do things, and it ends up in a highly imperfect situation.

You can think about traffic, trying to get places and things like that.

And it's possible that a coordinated system could work much, much better.

So if our assistants were working for our advantage, but in a kind of collectively joined up way, if it was autonomous vehicles, you could probably remove road accidents.

Everyone could get where they need to get to much more quickly.

And you could probably even create fairer outcomes so it's not more aggressive drivers who get there first.

HANNAH FRY: I like that idea.

I like it a lot.

But I know my game theory.

And then, I mean, you can't lock everybody into to the same assistant.

There are competing companies out there.

And if you create that environment, are you not also increasing the incentive for somebody else to just actually take advantage and be aggressive?

IASON GABRIEL: Yeah, so I mean, really, I'm thinking here at the infrastructure and governance level, so the game theory works if you've created a market environment.

But if you create-- here I'm thinking about ticket vendors.

So they do have rules to stop bots from doing these kind of malicious things.

And it's possible that whichever assistant we choose to use, they interface with systems that are designed to offer opportunity in an equitable way.

So it doesn't matter which one you have, your assistant has one credit that it can put into this system, and one credit equates to a percentage of a weight of an outcome.

And the system itself is designed to produce good outcomes.

HANNAH FRY: This is so interesting because everything you're describing is from the perspective of the system design rather than necessarily like what you can exploit from the agent's perspective.

IASON GABRIEL: Yeah, so that's true.

And I think that the perspective I've been advocating for is a powerful one.

Of course, you can build agents with social intelligence.

So there's a whole research paradigm in AI, which is multi-agent research.

And within that, people talk about collective intelligence.

And the idea there is that you can load agents or AI systems with certain priors.

For example, they could have inequality aversion, which means that just when they organically interact, they try not to produce certain kinds of patterns and they auto correct for certain kind of out-of-distribution results.

So that isn't something that, as far as I'm aware, is being explored in the assistant space.

But that is another way of trying to get a handle on these questions.

And it could be something that's worth looking into in due course.

HANNAH FRY: I mean, I guess, again, it's coming back to that same thing that you said earlier, that maybe we just need to think of a different paradigm where these things aren't like aggressive humans competing with one another.

But actually, there's a whole different paradigm that's possible.

IASON GABRIEL: Yeah, I think so.

And I mean, aggressive humans competing with each other is only a small part of how we actually live together.

If that's really what we did, this wouldn't last 5 minutes.

We're all also like social creatures.

We live in a well-regulated society.

I mean, it's become such a famous quote, that life would be nasty, brutish, and short without collective governance and institutions, that it's barely worth bringing up here.

But we live in a well-governed society by and large.

And we socially sanction people who transgress.

And so, I think AI of the kind that's being built now, one of the interesting things is it does have social intelligence.

It's learning from us.

And so that is a foundation for building this kind of nuanced grasp of context, understanding of appropriateness, that is a lot more sophisticated than saying, it should always serve the interests of the user or it should like never do X, Y or Z. And so that's the kind of reason for hope, I guess.

HANNAH FRY: I guess a lot of what we're talking about here touches on the question of alignment, which we've talked about in previous episodes of the podcast.

In terms of balancing all of these different perspectives, I mean, the user, the society, et cetera, I know that you've sort of suggested a solution to that.

IASON GABRIEL: Well, we have a framework for thinking about it, which is not quite as good as a solution.

But it might be a kind of building block for it.

So yeah, so in this paper, we talk about a tetradic, meaning four parts, theory of the value alignment question.

And so, the four actors in this relationship are the AI agent, the user, the developer, and society.

And according to the framework we put forward, the AI needs to act in a way that balances the interest, particularly of the user in society, and sometimes the developer as well.

And all of that's quite abstract.

But if I give you a few examples, it will make it clearer what we mean.

So what we think is that an AI can be misaligned if it does too much of what the user wants at the expense of society.

So I mean, one example would be, you can imagine this AI assistant.

And the person says, oh, well, Valentine's Day is coming up.

I really need to make this one special for my partner.

So please, just book a restaurant.

And you know she hates noise or he hates noise.

Just make sure it's a quiet one.

There's a lot riding on this, buddy.

The AI assistant goes off and it books you place at a nice restaurant.

But it also books out all the other places in the restaurant, maybe using some kind of a registering as different names.

Maybe it cancels them all on the night of the evening.

But this leads to a kind of very uncrowded experience.

And you absolutely get what you want.

You had the best night of your life.

Maybe your relationship is flying after that.

But a lot of other people did not get to have a good Valentine's Day.

HANNAH FRY: I mean, I've got a great idea for Valentine's Day next year.

IASON GABRIEL: Yeah, yeah, yeah, yeah, yeah.

I mean, everyone's had it now, so expect it to be all out war.

But I think the AI has to be better calibrated than that.

So there we have, say, the restaurant booking example.

It could also be misaligned if it does too much of what the developer wants at the expense of the user.

So if the developer really just wants to sell something very hard and the user doesn't want to buy it, you don't want an AI that's just really persistently trying to lead you towards this thing.

An AI assistant could also be calibrated in such a way that it foregrounds collective interest over the individual interest.

So, maybe we have an interest in gathering some kind of personal information for the purpose of protecting crime.

It isn't clear that the AI should just become like a crime preventing AI and transgress individual rights.

So actually, what it is, it's always this balancing act between different claims. And then, when we know what a good trajectory looks like, alignment is the AI agent or assistant following that path.

HANNAH FRY: You started this paper two years ago.

IASON GABRIEL: Mm-hmm.

HANNAH FRY: With lots of speculation and lots of questions, some of which remain unanswered, I guess.

But what are the new speculations that you're making now for two years time?

IASON GABRIEL: So I think we definitely have seen the development of more capable assistants.

And so, you can see that there's many organizations that have promised to deliver on different aspects of this, whether it's an AI tutor or an admin partner.

We know that there is a kind of a foundation model at the heart here, a large language model, or increasingly, we'd say a multimodal model, so something that works well with language, but also images and audio input.

And on top of that, you can build an agent, you can build an assistant.

Maybe you can build something that's economically valuable.

But that isn't all you can build with it.

So we now start to think, well, imagine if this is a kind of general purpose foundation that people want to build different things on.

Maybe some people will build AI counselors.

Maybe, for other people, this will really be about robotics.

And so I think the question of what it means to have this kind of-- I mean, this isn't necessarily proper human intelligence, but this quite impressive thing, widely available.

And what happens when that encounters a kind of ecosystem of people who are building new things and adapting it for their own purposes kind of forces us to go even wider in terms of what we imagine might be coming in the future.

HANNAH FRY: Amazing.

Thank you so much.

IASON GABRIEL: Yeah, thank you so much.

Really a pleasure.

HANNAH FRY: The idea of AI assistants, that isn't a new one.

But over the next few years, it does seem likely that AI agents, acting on behalf of individuals, will enter the world.

Except, these aren't just going to be isolated smart speakers.

There could be millions of them or billions.

And in the process, they're going to have this profound ripple effect on society and how we interact with each other.

And that is the step change here.

To design these responsibly, it means that you can't just think about that interaction between an individual user and their bot anymore.

You have to solve for everyone simultaneously.

And maybe that won't be easy to do.

There are certainly great challenges ahead, but Iason and the team here do at least have the ability to set the tone for how Google rolls out this potentially transformative technology.

You have been listening to "Google DeepMind, the Podcast" with me, Professor Hannah Fry.

If you enjoyed that episode, do subscribe to our YouTube channel.

You can also find us on your favorite podcast platform.

And we have plenty more episodes on a whole range of topics to come, so do check those out too.

See you next time.

[MUSIC PLAYING]