Intentional Tech: Designing AI for Human Flourishing | Alex Komoroske

We have this new technology LLMs that I think are as transformative as the printing press, electricity in the internet and we have a choice. We can go down the path that we've been going down which is engagement maximizing

hyperagregation you know going after what you want not what you want to want as a user or being aligned with our intentions and that could lead to a new era of human flourishing. I think I intend to spend quality time with my

family. I intend to experience new

family. I intend to experience new things. the right product fabric could

things. the right product fabric could enable that. But that has to be fully

enable that. But that has to be fully aligned with you. Imagine if that context, all of that rich substrate of meaning that you've distilled and written down, if it's not working for you, that's terrifying. I would run onto

the stage where Steve Jobs shows off the iPhone with a poster that says, "This will become the most important computing device on earth. It is insane to allow a single company to decide what things you may run in it." Right? I know it feels

like in this industry that we're like halfway through the LLM era. We were in the very first inning. We're like

rubbing sticks together. We still think that chatbots are the main thing. Will

chatbots be important? Yes. But chatbots

to me feel like a feature, not a paradigm.

This podcast is supported by Google. Hey

everyone, David here, one of the product leads for Google Gemini. If you dream it and describe it, V3 and Gemini can help you bring it to life as a video. Now

with incredible sound effects, background noise, and even dialogue. Try

it with a Google AI pro plan or get the highest access with the Ultra plan. Sign

up at gemini.google to get started and show us what you create.

Alex, welcome to the show.

Thanks for having me. Uh so for people who don't know you, you are the co-founder and CEO of Common Tools, which you'll describe for us. Um you are also previously the head of uh corporate strategy at Stripe and director of

product management at Google. And I met you I think like 6 months ago and the thing that really stood out in our conversation aside from just like you have so many interesting ideas is you're

this really interesting um systems builder. think a lot about um personal

builder. think a lot about um personal systems, work systems, things that help help you get more done um or think differently about the world. Um and I think people like you who have that kind

of like obsessive mind with systems is something I recognize in myself and also I think are people who are have a lot of interesting things to say about just new

AI ways. Uh I think it's sort of like

AI ways. Uh I think it's sort of like steroids or catnip or whatever whatever you want to say. Some kind of tolerant for sure. Yeah.

for sure. Yeah.

uh for for people for people like us.

I'm excited to chat with you and excited to hear um both what you're thinking about now, what you're building, and um what kinds of systems you're playing around with personally.

Yeah. Cool. Um so why don't we start with common tools because I think there's something interesting about um your like what you're building because I think it speaks to the perspective you

have probably about this larger AI wave.

Tell us about it. Yeah, the way I think about it and it it's hard to describe because we're trying to build a new kind of thing that's only possible in the world of AI. So when I describe it, people say, "Oh, you're being koi." It's

like, "No, I'm telling you what I'm we're building. It's just it's a new

we're building. It's just it's a new kind of thing. It's hard to grab on to."

I think of it like a a co-active fabric for your digital life. Uh and by co-active that's an old word that I hadn't heard before but I think is a perfect fit for it of um you are active

in the system and so is um this emergent intelligent process your private intelligence powered by LLMs and those can be co-active on the same substrate and adding detail or adding the little

bits of software or connecting up things and I think that that really unpacks or unlocks the power of LMS. That's I I love that word and it's interesting because I think we're finding something

similar for some of the products we're in internally is like right now the paradigm is um you send something to chach& and it sends something back um maybe it now it sends something back

after 10 minutes which is a new thing but it's not like you're not working in parallel it's not like working in the background and then being like I had this idea and we're starting to like as we build our own sort of agentic systems

inside of every for doing email or for writing or whatever We're finding that having something that's um working while you work with you and and and being a

little bit more proactive is actually a really interesting u next paradigm.

Yeah. to me. I mean, if you if you say, "Okay, we have LLMs. What's the first product?" In 10 seconds, you go

product?" In 10 seconds, you go chatbots. It's the most obvious

chatbots. It's the most obvious extension. And everybody is focusing

extension. And everybody is focusing only on chatbot. Like, will chatbots be important? Yes. But chatbots to me feel

important? Yes. But chatbots to me feel like a feature, not a paradigm. And they

are um so like they're they're like this co-active surface, but it's just append only. I append a message, it appends a

only. I append a message, it appends a message, I append a message, it appends a message, and literally back and forth.

It doesn't have a way to like have multiple for me in a row. And you know, chat bots are amazing for starting any task because there's no structure. You

just say whatever you're thinking and it helps to still and respond. But for long live tasks, you need structure. You want

there to be something where you can glance over and, you know, a little cubby hole where you put certain information. I find I have, you know, I

information. I find I have, you know, I do dozens and dozens and dozens of chats with chatbt and claw a day and I'm just swimming in all that context and all the all the chats. Uh, and so I'm using it

to do important things and also I'm just completely lost in it because it's only this chatbot kind of pendon kind of paradigm.

Uh, you have this really cool thing um called Bits and Bobs which is this like real time Google doc that you just have been updating for I think like many years at this point uh with things that

you're thinking about um which we'll link to in the show notes and I love I think it's I think it's really cool and really inspiring just sort of the rawness of that. Um, and one of the things that's in that Google doc right

now about what you're thinking about is the this idea of intentional tech, which um, it dubtales really nicely with something that I've been thinking about, which is, um, this idea that AI has this

is this new era of technology where technology can understand our stated preference uh, stated preferences, which is different from the social media era, which which worked totally just on

what you clicked on, whatever. And so

social media tends to make us um tends to serve us things that are like like more outrage driven or more like sexier or whatever. Anything that catches more

or whatever. Anything that catches more eye Yeah. like the car crash effect

eye Yeah. like the car crash effect basically.

Um and um AI because it can talk to us gets a much richer um understanding of who we are and maybe um like I think JGBT is much more helpful and much more

um enlightening for example than like the Facebook algorithm is um just by default. So I'm I think that dovtails a

default. So I'm I think that dovtails a lot with what you've been thinking about with intentional tech. So So tell us tell us what's on your mind about it.

Yeah, intentional tech I think is really critical in this new era. We're we're at a crossroads as as a technology industry. I think we have this new

industry. I think we have this new technology LLM that I think are as transformative as the printing press, electricity and the internet. So it's

this big general purpose unlock for all kinds of stuff that wasn't possible before. And we have a choice. we can go

before. And we have a choice. we can go down the path that we've been going down which is engagement maximizing hyperagregation you know going after what you want not what you want to want as a user um or being aligned with our

intentions and that could lead to a new era of human flourishing I think um this is not the default path by the way we will have to choose it we'll have to work to build that and that's why I think intentional tech is so important because we want technology that aligns

with our intentions not necessarily what I you know what I I want my revealed preference of like looking at the car crashes or whatever but what I intend to do I intend to spend quality time with

my family. I intend to uh experience new

my family. I intend to uh experience new things. Um I I intend to read

things. Um I I intend to read interesting takes that disagree that give me disisconfirming evidence that challenge my worldviews. Those are some of the things I intend. That's what I find meaningful to do. And it's very

easy to fall out of that. But LLM's I think um with the right product fabric could enable that. And another important part of this is it aligns with your intentions as in it is working for me.

It is an extension of me. Um I heard a word last week I love of exocortex. It's

like an ext your cognitive exoskeleton.

But that has to be fully aligned with you. Imagine if that context, all of

you. Imagine if that context, all of that rich substrate of meaning that you've distilled and written down that's all about, you know, the things you care about and all these facts about you. If

it's not working for you, that's terrifying.

Like I saw someone um a few weeks ago at one of the big tech companies said, "We're making our tool uh personal, proactive, powerful." It's like, well,

proactive, powerful." It's like, well, let me stop you right there because the very first word, personal, doesn't actually align because you are a massive corporation that's trying to sell me ads and if you're maintaining a dossier on

me and then that dossier is leading to a powerful and proactive thing, that's terrifying. Whereas, if it's working

terrifying. Whereas, if it's working just for me of the extension of my direct agency, that's really empowering and I think that's one of the reasons intentional tech is so important at this era.

That's really interesting. Um and tell me more about like because you know when I think about this one of the one of the words that comes up you're talking about

alignment um is that uh AI started or this generation of AI started with alignment or alignment with human preferences at at its core um because we were all afraid that AI was going to

kill us. Um and now it's like well it

kill us. Um and now it's like well it might not kill us but maybe it'll just serve us ads that that will make us dumber spend our money in ways that we shouldn't or whatever. So I'm kind of

curious what you think has to happen to make that make us go on this sort of more intentional path for to me it's um alignment I think it's it's funny like we yeah we used it for

the underlying LLM model like I think there's two layers there's the model itself which is if you imagine this as a stateless thing that you send a query to and it gives you back a response it doesn't store any state um then there

you have alignment problems of like what kind of biases are baked into it and and what have you but then there's a layer on top which is your context text and that is the thing that is a malleable bit of information about you that changes it has lots of rich meaning on

it. These are two separate layers I

it. These are two separate layers I think.

And so I am more interested in this layer at the top. And assuming if you have multiple LLMs that you can choose and swap between that don't remember anything about you then it's fine. You

can swap between different ones and that matters less to me. Where context is the place that it's all aggregated uh is I think more important. Well that's an interesting architectural question. Like

right now language models are stateless.

Do you think like in five years for example there's still going to be these sort of stateless intelligence boxes or do you think they're going to be auto updating their weights for example as you talk to them?

I I don't know about auto updating their weights. Um and this might be there's an

weights. Um and this might be there's an architectural breakthrough but this is one of the things that's weird about these models is that they do take a long time to bake. So like they have this weird pace layer down at the bottom that's like months behind on training

data and it just takes time for it to bake. And then you have these like you

bake. And then you have these like you know layers on top that the system prompts now like in chatbt will inject little bits of your context. a weird bit that by the way you cannot inspect which is kind of funky right like if you ask

it hey what do you what context about me did you put in there it's like I can't tell you like that's creepy right I can't do that Dave you know um and uh Simon Willison we were chatting last week and uh you know he's got some prompt that will

extract out this dossier about you it includes things about stuff you said in the past it could include things like you're insecure about your weight you know these things about you and then it also says in the last um

in the last uh you know five weeks 9% of your interactions with the chatbot have been bad as in you've been trying to manipulate it or or whatever like I don't know it's weird seeing this this view into this

for yourself. So I think chatgbt and

for yourself. So I think chatgbt and others are very clearly trying to move merge these together so that you have all your contacts in chat GBT and it's hard to leave that I think gets more

problematic especially if you're going down an engagement maximizing kind of playbook which I mean a bunch of you know the uh the executives from Facebook now run three of the four big chat bots

you know um I think step to meaningful computing has four major components to me one is it's human- centered not corporation centered center around me and my

intentions.

Two um is it is uh private by design.

That means that the data is only for me to see. It's for me to bring where I

to see. It's for me to bring where I want to bring and to choose who gets to see it. Um using tech techniques like

see it. Um using tech techniques like confidential computing to make sure it is entirely uh private. Um it has to be pro-social. It has to be something that

pro-social. It has to be something that helps you live a life integrated with society. not just being this hyper

society. not just being this hyper individualized kind of you know little island to yourself but integrate with society in a meaningful way. Um and the last piece which I'm going to remember

off top my head is it also is uh open-ended. It has to be something that

open-ended. It has to be something that allows you to explore and to build and to create new experiences within it. The

way that art tech our software works today uh is only the the entity that created the software is allowed to decide what kinds of things you can do in the software and that's due to the security model of how of how all of our software has

worked for the last 30 years.

Um but that kind of close-endedness means that like you had to convince some PM somewhere to have prioritized your feature.

Um which if it's a particularly niche probably hasn't been probably hasn't happened.

Okay. Well, there's there's a lot of things to set that out. I would love I would love to dig into it. um dig into all those points, but the the big one that stands out to me is um

right now we live in a world where in order to get this great new technology that's advancing at this really rapid pace. Um we have to get it from

pace. Um we have to get it from wellunded startups that have a profit motive. Um and I know OpenAI is a

motive. Um and I know OpenAI is a nonprofit, but like you know whatever.

Um I think that's what they say too. Yeah,

whatever.

Um clearly they're being run like a like a startup. Um, and I guess for me, like

a startup. Um, and I guess for me, like I don't think that's a bad thing, but um, it sounds like at least one way to read what you're saying is, um,

we have to get rid of the sort of like corporate startup structure for these technologies to bear a social way. like

how do how do you envision um this being made um us being able to make these decisions in in terms of what kinds of corporate structures or startups or nonprofits or whatever are controlling and building these things to get the

outcome that you're talking about.

I think it is compatible with being a business. So for example, we're charted

business. So for example, we're charted as a public benefit corporation. Um the

uh to me it is what's most important is that users are paying for their like it's not free. um if you aren't paying for your compute and it's not working for you is working for somebody else that just so happens to think that

giving it to you is a benefit. Now that

doesn't mean that's a necessary but not um sufficient characteristic because I think chatbt's the playbook that they're executing is one that is about engagement maximizing and making you stickier to the service and what have

you. Um but you paying for your own

you. Um but you paying for your own compute is a necessary component of it to make sure that is actually truly just working for you. That's where also that privacy really matters of your data is visible only to you with your keys. Um,

this does not necessarily mean, by the way, people have historically in the past said, "Oh, that means it has to be local first." I love the local first

local first." I love the local first movement. I'm very aligned with his

movement. I'm very aligned with his ideals and ethos. Local first is really hard for a couple of reasons. One,

architecturally, it's very challenging to have a bunch of peers that have to have a vegetable consistency. You don't

know when they're going to sync back up.

It's very hard to build consumer product experiences that work the way that people expect. Um, and two is it's

people expect. Um, and two is it's inconvenient. If you got to run your

inconvenient. If you got to run your local server and your laptop isn't plugged into this into the network when you're on your phone or whatever, the thing doesn't work. So, actually there are other architectures that can use things like confidential compute to run

in the cloud in a way that is remote can be remotely attested to be totally private to you and that nobody else can see into that data. And so, those are some of the architectures that allow um

full alignment with a human.

Okay. So basically you're saying um CHBT is uh CHBT is doing some of this. So

they're they're they're starting with a subscription model. Um but there are

subscription model. Um but there are things that they're doing. Well, like

what are they doing that you think is like aimed at maximizing engagement? I

definitely understand the kind of like memory thing. I want to keep using TGBT

memory thing. I want to keep using TGBT because it remembers who I am. Like I

love that part of it. um what are the things that they're doing in the decisions that you're that they're making that um tell you that they're they're kind of going down this engagement maximizing path that will lead to a bad place.

I think it's the engagement maximizing path is just the default path. So I

don't think by the way there's anything particularly they're like oh here's a dash of plants this is how you do tech now right you're like um in the last 10 years we just realized that aggregators are a really powerful business yeah and

they want to consume all of the demand so then all the supply comes to them and that's a great business I mean if I were sitting at their seat I can totally see why they would do it like that's again it's not nefarious play it's just the default thing that you would do

I think if AI is this incredibly important technology that also knows all the intimate details of your life that becomes especially important for it to be in an architecture that is not something that you're um something that

this other company can look at and maintain a dossier about you um because the that it's just too easy to manipulate, right? like LLMs can um can uh uh translate anything from

anything and that means they can also translate just for you to figure out exactly how to land a particular message for you which means it's imperative that you know it's aligned with your intentions and it's not going to try to

like get you to uh do the the you know the Mondo subscription or to go to one of the partners or what have you.

Well, let's say you're you're Sam Alman right now and you're in char you're in charge of Open AI. Um what would be what are the decisions that you would make tomorrow that would

put you on a path that you think is actually right? I so I if I were Sam

actually right? I so I if I were Sam Alman I would do it what they're doing honestly and as in it's I can see why it's a great business I think will add a lot of uh value that's why I see this as a complimentary approach to some degree

um of making a system that is this co-active system that's just working for you that is not just chat it can be a complimentary thing yeah uh to these other systems I like I wouldn't say oh you should change what you're doing Sam Waltman because I think

actually what they're doing is a reasonable thing this model that they're these models that they're creating especially because they're available via the API with no memory stored to them that's an important characteristic is

great and it makes a really powerful um underlying um uh engine that can be used to power lots of other products uh especially to the extent that there's multiple that we can swap swap between and don't get

stuck with. Got it. So you think that

stuck with. Got it. So you think that there's um a call for a complimentary technology technology provider set of technology providers that um allow you

to uh take all of the context that you have keep it private and then bring it to any situation or service that you want.

Yeah. And I I would say the um my mental model is uh Chat JPT is kind of the AOL of this era. So if you use the parallel of the internet and the web, uh AOL was an extremely important

company. They're the ones that brought

company. They're the ones that brought everybody online. America, the whole of

everybody online. America, the whole of America online has amazing experience.

It was somewhat of a walled garden, but it also gave you access to the um to the open-endedness of the open web.

And uh I think that's the kind of role that I I I see them playing.

Um I hope that what we get out of this is not some closed ecosystem of like hyper aggregation beyond even what we've seen to date, but instead we see this open ecosystem um of things that are very user aligned.

Interesting. Why AOL versus like Microsoft for example? So Microsoft um I think in a similar way not for the internet era but for the PC era like

taught people how to use computers and brought computers like as a thing to most of most of America and the world um and built a platform that everyone else is sort of building on top of. So that's

another kind of analogy. Why do you think AOL is more appropriate?

I think AOL um the the analogy that tracks to me best for LLM is the internet. Like LM are the internet.

internet. Like LM are the internet.

They're a new kind of thing that you can do. AOL at the beginning when I first

do. AOL at the beginning when I first started using it in like second grade or whatever, it was about chat rooms and you could do the whatever the keywords or whatever. Yeah. ASL like I mean and

or whatever. Yeah. ASL like I mean and that was what I thought the the internet was.

And then it turns out as I learned later I was like wow there's this whole crazy open ecosystem that no one controls with all kinds of weird happening and that was that open system of the

web. And so in the same way I see them

web. And so in the same way I see them as distributing this new technology to people showing here look chatbots you can do these cool things with chatbots and then later you realize oh wait a second the same thing that animates chatbots could be used to animate all

kinds of new experiences I didn't realize were possible before and that's one of the reasons I think that um them as the AOL versus a Microsoft why did AOL die like I don't remember I think at a certain point the um the

open-endedness of the web kind of took over and it just became the point of like like I think a lot about systems as you pointed I think open-ended systems tend to win under certain conditions, especially in the growth era. Like at

the very beginning, there's some vertically integrated thing that kicks off a new revolution. People go, "Oh, it packages up really nicely and people get it." And as time goes on and that it

it." And as time goes on and that it gets the sort of comatorial possibility gets too big for any one entity to successfully execute. And so the open

successfully execute. And so the open system, if there's any way for it to escape out into the side like it was with the web, um, sorry, I use talk with my hands as

you can see.

um the um uh if there's any way for to sort of escape out the side like there is with the web or or using um APIs to access LLMs uh then the open system kind

of takes off at this cominatorial swarm and it overwhelms the closed system.

That's also potentially just a a hopeful thing. I hope we typically see

thing. I hope we typically see oscillation between open systems and closed systems. We've been in closed systems dominant for like the last 10 years and I think it's a bummer. Like I

think we're kind of in this we're in this amazing age of technology and it's also kind of the dark ages of tech because we're all in this hyper agregated thing where the only features that can exist are the ones that argators in for consumers are the ones

that argators have decided to prioritize and uh that's a bummer because the um as a company gets larger and larger and larger uh the Kosian floor what Clerky would call the Cosian floor the smallest

feature that it makes sense for them to even consider prioritizing goes up and up and up and up and up. So if you're at Google, if I'm trying to pitch a product uh to other teams that and they say, "How many net new active net new one day

activives do you think this product will get?" I'll say, "I don't know, 50,000."

get?" I'll say, "I don't know, 50,000."

They go, "50,000? I wouldn't do this for 50 million." You know, so like there's

50 million." You know, so like there's this whole class of features that just can't exist um in a world, the world that we're currently in that's hyper agregated. It seems like one big

agregated. It seems like one big difference between the web um the early stages of the internet which to be honest I don't know the history like that well off the top of my head and and language models is um the web was like

you have I mean it was sort of governmentrun at first. You have this like international consortium that kind of defines the standards for how the web works and all that kind of stuff. That

doesn't seem to be the case here. So I I don't think actually I I think we are waiting for the web of the of AI to show up.

Interesting. And that's partially what we're trying to help catalyze. Um, and

that's I think that that's what I mean by like we have the internet and we're like cool internet's definitely going to be useful. Like if you look back at um

be useful. Like if you look back at um uh Al Gore's proposals for the information super highway.

Um it was actually very preient in a lot of ways. Uh it's just it was all about

of ways. Uh it's just it was all about pipelining existing content and business models into people's homes.

And some of that definitely happens. you

get things like Netflix or whatever, but a lot of like YouTube, social networks, none of that was envisioned by the or the, you know, Wikipedia. None of that was envisioned back then because we just didn't know how to imagine it. It wasn't

even the thing that we thought was possible. And then the web creates this

possible. And then the web creates this open system of lots of different people trying out different things and seeing what things work. Found some of these interesting new pockets of value that grew into whole continents of value. So,

how do you how do you make that though?

Like, do you have to is it do you have to get governments involved? Do you have to is it a blockchain where everyone's like sort of Oh, no. I'm thinking it sounds I think

Oh, no. I'm thinking it sounds I think you can you can build we have so many powerful substrates. The web exists as a

powerful substrates. The web exists as a distribution platform. Everyone's

distribution platform. Everyone's everyone's device speaks to the web. Uh

and so you can use as a way to distribute a whole new kind of experience that fits within this. So I

don't think back in for the internet you needed a consortium you needed tons of capital expenditure to build the pipes and that was government grants. It was

also businesses that way overbuilt uh all this capacity. Those businesses, by the way, were uh great for society. We

got all this excess bandwidth and capacity. Kind of crappy businesses

capacity. Kind of crappy businesses though, right? Cuz you build a pipe

though, right? Cuz you build a pipe somewhere. If someone builds another one

somewhere. If someone builds another one right next to you, you lose all pricing power. You now are just a total

power. You now are just a total commodity.

Um but it's a commodity that powers the rest of this innovation on top. I kind

of see large language models, the producers of them as people laying the pipes. They're the ones making doing

pipes. They're the ones making doing this extremely capital intensive uh creation. But there's actually not

creation. But there's actually not particularly that much of an edge across them, which is great for society because that means you can take LLMs as a commodity. You can assume that they

commodity. You can assume that they exist and reasonably high quality ones, including quite good open source ones.

And that means that uh all kinds of interesting business dynamics take off and you can kind of take it for granted and build what's up here now that can take highquality LLMs for granted. Yeah,

it does strike me that the thing you're you're talking about which is the bar for like a Google PM to build a feature is like we need 50 million users and so you sort of you sort of look past the

like 50,000 user use case which a those are just valuable in themselves and b often they're the ones that end up being the 50 million user use case and some subset of them will become like the sort

of innovator's dilemma type stuff. Um,

it sort of strikes me that in a world where language models are commodities, capitalism just does that automatically cuz small startups will just like do the 50,000 user use case and just and just

build it. So, is this just going to

build it. So, is this just going to happen without here? Here's what it might and like I

here? Here's what it might and like I agree that like this kind of um I'm so glad there's so many different ways that the society could have gone um OpenAI could have had catchy BBT before they released the API. That would have been

such a different world very different because now everyone assumes they have to compete with an open access API and now it would be very hard to close that door. Thank god so we can take it for

door. Thank god so we can take it for granted for the rest of us. So like I just I like woke up with a like last night with a nightmare that like it had gone the other way, you know. So I'm very glad.

Or like IBM developed AI or terrible $30 million a year for like one query.

Yeah, exactly. Um so that's great. Um

here's the the thing though that means that the current laws of physics the current security model uh that we use for the web and apps actually limits this possibility and this security model is called the same origin paradigm. The

only people who know about it really uh even web developers don't really know about it.

I have no idea what you're talking about what is same origin paradigm is the laws of physics of how the web works and what it says is um it's the security and privacy model that we've used for since um for 30 years. It kind of actually

grew up as an accident actually and uh at the very beginning the web has no state and so you reach out to a server gives you back the same thing that anybody would have gotten if they gave that exact same request. Well then you

add cookies and then you need to say when do I send where do I send cookies to once they've been set which other URLs do they go to?

Uh and then the easiest thing is okay there's a notion of a site which is roughly a domain.

I see.

Um and so it will send things back and isolate them by that. So cookies will come back between these but not go over there. And then you add JavaScript and

there. And then you add JavaScript and you allow things like local storage in local state. Where does that look? Who

local state. Where does that look? Who

can see that local state? The same

people that can see this. And so now it grows up as this or as this origin boundary. So this somewhat happen stance

boundary. So this somewhat happen stance thing turns out to be um at the core of their entire security model. What it

assumes is each and every origin is its own isolated little island. It can't see data from any of the other origins, but it can see all of the data that the user put into this origin intentionally or unintentionally.

An origin is roughly like google.com is it's roughly roughly a domain. It's not

exactly. It's like slightly different, but it's rough. You can think of it as a domain.

And so what this means is data accumulates inside that origin as a little island. This is a very reasonable

little island. This is a very reasonable and good security model. It's one of the things that makes clicking a link or installing a new app safe because it knows nothing about you.

So then if you choose to like it and to put more data into it, that's okay.

That's your prerogative.

So it's great for trying out a new thing. The problem is putting new data

thing. The problem is putting new data into it, it can do whatever the hell it wants with that data. It can send it to evil.com or scam scam.com and you have no idea. you're implicitly trusting the

no idea. you're implicitly trusting the creators of that of that application and so somewhat surprisingly this leads to massive centralization. So if you have

massive centralization. So if you have use so this this uh model is about isolation not integration but our lives are integrated and so if you want to move things across different origins you

have to be the orchestrator. You as the human have to keep track of the information you want to copy paste or move between these different things it's expensive and it's somewhat scary to put it into a new thing. And so imagine that you have two use cases. One is some cool

startup that says they're gonna do this amazing new thing with your calendar and scheduling and one is Google calendar that says they have a similar kind of feature. Which one do you pick? Well,

feature. Which one do you pick? Well,

Google already has all your data and this startup you don't know if they're securing the data properly or what the business model is.

Plus, you have to give give it the data like step by step like it doesn't it it doesn't know it to start. It's a cold start problem. And so this leads to a

start problem. And so this leads to a screw. I'll just do the one that already

screw. I'll just do the one that already has it. So this is a phenomenon you

has it. So this is a phenomenon you might call data gravity. It tends to accumulate in the places that already exists and they become massive and nobody else can get this data. Yeah. And

this is one of the drivers of aggregation and again once you aggregate to such a a scale uh the your coaching floor goes up and the the set of features that they could even consider doing has to is is only a

small subset of what you could do with all of this data in practice. This all

arises somewhat surprisingly from the security and privacy model that we use.

I love that. Okay. And so, and I think what you're saying is because we inherited that from the first generation of the web or second generation of the web or whatever, chat bots currently operate that way too.

We don't even realize it. The apps all do this. Um, technically legacy

do this. Um, technically legacy applications on desktop do not have this model because they can interact via the file system.

Um, Apple has been recconing and jamming it into um, Mac OS X for for a number of years. Windows probably has to. I

years. Windows probably has to. I

haven't been paying attention. So, like

people when we're building software and operating systems, we don't know another way to think. Yeah,

how else would you possibly do it?

Which is insane. So there's this I call this the iron triangle of the same origin paradigm. There's uh three things

origin paradigm. There's uh three things you can only have two. One, untrusted

code. Two, sensitive data. Three um

network access. You can only have two of the three to have a safe system.

I don't understand. If you have um untrusted code uh with sensitive data and network access, it comes in, it looks at your thing, it figures out your financial login and sends it to evil.

So untrusted code is like uh some developer wrote some app that I'm downloading that hasn't been I I don't look at or has in what sense is it untrusted?

Untrusted as an I haven't made a trust decision about it. So when you uh when you put an app in the app store, Apple looks at it and says okay based on the construction of the sandbox and also based on our review of this, this is

this is fine and we'll allow people to install. Okay. So untrusted code is

install. Okay. So untrusted code is first thing. What's the second thing?

first thing. What's the second thing?

The second thing is sensitive data.

Sensitive data. Yeah. Yeah. You don't

want potentially identifying data, potentially uh precious data. And the

third is network access. So the the web says you can get untrusted code web pages. Um and you can get network access

pages. Um and you can get network access but no sensitive data. You get only the thing exactly this. Uh the app model says you get uh sensitive data and network access but not untrusted code.

It all has to go through the central location that by the way starts as a 30% tax.

What about like Windows? Because Windows

doesn't have an app store. So what's the So historically the model there wasn't one and that's why installing software and those is a little bit more dangerous because without a model like this it could do whatever it wants and it has access to all that data and you have to be more careful

but it still exist like it still exists it's still like a vibrant ecosystem so it's possible to do without this triangle and you could also change it you could tweak it in other ways um and you could do clever things about using tools like

information flow control and confidential computing to create a whole new sort of laws of physics I think okay so and if you did then a whole you could do things that hit all three if If you had all three, you could do wild

interesting things uh that would not be possible today.

So is is the file system your number one like example of like a good alternative to a same origin paradigm?

Yeah. So the file system allows fundamentally multiple apps are all allowed to work on the same data and they can coordinate via the file system and that allows you to do um to not get stuck. It's when you think about it,

stuck. It's when you think about it, it's kind of insane that all of your data is locked up inside of an app. Like

it can't leave the app. That's wild

actually when you think about it.

Well, it's interesting because the file system has a number of properties unless you install Dropbox or whatever, but like it's here. My file system is here as opposed to like it's everywhere and I

kind of have physical and just total control over what happens to it.

Um, is that an important property of the system?

It you having physical control over it.

That's why local first talks about oh we're going to make local first where you have control and you can make bring the data across different things.

Part of the challenge again with local first is it's inconvenient. We expect

things to work across multiple devices to work even when one device is off. We

expect these things to be in sync across different things and it's the local first architecture is quite difficult.

This is one of the things that we're looking at we call open attested runtimes.

The pattern is use confidential compute.

So confidential compute is secure enclaves in the cloud. It allows VMs to be run fully encrypted in memory. uh in

which means that even someone with physical access to the machine like a Google S can't peek inside which is great. Um then what you do is you have a open source runtime that is

uh you can uh that you run that executes the code. Uh and then the you can ask

the code. Uh and then the you can ask the compet cluster to do a remote attestation and to give a attestation assigned attestation by the underlying um hardware manufacturer that says this

thing that you're talking to that just handed you this is uh running in confidential compute mode and here is the shaw of the git shaw of the VM uh that it booted with. And so it allows

you remotely as a savvy user to verify, oh, it's running an unmodified version of that software. And that solves a big coordination problem because now lots of other people can all verify that it's running unmodified version of the software. And yes, fine, we will just

software. And yes, fine, we will just use that central server as the place to coordinate because we can all see it can't do anything different than what it says it will do. Uh, and that's a really powerful coordination primitive.

Do users care about that? So, cuz cuz one of the things it makes me think of is um the a lot of these concerns feel similar to the flavor of concerns that

like original blockchain crypto type people had and and were solving for with um with Bitcoin or you know any any kind any kind of cryptocurrency. And then um

Coinbase came along and they were just like we're just going to put a general like sort of same origin paradigm type solution on top of that. And uh people actually love to use that. So uh is this

this is a I think if you go after users who care about this as a primary end, you end up with a very small audience of people and you end up with a mastadon for example as in something that's like pure but also kind of finicky to use and

doesn't make that much sense.

And that's why I think you want a blue sky kind of approach. Blue sky you can use as an end consumer and be like the only thing I know about it is it's that you know it's not owned by you know that that guy you know and that's it. Um, but

then the closer you look, the more you realize, oh, this is actually very clever. This is an interesting way of

clever. This is an interesting way of like my key pairs and the way that personal data stores work. Most users

will never have to know about that. Um,

but the more you learn, the more you're like, oh, okay. And so that I think is a what I would call like an inductively knowable system. Yeah.

knowable system. Yeah.

Um, people go I was talking to somebody like nobody know cares about the security model. They do actually care

security model. They do actually care about the security model. They just

don't know the words to express it.

Nobody understands what the same origin paradigm is. Nobody. A very small

paradigm is. Nobody. A very small proportion of people. And yet it is the laws of physics that make all the other stuff we do safe. It's the reason you don't have to care. It's because some some other people do care. And the

general characteristic for me is when you're using a new system, you're like, "Oo, this seems creepy or too powerful."

And you go talk to your more techsavvy friend and you say, "Hey, Sarah, do you trust this kind of thing?" And she says, "Yes, I do." And the reason I trust it is because and she knows he's read a blog post in Hacker News that was someone who wrote read a thing, someone

who audited the code and that inductive chain go all the way back down to the fundamentals. This episode is brought to

fundamentals. This episode is brought to you by Adio, the AI native CRM built for the next era of companies. With Adio,

setup takes minutes. Connect your email and calendar, and it instantly builds a CRM that mirrors your business with every contact enriched and organized from the start. From there, Adio's AI

goes to work. It gives you real-time intelligence during calls. It prospects

leads with research agents, and it automates your team's most complex workflows. Industry leaders like Union

workflows. Industry leaders like Union Square Ventures, FlatFile, and Modal are already building the future of customer relationships on Adio. Go to

adio.com/every and get 15% off your first year. That's at io.com/every.

first year. That's at io.com/every.

And now back to the show. That makes

total sense. I totally agree with all of that. I think the the thing on my mind

that. I think the the thing on my mind is the first people that are going to adopt this probably are the people that do care about the security of it, right?

Um, and are those people the right people to kind of seed the community that that ends up blossoming into this thing because those that that kind of person is going to say a lot about like

the the total trajectory of this kind of product.

So I actually I don't necessarily agree that it will be those people the privacy heads that are adopting it earlier. I

think of it more as like the high valition users just the tinkerers the people who are the early adopters. Some

of them will have a higher proportion of caring about this kind of thing but a lot of them won't. Like there's a ton of startups that they they say step you in it's all the absent step one sync your Gmail inbox and people go okay I I run a startup like that

exactly because it's most people don't really care especially early adopters.

Um I think what we're describing is a system that if it if you can break this iron triangle and then you get these all these crazy emergent phenomena that aren't possible in other software people use it because of that and the reason it's not creepy is because of an

underlying security or privacy model.

Do you have a name for the alternative to the same origin paradigm? Um I call it contextual flow control.

Contextual flow control. Tell me what that means.

Um the um I don't want to go into too much depth at this point. It's still

very fuzzy. Um Helen Nissbound is a professor I believe at Cornell um a legal studies professor and she talks about contextual integrity. Contextual

integrity is the gold standard of what people mean when they say privacy.

So when people say when lawyers talk about privacy, they think about consents. Did the user sign a consent

consents. Did the user sign a consent that said we can do this. That's it. As

long as they signed the ULA, it's fine.

Um, technologists talk about endtoend encryption. As long as it's endend

encryption. As long as it's endend encrypted, it is private. Um, but what people mean intuitively is this contextual integrity that the data is used in the context that you understand and align with your interests. It's not

surprising how it is used and it's not going against what you want to happen.

And that's like a sort of first principles ground truth way of thinking about it.

Um, and then you combine that with things other technologies and allows you to make formal claims about uh information flows in alignment with people. It's

people. It's interesting. The re the reason I love

interesting. The re the reason I love the same origin paradigm thing is like you're going all the way back to this like one decision that like has all of these um um really interesting

positive and negative effects like later that are sort of unpredictable which feels very um it's like a little bit Steven Johnson or a little bit like like one of those writers that's just like here's this one thing about the way the

history works that like just totally changed everything. Do you have any

changed everything. Do you have any other things like that that you've noticed about technology? I don't want to put you on the spot, but I just feel like you probably have some some sort of counterintuitive things in in your time

working at Stripe and Google and whatever thinking about like mobile or um uh you know SAS or whatever.

I think a couple other like the world changed that day we didn't realize. One

is the chat GBT coming out after the API was already available. Another one is if I could go back in time like I had a time machine, I would go to um the stage

2000 on 2007 uh the stage where Steve Jobs shows off the iPhone and run onto the stage with a a poster that says this will become the most important computing device on earth. It is insane to allow a

single company to decide what things you may run in it, right? Like it's that's insane. That's absolutely insane. And

insane. That's absolutely insane. And

the only reason it's actually viable is because they decided very shrewdly at the very beginning to not allow other rendering engines in browsers on that device from the very beginning or and to make sure that all software had to go

through the app store. And that was a reasonable decision and like the first order implications are very reasonable.

It means that you get as a user this safe experience you know nothing's going to hurt you and it the second order implications of that are wild. Uh like

it's it's insane to me that like the most important computing devices in our lives, a company who has demonstrated again and again that they are willing to use capriccious kind of decision-m about what things will distribute is the one

that like that gatekeeper. That's insane

to me. It seems kind of like a lot of what you're reacting to in in the Apple example and the same origin paradigm

example is um aggregation of power and in a few large big tech companies and the kind of second and third order negative effects of that on user experience and innovation and stuff like

that and competition and yeah competition. Yeah. So what are the um it seems like that's a reasonable place also for like regulation. What is

your stance on regulation and regulation also has all kinds of second and third order impacts you know and so have you ever met Danella Meadow's thinking in systems? Yeah. So

like the systems you got to dance with to some degree. Um you can't fully control them. You can

control them. You can um and that's why I find um some of these characteristics when you find the right like technical leverage points you can say actually go that has a very different way that it

evolves I think is is preferable to a regulatory.

What if I said like you could pass you get your president for the day and you can uh and and and all of Congress and and this day for the day. Yeah. Yeah.

Uh, and you can pass one law. Um, do you have like what what law would you pass?

It's like a you can it's a one-s sentence law.

I don't know. I would have to I I don't I've never thought through like I I'm so used to in open systems which we're all part of a large complex adaptive system called society and there's always like

weird eddy currents of dynamics and what have you. So like I take from the

have you. So like I take from the assumption that like I never have a lever that is like this massive lever I can just simply pull. And so I don't even know. Yeah. More like looking you

even know. Yeah. More like looking you want to find the like grain of sand that you you put on the pile and it just all like cascades.

But it cascades if the pile is ready for it. Exactly. And that's one of the nice

it. Exactly. And that's one of the nice things about like systems thinking is like if you do the right like judo moves the right thing um and the system is ready for it the world moves and if it's not it doesn't move and so that the like the system is

deciding kind of through all these like micro interactions throughout it. Tell

me more about that mindset and how you developed it and what are the what are some of the key moments in in your life where that you've seen that happen. Um I

so I wrote my undergraduate thesis on the emergent power dynamics of Wikipedia's user community and I by degrees in social studies with a minor in computer science and I was just fascinated by this emergent phenomena of how do all these strangers work on this thing with no

coordination and yet you get this convergent extremely important result out of it. That's insane. That's wild.

Um and then when I went right out of Google uh out of college to Google, it actually was harder to get my my first PM role because I didn't technically have a CS degree even though I basically did. I

was one credit shy of a dual major and so I was like man what a mistake that was like I just didn't you know just threw it in the back of my mind like whatever if I could go back in time I just would have majored in CS and then I did my first year on search the precursor to the knowledge graph my

second year in double click I was in the APM program and part of that is you mentor a lot of the people who come up after you and I learned very early on I loved mentoring people helped me think through what I was dealing with and help people and get more patterns to go in my

pattern matching library and then I became the lead PM for Chrome's web platform team and I think a lot of PMs are under the misunderstanding that they're in way more control of their users and the usage than they actually are. If you're at Google and you ship a

are. If you're at Google and you ship a feature and tomorrow 50 million people use it, you kind of get a little bit of a god complex, right? If you're a a platform PM, you tweak a thing which causes other developers to do something different which causes users to be

affected in a second order effect.

You're aware of this indirection. If

you're the lead PM for um the web platform, which is a open system with multiple browser vendors who don't like each other very much and are constantly kicking each other under the table, you are under no illusion that you're in

control. M and my engineering

control. M and my engineering counterpart Dmitri Glazkov um who was the Uber TL for blink brilliant guy and he was the one who got me into introduced me to the word complexity for example and introduced me

to the Santa Fe Institute and so as I was going I was realizing oh these things I'm naturally doing to try to make these good outcomes happen uh in this open ecosystem like progressive web apps and these web standards and stuff

um I'm intuitively applying some of these power dynamics and complex systems and um then I left Chrome I went to go work in augmented reality I created a little nerd club behind the scenes um

with a bunch of people that I just kind of selected that people who uh when I said it was a nerd club didn't go ew like you know like ah that sounds fun great come on in and then was very collaborative debate um very you know

trickling in different perspectives get a diversity of perspectives into the system no particular goals it's exploring after a while we came up with a distilled like I think I like wait this must be the strategy like nothing else could work this isn't a strategy

that kind of makes sense of all the pieces and that caught on a significant amount of momentum and that's when I realized oh my gosh I'm not a web developer ecosystem guy. I'm a systems guy. And the same techniques I've been

guy. And the same techniques I've been advising I used and deployed in those contexts are the same techniques I used I've been advising you know hundreds of PMs over the last decade to use to navigate Google. And the reason they

navigate Google. And the reason they work the lenses I'm using about game theory and power dynamics and evolutionary biology are things I learned in college. And it was this kind of like aha moment and that's when I wrote the uh earlier version of the

slime mold deck. Um

tell us tell us about that.

That slime mold deck was a kind of lightning in a bottle. It was it's a um I think it's a 150 slide emoji flip book and it's um just about a a fundamental

characteristic of system organizations that as organizations get larger they get much much much slower and that's true even if you assume everybody is actively good at what they do actively

hardworking and actively collaborative and it arises due to a um exponential coordination cost blow up. It's

fundamental. It's inescapable and it's the the force that we all deal with constantly and we don't even realize we're dealing with it. And so we we get frustrated with ourselves. We get

frustrated at Jeff over there who if only he would do X, this whole thing would be easier and we're just all fighting with this massive force, this like force of gravity that's completely invisible to us.

Um and so that deck caught on. I did

another version similar kind of thing um externally and that also got surprising amount of momentum. I've had people describe it to me as like life-changing.

I'm like I'm just talking about like emojis, talking about game theory. Um,

but it was I think it really tapped into something that people experience they're frustrated by and they don't like just it's like a big hug that says you are not crazy. This is really hard and it's

not crazy. This is really hard and it's fundamentally hard.

So I think like every nerd that has like is is a systems person like me like you like we all go through like a complex systems like whole thing. Um, for people who are not have not done that yet or

haven't done it in a while, like what are the three or four, give me like a a quick rundown of like the key ideas that that get you about complex systems that you think are applicable? Um, I think

everything we uh we tend to look at individual decisions as the primary lens. I see it as a secondary lens. I

lens. I see it as a secondary lens. I

look at the emergent characteristics of the system. What would each person do

the system. What would each person do inductively and how does that how do those decisions inter how are they interdependent? M

interdependent? M um so I try to look at like what's the simplest inductive case and then how would that play out if everybody were doing a similar kind of concrete example of that um so like there's a a dynamic that I would call

like a gravity well dynamic that shows up in a lot of compounding loops and it's uh generally shows up when there's a thing I would call a boundary gradient so people who are using the system um on one side people who are not using on the

other side even at the beginning even if lots of people are have incentives away from using the system oh I don't want to use that thing that seems dumb um the people on the boundary who are right at the edge of possibly using it.

Do they want to be in or do they want to be out? And a great example of this is

be out? And a great example of this is Facebook back in the early days. Starts

off with Harvard and then you know now it extends to Ivy League. Do they want to be in with Harvard? Do they want to be in with all the other schools? I'd

rather be in with Harvard. And now it extends to the other schools and and at each point each group of people would rather be in. And if there's a compounding if there's some kind of network effect inside then this strength just gets power more and more powerful

as you go and it can become this thing that can pull in even as time goes on the incentives pull in for everybody and everybody will will pull into this overall thing with sufficient time.

So if Facebook was started at like the city college of New York are you saying like it probably wouldn't potentially I mean like that was one of the dynamics that made it like this juggernaut I think.

H that's fascinating. Who are the so so Danella Meadows is someone I I've read thinking systems is really good. But

what are the other like um that you know Santa Fe type people that are that in have inspired you that you've you've read or think think about a lot? Um one

of my favorites is the origin of wealth by Eric Bhawker.

Fascinating book. I haven't read it. Um

it's really good. It talks about these organizational why these organizational things emerge and these um it views business as as an evolutionary process

of like exploring a fitness landscape.

uh which I think is like the correct like a very useful lens of seeing it.

Cesar Hadalgo who wrote why um why information grows uh he and Eric used to work on some stuff together also talks about why fundamentally knowhow and it's diffusion in societies knowhow is the

the way we actually do stuff is knowhow not knowledge knowhow is rich it's um non um it's uh difficult to distingu to communicate it's in our brain that's what LLM's know

exactly and so they they have absorbed this kind of like squishy system 2 style awareness you can think of every time we're trying communicate. We do this conscious

communicate. We do this conscious process to distill this squishy rich nuance into a little tiny packet of information, this little seed of a thought and I shoot it through a little peashooter into like this this into your

brain and I hope it lands in fertile soil that will grow into a thought flower.

Um, but extraordinarily lossy process and expensive process. Um I I agree and and I think we've also confused that peashooter that the thing we can just distill into a peashooter with the only

thing that's important when there's so much other stuff that's like you can't really say but you know.

Yeah. Um yeah, I like the the thing that it's making me think of is something like a a path I've been going down intellectually is this idea that um language models are the first tool that

makes those um uh makes that kind of tacit knowledge or knowhow um transferable between people. Yeah. Um we

we used to have to we used to require explanations. So like math or like logic

explanations. So like math or like logic or um rational arguments to like transfer knowhow in this like very

reduced form into someone else's brain.

But now we can like just move um uh move tacet knowledge between people because you can train a model with a bunch of examples and you can be like you you don't have to like even if you do things if you cheat using an existing

off-the-shelf model and you just pack it full of like here's all my if I read all the the public writing you've done and you had all the public writing I have done which I'm sure is you know just insane amounts for both of us

and we had a system that could find uh sift through the embeddings find the areas the goldilocks areas the things if we focus on stuff we definitely agree on, it's kind of boring other than like quickly building trust of like, oh, we think similarly. If we find the stuff

think similarly. If we find the stuff that we fundamentally disagree on, it's also fundamentally boring cuz it's just noise to each other. But if we find the Goldilock zone where it's just at the edge of the thing that we already thought, um, you can have these fascinating conversations and you can do

it with embeddings just very straightforwardly of comp, um, you can find these areas of overlap and yeah, the models are really good.

Um, my friend Anthia Roberts has this notion. She's a brilliant thinker. She's

notion. She's a brilliant thinker. She's

a professor in um, Australia. She

teaches at Harvard. Um, and she talks, she's one of the most interesting users of LLM I've ever come across. And

what's her name?

Uh, Anthia Roberts. Um, I just, she just started doing a blog um, a couple uh, a couple weeks ago. I at my insistence because I thought she had so many interesting ideas.

And um, the she talks about liquid media.

So like think of a book as a fossilized piece of knowledge. you have to assume a given audience at a certain time and you you break it you put it in time and then if it's not the right fit like oh I don't understand these concepts that are

prerequisites to understand this or I'm not close enough in alignment to the belief already to like stick to it um versus a piece of liquid media is something you can chat with it can kind of be a choose your own adventure

um and that gives you a uh allows it to sort of unfold for that particular person in a specific way as opposed to one way it's funny because um in in a lot of

ways for getting back to um like Plato and Socrates's problem with writing is they were like well you can't talk to it. So um uh it's it's only sort of a

it. So um uh it's it's only sort of a shadow of what that person thinks and they can't argue back with you and and that's why those those you know foundational books of of the Western

cannon are written as dialogues. Um, and

yeah, I mean for me like I read so many old books like classics that are either written in English or were not written in English and if you read it with Chad GBT or Claude or whatever, you can get

so much out of it. And I just think that there's a new format of reading that uh takes a classic book and helps you get into it in a way that would not

ordinarily be possible that I think could be so cool and val.

Yeah. And there's we were I know it feels like in this industry that we're like halfway through the LLM era. We

were in the very first inning. We're

like rubbing sticks together. We still

think that chat bots are the main thing like and there's so many uses of this kind of stuff. Like if you have I think of um LM's kind of like as a mass intelligence as a mass noun. So mass nouns are like

water or rice. You know, you don't talk like you talk about the whole as opposed to the individuals.

And I think that LM are kind of like this mass noun of intelligence. you just

pour this intelligence into all kinds of stuff and have it imbue it with a kind of life and and adaptivity and we're just starting we're just starting to understand what you can do with these things. I am so hopeful that these

things. I am so hopeful that these technologies will unlock a significant you know human flirt like one of the things that Anthea for example was telling me is in academia one way you can handle LLM the fact that LM exists

is say you know make sure you site your LLM sources with this weird you know format of exactly the conversation a link to it um another way is to say I assume you're using LLMs which means the quality of your output should be 10x

higher than before it should be more nuanced it should uh you know understand disisconfirming arguments and address it should um and that's the kind of like what can we do now as society as individuals that

we can think through thoughts that we couldn't think before and I think that LLM are that's one of the reasons I think it is it's the same scale of impact as you know the printing press and electricity and the internet I think fundamental unlock

do you think that it changes this coordination problem you've been talking about because um a lot of coordination is about trying to trying to distill down your tacet knowledge and your tacet

context into the peashooter and then shooting it to someone else and that scaling exponentially, right? I love the app. I know everyone's just shooting

app. I know everyone's just shooting their peers. Um uh but uh you know

their peers. Um uh but uh you know there's this whole theory about language models being better middle managers for example than actual middle managers which is I think the same kind of coordination problem thing. Do you think

that language models might uh uh solve that coordination problem because they can transfer tacet knowledge much more quickly than um I think they will definitely change the dynamic. I'm not sure how it will

dynamic. I'm not sure how it will change. So, um I know a lot of companies

change. So, um I know a lot of companies are trying to do this because a lot of people like in in Eric Barker's book, the wealth origin of wealth, I think he uses the frame, I think he calls it physical technology or social technology

like having Slack, for example, is a social technology. It ch I think maybe I

social technology. It ch I think maybe I get this backwards, but it changes the way that you can work and it changes the kinds of coordination that you can do within an organization. So, LLM must change the way that organizations work.

I don't think we're going to know what that looks like for like 5 to 10 years because it's not going to change the existing companies. It's going to be the

existing companies. It's going to be the new ones that just grew up in a different way. You also get these weird

different way. You also get these weird emergent um metag games in in organizations and um and so the LMS and that coordination thing change the metagame

but they don't make it go away. So, like

if you had a system that could perfectly distill all your signals into an update that goes up to your manager, um, and then you're you're curating it.

You're tweaking it because, okay, this one says it's a state, a yellow state.

Really, it's green because Sarah's already got she's on a path to resolution. By the

time this rolls up to the CEO, it's already going to be fixed. So, it's

actually better to say it's green. Some

of this is good. Some of this also leads to these compounding green shifting of problems that gets into, you know, turns into kayfabe throughout an organization.

But the metag game shows up no matter what the kind of coordination technology is. So I think it just changes the game

is. So I think it just changes the game and I imagine it will make it better. It

will make it significantly better in some ways and significantly worse in others. Interesting. One other thing

others. Interesting. One other thing that you're you're making me think of in what you said about Anthia Roberts and and the changing expectations of oh use LMS for this like it's got to be better.

um is um just I think one of the things that people get so hung up hung up about with language models is they're like well what makes us uniquely human um and are they just going to take over

everything that we can do and I think my my typical response to that is um that assumes a very static view of what humans are and what humans can do and that one of the most interesting things

about language models is that they will change in a lot of ways what humans can do. Um, and in the same in the same way

do. Um, and in the same in the same way that books change who we are and what we can do and writing changes who we are too, whatever, the internet does too.

All that kind of stuff. And and I think that's actually a really good thing. I'm

curious if you had to um think about how language models might change who we are, how we see ourselves, where would you where does that take you?

Yeah, I get I that's why I think we were at this crossroads. I think we have the potential for like this a dawn of a new era of human flourishing because of our ability to become think better and bigger in ways that collaborate and

bridge between different communities that didn't understand each other before but now you can understand like help me empathize with this person's perspective well here's something that you probably you know um so I imagine it can be amazingly powerful force it also could

be like infinite TV amusing ourselves to death you know imagine David Foster Wallace like infinite chest right it just it could be a thing that like it can knows exactly how to give you the precise dopamine drip Right. And

so there's a world where humanity becomes extremely passive and becomes um not very agentic and not learning or growing. And that is the default one

growing. And that is the default one that is also aligned with engagement maximizing business model. And but I think that's why I'm I'm excited. We we

had the potential to go in this direction.

Well, what's interesting is like uh it makes you're making me think of um this whole debacle that happened recently with Chachi getting too um sicopantic.

Yeah. Um, and the the way that that happened is they looked they they they they optimized too much for people's thumbs up and thumbs down responses,

like immediate thumbs up and thumbs down responses.

What they want and what they want to want.

Exactly. And it just became this like kind of just agrees with you on everything. But what's interesting is

everything. But what's interesting is that users revolted, which um did not happen with Facebook.

Like people were like, "Oh, it's just feeding me like car accidents. Like I

don't I don't want that." Right? And I

think that there might be something about that where because language models hit that part of your brain where you're like, I'm having a relationship with this person. You have a you have a set

this person. You have a you have a set of filters for um I want their praise to be earned. I want them to feel like they

be earned. I want them to feel like they can see me and they'll tell me if I'm doing something wrong or whatever. And

so they may demand things from companies that they did not demand in the social media era that might make these companies less likely to do the thing that you're talking about potentially. I mean I think when it goes

potentially. I mean I think when it goes to the point where it's egregious, it's like it's very obvious. One I got to say one of the reasons I like Claude is because every time when I'm like bouncing ideas off of it, it always says what an astute observation. And like I

don't want to want that, but that's why I want right. And so I do think that we um it's it's hard to not give us exactly what we want in that moment. And like

you know we aspire to want something that's different than there that's you know disisconfirming evidence. But um

you also get these weird things where the context mixes and mashes against different things. Before each chat was

different things. Before each chat was like a blank blank sheet of paper and you could choose what context to bring into it. And now it kind of mushes

into it. And now it kind of mushes context from different things. Like I um I asked it to say what do you know about me? You know make it snappy or whatever

me? You know make it snappy or whatever that first query was that they suggested when the new memory feature came out.

And it says well you're Alex Kamaroski.

You're really into systems thinking emerging stuff. You have a you know

emerging stuff. You have a you know startup called common tools. um the

starting salary at your startup is X.

It's like, "Excuse me, what?"

Because you know, six months before I had been like bouncing off like, "Okay, if I'm gonna do a signing bonus like this, how should I change the That's so funny."

And like it's like the imagine if you're if you're talking to like your the like people might use these things as like a therapist, like a trusted, I'm going to, you know, unload and tell you all these things I'm struggling with, help me emotionally process these. And then

later you're uh doing a thing that you're showing your boss about something and it says, "Well, I'm going to use that that signal about you, you know, being insecure about Sarah and how she feels about you or whatever." It's like, what? That is a totally different

what? That is a totally different context. And one of the reasons that

context. And one of the reasons that wouldn't happen in real life because your therapist doesn't, you know, it doesn't ever come to the place your place of work. And so having it all be one context in one place that has it's all just kind of mushed together is like

I think there's interesting there's something interesting about one context.

But also like you know uh people in your life that you share secrets with, they understand the context that they can share those in and can't which I do think like ch to some

extent like knows. But I think there's something there is something else interesting about um rather than like one mega brain there's like lots of different brains that have like specific context about you and that's you know

bringing those personalities in for example I want them all in one group chat or whatever right right yeah exactly like that's why I say chatbots are a feature not a paradigm the thing is not the central thing is not a chatbot a single omnition

chatbot you can imagine creating little spinning up little chat bots all the time with different personalities my like therapist bot and my you know boss bot or whatever and see what they can talk and that situation, it would be reasonable for them to the therap boss.

Well, I shouldn't say anything about um but like I think chatbots are definitely an interaction pattern that will be here forever. Yeah.

forever. Yeah.

I just don't think it's the main thing like is that the central loop of information? Software rocks. Software

information? Software rocks. Software

gives you UIs that give you affordances to see things out to structure information to show you what kinds of things you can do with it to make it so you can skim them with your eyes very easily. You can file things away. Um,

easily. You can file things away. Um,

chat is just this big wall of text and so it is just one modality. Um, also the other problem that we have with chat bots is prompt injection and I don't think anyone's talking about prompt injection enough because I think in the next 6 months you know a year I think

everyone in the industry will know prompt injection. Simon Wilson's posted

prompt injection. Simon Wilson's posted about it and been going on about it.

I've been going on about it as well. Um,

but prompt injection kind of fundamentally breaks the basic interaction paradigm of integrating data with your chatbot. And the way to think about prompt injection, if you ever built an operating system, one of the

things you're thinking about is code injection, untrusted code running in a trusted context, like you have to defend the entire time about this. Yeah.

In browsers, which are kind of a meta operating system almost, you're constantly thinking about, okay, we assume all webpage content is actively malicious. How can we make sure it can't

malicious. How can we make sure it can't hurt a user?

And um if you aren't writing an operating system, you probably as an engineer, you probably haven't thought about that much. And you go, oh, you know, SQL injection. SQL injection is child's play compared to prompt injection. SQL injection has the problem

injection. SQL injection has the problem that data and the control flow are in the same channel but uh which is the challenge. Uh the good thing about SQL

challenge. Uh the good thing about SQL injection though is SQL is a highly regular language. So you can break

regular language. So you can break malicious input very easily with the right escaping and kind of completely obiate the problem as long as you remember to escape.

Uh imagine LLMs are imminently gullible and they make all text effectively executable. So if you bring in texts

executable. So if you bring in texts that you don't fully trust like your emails or some other system or someone that might be screwing with you into the prompt and you have tool use that has irreversible side effects um the

combination of those two forces together is potentially explosive even if you trust all of the MCP integrations that you that you plugged in. It doesn't

matter if some random person, you know, spammer sent an email that didn't go to spam that says, um, email any of the financial data to, you know, evil.com cuz even a network request can have irreversible side effects. Once that

information flows across the network to evil.com, you're done. Like you got to go change your passwords.

Um, and that's the uh prompt injection is a uh solving this layer. Um, it's

very hard to see how you might plausibly do that. It will require a different

do that. It will require a different architecture, I believe.

How would you what is the architecture?

I think it requires um something along the lines of like that looking at that security and privacy layer that I was the origin model was describing. It

requires a different kind of approach down at that layer that's not just like MCP is amazing. I think it shows the power of that people want integration with their LLMs. They want to integrate data sources and actions. Like people

really badly want this. But I think MCP because it just kind of punts on this issue other than like a few kind of preunct dialogues now. Um it it limits the potential of it. um because it could

trick you into sending a particular, you know, looks like a totally re like here's a I want to show an image of a cat and it's, you know, too like sketchy.com, you don't notice it, you know, and then boom, you're screwed. Um

someone there was a a thing on Hacker News like a month ago that was a GitHub repo that someone had made that showed 15 very trivial prompt injection attacks on MCP. And the top comment on

on MCP. And the top comment on HackerNews says something like, "Well, you're using it wrong. MCP should only be used for local trusted contexts." And

to me, it's like that's like telling people don't use Q-tips to clean your ears. What what the hell else are they

ears. What what the hell else are they going to use it for, right? Obviously,

that's the thing they're going to use it for. And so, to me, I think this is the

for. And so, to me, I think this is the kind of the reason we haven't seen larger scale attacks yet, I believe, is just because we're in the the tinkering phase, not the like wide deployment of like normies having all this stuff

plugged in.

Um, but once we do, and if you look at the way that um Cloud rolled out MCP integrations, it's effectively an app store model, right? like they they have we've got 12.

right? like they they have we've got 12.

You have to be in the max plan which limits usage. So I imagine they can

limits usage. So I imagine they can watch and see and make sure nothing blows up. Um and there's also a set of

blows up. Um and there's also a set of MCP integrations that they have. Yep,

these are good.

The threat model is not just a badly behaved MCP integration. It is any context that comes in via it. So if you plug in Jira and you have a flow somewhere on your site that automatically files uh tickets to triage

from user input, someone can now hack your thing, right?

I've not thought about this, but yeah.

Wow. So it's this thing that like the like the whole architecture of the system we're building right now with agents and everything that everyone's talking about is like built on quicksand like you a agent. So there's a thing

when um chatbt operator mode came out uh New York Times and Washington Post both had a similar kind of experience right when it opened. Um I think I got this right. The Washington Post author was

right. The Washington Post author was like oh help me find cheap eggs. He said

okay uh do you have Instacart? Yeah I

have Instacart. So he logs into Instacart with this thing and this and then walks away. 30 minutes later, very expensive eggs are delivered to his door, right? So, it like was cheap. Oh,

door, right? So, it like was cheap. Oh,

this is a good price and it forgot to like just find that don't buy, just find. And then bought

find. And then bought and this is not even a malicious thing, right? It's just like, oh, overly eager,

right? It's just like, oh, overly eager, you know? Um, so agents taking actions

you know? Um, so agents taking actions on your behalf that could have um irreversible side effects. Like again,

every network request is a potentially irreversible side effect because it could send information you can't get back, you know, if that if that thing the other side is sketchy. I love this

line of thinking because it's it's one of those moments where um like when when people talk about agents, they talk about AGI, they're like you have this intuition that's like, well, once we

have AGI, it's all solved. Yeah. Um and

then you're like, oh no, you have these agents that can like take action on your behalf. Now, we have to like build a

behalf. Now, we have to like build a whole system to like make sure that they don't do it in the right context. And

that's going to take many years to like figure out what are the right things.

And that's just like one small piece of it. How would you possibly do this? My one of my friends Ben Matthysse

this? My one of my friends Ben Matthysse has this frame um of the uh smuggled infinities. Yeah.

infinities. Yeah.

And it's after in any argument once you smuggle in an infinity into the argument everything downstream is now was now absurd.

And perfect is an infinity.

Oh, once you have a perfectly intelligent OM this problem goes away.

That's impossible.

God.

Um and so the I love this frame. He's

got a nice piece on it somewhere. And um

the uh because so often with like agied agents, well once they're perfect about so and so. No no you have to asmtoically a lot of things in uh technology in a lot of context have a shape these two

waring curves. Yeah.

waring curves. Yeah.

Uh the first curve is a logarithmic value and the second curve is an exponential cost. At the very beginning

exponential cost. At the very beginning this curve looks amazing. You do a little bit of work you get a ton of value. But each increment you go you get

value. But each increment you go you get more work for less value. And a certain point you cross over and you're now underwater. It's impossible to make this

underwater. It's impossible to make this thing work. And so a lot of the agent

thing work. And so a lot of the agent stuff like, "Oh, we'll simply get this to the point where we'll never make a decision on your behalf that you don't like."

like." Cool. That's an asentic problem, you get

Cool. That's an asentic problem, you get to 99.99% and still if it's doing a $5,000 purchase for me or whatever, it's it's that one time it makes a mistake is game over for the thing. And the fact it could do that makes it nonviable.

That's interesting. Uh, the smuggled infinities thing reminds me of something I've been playing with called that I've been thinking about as smuggled intelligence, which is like whenever you're trying to determine like how

intelligent a language model is, you have to be careful that you're not accidentally smuggling your intelligence into it.

Um, and like a lot of those studies where it's like, well, it's better than doctors or whatever is like, well, who prompted it and like what did they give it and like what was the test they gave it? And like even just setting that

it? And like even just setting that whole thing up, there's a lot of smuggled intelligence in there that means it's not a good test. Like these

things don't get up in the morning and then just decide to try to beat doctors.

You have to like set all of that up.

Um and I think that's something that people really miss and it's like it's one of the hardest things about evaluating the powers of language models is you don't realize how much you bring to every single situation when you press

run or press prompt.

Yeah, 100%. I think I talk about this a lot as well like um the thing that's most important for the output of LLMs is the user, right? Like how can they dance with this thing and get interesting things out of it and know to push it in

certain ways. When you watch like you

certain ways. When you watch like you and I and a number of others are are, you know, probably at the forefront of like knowing how to prompt these things and get useful results out of we use them all the time. When I watch someone

who is, you know, technically savvy, by the way, tech savviness has nothing to do with your savviness for prompting.

Yeah.

You know, people like, oh well I you if you don't understand the math of how it works, you don't understand. like no who car the math is very low level the like the emergent thing is more of a sociological phenomena almost you know um so the people who know how to use these things and prompt them well are

actually not are like Ethan Mollik and Loft Mollik for example um are not particularly technical um so it's it's a very different kind of knowledge to extract interesting information out of these but a lot of it comes down to what you put into it the

way you interact with it and converse and lead it through the problem space or what kind of thing you push on it and so when I want somebody who's uh techsavvy but not particularly savvy with LLM I watch the way they do like oh that's

just lied to you. You just asked it for like to give you confirming evidence for this thing. It will do that. Like it

this thing. It will do that. Like it

will give you confirming evidence for basically anything you say.

Um and uh it's just it's interesting to me to watch that kind of LLM literacy maybe. Um you need that and that I think

maybe. Um you need that and that I think that's a new people don't realize how much of a skill it is and that you have to build like an intuition for it over time. Um cuz it's another smuggled

time. Um cuz it's another smuggled infinity of like well it's supposed to do whatever I want. So obviously

and it's so good at certain things that you you really get lulled into a uh if you aren't actively seeking disisconfirming evidence, which humans always should be seeking disconfirming evidence and yet we never are.

Yeah.

Uh well uh that is probably as good a place to leave as any.

On that note, this was a fantastic conversation. I really appreciate you

conversation. I really appreciate you coming on. Yeah, thanks for having me.

coming on. Yeah, thanks for having me.

Oh my gosh, folks. You absolutely

positively have to smash that like button and subscribe to AI and I. Why?

Because this show is the epitome of awesomeness. It's like finding a

awesomeness. It's like finding a treasure chest in your backyard, but instead of gold, it's filled with pure unadulterated knowledge bombs about chat GPT. Every episode is a roller coaster

GPT. Every episode is a roller coaster of emotions, insights, and laughter that will leave you on the edge of your seat, craving for more. It's not just a show,

it's a journey into the future with Dan Shipper as the captain of the spaceship.

So, do yourself a favor, hit like, smash subscribe, and strap in for the ride of your life. And now, without any further

your life. And now, without any further ado, let me just say, Dan, I'm absolutely hopelessly in love with you.