Notion’s Sarah Sachs & Simon Last on Custom Agents, Evals, and the Future of Work

Broadly speaking, I'm really bullish on CLIs. I'm still bullish on MCPS in in a

CLIs. I'm still bullish on MCPS in in a certain environment. I think in in

certain environment. I think in in particular, MCP is really great for when you want sort of like a narrow lightweight agent.

I think there's there's definitely a lot of use cases where where you don't want like a full coding agent with a compute runtime and also you want it to be like more tightly permissioned. MCP

inherently has a really strong permission model. Like all you can do is

permission model. Like all you can do is call the tools. MCP is just like the dumb simple thing that works and it it's pretty good. Notion is dedicated to

pretty good. Notion is dedicated to being the best system of record for where people do their enterprise work.

So we will always support our MCP in so far as other people are using MCPs, right? So regardless of our perspective,

right? So regardless of our perspective, we've put a lot of effort into our MCP and we have a fantastic team that we're building.

Before we get into today's episode, I just have a small message for listeners.

Thank you. We would not be able to bring you the AI engineering, science, and entertainment content that you so clearly want if you didn't choose to also click in and tune into our content.

We've been approached by sponsors on an almost daily basis. But fortunately,

enough of you actually subscribe to us to keep all this sustainable without ads and we want to keep it that way. But I

just have one favor to ask all of you.

The single most powerful, completely free thing you can do is to click that subscribe button. It's the only thing

subscribe button. It's the only thing I'll ever ask of you. And it means absolutely everything to me and my team that works so hard to bring the Inspace to you each and every week. If you do

it, I promise you we'll never stop working to make the show even better.

Now, let's get into it.

Hey everyone, welcome to the Lady in Space podcast. This is Allesio, founder

Space podcast. This is Allesio, founder of Colonel Labs, and I'm joined by Swixs, editor of Elen Space.

Hello. Hello. We're back in the beautiful studio that Allesio has set up for us with Simon and Sarah from Notion.

Welcome.

Thanks for having us.

Thanks for having us. Yeah,

congrats on the launch recently. Custom

agents. Finally, it's here.

How's it feel?

We ship things slowly. So, it had been in alpha for a little bit and at the point at which is it's in alpha. Um,

there's a group of people that are making sure it's ready for prod and then there's a group of people working on the next thing. So, sometimes some of these

next thing. So, sometimes some of these launches are a bit delayed satisfaction.

So, it's quite nice to remind yourself all the work you did because we do have a habit of like being two or three milestones ahead just because you have to be, you know, you can't get

complacent. Um, but it's been great that

complacent. Um, but it's been great that people understood how this is helpful.

And I think that's just easier in general building AI tools today than it was two, three years ago. People kind of get it. And so that user education um

get it. And so that user education um there's just it was our most successful launch in terms of free trials and converting people and things like that.

It was really successful. So yeah, but there's a lot to build.

Making it free for 3 months helps.

Yeah, it was definitely super exciting for me because it's probably the fourth or fifth time that we rebuilt that.

Yes.

And I mean, you've been building this since like 2022.

Yeah. I mean, like it was even right when we got access to like GPG4 in late 2022. One of the first ideas we had is

2022. One of the first ideas we had is like, oh, okay, let's make an agent that we use the word assistant at the time.

There wasn't really the word the word word agent yet, but oh, we'll give it access to all the tools that notion can do and then it will run in the background like like do work for us. And

then we just tried that many times and it just was too early. Um

I need to force you to like double click on that. What is too early? What didn't

on that. What is too early? What didn't

work?

We were find like before function calling came out, we were trying to fine-tune with the Frontier Labs and with Fireworks like a function calling model on notion functions. This is right

when I joined. I joined because um we needed a manager. Simon was needed to be able to go on vacation. So uh that's that's around when I joined. So you can speak much more to it. Yeah, we did partnerships at both enthropic and open

AAI at different times uh to try to at the time the I mean when we first tried there wasn't even a concept of like tools yet. We we sort of designed our

tools yet. We we sort of designed our own like like tool calling framework and then we tried to fine-tune the models to to use it over multiple turns. Um and

because it it didn't work well out of the box I think. Yeah, the models are just too dumb and the context length was also way too short. Yeah.

Um and yeah, we just kind of banged our head against it for a long time. Uh

unfortunately, it was always like there was always like sort of glimmers that it was working, but um it never felt quite robust enough to be like a useful,

delightful thing. Um until I would say

delightful thing. Um until I would say uh the big unlock was probably like Sonic 3.6 or 7 uh early last year and that's when we started working on our

agent, which we shipped last year. Um

and then and then uh uh custom agents kind of a similar capability and that that one just took longer because we we just wanted to get the reliability up a lot higher because it's actually running in the background and the product interface of like

permissions and understanding you know this custom agent is shared in a slack channel with X group of people and has access to documents that are surfaced to Y group of people and the intersect of X versus Y might not be whole and so how

do you build the product around making sure administrators understand that permissioning took multiple swings.

Everything is our back at the end of the day. Yeah. I'm curious like when the

day. Yeah. I'm curious like when the models are not working, how do you inform the product road map of like, okay, we should probably build expecting the models to be better at some reasonable pace, but at the same time,

we need to, you know, you had a lot of customers in 2022. It's not like you were a new company with like no user base.

Yeah. I mean, I think there's always the balance of, you know, like you want to be AGI pled and thinking ahead and building for where things are going. Uh

but also you want to be like shipping useful things and so we always try to like like keep a balance there. You

know, we we try to take like a portfolio approach. You know, we're always working

approach. You know, we're always working on multiple projects and and we're always trying to work on, you know, maintaining things where they've already shipped like like shipping new things that are like eminently working well and make them really good and and then we

want to always have a few projects that are a little bit crazy. And

yeah, what are the AGIP projects that you have today? I'm curious what you don't have to share exactly what you're working on, but I'm curious what are things today that maybe in 18 months people will be like, "Oh, obviously this was gonna work."

18 months.

Yeah.

18 months is, you know, it's a long time. And yeah.

Yeah. I mean, there's a number of things happening. I think one thing that's

happening. I think one thing that's becoming more clear is I think like like coding agents are the colonel DGI sort of everything is a coding agent. I think

that's that's sort of one one direction. Um,

and then yeah, the exciting thing about that is sort of your agent can sort of bootstrap its own software and capabilities and actually debug and maintain them. And so, yeah, we're we're

maintain them. And so, yeah, we're we're we're thinking a lot about that. And

then, yeah, like like another category of things that I'm really excited about is like uh we call the software factory.

Lots of people are using this this this sort of word. Um, basically just means can you create sort of like a as automated as possible a workflow for developing,

debugging, merging, reviewing, and maintaining a codebase and a service where there's a bunch of agents working together inside and like like how does that work?

If you think back to your initial question like why did this take so long?

I think something notions I didn't say that but yes. Okay, go

ahead.

Why what what changed over the three and a half years of trying it, right? Because most people always say

right? Because most people always say like it didn't work yet, then reasoning models came, then it worked. I was like, okay, let's go a little bit.

I mean, that's part of it, but I think the other part of it that I actually think is really what will set notion apart for every new capability is we have like two skills that are crucial

when it comes to frontier capabilities.

One is not letting yourself swim upstream. So like quickly realizing if

upstream. So like quickly realizing if you're just pressing against model capabilities versus not exposing the model to the right information, not having the right infrastructure set up.

That in of itself is a skill of intuition. And then the second is to see

intuition. And then the second is to see okay you're not swimming upstream. Which

direction is the river flowing? And what

is like how do we think ahead about the product and start building it even if it's not great yet so that when it is there we're ready for it, right? And

like those can sometimes feel like counterintuitive things like we can be trying to fine-tune a tool calling model when they don't exist yet. And the the trick is to not do that for too long, but realize that there was something there. And we've had a lot of things

there. And we've had a lot of things which like um were just like not swimming in the right direction with the streams. I think we had multiple versions of transcription before we got meeting notes, right?

Oh, I got to talk about that. Yeah.

Yeah. Um and so I I think that like we we really closely partner with the Frontier Labs on capabilities and we also have to have strong conviction on as those capabilities move. Notion is

about being the best place for you to collaborate and do your work. And how

does that narrative change if the way that we work changes?

Yeah. Yeah. You told me you were a fan of the agent lab thesis and this is this is kind of it. Right.

Right. I show that thesis to so many candidates. Like I have it as like my

candidates. Like I have it as like my Chrome autofill um at this point. Like

it's one of my most visited.

Like is this the here's why you should work in notion and not open your eye.

Here's here's what's different about it and here's why it's not just a rapper. I

actually think more and more people understand it's not just a rapper. Yeah.

Um, and by the way, like in the beginning, parts of what we build are rappers on functionality that works well, but that's not really the most um I would say that's not the product that that drives revenue and that's not

necessarily always what users need.

I mean, you know, notion is the AWS rapper, but like the the rapper is very beautiful and like very well polished.

So like the like the analogy that I've been coming back to is data dog in AWS. Yeah.

So, uh data dog could not exist without cloud storage, right? that it's kind of fundamental

right? that it's kind of fundamental that that works. Um, and AWS has like a cloudatch product, but Data Dog is an expert on understanding how people want observability on the products they launch. And we're experts on

launch. And we're experts on understanding how people want to collaborate and that's really where our expertise lies.

Totally.

Um, regardless of the tools that we use, I'm kind of curious how you think about implicit versus explicit expertise. I

feel like data dog is half and half implicit and explicit. It's like they understand across markets and industries what engineering teams usually look for.

With notion, it's almost like more of the expertise is at the edge because you as a platform you're like so horizontal that the end user is not really the same. Like with data dog, the end user

same. Like with data dog, the end user is always like an engineering lead like SR related person. With notion it can be anything. So I'm curious how you put

anything. So I'm curious how you put that expertise into a product versus you know obviously AWS cannot build notion.

that doesn't quite work in this case, but it's it's a little bit differently shaped. I think you know a classic

shaped. I think you know a classic vertical SAS like like data dog is kind of like that. They understand their individual customer very deeply. It's

kind of a narrow slice. Um notion has always been super horizontal and our our task has always been to sort of balance these two somewhat opposing forces of like we're listening to our customers

and what they want us to build. It's a

broad slice. And then also we're thinking about like okay, how do we decompose what they want into nice primitives that are that are really nice to use and will will get us like as much

bang for the buck as possible and then you know maintain the whole system make it all like like super clean and nice to use.

We still have easier journeys. I mean we still focus on like core I actually think the failure of our team is when we focus too much on what are tools that are what are tools that are cool tools.

I actually think that's when we make have the least velocity because you still need some sort of focus on a user journey. So like for instance, we'll all

journey. So like for instance, we'll all sit down every Friday and look at the P99 of like the most token exhaustive custom agent transcript and just look at why it didn't do well and cut a bunch of

tasks. Like we still focus on like this

tasks. Like we still focus on like this has like this should work. Email

triaging should work, right? And

similarly like when we're talking about before building um chatting um before we started filming about okay how can I do PDF export well that's functionality that then merits maybe we should build a

tool that has access to a computer sandbox and a file system and the ability to write code right um but it's because we're thinking about the fact that our users to do their to do their

daily work need to export PDF's not because we're like I think a computer tool could be cool like let's just see what happens like we we have to focus on some user journeys otherwise we just don't have like enough strategy to to

prioritize.

I think there's a lot of like really strong opinions that you've had. Do you

have like sort of like a towel of Sarah Saxs like you know like what how do you run your team like I feel like you just have accumulated all these strong opinions. Obviously part part of this is

opinions. Obviously part part of this is your your token town thing.

I think the to working with Sax is um you'd have to it depends who you ask. Um

I think it depends if you're on my team or a partner right or a vendor. Yeah,

there other people want to run their teams the way that you're you're bringing these things. And then also similarly, uh, Simon, when you did the custom agents demo, you had like, well, we've been using custom agents and here's the super long list of everything that we do. No humans ever read it,

right? That's what you said. I was like,

right? That's what you said. I was like, yeah. So, I think for for me, um,

yeah. So, I think for for me, um, something that I learned very quickly and became very comfortable with was that my job was not to be the ideas p person or the technical expert. My job

was to make it so that everybody understood the objective, had a resource to help prioritize what they should work on, and had an avenue to prioritize what they thought was important. And I think it's true with all leadership, but I

think especially on the AI team, almost all of our best ideas come from prototypes from people that have a cool idea because they saw a user problem.

And it's a huge disservice if all of those ideas have to pass like the sniff test of what me and a product partner or Simon and Ivan decided were the direction, right? because a lot of what

direction, right? because a lot of what we're doing is leaning into capabilities. So I think that's the

capabilities. So I think that's the first thing is like I don't really view like the role of engineering leadership as like uh hierarchical nor has it ever

been but especially now like very willing to change direction based on um like proof is in the pudding and like and I think we have rebuilt our harness

three or four times and when you do that then the second rule of engineering leadership is like you need to build a team that's comfortable deleting their own code and is very low ego and is driven by what's best for the and um

doesn't write design docs because they think it's their promotion packet, right? And that's a culture that notion

right? And that's a culture that notion had long before I joined. But like our willingness to just swarm on different problems and um redo things that we've built before because something has changed like there's a lot of friction

that can happen at companies when you do that and it doesn't happen at notion and because it doesn't happen when new people join like they don't want to be the ones that are saying we shouldn't do this. I wrote that code. So then it's,

this. I wrote that code. So then it's, you know, you create a culture that everyone adopts and that culture comes directly, I think, from Simon and Ivan though, um, because they're very open-minded.

Anything you you'd add?

I'm not a manager like like like Sarah is. Um, a lot of my role is really to

is. Um, a lot of my role is really to try to think a little bit ahead, make sure that we're we're building on the right capabilities and then like the prototyping stuff. And yeah, it's really

prototyping stuff. And yeah, it's really really critical to always just be starting again. It's like, okay, this is

starting again. It's like, okay, this is new thing. What does this mean? What if

new thing. What does this mean? What if

we just rethought everything, rewrote everything? Hence, I I'm basically just

everything? Hence, I I'm basically just doing that in a loop every six months.

Yeah. Do you believe in internal hackathons for this stuff?

I think there's like two different versions. So, one is like we just have a

versions. So, one is like we just have a a a solid bench of senior engineers that come and go on what we call the Simon vortex and productionizing what we built, right? Cuz when you're in the

built, right? Cuz when you're in the Simon vortex, the velocity is super high. The direction changes daily. And

high. The direction changes daily. And

it's meant to be like the equivalent of a skorks lab. We don't need to do hackathons for that. We need to have senior engineers that we trust to come in and out of those projects. For

instance, like management boundaries are really loose. Like you report to him,

really loose. Like you report to him, but you work for her right now. Like

that is something that when we hire managers, it's important they don't care about because we tend to form more structures.

Yeah. Don't be too territorial.

We form or structures after we ship things, not not before, just historically. Um the second thing is we

historically. Um the second thing is we do have companywide hackathons.

Actually, we just had our demos day for the hackathon we had last week this morning. That's more for people that

morning. That's more for people that aren't directly working on the project feeling like they have the time to pause and learn how to make themselves more productive or how they would use notion custom agents to build something or part

of the hackathon was actually encouraging everyone across the company to build their own agentic tool loop calling from scratch following like an every blog post on how to do it. I think

because we want is that the compound engineering one?

Yeah, we want everyone to use cloud code in the company or whatever the coding agent they please and understand that fundamental. So we set aside a day and a

fundamental. So we set aside a day and a half where all leadership encourage everyone on their teams across the company to do it. So we have hackathons like that. I would say like kind of

like that. I would say like kind of facitiously like everything we build is a little bit like a hackathon until it graduates and puts on big boy pants and has a product ops roll out later and has a assigned data scientist and stuff like

that. But

that. But security review enterprise stuff.

Actually, security review is one of the things that we bring in first because it just slows us down way more and um causes a lot of tension and they build better product if they're involved early. So, um that is probably the first

early. So, um that is probably the first person to get involved in something.

The right PR approved answer.

No, but it's not just PR approved. It's

like it's actually real. It's actually real.

It's like dark tissue.

Yeah. because like you know my background is also I worked at Robin Hood for a number of years. So like uh compliance and things like that um are a little bit more you learn the hard way when it doesn't come naturally.

Yeah. I think the the hackathon is really important for uplifting the general population but like if that's the only way you can build new things you're kind of toast. I mean it it has to be like the daily processes like you

know building these new things. Um, and

it has to be about I think like I think in the AI era a lot more leverage accumulates to the most curious and excited people. And so it's like we're

excited people. And so it's like we're all about just like activating that energy, you know, like if someone's proming something on the weekend that they're excited about and it's important. That should be the main thing

important. That should be the main thing that we're doing. Um,

it's not a hackathon that we schedule once a quarter. It's just like daily process. It's part of the culture.

process. It's part of the culture.

Yeah. I mean, that's how we shipped image generation in notion now. It was

always this thing that would be kind of nice to have, but it wasn't really clear where that was necessarily aligned in product priorities. It'd be a lot of

product priorities. It'd be a lot of work. And we had someone on the database

work. And we had someone on the database collections team, Jimmy, who was like, I really want to do image generation for cover photos and inside notion. And

we're like, if you want to build it, like it's do it, please. Like, we

encourage you. We gave him all the resources of working directly with Gemini and being able to like track the token usage and it working through our endpoints. We gave him email support,

endpoints. We gave him email support, everything. And then it became a full a

everything. And then it became a full a full project. Yeah,

full project. Yeah, that's why you can't have like ego as a a leader. Like that's that's how we

a leader. Like that's that's how we work.

What's the size of the team today? Both

engineering and overall.

I manage uh the team that's what we'll call like core AI capabilities and infrastructure. That's about 50 people.

infrastructure. That's about 50 people.

But then we have part I partner teams that do packaging. So how it shows up in the corner chat versus custom agents versus meeting notes that's another 30

40 people. And then every team that has

40 people. And then every team that has a product service at notion that a user can interface with owns the tool that the agent interfaces with. The editor

team, the team that did CRDT for offline mode is the same team that handles how two agents um edit competing blocks, right? It's the same problem. The team

right? It's the same problem. The team

that built the underlying SQL engine is the same team that owns how the agent ask it to run a SQL query and it does it performantly. And so from that regard,

performantly. And so from that regard, anyone working on product engineering is tasked with making them work for customers that are humans and agents because over time the majority of our

traffic will be coming from agents using our interface, not humans. And so our objective is to make it so that the whole product or is building for agents.

Yeah. How has it changed internally? The

activation bar is kind of lowered a lot like anybody can kind of create a prototype very somewhat easily especially if you're like in an existing codebase. Have you raised the bar on

codebase. Have you raised the bar on like what type of prototype people need to bring forward to gonna be taken not like seriously but like you know what I mean? Yeah, I think the bar is lowered

mean? Yeah, I think the bar is lowered in many ways. Like one thing that uh that our team built that was really cool is our uh our our design team made a whole separate GitHub repo called the

the design playground and it's basically just they created a bunch of like like helper components and you uh for for quickly throwing together UIs and it's become like actually quite sophisticated like it has like an Asian in there and

like uh that's pretty fun. So like we pretty much like they don't do mocks they just make like like full full prototypes.

Here it is. It works. they give you like a URL. They're like, "Okay, all right.

a URL. They're like, "Okay, all right.

So, we have to make like the real production version of that." Um, and then for engineers, a prototype looks like just making it a feature flag that actually works. Like that's sort of the

actually works. Like that's sort of the bar.

Something to understand that's really unique about notion, one of the reasons I joined, we're super lucky, is no one uses notion in their job as much as people that work at Notion, of course.

So, I think there's very few companies, maybe if you worked on Chrome, I guess, but like everything that we ship, we ship internally first and get a lot of really quick feedback. And also

sometimes our dev instance is totally borked and you have to change a bunch of flags to get things done and that's kind of like but everyone so people that do it ticketing people that do supply chain procurement recruiting everyone is using

the same instance of notion with like a lot of flags on for these prototypes people build and so we have this Brian Leven one of the designers on our team I think evangelized this concept of demos

over memos um which has been uh very good for building demos and I think it's put a big pressure point on us to have really

strong product conviction because if anything can be demoed, you really need a strong filter of making sure that if you know you're doing X amount of work, you're making the you're you're focusing

on one tower. You're not just building a really flat hill, right? That's actually

where I think there has to be more conviction from our PMs um and our designers and and well the company really to have conviction of what journey we're going on.

But overall, I feel like it works pretty well. like people almost all the

well. like people almost all the engineers have good enough taste to realize that like this prototype doesn't actually make sense in the product or or it does. So it's not that common that I

it does. So it's not that common that I would see a prototype it's like oh this makes no sense. It's like you know people are doing reasonable things and and and then it's just a matter of which things we build first and then often

just just figure out how to turn it on and off. There's our in the in our like

and off. There's our in the in our like experimental chat UI there's this there's probably like like a hundred check boxes in there. It kills me.

You could turn on and off.

But I think that Okay, so that is kind of true, Simon. But like being the person that manages the eval team, like there is a level of intensity that it adds to the platform team. So you know,

if we're going to do image generation in notion, all of a sudden the way that we do attachments and the way that we um are LLM completion like Cortex talks and expects tokens back and now it's getting

images back. Like there's a lot of

images back. Like there's a lot of platform work that we do need to like solidify a little bit. So sometimes

it'll be in dev for a couple weeks before it makes it to prod just because we still have to like make it robust, make it HIPPA compliant, ZDR compliant, figure out the right contracting with

the vendor, whatever it is. And we need to eval because we want the team to still maintain what they build. That's

the one thing is like if we have a bunch of prototypes, it can't just be like a small group of people that then maintain whatever in prototype. So we have invested a lot of people in an eval and

model behavior understanding teams that we call it agent dev velocity. So your

dev velocity building agents can be faster if we invest in that platform.

And so we have a whole org dedicated to agent um platform velocity so that you can build your own eval and then maintain it once you ship a so if a new model release comes out and we have every team maintains their own eval.

We maintain the eval framework. Every

team owns their own evals and a lot of them we've integrated to opt into CI or we run them nightly and we have a team uh a custom agent that triggers to a team to look at the major failures.

That's really critical because if we have like all these different services now a lot of it's on the same agent harness. So it's easier to maintain.

harness. So it's easier to maintain.

It's just packaging of different agent harnesses but new functionality of the agent. Let's say that like we want to

agent. Let's say that like we want to update like uh you know they deprecated sonnet um four or whatever it is and we need to auto update already. That's so okay. Yeah. It wasn't

already. That's so okay. Yeah. It wasn't

that long ago or just 3.5.

3.5 37 just got deprecated. 37 5.2 or Yeah.

No 5.2.

Yeah. No 54 is 40% more expensive than 52. So if they deprecated 52, you would

52. So if they deprecated 52, you would hear you would hear from me about that one.

Um but uh another conversation to have I have a cheeky evals question for you.

Have you noticed any secret degradation from any of the major model providers?

Secret degradation like during the war when it's high traffic it suddenly gets dumber.

Yeah I mean not just between the I mean we definitely noticed flakiness. We've

definitely noticed particularly for some providers that things are slower during working hours and but that's a latency argument not a quality argument. No, I think the

quality argument. No, I think the quality difference that's interesting is um even though companies that say they're selling the same it's related into like quantiz quantization, but like companies that say they're selling the

same model through different vendors, whether it be through first party or Bedrock, Azure, etc. We do see different qualities sometimes and that's not necessarily what's advertised.

Yeah. Kney went to the point of like we they shipped like this like eval across all the providers and it was like very obvious who was secretly quantizing and it was yeah but that's you know um we hire

subprocessors to figure that out for us.

So we just want to understand where it's regressing or where it's optimized and sometimes we're okay with regressions that optimize latency if they're the appropriate regressions. Our job is to

appropriate regressions. Our job is to make sure we have the eval to understand the changes that are important to us.

And even like when we're partnering with labs on pre-releases of models, they'll send us multiple snapshots. And this is less about quantization but more just regressions. Like they have shipped

regressions. Like they have shipped models that were not the snapshots that we wanted. And they have changed the

we wanted. And they have changed the snapshots that they shipped based on the feedback that we give because our feedback tends to be more enterprise work focused and not coding agent focused. And definitely those can be

focused. And definitely those can be bummer like you know oh we know that this wasn't the version you wanted but we'll help you make it work. I mean, we always make it work, but that definitely happens.

Yeah. Do you have um failing emails that you're just hoping that we'll have success eventually when a good model comes out?

I mean, yeah. So, I think I mean, I could talk about this for 60 minutes, so I will limit myself. I think it's a real issue when people say evals and it's just like that's quality. That's like

unit I mean it's like saying testing.

It's not just unit tests, right? So,

we have the equivalent of unit tests, regression tests, those live in CI.

Those have to pass a certain percent you know within some stocastic error rate.

Then we have as you're building a product eval of these aren't passing right now and this is launch quality. So

we have a report card and we need to on these categories you know be at 80 or 90% of all of these user journeys to launch and then what we have what we call frontier or headroom eval where we

actively want to be at 30% pass rate and that's actually been a effort that we took in partnership with enthropic and open AI in the past maybe two or 3 months because we actually hit a point

where our evals were saturated and we weren't able to really give insightful feedback other than it wasn't worse and not only is that not helpful for our partners it's not helpful for us to understand where the stream is going.

You know, going back to that analogy.

And so we spent a lot of time thinking about what notion's last exam looks like, right? Not just humanity's last

like, right? Not just humanity's last exam, Notion's last exam. And um there's a lot of, you know, dreams about what that would look like. I know we've talked a lot about benchmarking um Swix, but

uh yeah, Notion's last exam is a big thing inside the company, and we have people full-time staffed to it exclusively. We have a data scientist, a

exclusively. We have a data scientist, a model behavior engineer, and an full-time um evals engineer just dedicated to the evals that we pass 30% of the time.

What you're hiring for MBEs?

I am hiring.

What is an MBE?

A model behavior engineer. Model

behavior engineers started with a title data specialist before I joined when they were working with Simon on like uh Google Sheets and like Simon just needed someone to look through Google Sheets and say yes, no, this looks bad, this

looks good. Right? And so we hired

looks good. Right? And so we hired people with kind of diverse linguistics background. We had like a linguistics

background. We had like a linguistics PhD dropout and a Stanford complate new grad and they're amazing and they formed a new function basically and over time we've built a whole team um with a

manager who's now kind of reinventing what that role is with coding agents. So

they used to be kind of manually inspecting code. Now they're primarily

inspecting code. Now they're primarily building agents that can write evals for themselves or LLM judges. There's a

really funny day I can send you the picture where Simon about a year and a half ago was teaching them how to use GitHub. um and they're on the whiteboard

GitHub. um and they're on the whiteboard and it was like okay I think we'd be so much faster if our data specialists learned how to use GitHub and like learned how to commit these things into code and and that was then and now I think you know coding has been a lot

more accessible um but moving forward it's this mix of like data scientist PM and prompt engineer because there's craft in understanding like even like what models can and can't do things how

do we define like that headroom how do we define like what a good journey is um is this model better or not why is this failing there's some qualitative work, but then there's also like a lot of instinct and taste to it and that's not

necessarily software engineering and so we have like very firm conviction and we have had for a number of years now that that is its own career path and we have always welcomed the misfits so to speak.

So we really firmly believe that you don't need an engineering background to be the best at this job and that's what's quite unique about this particular role. This is something that

particular role. This is something that I've been pretty excited about recently is we made an effort basically to treat the eval system as like an agent harness. So if you think about it like

harness. So if you think about it like you know you should be able to have an agent end to end download a data set run an eval iterate on a failure debug and

and then implement a fix and ultimately you should be able to you know drive the full end tom process with a human sort of observing the you know the outer uh system. So yeah we went pretty hard on

system. So yeah we went pretty hard on that. That's worked extremely well so

that. That's worked extremely well so far. It's like basically just to turn it

far. It's like basically just to turn it into a coding agent problem.

Your coding agent or just whatever agent.

It should be totally general. Yeah. I

think it would be a mistake to like like fix it on any particular coding agent.

At the end of the day, it's just like CLI tools.

It's like the same way that you would have a coding agent write the unit test, you should have a coding agent write the eval.

Yeah.

But there's a lot of supervision in that still. We just don't believe that

still. We just don't believe that supervision has to come from software engineers because a lot of it is like um kind of UXRE and whatever. And these are the people that also triage failures and tell us where we should be investing

next.

Yeah. I'm going to go ahead and ask a spicy question. Is there a day there are

spicy question. Is there a day there are no software engineers at Notion?

Um, what does it mean to be a software engineer?

Exactly.

I mean, I think the way things are going is like we're on some continuum where if if you look back 3 years ago, humans were typing all the code and then we had autocomplete. You're typing a little

autocomplete. You're typing a little less of the code. we had sort of like filling agents filling lines and now we're getting into like agents doing longer range tasks where you can debug and implement a fix and then verify it

works and you know get your get your PR even like like merge and deployed. I

think we're sort of just moving up the abstraction ladder and then the human role becomes more about observing and maintaining the outer system. There's a

stream of agents flowing through like merging PRs what's going off the rails like what do I need to approve? Is there

like a learning or memory mechanism that that works? So it's kind of a hard

that works? So it's kind of a hard engineering problem. There's a, you

engineering problem. There's a, you know, there's a lot to do there. I think

we're just sort of moving up the stack.

The same transition machine learning engineers have made, right? Like I

haven't looked at a PR curve in a while.

Yeah. You used to do this stuff and now auto research can do it, right? Like I think it depends on what

right? Like I think it depends on what you define as a software engineer.

Yes, that's changing for sure. I think

every software engineer at Notion this summer went through like this um shear um one of our engineering leads at the company called it like every software engineer is going through the the

identity crisis that every manager goes through where all of a sudden they realize their ability to write code is less important than their ability to delegate and context switch and I think that is a transition out of being a

software engineer but yeah there's a critical difference to being a manager which is that like It is actually very deeply technical. The

problem of you know humans are very like like like fuzzy and you you can't like treat a team of humans like a like a rigorous system where like you know PRs like like flow through and can be in like a blocked status and then what

happens when they're blocked right with a set of agents you actually can do that and and I think it's actually there's a lot of interesting technical rigor that that goes into that. It's a

it's a technical design problem ultimately.

What is the design of the software factory that you're building? Yeah, I

mean I think we're trying a lot of different things. I mean ultimately you

different things. I mean ultimately you want to design a system that requires as little human intervention as possible but like still maintaining the invariance that that you care about. So

yeah, we're exploring a lot of different ideas there. I mean I think I can talk

ideas there. I mean I think I can talk about like a few things I think are important there. Like one thing I think

important there. Like one thing I think is really important is um having some kind of like specification layer. You

can just commit markdown files. That

works pretty well. It's nice to be notion man. I'm just saying like the

notion man. I'm just saying like the spec like the natural home for specs is notion.

Yeah. Right.

It can be a database of pages. Yeah. I

mean it needs to be something that is you know human readable and and viewable and I think that's pretty key. Another

really key component is like the the self- verification loop. You need a really really good testing layers basically. And that's a really deep uh

basically. And that's a really deep uh problem, but but getting that right, you know, and then and then it's kind of like the workflow of like what happens when there's a bug, how does it flow into the system? Like is it like a sub

agent working on it? How does it make a PR and how does that get reviewed and merge and then you know, so there's like the the flow or process.

Yeah. Cool. Uh you know, one thing we did work out before you guys came in was this demo or this agents agent demo.

So every every time we do an episode, we tried a product, right? I don't think there's ever been an episode that I haven't tried. Um when try try is a a

haven't tried. Um when try try is a a big word like since day one lane space has been on notion but this is the this is the net new thing. Yes.

So this is for kernel labs which is the space we're in. So next week we're opening applications for tenants. So

there's a web form. Let me we got this form done here. Uh so before the workflow would be I get an email then I look at the person was like I should have spent time talking to this person

then I respond they respond back so I build this. So the name it came up for

build this. So the name it came up for on its own. Can you maybe how do how does it come up with its own name?

Yeah that's a pretty apt name. It's it's

just a random it's a random name generator.

Oh okay that's funny. It just came the fact that it picked that is is kind of hilarious.

I I'm pretty sure it's just resilient collector. I I think I've

resilient collector. I I think I've never looked at the code for that. I've

never second guessed it. I think it's kind of like a Mad Libs situation.

Yeah, it's it's totally a deterministic.

Oh, I I thought it was great.

Although although when the if you use the AI to set itself up, it can update its own name. So,

okay.

Um, how did you create it? Did you just do class?

Okay, I did. Yeah, I'll say just check my

I did. Yeah, I'll say just check my inbox for applications or co-working space keep people. So, it created a database for me which I have here and I guess database is like a notion table

because everything is notion. Um, and

then whenever an email comes in like here, it just creates a new row for the person and then it uses web search to enrich the the profile. So, it kind of

like searches the web and it's like this is who this person is. This is when they want to move in and kind of updates everything else. This is I mean it's not

everything else. This is I mean it's not AGI, but to me I don't want to do this work. So, it feels like I mean it took

work. So, it feels like I mean it took me maybe like 15 minutes to set up the whole thing. Um, and I really like that

whole thing. Um, and I really like that most of the information should live here, you know? It's not like some other tool asking me, yeah, to like bring my stuff there. It's like

I would have probably already created an ocean thing.

So, most of our biggest use cases and gains are from that extra layer of human involvement in the process to make it so, right? And so, like one of our

so, right? And so, like one of our biggest use cases is bug triaging. So if

someone posts something in Slack, can you just have a custom agent that lives there that has its own routing constitution of what team this belongs to, creates a task in your task database, and then posts in that Slack

channel, right? Like that's like one of

channel, right? Like that's like one of the first things that we built internally, I think. And it's completely changed the way that notion functions as a company. Nothing falls through well,

a company. Nothing falls through well, most things don't fall through the crack. We don't know what we don't know.

crack. We don't know what we don't know.

But it's not replacing people, it's replacing processes.

Yeah.

Right. And I'm curious how you think about composibility of these things. So

the other one I was working on is like a piece filler. So whenever somebody signs

piece filler. So whenever somebody signs up as a tenant, kind of fills out the lease for them. There should probably be some agent that is like office manager agent that can handle the request, make

the lease, and then uh give them a verata access to the office and all of that. How do you think about that

that. How do you think about that feature?

Yeah. So I mean there's there's two ways you can compose. One way is by using like the data primitives. So you can, you know, you could give you have one agent uh be writing to the database and

there's another agent that's walking the database. So that's that's one way that

database. So that's that's one way that they can coordinate that's like a little bit more decoupled and works really well. Or you you can couple them. So I I think it's actually not

them. So I I think it's actually not released yet releasing it like next week is uh in the settings for an agent you can give access to invoke any other agent.

So you can have them just just uh talk directly. So is there a limit on like

directly. So is there a limit on like number of recursions or just um probably you know what I mean like you can just get an infinite loop that way some kind of yeah I think it's there is actually a number

somewhere I believe I'm just you know like someone's going to screw it up you should you should try it and see I mean everything's going to be paper clips so yeah but uh but but that's really useful

yeah so you know like I just I I helped uh someone internally the other day they had they had built like over 30 custom agents for for our go to market team doing all kinds of different things, you know, for example, like researching, you

know, like like filling information about about a customer or like like triaging customer feedback or like something like that. Literally over 30 of them. And and then and then he made

of them. And and then and then he made like a database of all the agents. And

then he was like, "Okay." And and now I'm getting 70 over 70 notifications per day with just the agents are blocked on various things. Uh and then I was like,

various things. Uh and then I was like, "Oh, okay, cool." You know, the obvious thing to do there is to make a manager agent, right? that's going to sort of be

right? that's going to sort of be another abstraction layer in between your your 30 agents. Uh so yeah, we we set them up with like a manager agent and then has access to invoke all the other agents and it's sort of like like

watching and observing them and then it sort of yeah just creates a layer of abstraction. So instead of 70

abstraction. So instead of 70 notifications per day, it's like like five and then and then the manager agent can help like debug and fix any any problems with the this is a concept like an inbox or

something like you're basically saying that they can message each other. Well,

they use a system of record which is notion.

Yes, we actually Yeah, we didn't make any special concepts at all. They're

interested notifications that I would have got.

They can just like write a task to a database that the other agents tasked to listening to or they can actually call a web hook up to the agent like they can just at the agent.

Okay. Yeah. I mean, this is something that that we're still working on. I

think we you know like like generally generally the way we do these things is you know you first make it possible maybe like a sort of janky way. So I I I think the way I set him up is like, you know, we created like a new database that was sort of like

issues that the custom agents were were were experiencing and then gave them all access to file an issue and then the manager has access to to read the issues. Um and that worked pretty well.

issues. Um and that worked pretty well.

Essentially like like give it its own like internal issue tracker just for the agents and then you know if that becomes a a concept that seems useful generally maybe we'll think of how to package it

in. But I mean generally we try to just

in. But I mean generally we try to just keep it to composing the primitives if we can. You know, another example of

we can. You know, another example of this is we have no built-in memory concept. Memory is is just pages and

concept. Memory is is just pages and databases. And so if you want to give it

databases. And so if you want to give it memory, you just give it a page and give it edit access to that page and a human can edit it. Agent can edit.

Yeah. And so that works that pattern works extremely well and and you know depending on the use case you can have it be just a page or it could be an entire database with you know or you know can have sub pages is is pretty what you can do with it.

So when I was setting this up uh I connected my inbox and it was like do you want to use Gmail or notion mail?

And I'm like I don't want to use either.

I just want you to do it. I'm curious

how you think about you know notion mail notion calendar all of these kind of UI UX interfaces notion yeah when like at the same time you have the agents abstracting them away from you in a way you know how do you spend

like the product calories so to speak yeah I mean I think it's pretty important that you don't have to use notion mail to connect to the mail capability so we can just connect to Gmail or or whatever you want uh to use

and we're thinking of the mail service as being really great to the extent that it's really agent impelled, right? So

maybe the mail app is just sort of a prepackaged agent that helps you automate your your inbox.

Yeah, the auto labeling is great. I

think the when we um integrate with Gmail, for instance, we have a series of tools available that are available via MCP or API to Gmail. When we integrate with

notion mail, we have the notion mail engineering team to build us the um exact right tools that optimize latency, optimize performance and quality. they

own that quality. Um there's product leads there that they're directly thinking about the user problems that happen in mail. So it tends to be when we build integrations and connections, we build natively first um and then

think about um extending them generally just because it's also easier um to build natively first. Um so that tends to be how we phase things out.

Talking about integrations, you prompted me so I got to ask MCP CLI, what's going on? What's the opinion? Yeah, I think I

on? What's the opinion? Yeah, I think I mean I'm I'm definitely bullish and excited about CLIs. I think there's a few really cool things about CLI. So,

one really cool thing is like um is that it's in the terminal environment. So, it

gets a bunch of extra power. So, you

know, for example, I can like like pageionate and cursor through like long outputs. Um and it has a progressive

outputs. Um and it has a progressive disclosure inherently. Uh so, you know,

disclosure inherently. Uh so, you know, you don't see all the tools at once.

It's just you see the CLI wrapper and you can like use the the help commands and and read files. And then I think the most important thing that's that's super cool is that there it's also inherently

bootstrapped. So if there's an issue,

bootstrapped. So if there's an issue, the agent can debug and fix itself within the same environment that it uses the tool, right? Like you know, I think I saw a

right? Like you know, I think I saw a tweet this morning. Someone said, you know, my agent didn't have a browser, so I asked it to make itself a browser tool. And within 100 lines of code, it

tool. And within 100 lines of code, it gave itself a little browser like like wrapping the the Chromium API. Um,

that's pretty incredible. And then if there was a bug, it would just immediately try to fix it. Right. On the

other hand, if you use, you know, if you use like the Chrome Dev Tools MCP, I've had this issue where like sometimes the transport gets like messed up. If it

gets messed up, the agent has no way to fix itself. It it no longer has a

fix itself. It it no longer has a browser. It's it's now broken, right? I

browser. It's it's now broken, right? I

think that's that's pretty fundamental.

But I would say like a lot of the the bad things about it can be fixed. Uh so

I think like the progressive disclosure can be fixed with with red harness. Like

it it obviously doesn't make sense to show it to all the tools all the time.

That's not really inherent to the MCP protocol. just like how you wrap it and

protocol. just like how you wrap it and use it.

There's many poorly implemented FCPs because we didn't know.

Yeah. Yeah. I mean it was just early like like the obvious thing is you know to start with is is to just show it all the tools and it's like okay now we have 100 tools and like the tool calling actually works so let's victim success give it a way to like filter the source

the tools. So I would say like broadly

the tools. So I would say like broadly speaking I'm really bullish on CLIs.

I'm still bullish on MCPS in in a certain environment. I think in in

certain environment. I think in in particular MCP is really great for when you want sort of like a narrow lightweight agent. Mhm.

lightweight agent. Mhm.

I think there's there's definitely a lot of use cases where where you don't want like a full coding agent with a compute runtime and also you want it to be like more tightly permissioned. MCP

inherently has a really strong permission model. Like all you can do is

permission model. Like all you can do is call the tools. A CLI is a little bit murkier. It's like can I access the API

murkier. It's like can I access the API token? Are you like properly sort of

token? Are you like properly sort of like re-encrypting the token so it can't like exfiltrate it? It introduces a lot of

exfiltrate it? It introduces a lot of like like new issues which are real and hard to solve. And MCB is just like the dumb simple thing that works and it it is pretty good.

I'll add two more perspectives, not from it working well for notion, but how notion like commits to both platforms. Notion is dedicated to being the best system of record for where people do their enterprise work. So we will always

support our MCP in so far as other people are using MCPS, right? So

regardless of our perspective, we've put a lot of effort into our MCP and we have a fantastic team that we're building um to do more there. And the second thing I'll say, I think um we all think a lot, but lately I've been thinking a lot

about making sure there's a value alignment and pricing um with capability.

Literally on the expression and needing language to execute deterministic tasks feels wasteful and requiring on a language model to interface with third party providers seems wasteful for tasks that don't

require it. and particularly because our

require it. and particularly because our custom agents are using usage based pricing. We think of pricing as like the

pricing. We think of pricing as like the barrier of entry for use of our product and we're quite committed to making sure that it's not wasteful. Um not just because it's a bad deal for our customers, but it's also bad business.

We want as many buyers. Like there's a there's an elasticity of demand. And so

if we can have our agents properly execute code that calls on CLI deterministically, it's a one-time cost, right? versus constantly having a

right? versus constantly having a language model integrate with an MCP over and over and over and paying those like repeated token fees and if it's happening outside the cash window then you're paying for it over and over and

over and it's just kind of unnecessary and less deterministic when it doesn't have to be.

Yeah, the open-endedness I think is like the main thing. It's like, well, if I could write code to just call an API, I would never use an MCP. But then you need an MCP sometimes when you know what to call, but you don't want it to

restart versus like I think the it built a browser from scratch. It's like

it's great when you're doing it on your own, but like if your customers were having your AI write a browser from scratch every time and you had to pay the token cost of that, you'd be like, "No, no, the Chrome DevTools MTP is actually pretty great. Just use that."

I'm curious, how do you make that decision? Like, should it be just

decision? Like, should it be just straight API call, very narrow? Should

it be an MCP? Should it be super open-ended?

Do you mean for when we ship notion capabilities or when we add capabilities?

You might have a capability that the only way to do is an open-ended agent like an agent with a coding sandbox.

Yeah. In notion AI they're not explically like is there ever a discussion like we're not going to ship it because we're not able to tie it down or are you happy to just like

um No. Well, I mean there are a lot of

um No. Well, I mean there are a lot of things where we choose not to use MCP because we want to add more hightouch to quality. I think search and agentic find

quality. I think search and agentic find is like the largest instance of that where we have um Slack and Linear and Jira search and notion that is not using

necessarily the search MCP functionality that is provided by those companies. And

that's because it's quite critical we think to how our agent trajectories work is for us to have a little bit more control on the functionality of the search journey. And so it usually comes

search journey. And so it usually comes from quality and there's a long tale of things and that's why we built an MCP client or an MCP server, excuse me, so that people can connect whatever they

want. There's that long tail, right? But

want. There's that long tail, right? But

we for search particularly, I would say that's like the primary entry point, but there are other connections as well that it's a little bit of secret sauce about when we are okay with like MCP functionality and userdriven off and

when we actually want to want to carry along ourselves.

I think that there's not really a conflict here. There's just like

conflict here. There's just like different layers of the stack and different abstractions. I mean, if I

different abstractions. I mean, if I were to like map it out, it's like, you know, you've got MCPs give you a way to it's a protocol for gaining access to

tools. It's an open protocol. So, you

tools. It's an open protocol. So, you

can you can easily get like a longtail many things. So, if you open up our like

many things. So, if you open up our like in the tool settings, that's not the trigger actually. That's something that

trigger actually. That's something that MCP can't do. So, if you scroll down and you and yeah, the the tools and access so you're now a connection. Yeah, MCP is a really great way to gain access to

tools. It works really well, but you

tools. It works really well, but you just looked at the the trigger Y for example, there's no trigger protocol and so those things we had to build ourselves and then there's there's some

integrations where we use MCP like so for example I think that you know the linear and the GitHub they use MCP but but the Slack mail and calendar those are actually ones they built in house and we spent a lot of

time really fine-tuning all the tools to make them really good and also like building out the triggers. It's just

like different layers of the stack. Some

things make sense at sometimes and then you know we just have to like like harness the right tool at the right time. I don't think there's an inherent

time. I don't think there's an inherent like strong conflict between these things.

Do you have a canonical representation of these tools internally where like you wrap these things together? The MCP plus the custom build.

Yeah. Yeah. We have like internal abstractions for like what is a tool, what is an agent, what is a completion call. Yeah. We even have internal

call. Yeah. We even have internal obstructions for like what is a chat archetype, whether it be from Teams or Slack.

Yeah, it's like the only way to build with with AI because everything's moving so quickly. You'd have to abstract it so

so quickly. You'd have to abstract it so that you can swap things up.

Yeah, I mean there's always a dance. I

mean, we we probably rebuilt our our framework like like I said like like five different times. Um

it's always a dance of like okay, how does this new thing work, right?

What should the abstraction be? Like

what is OpenAI giving us? What is

Enthropy giving us? um you know like trying to wrap over it. I think I think we've been pretty successful with that.

It's just a matter of like like staying nimble and making sure that you always have like the simplest dumbest abstraction you can that you know that the maps over different things. Yeah. So

so we have like a tool integration abstraction for example and then MCP is like a type of integration.

Yeah that's that's one of this might be a big ask u but I'm going to try uh which is you said you've said multiple times you rebuild a few times like five times through I don't know if the what the right number is. Is there

like a brief history of what was the each rebuild doing? And yeah, I know it's I can try to do that. I mean,

yeah, there's you need you need to rag over archaeology. Yeah, I mean the first

archaeology. Yeah, I mean the first version the first version that we started building in like late 2022. Oh

my gosh. Well, there have been many versions actually. Okay. Well,

versions actually. Okay. Well,

the highlights the the like wow.

The the first version we built was actually a coding agent. Yeah.

So, we're like, "Oh, instead of building tools, let's make everything be JavaScript and then we'll just give it JavaScript APIs and it'll just write code and that's how it speaks the tools." Um, but at the time it just

tools." Um, but at the time it just sucked at writing code. It wasn't that good. Uh, so then we moved to uh more of

good. Uh, so then we moved to uh more of like a tool calling abstraction. A tool

calling didn't exist yet. So, we created this whole XML representation. And a big a big learning

representation. And a big a big learning in that version is we were catering way too much to what made sense for notion and notion's data model versus what the

model wants. So as an example, we

model wants. So as an example, we created this whole uh XML uh format that can losslessly map to notion blocks and the transformation between them is super easy to do. Uh and then we created these

sort of like mutation operations to to edit pages. Um, but it sucked because

edit pages. Um, but it sucked because the model didn't know the XML format and also the you had to prompt it in.

Yeah, you had to prompt it in and the just weren't convenient. And so, yeah, we're like, okay, well, it has to be markdown. The model's no markdown, you

markdown. The model's no markdown, you know. So, we did a whole project around

know. So, we did a whole project around basically creating a notion flavored markdown where, you know, the whole goal was like it has to be just simple markdown at the core and and then we can add some

enhancements and it doesn't have to be a a full lossless conversion. That was a big one we did. and and then we did a whole similar learning to uh the the database layer. So so so to querying a

database layer. So so so to querying a database I mean in the notion API the way you query a database is there's a crazy JSON format and it's you know kind of limiting but it maps nicely to like

how we represent things internally. We

scrapped all that we're like okay let's just make it SQLite everything is a SQLite database you can query it just like a SQLite query and the models are super good at that. So

give the models what they want.

That was another one. Yeah. Yeah. Give

models what they want. I mean that was I would say that was a big learning is just you know really be be savvy and really careful thinking about what the model wants in terms of you know its

environment and and and cater around that and really try so hard not to expose it to any complexity about your system that that's unnecessary.

Notion's underlying database is Postgress right not SQLite.

Yeah.

So I don't know if there's any mismatch there. That one was kind of a fortuitous

there. That one was kind of a fortuitous thing because we actually already um had a big project uh going where so so we

had this um when you query a notion database it's actually querying this like uh cluster of SQLite databases.

That's something that we had already been working on even before the agents came around.

Yeah. You know you guys had a fantastic blog post about it and like it's it's actually really good database engineering knowledge to have that from you guys because where else will we get it? Yeah. Yeah. It's it's a crazy

it? Yeah. Yeah. It's it's a crazy engineering problem when you want to have like millions and billions of tiny databases or where where some of them are tiny but some of them are are very large and you want everything to be very fast.

Yeah. And also like not that hierarchical sometimes, you know, uh so somewhat of a graph.

Mhm.

I do like that history because I think that shows the evolution that you guys went through and the work that went into it.

He just ended you a year and a half ago.

Oh, okay. Okay. Well, let me I need to hit continue.

If you're curious, I mean, we can keep going. I'm just saying like that's

going. I'm just saying like that's really that's another one. Yeah.

Well, no cuz there was tool calling and then there was research mode which wasn't a fully agentic tool calling. Um

then we moved away from fot prompting entirely to tool definitions. Um and now we're thinking about agent 2.0.

So no fus prompts ever, right?

Uh okay. No.

okay. No.

I don't know if but yeah, that kind of went away. It's an interesting thing,

went away. It's an interesting thing, right? Yeah. I mean

right? Yeah. I mean

they just instruction follow really well. I I would say there's been like a

well. I I would say there's been like a general arc where you know it's like you gradually strip away everything and it it looks more AGI like and so you know it it it started out as like it's a one

shot one prompt there's few shot examples and it became like okay actually let's give it let's give it tools but it'll still have few shot examples and then it became actually like no no let's just give it a whole

bunch of tools. One big big shift that that we I've been working on recently that's about to ship is um you know what happens when you have a lot of tools.

Yeah.

So then yeah so then a progressive disclosure becomes really important. So

you know we were we sort of hit a bottleneck where our our agent worked really well. Um we hit a bottleneck

really well. Um we hit a bottleneck where um it it became pretty hard to add new tools and and we became sort of worried about it like like breaking the model. looks like. Okay, someone

model. looks like. Okay, someone

No, I just heard it was like saying hello was like thousands and thousands and thousands of tokens.

It was really slow.

I can see you're the efficiency person here.

It's it was too many tokens, but also it's a quality issue because it meant that like any engineer could introduce this this new tool for some like like niche feature and it would kind of like like nerf the overall model by like causing it to call the tool too much and

stuff like that. And so um it uh yeah, so we uh we had an effort basically to to make our harness uh implement progressive disclosure in a nice way. M

that's a big shift you said earlier like everyone says reasoning models was the big shift like what's more there when we went away from few shots to describing the goal of the

tool in like goal driven basically moving from a DAG to like a a true system with feedback that's when we could distribute tool ownership to the teams much better because when it was

all a few shots it was everyone truly editing one string and things would would compete and like the order there were all this all these papers about oh you know Not all context is created equal. The higher up it is in your

equal. The higher up it is in your examples like the more the model listens and we're trying really hard to like fight against the order and the selection of the fuchan that really had to be a center of excellence and it

didn't scale with the number of people for the need the company had. It was

really just five or six people that were allowed to even touch that or had to approve it rather in our codebase. And

then now we can actually with the right email setup distribute um so that everyone owns their tool and their tool definition. And sometimes we have crazy

definition. And sometimes we have crazy things where like we write two tools that have the same title and the agent crashes and stuff like that. So like you know there are issues actually believe it or not um Enthropic couldn't take it.

Sonic couldn't handle two tools with the same name in OpenAI GPT 5.2 was like I can figure this out. So that was an interesting one that we learned by accident through a a SE.

I mean then you know the underlying representation is that's a dict right like that's a sik.

Exactly. Exactly. Um but so that was like a big shift for the company in velocity not immediate because the AI team that was the center of excellence team that owned you know that one file

of fuchot promps had to become a platform team overnight and that wasn't natural but I would say that in terms of like the velocity of how we contribute to the agent beyond coding tools obviously

being a big velocity lever um being able to distribute tools and not have to all collaborate on like one very select string of system prompt is truly I would say the biggest lever on how we've scaled.

We're just fighting to keep the prompt as short as possible now. And then yeah, it's in the latest version of the agent.

It's not in custom agents yet, but it will be like like next week, a week after or so. Um there's now like over 100 tools just for all all the crazy notion stuff. So we're able to really go

notion stuff. So we're able to really go deep and like would you list those tools publicly? Is

this like IP or uh No, no, no. It's it's totally public.

Uh you can ask you can find just ask you can just ask the agent and and it'll tell you we're going to post a bench. I

mean like you got we don't think our system prompt is our secret sauce.

Yeah. Mhm. Great.

We don't try to hide the tools at all. I

think it's I think it's kind of important actually as an operator, you know.

Yeah. As a power user, I want to be like, "Oh, I can do this. This is this."

Great.

Yeah. Yeah. I mean, one thing that one phrase we say internally a lot is to to teach at the top of the class, you know, want to build like like customation is kind of like a power tool. I mean, we try to make it as easy as possible to

set up, but we want it to be pretty deep and sophisticated. And I think a huge

and sophisticated. And I think a huge part of that is the operator needs to be able to interrogate the way the system works. And a big part of that is like

works. And a big part of that is like what are the tools? How do they work?

You know, like like how should I prompt it to use the tools in the right way?

I'd actually say we don't try and make it as easy as possible to use because the more we do that, the more we abstract away that interpretability that Simon's talking about that basically nerfs the model or nerfs the agent from

being super capable. So a huge I would say turning point. I can think about like the week and a half that we all came together on this as we were building custom agents was that alignment that we're not trying to build for everyone here. We're not trying to

build the model that um or build the user experience that anyone can figure out how to use cuz the more we do that the more we just diminish its capabilities and that was a big you know everyone in a couple Slack messages

aligned on that that actually made us all work faster again right cuz we all were like more centralized on who we were building for.

What does the metaprom generator look like? Okay. So, I looked in the system

like? Okay. So, I looked in the system prompt that it gener for example uses emojis. That's not a you know obvious

emojis. That's not a you know obvious thing to be doing.

Wait, did you just ask it what's your system prompt? Oh, no. This is how to

system prompt? Oh, no. This is how to generate prompts. The pros set up. It's

generate prompts. The pros set up. It's

a set.

Yeah. Well, well, so this is actually just the agent. So, so one thing we did that that I really like with custom agents is it can set itself up. So, we that only gave it access to use the tools it has

access to like send your emails or whatever. Uh, but it has more tools to

whatever. Uh, but it has more tools to set itself up and to debug itself. And

so when you ask it to write a system prompt, it's just your agent itself is doing that.

So this is just the model preference.

You're not really injecting into the model too much.

I think what makes a good custom agent and and things like that. And then and it's really nice too because like if it fails, you can ask it why did it fail and then say okay update your instructions so it doesn't fail again.

Obviously we should build product of self-healing. That's that's next on our

self-healing. That's that's next on our road map. But um it actually it creates

road map. But um it actually it creates a nice system. Yeah, we do essentially give it like a development guide.

Here's, you know, here's how to make a custom agent. Here's how to like like

custom agent. Here's how to like like help the user test it end to end, you know, to to help them gain confidence that it works, stuff like that.

Mhm. Yeah. Yeah. The fixing thing worked. I mean, it wasn't automatic, but

worked. I mean, it wasn't automatic, but I I miss set something up and then there was like a fix button and then just Yeah. Yeah. One thing

Yeah. Yeah. One thing

agent it's it's actually it's an interesting sort of permission problem. So like the thing about custom agents that is that by default it has no permission to do anything and then you have to explicitly

grant it all its permissions and that's what lets you trust it can work in the background right like you can know like oh it it can read my email but not send email okay I can trust that right if you let it fix itself

you know you're you're breaking that that there it's not allowed to edit its own permissions but so you know in the current product you can sort of click a button to fix but now you're entering sort of an admin mode where where where you're in a synchronous chat And you can

and you can see what it's doing.

Yeah.

And it and it confirms before it changes.

The thing I really like that most people don't do is like the editing chat is the same thing as the using chat. Like you

can message the agent to both edit it and use it versus a lot of other products are like I think that's really key. I think

I think a lot of designers will feel so happy you said that cuz we spent we we called this Flippy. Um

what is this? What do you mean this this? Well, yeah. So if you sort of if

this? Well, yeah. So if you sort of if you close that and like open settings, you can see sort of Yeah. This is we we call it flippy because you know we started with sort of like the settings

were the sort of the main page and then you can test the agent. The agi pill to think about it is like oh it's just the agent. Everything is the agent. It can

agent. Everything is the agent. It can

set itself up. It can test itself and it can run the workflow that they want to run. Uh so we flipped it. So the main

run. Uh so we flipped it. So the main view I was looking at is the chat and and then the settings is more just like a side panel at sort of previewing the changes that it's making. So you can introspect on them or or you can also

make changes manually if you'd like. But

but we want to design the experience from the get- go so you don't have to ever any of the settings manually. You

can just talk to it.

And the inside baseball is like how this works was probably the launch blocking part of this build. Um especially

because we had a lot of early adopters that were used to the old way. And

that's like the benefit of adopting in public. But then changing how people

public. But then changing how people think about setting up custom agents when they already had this flow in and of itself was difficult. Um,

I mean that's really fun because the we we ended up sort of painfully delaying the launch by a few weeks. Yeah, definitely like like a month or so. Um, but the whole team

was super enthusiastic about it though cuz it was just so much better. It was

like, oh yeah, obviously you have to chat with itself up and everyone was super bullish on that.

It was like like painful for a second, but then everyone was like, "Right." And like back to, you know,

"Right." And like back to, you know, organization design, which I probably care about more than Simon, but like the people that built this are three engineers from three different teams because we're like, "We need to launch this and we need to fix this." And then

we've just built a company where then we just put people on it and no one complains. The manager doesn't complain

complains. The manager doesn't complain and we were able to unblock and just ship it.

Yeah. Yeah. But being in a failure chat and asking it to just fix yourself is amazing versus I got to copy this and put in the settings chat to do it. So

yeah, there's an interesting like trade-off in there that that we're trying to explore which is you know we want to be like a business enterprise safe agent where you can delegate something and and trust that it's going

to work. But also we want to get some of

to work. But also we want to get some of that sort of bootstrapping power that that you feel like when your coding age is making a browser like for itself, right? There's something there. I think

right? There's something there. I think

that's that's really important. So,

we're trying to sort of navigate that that that trade-off and try to get you both.

Now, it's free.

Yeah, it's amazing. Uh I'm worried about when

it's amazing. Uh I'm worried about when I have to start paying. How do you think about So, you have notion credits as a payment for this, which is like separate from the usual tokens uh that the model

generates. How do you design pricing,

generates. How do you design pricing, value based pricing based on the task and things like that?

So, they are um the credits and payment structures associated with the token usage. reason that we had to make it not

usage. reason that we had to make it not just throughput of tokens is that it's not always priced that way. Like our um fine-tuned open source models are served on GPUs, web search is priced differently. You know, if we were to

differently. You know, if we were to host sandboxes, those are priced differently. So, we had to think of an

differently. So, we had to think of an abstraction above tokens. And it's also not just tokens, it's the token model um and serving tier trade-off, right?

Because we can have priority tier processing, we can have asynchronous processing, the cache rate could be different um depending on who uses it when, right? And so we wanted to um from

when, right? And so we wanted to um from the get-go commit to making sure that customers were getting the fair deal, not necessarily that we were making a ton of money off of it, but that customers were paying for what was

reasonable. That's the fundamental of

reasonable. That's the fundamental of where we started. And also, you know, we're selling enterprise SAS, so if we sell credit packs, then you get discounts if you're an enterprise and you buy a certain amount of credit packs and things like that. So it also just helped the sales motion um work a little

bit easier. So that's the answer on the

bit easier. So that's the answer on the abstraction of credits to dollars. Now,

was the question how we decide how to price it or Yeah, like I mean I think there's all tokens are not made equal, but we obviously get charged mostly equal. Like

you can ask uh codeex to create you a dumb tool for like I created one for a Starcraft 2 land for people to like find the game. Uh but then people create it

the game. Uh but then people create it to build features and like billion dollar companies, but the token price is the same. Yeah. Like for you, I can ask

the same. Yeah. Like for you, I can ask this to update my favorite recipes doc and it'll do it. Or I could ask it to like respond to an email from an investor and like the value is like very

different, you know, and you could charge more, but you're not necessarily doing it. So I'm curious if there was

doing it. So I'm curious if there was any discussion.

I think that um that's not where the market is right now. Um number one, the second reason that we're not doing that is it ended up being kind of complicated to figure out what was complicated or

not. So we at first were like let's just

not. So we at first were like let's just charge on agent runs and you know you went through all the different versions that ultimately just brought you back to a lot of complexity that map directly to token throughput and so it it's also

just simpler um it's quite difficult um to build those pricing systems and um I actually think that one of the biggest reasons we want had usage based pricing for this capability is we've had our

core agent for a while with a model picker and there were certain models um or certain functionality that we had margins to maintain And if we wanted to ship this functionality uh you we couldn't afford it. It would bankrupt

the company if we let for instance like autofill or the database autofill feature will soon be agentic. That will

be associated with usage based pricing because if every single autofill action was an agent running on OBUS on every single database cell it would be billions of dollars, right? And so we

had to find a way for the customers that wanted to do more and wanted to give us their money and pay more to find the outlet for them to do it that we didn't have to apply to the lower end of the curve. And also not all knowledge work

curve. And also not all knowledge work is equal. Like there's different points.

is equal. Like there's different points.

A lot of the agent workflows here really saturate model capabilities. Like you

don't need a complicated model for it.

And so charging based on token usage um we couldn't just decide for you that you wanted your email client to be dumb or not. like we want you to decide if you

not. like we want you to decide if you want to have Opus auto triage all of your emails.

We will actually give you nudges in the product to rethink if that's the right choice. Um because also not every user

choice. Um because also not every user um you'd be surprised in user interviews people be like, "Oh, I didn't know that." So now we actually have a little

that." So now we actually have a little hover that tells you like if it's expensive or not.

Yeah. I mean it's also slower. So the

thing that's interesting is like people don't care about speed in custom agents.

And so the incentive of like uh ha coup being faster, people don't care when it's asynchronous. Um and so we want to

it's asynchronous. Um and so we want to only provide the service of extra extra benefit that people want and the best way to do that is to incentivize them because it's their own money.

Must be confusing for people that are not familiar. It's like why is there no

not familiar. It's like why is there no 5.3? You know, you open this thing and

5.3? You know, you open this thing and it's like is there something missing in my fault? Not their fault. Yeah. That's

my fault? Not their fault. Yeah. That's

just the world we live in now.

Yeah. I mean, it's just randomly John's point, too. It's like cloud had that.

point, too. It's like cloud had that.

I mean, but auto is heavily I think what's actually been hard for us is to convince people that auto is not just our cheapest, dumbest model, but actually the model that's best for the task that you want to do.

All right, Steve.

I'm in.

Exactly. Nice. Um, and a lot of our job is actually figuring out auto because it's this is the agent lab. Every agent lab has an auto.

Yeah. And

that's the job.

Exactly. Because if you think about like I said I come from Robin Hood like you could spend a lot of time keeping up with the markets or you could have a auto investing right and you can have an

index fund or you can have robo adviser robo advisor. So like at a certain point we also can be robo advisors and like we have a lot of people figuring out what model is best

for the right task and right now we're not using auto as a as a margin maker.

We're just using it to kind of reduce stress. It's not opus, that's for sure,

stress. It's not opus, that's for sure, because a majority of the tests people are doing aren't opus level um intelligence.

The other thing I would say is the um you know, unlike a lab, we aren't fully incentivized just for you to use as many tokens as possible. We're actually

really interested in getting you the right tool for the job. A lot of the time, the right tool for the job is actually just writing code and not even using agent at all. So that's that's something that we're investing in a lot is like, you know, imagine your your

agent can actually automate itself out of a job, right?

We would love if that were true.

I feel very strongly about this because I don't necessarily feel like that's the SKS that Frontier Labs give you. I feel

like they are just getting more and more capable and more and more expensive, which is fantastic for the use cases of when people want to do really complicated things on notion. Um what's

difficult is like that market that I think right now is no man's land of where reasoning models were six months ago that the nano haikus etc haven't caught up to because now we're just

paying more for those um for like extra capability that we didn't necessarily need and so are our customers and um labs aren't necessarily incentivized um right now with how few players they are to be meeting the market everywhere.

They just need to be the cheapest. They

don't need to be at value that the customer wants. And if no one's cheaper

customer wants. And if no one's cheaper than them, then they're the cheapest and that's good enough, right? And so we're doing a lot to make sure that we have the right optionality um to switch between models and also invest in open

source because the open source models actually are um getting to be the place where reasoning models were 3 four months ago and um that's what's filling that gap right now. So you'll see we offer mini max and um we're

collaborating a lot with different open source labs to think about notion's last exam and how they can do better on these types of tasks so that we can offer them for that intelligence to price to

latency trade-off because you know in that triangle of intelligence price um intelligence price and latency excuse me um users get to choose where they are

but right now um there's not the whole triangle isn't filled with models right And the more that different models fill that triangle, everyone's clustered in capability or everyone's cluster. I

mean, haiku is not that much cheaper. No

one's really in the middle. Like people

really tend to cluster around too. Like

this is really capable and it's really fast, but it's really expensive or whatever, right? And so we just want to

whatever, right? And so we just want to make sure that that triangle is filled.

Um, and we want to offer the models that fill it and we want to um get guide users to understand when they need it.

Um, which one I mean all I'm hearing is that someday you're going to train your model. You

have lots of tokens.

I don't know if What do you mean by train your model? Train your own money to train a found. I mean,

you go raise it.

Yeah, you you can raise it.

That's your job, Simon.

No, I I don't think that that needs to be our core competency.

This is usually the the thought process that leads to like, well, no one else is doing it. We'll take a crack, you know?

doing it. We'll take a crack, you know?

I think I'm Yeah. I mean, I feel like to the extent that we do anything like training and the the area I'm actually most excited about is um less of like one big model for all the users, but

like as as it becomes more possible to do, you know, to make like a specific fine-tuning that's like really knows your context of, you know, your company, the people that work your company, what's going on. I think that's that's

pretty interesting because if you if you had a model that really knows your company, I think that would be like a huge quality uplift. We actually have some enterprise vendors that kind of ask about this um along with bring our own key like if I have a model that really

understands like my enterprise that we're training for all these reasons.

These tend to be like quite large institutions thinking about how to let people bring their own models but those models have to function with like right understanding how to call our tools and

that's where again having um more public system prompt is like beneficial to notion right um we want all models to plug into notion as as as well as they can. Um that being said like of course

can. Um that being said like of course there are certain aspects of notion where we do fine-tune and do reinforce and fine-tuning on our own capabilities.

Um but that's not necessarily trained on user data. Um you don't need that that

user data. Um you don't need that that much data um in the first place. And

that's where when we have like a data scientist and a model behavior engineer really understand where the capability gap is. That's when we invest there.

gap is. That's when we invest there.

I personally burned a lot of time trying to train models. Uh and it's tempting right? It's so training retraining every

right? It's so training retraining every day.

I was doing crazy amount. Yeah, I was doing a lot of different things. Um, and

I was the budget person that came and pushed out and I showed up and I heard that that was happening.

Time out.

You know, like a a funny thing that a sort of an arc that like looped on itself is uh you know, back when I was doing tons of training stuff, it takes a long time to do any kind of training run. And so you end up operating like

run. And so you end up operating like like 24/7 around the clock. like it

becomes very important that before you go to sleep like everything is watch the experiments are started and then as I stopped training that kind of went away but now the coding agents have totally brought this back so now every night before I go to bed I'm like

okay did I start enough agents you know to get them done I get everything done so it's yeah like you have to try polyphasic sleep so you can wake up every day

yeah we uh yeah I have not gotten there yet but my goal these days is just to before I go to bed the agents are running and I'm confident ident that they won't be done by the time I wake up.

Really 8 hours.

There was a I won't say which coding frontier lab, but there was a point where he had like outlived like the thread length and context length that that coding agent provided.

And I DM you DM'd them being like, hey, I need I need more and rep DM'd me directly and they're like, is Simon trying to prove string theory? Like what

is he doing?

Yeah, I I had a single coding agent thread going for I think it was like 17 days.

Uh pretty much continuously.

Don't they just compress? I mean

Yeah. Yes. It it was actually just a bug. It was a harness bug. Yeah. It had

bug. It was a harness bug. Yeah. It had

done compaction like a hundred times probably or the other thing that um reminded me about fine-tuning that I think you and I have aligned on is that our tools change really frequently and right now we spend

a lot of time rethinking and building tools for capability and fine-tuning a model um to understand your tool like we don't have legal expertise or coding expertise. So if we were to fine-tune a

expertise. So if we were to fine-tune a model, it would either be expertise about the enterprise and you know we have ZDR no data retention offerings for those enterprises. So we'd have to

those enterprises. So we'd have to really rethink how we structure if an enterprise wanted to opt into that or it would be fine-tuning and better capability on navigating our tools. That

doesn't match with the velocity with which we create new tools. And so it would actually really slow us down um to have a model that was fine-tuned on our tools because we'd have to retrain and cut a new model every time we did that.

And that's not how we're set up right now. Um, particularly with the way that

now. Um, particularly with the way that we're changing our I guess we could fine-tune a model to like search for tools. It's just the the amount of time

tools. It's just the the amount of time it takes to do that, ship it, have the right system. You're basically making a

right system. You're basically making a bet against a frontier capability not serving that and the time it takes you to build it. And that that time lag hasn't happened for us yet. It hasn't.

Yeah. It's just the wrong trade-off. I

think it's just like you want Yeah. We

literally change our tools every single day. And if we notice an issue, we'll

day. And if we notice an issue, we'll we'll we'll fix the problem. I think a a good way to think about it I think is pretty fruitful is like don't focus too much on training. I would think of that as like that's an implementation detail

like what's the outer loop right like if the outer loop is you have a model and then some harness or or system where it's interacting with the system that needs to work and you know if there's a

problem the way to solve the problem isn't necessarily to train a model it's like oh maybe there's just a bug in one of the tools right and actually 99% of the time it's a bug in one of the tools and so just fix the bug and then the

outer loop thing that's really fruitful to think about is like how can you improve your your velocity and robust and making really good tools, making a good harness, you know, like like verifying it works.

The one place that we do invest more in model training now necessarily though is actually in retrieval because um we're at a point right now in our business and enterprise or AI enabled plans where the search load and the search traffic, a

majority of it's coming from agents, not humans. And so for every query that's

humans. And so for every query that's hitting our elastic search or vector indices, they're not coming from humans.

And the queries are structured differently and what's returned has a different requirement. positional

different requirement. positional ranking matters less, but top K retrieval mode matters more, right?

Isn't top K a form of position?

Of course, it is, but um when you're trading on like click-through rate, it's really, you know, number one through number six is very different that it needs to be in the top 100.

Like the slope is just higher.

It's a different optimization function for retrieval um model. Similarly, uh

what snippet you include matters more or less, right? So we are rethinking a lot

less, right? So we are rethinking a lot of that functionality um to work with how the agents like to write queries and how um they want to uh receive information. So we are doing like

information. So we are doing like another kind of reinvestment into rethinking not only search for um how do agents do searches versus how humans do searches. Um but we're also investing in

searches. Um but we're also investing in like indexing different things now because uh how are how do you index uh the setup generator for notion agent? it

kind of breaks our block model entirely.

Um where all blocks are nested in each other. Same with meeting notes. Um and

other. Same with meeting notes. Um and

so we do we I mean so we're hiring ranking engineers and model training engineers but it's primarily on ranking.

Yeah. Does ranking map to Rexis for you?

It does. Right. Recommendation system.

Yeah. Um yes.

Right. Okay. Um same same this but I'm trying to promote Rexus more in general because I is weirdly unpopular.

I don't know why. Um but the other thing is that like I I was just talking about this with a peer like how much is ranking important versus like uh being able to do parallel exhaustive queries,

right? Um so

right? Um so they're both important but like they're both two tools to the same user outcome or the same agent outcome, right? And so

um that that's something that we're also rethinking a lot even on we just did an experiment on um notion ranking at this point um for notion retrieval vector embeddings are less and less. Did you

see that?

Yeah.

Notion just notion over to night mode.

So long it became dark mode.

We're working the night shift for you.

Right.

Looks pretty new. I'm not seeing any bugs.

You know I worked on this like parallel search thing where you you fan out to eight different queries, right? And so

you actually need to use the model to work on query diversity so that you get maximum search space.

And so like the people that are working on um ranking and retrieval are the same people working on what query generation is. It's all one journey. We call it

is. It's all one journey. We call it agentic find. And we're actually

agentic find. And we're actually realizing for instance that it's less about selection. Like we don't spend a

about selection. Like we don't spend a lot of time trying to optimize what vector embedding we use anymore. That

was a period of time, but that's just not the right level of optimization.

Right.

Okay. Uh we've gone long. I have to talk about motion meeting minutes and then we'll we can call it there. Uh you you you just have a lot of comments. Uh you

uh I don't know where you want to start.

Um is it the audio side? Is it the summarization? Yeah. as sort of like

summarization? Yeah. as sort of like what makes it work or No, just like anything sort of interesting technically, right? Like I

think you had you had some uh book points. I always call these like check

points. I always call these like check marks along the way. When when a guest says something that they want to return to later, I just like check marking and like, okay, we'll go back to it.

Meeting notes was one of those things where at first we were nervous that we'd have to teach people a different way to work and we were nervous that that was a lot of user friction. I think one of the reasons why I mean they're one of our

biggest growth levers. I think they're one of the most like in terms of verality of adoption and retention quite strong. Um and so we've invested more

strong. Um and so we've invested more and more as we did that. I think what's really powerful about it is again notion is the system of record of where and how you work. The way that I use meeting

you work. The way that I use meeting notes is every oneonone meeting I have is meeting notes. When I do my performance review for myself, my self-re primarily look at all my conversations

with my manager and like write up what I did this year, right? because if I didn't talk about it in my one-on-one with my manager, it probably wasn't relevant for my performance review. So,

it also just adds a ton of signal on prioritization that's really helpful for a good system of record that's really helpful for like our agent. It's also

like caused a lot of scaling for search and for the agent. Um, and you know, it's it's just an explosion of content when you have transcripts like that. Um,

how we do compaction, a lot of that was triggered by meeting notes passed into context, things like that. Um, so it's been a good impetus for us to think about longer form um, content when you think of it as like a priority

primitive, but it's been one of the most powerful signals for our agent. Um,

because it unsurprising, right? Like you're

unsurprising, right? Like you're capturing a whole new thing.

So it's like our own data like we want users like are they creating their own data flywheel, right?

Like it serves me to prefer notion uh to put all my stuff because it has my other stuff.

Totally. I mean, the way that the way that like our team's run right now is, you know, there's a custom agent that does a pre-eread before standup. It

looks through all of Slack and GitHub and just says, you know, it it it creates a summary and it creates a meeting note and it says, "Everyone do this pre-eread." Then we just press

this pre-eread." Then we just press play. We have the meeting. We talk

play. We have the meeting. We talk

through the pre-eread. We talk about what needs to happen next. And then we have a custom agent integrated with our calendar and triggers that then files tasks for tomorrow or today based on what we spoke about and um sends off

Slack messages that we decided in the meeting needed to be follow-ups. like

our meetings are hands-off keyboard and we're focused on um the root of the problem, not the bookkeeping around the problem.

One thing that the meeting added recently that was that have been blowing my mind is they we made it so it actually when it makes a summary will actually appment mention the people that were referenced in it. So I I I now get

notifications whenever someone talks about me.

I feel like that one it's like it's like oh you know Simon is working on this.

Okay.

It's actually amazing how because then I'm like oh okay cool I'm going to go talk to them about that. What what if there are two assignments?

Um, no wait. So it's powered by the agent.

no wait. So it's powered by the agent.

So it's doing agentic. So if you look at it thinking I don't know if this is shipped yet.

It will be. When you look at it thinking when it's doing the summarization it's saying figuring out who Simon is.

Most probable Simon.

And we also have like a peopleto people similarity cache and stuff like that on the attendees. Like there's ways that we

the attendees. Like there's ways that we sort of like we also like generate a profile for each person and like and use that. But I mean of course I can get it

that. But I mean of course I can get it wrong but the goal is for not to get it wrong. Meeting notes is just like the

wrong. Meeting notes is just like the agent primitive packaged on top of a transcription primitive and then a vertical team. It's probably one of the

vertical team. It's probably one of the only teams at notion that's completely a vertical team around quality and product like UX design because it's still a tiger team um with a fantastic manager

Zach that joined recently um from Ember but um Zach Tatar. Yeah.

Zach Tatar. Yeah.

Yeah. I uh chatted with him when he was talking about who's working on Ember.

Yeah. So he's he's managing that team now and thinking about it as data capture. That's what MIDI notes is is

capture. That's what MIDI notes is is data capture kind of reframing um where MIDI notes are valuable as a data capture problem and then working inside um like the summarization used to not be

agentic. Now it is because it does all

agentic. Now it is because it does all the things like figure out who the right Simon is and one day you can have a custom agent directly integrated in it that knows like what task database the meeting is referring to and as you're

having the meeting perhaps update the task and things like that like there's a there's a lot of that experience of where we do our work in meetings that we want to invest in making more seamless.

Yeah. Uh, open eyes doing new hardware.

Uh, would you ever ship one of these?

Yeah, probably not.

But, you know, this is meeting notes in person.

Yeah. Yeah. I mean, I'd be excited about I mean, I'm excited about that that product category in general for sure.

Yeah.

I think it's like it's a it's a mechanism and it it one of those needs to work really well with notion. We

would partner with whoever is building one of those.

B they they bought Amazon. I don't know.

I can't refer you. And there's like there's some wild companies doing like really cool things that come to our partnerships team that I like to sit in on the demos of of wearables. I always

like to sit in on the demos because I think they're pretty cool and all of them want to make sure not just notion but like you can imagine the ones that talk to you um being able to do search and build context. So like if you're

entering like a conference um being able to like do like look at your CRM and do things like that um and you can utilize the notion agent to do that. So we are in like the very beginnings of those partnerships. I think what's unique

partnerships. I think what's unique about that particular technology is it goes against what I talked about with custom agents right now which is the more simple it is the harder it is to have like advanced controls over its

capabilities right and so that would be a great investment for data capture but not necessarily like our agent who's workless it's a little bit of a different slice of the problem I would say like that's going to be deeply personal like like

your company's not going to force you to wear a wrist wristband right I think it's good to hear that for me from you yeah the the CEO is going to force everyone to respect. I mean, the slice of the

to respect. I mean, the slice of the problem that that we care about is like, you know, can the company have all the context of what everyone said at every single meeting and then use that to to to derive value for themselves.

That kind of reminds me, I remember once you very strongly reminded me, our job is to not make the best harness for agentic work. Our job is to be the best

agentic work. Our job is to be the best place where people collaborate. It's

like our job isn't to build the best wearable to capture meeting notes. Our

job is to build the best place where meeting notes live, right?

Yeah. So, basically, you're saying everyone else can just pipe to you and that's fine, right? Yeah.

Yeah.

Yeah. That's that's a reasonable thing.

All I will say is that people there's people walking around with notion tattoos on them. They they'll wear Notion anything. So, just I don't know,

Notion anything. So, just I don't know, do a limited run.

Yeah. Yeah. No, I mean, well, we have such understated swag that the idea like our swag has so few Notion logos on it. The idea that people have notion tattoos is pretty antithesis to

our design principles. That's pretty

funny.

Do you have one?

No. No, they're not. I do not have notion that do. I've I've seen them.

Yeah.

Cool. Uh well, thank you so much. This

is such a great deep dive. Actually, the

chemistry between you two is amazing.

Like I I can't believe like we work together a lot.

Different jobs work closely.

Yeah.

That's it. Yeah. Thank you. Thank you.

Thank you.