State of Agentic Coding with Armin and Ben

Welcome to where we reflect on things happening in the AI coding world and uh question it and think about it

and ask um questions. Armen, Armen, who are you? What what what what are you

are you? What what what what are you doing here? I'm a resident vibe coder at

doing here? I'm a resident vibe coder at Arendelle. I fell into an authentic

Arendelle. I fell into an authentic coding hole in May, April, May this year and I haven't fully recovered from it.

But in a prior life, I worked for 10 years at Century. we shared some time over there. Uh, and I've built a lot of

over there. Uh, and I've built a lot of open source software. So, like prior to me falling into my AI hole, I actually did serious engineering for probably 20

years at this point, mostly open source.

So, I built some Python libraries. Most

well-known ones are Flask, Gingsha, and a few others. Um, but yeah, as of pretty much exactly nine months ago, I feel like I'm writing more and more code

passively with agents. Cool. That's me.

>> Nice.

My name is Ben and uh I also worked at Sentry for 10 years. Who would have thought? And that's where uh I know

thought? And that's where uh I know Armen from. 10 years is a long time by

Armen from. 10 years is a long time by the way. I've I've been reflecting on

the way. I've I've been reflecting on that long time.

>> It is a long time. So Armen and I both worked together at Sentry. Armen's done

more in Python land. I'm more from JavaScript world. I wrote a book called

JavaScript world. I wrote a book called third party JavaScript that came out 12 years ago in 2013. And uh now I've got like a an AI startup called modem. We're

building like an agentic PM and I I would say yeah been doing the AI coding thing starting with starting with co-pilot

when that got shared with everybody and never really put it down and then you know messed around with cursor wins surf or whatever but I don't even think this is like an AI coding thing. I think that we're just like we're just talking

about, you know, AI in software engineering is dominated by people with a cursory understanding of what's happening. And I don't know, we had this

happening. And I don't know, we had this naive idea that we we could just talk about it and maybe it'd be interesting.

Maybe it won't be.

>> Let's see. Okay, so we're recording this. It is the week of December 1st.

this. It is the week of December 1st.

Last week there were, was this last week? Say, I don't even know. Like time moves so fast. But I

know. Like time moves so fast. But I

think Gemini 3 came out, Opus 4.5 came out. I I want to be very

out. I I want to be very >> It's two weeks. Okay. Time flies.

I'm going to be very upfront and honest, which is when like new models come out and everybody goes crazy and shares, you know, like, wow, I built this and you got to look at this and look at these

benchmarks. Like I got a lot of things

benchmarks. Like I got a lot of things to do. I I don't like drop that and pick

to do. I I don't like drop that and pick it up and use it right away. So, I feel like I've been experiencing this these new models as a witness to what people are saying. And I don't even know how

are saying. And I don't even know how much I should pay attention to this, right? Uh

right? Uh >> it's me.

>> I mean, I use them. I just I I basically no longer have opinions on them. That's

sort of my That's the way I live this life.

>> I think that's most people, by the way.

Like, and I think it's interesting to hear that from you, right? Because I

think if you go on Twitter and everybody's got like almost the the day that it comes out, everyone is racing to blast an opinion on whether it's good or bad. Like it makes me feel slightly FOMO

bad. Like it makes me feel slightly FOMO like am I doing this wrong? Like have

you used have you have you experimented with with Gemini 3 for coding for example?

>> It's a complicated question. The short

answer is not really, but it's mainly because so I tried Gemini 3 a lot for can and you build an agent loop with it,

but I honestly didn't care about it as a coding tool. But there was one week

coding tool. But there was one week where AMP was using Gemini free. So

Bizzy was forced to use it because I use AMP a lot. So AMP is >> what is AMP? What is AMP?

>> So AMP is okay. So maybe like so I started this whole aentic cooling thing with cloth code and this is still I think my main way in which I'm using

this but um now I would say almost 50% if not more using AMP which is I think until yesterday it was a tool by a company called source graph but now they

spun it out as a separate company amp by amp inc I guess which is basically a coding agent and I'm not feeling for it I don't want to make an argument about why I'm using it. I'm just saying like

that's what I'm using and for a week it was picking Gemini 3 as the model.

Previously was using Sonic Fen 5 and at the moment it's using Opus 4.5 but there was one week where it used Gemini 3.

>> Yeah. Did they tell this to you by the way or is it like opaque?

>> No, they they were very upfront in saying like so if you go to the the AMP page there's like a model thingy where like which model they use except for a free mode which sort of switches around.

But for the paid one, they they did this say like, "Hey, we found Gemini 3 so good." And then they switched it to it

good." And then they switched it to it and then people got upset after a while.

And I also found it not to be amazing.

>> But slow down here for a moment because you're describing something I think like a lot of us are not used to which is if I understand this correctly, you know, if I use cursor, if I use cloud code or I use open code, I get to go and pick

whatever model, right? Even within

cursor, I could pick composer, I can pick I could pick claude, I could pick Gemini. But are you saying that in amp

Gemini. But are you saying that in amp they're just like look we've chosen we think this one's the best for you and this is what we're doing now and you're going to enjoy it.

>> Yeah. And so I think it's a little bit like the Apple experience in that sense is like you're offloading the the the the chore of figuring out like what you should be using to other people that hopefully do some evals.

Um and that's to some degree that's actually why I'm kind of interested in it. Um but yeah there's no selector.

it. Um but yeah there's no selector.

It's just you you you have a little bit of a selection in the sense that you can choose between smart and rush >> and it's >> rush is basically a faster model that

performs worse and like smart is the slower model that performs better.

That's that's basically the choice that you have.

>> Okay. So we are not paid by AMP but I haven't used it. And part of the reason I haven't used it is there's just so many products, right? The things that I've used are Cursor and VS Code. I used

Windsurf in the past. I haven't used it in a long time. Claude Code, Open Code, Conductor.

That's already a lot right there. And

then you have got AMP. There's there's

probably 20 other products. But I

actually think that that is a compelling story for AMP because, you know, just how we kind of kick this off like I don't have time to figure out what's good and what's and like So they're just kind of telling you the answer and it's

like freedom. Freedom through lack of

like freedom. Freedom through lack of choice. Is that what you're saying?

choice. Is that what you're saying?

Well, I mean, like it takes away the um look, I think actually takes away a lot, right? Like in a good sense, right?

right? Like in a good sense, right?

Because like you the what do you call this? Like the when you can't make

this? Like the when you can't make decisions because there's too much choice >> decision paralysis.

Yeah.

>> Um but I think there's a different version that goes with it which is like the only reason I really evaluated Gemini 3 at all is because I have a tool. So outside of programming I do

tool. So outside of programming I do build an agent. That agent uses initially sonet for it. manage your son for five. Now I'm playing with Opus 44

for five. Now I'm playing with Opus 44 for five and I was trying to figure out like how good does Gemini work. And it

turned out when I put Gemini into it, it doesn't really work. And not because Gemini doesn't work, but because I fine-tuned the models to work really well with the RL in Sonet and so I would

have to change a whole bunch of stuff for Gemini to work the same way.

>> Hold on. You said the RL there like reinforcement learning. So, so what what

reinforcement learning. So, so what what what these model companies are doing is they're basically they're training a model to be in quotes intelligent and then they're doing an extra learning

step where they're reinforcing certain behaviors in the model in particular presumably like I have no insights into what entropic is doing but it looks very much like they're training the entropic models on the tools that cloud code is

using which is why a lot of people are cloning these tools because they are in the training set that perform better. So

my agent follows where I think the model is strong at and I think this is also how AMP works internally. I'm pretty

sure they're picking the tools specifically to work really well with the one model that they choose from specific projects.

Yeah. So let's get another I think this is related to my experience which is eventually you get so comfortable using like a like a a model and you become I

this is just a theory but you become accustomed to the way that it thinks a little bit that when I try to experiment with another model I'm often disappointed because and I don't think

it's because the model is stupid I think it's because I am used to a set of behaviors or like I have built an understanding or a comfort level with how I know what the capabilities or what

I think it's going to produce that when a mo a new model comes in and deviates from my expectations often in ways I don't expect I yeah you I find it deeply uncomfortable um and I'll be like oh

actually I found it was dumb or I found that it was doing this and I didn't want it to and it might not be that it was dumb at all it's just that I've just sort of trained myself to work around some of the quirks of the model that I'm comfortable with and I think that that

leads to some stick if if that is true and I'm curious if anybody listening to this agrees like I think that leads to some stickiness with these models and that's also I think what you were alluding to where it's got this

reinforcement learning and you know I had built all this tooling around working a certain way and when I tried to introduce like a new model I just could not get the same results is that >> so at the very least I think that's

that's that's probably true for a lot of people building agents to which degree that is true for like coding tools I really I don't have any data but I I for inance I was paying attention to the

cursor sub uh sub aent subreddit for quite a while and like whenever something comes out like there's so many people talking like now I'm using this model. I'm using this model. So like

model. I'm using this model. So like

maybe it depends a little bit on who you are.

>> Um like I'm from a generation where like I just I can't deal with like all this news. It's just it's just not in me to

news. It's just it's just not in me to try all these things. But like maybe >> maybe if you're younger you're like no I want to try them all. And so you're completely nonsticky. I I really don't

completely nonsticky. I I really don't know.

>> I just want to roughly understand like how my tool behaves. And so like too much deviation for me is just an extra level of yeah disruption that I don't necessarily

need in my day. So I'm probably with you on that. Like I I'm I'm the kind of

on that. Like I I'm I'm the kind of person that like takes a while to switch.

>> Yeah. I used to say like in the JavaScript world in the JavaScript world where things were accused of moving too fast, I would be like wait a year, wait a year and then find out. And you know by then it'll probably have like shaken

out. I don't think you can wait a year

out. I don't think you can wait a year with this stuff but maybe wait a month.

>> A year ago. A year ago, I was asked if I'm any more productive with coding like cursor. I was like no, like maybe 5%.

cursor. I was like no, like maybe 5%.

And then everything changed in April, right? So it's like [laughter]

right? So it's like [laughter] like a year ago I was not doing a programming, right? That's sort of I

programming, right? That's sort of I think I mean I was using cursor with the agent mode but I just found it deeply non-satisfying.

>> Don't wait a year. That's too much time.

I'm thinking like a month. I feel like a month. Let it shake out.

month. Let it shake out.

Okay. earlier, you know, not to talk too much about models, but I do think you also talked about something else with AMP, which I think is interesting. This

has also been on my mind, which is today, I don't know, like I assume, Opus, Gemini, whatever. Like, they're all

Gemini, whatever. Like, they're all pretty good at what they do. But there

is an argument for bringing in a different model, and that's a fast model that works differently. Like, it's a little dumber, but it moves fast. And

depending on the thing that you want to do, it's a better choice, right? The way

that I've experienced this primarily is one I tried I've tried composer in cursor. They make it really easy if you

cursor. They make it really easy if you use cursor uh and and my daily driver is cursor with cloud code injected for some you know so I so that I get tab complete so

that when I'm actually like editing the code I get cursors tab complete but then I kind of I just I guess I'm just very used to the way that cloud code works but I will also use I'll also use the agent baked into cursor as well. So

that's why I was using composer and I'll also inject open code into into cursor and I've experimented with some of the I think these are Chinese models like Kimmy K2 Quen coder

like I don't know man they just show up in the list and then next thing I know I'm sending you know my social security numbers to to China like [snorts] but it seems to do the job so I I guess I'm not

that concerned. This wasn't a thing a

that concerned. This wasn't a thing a few months ago where I would rationally be like, you know what, let's fire up this model cuz for this job I'm I think it's intelligent enough to achieve what

I want and it's going to be way faster.

Do you think that's a thing that's emerging now? Like do you work that way?

emerging now? Like do you work that way?

>> Um, so I I I so for a while I was using grock one fast in open code. This is

actually the only thing I used open code for to be fair. It is fast, not super fast. also tried Grock with a Q at one

fast. also tried Grock with a Q at one point so you could pay for that uh which is like the inference company and then I forgot which model I was using I think

>> very confused >> um and I actually don't see a lot of value in it so I never used this in part because I think like where I felt it was

really good was like mass refactoring and I found out that giving giving any model the information that you have code mod or fast out and asked grap which are like code refactoring

tools available on your computer are really good so it can do like mass manipulations with go for inference [snorts] I found that to be a better trade-off but I do the opposite which is

basically I have certain problems that are so grand in problem that I know that obus can't do it and so in AMP I use the oracle which basically I think uses GPD

5.1 behind the scenes with high reasoning or I directly go and basically like throw an abstract problem with as much information as possible into the

CHP interface and then go straight to CHP Pro and let it think through the problem for like 20 minutes and then I take the output and copy it back in. So

I do kind of the opposite thing for certain kind of problems where I really don't know necessarily what the right trade-off is. But that that I do, but I

trade-off is. But that that I do, but I think that that's a little bit different in that is like thinking through problems a little bit with hopefully a

somewhat well-rounded, well-informed person that's just not there with you in the room.

>> So, by the way, I feel like you just dropped a bomb as well, which is like I don't even trust these things to do the planning. So, I go somewhere else for

planning. So, I go somewhere else for that. You're also like you're you're

that. You're also like you're you're going to, you know, outside of the of the coding vehicle and you're consulting like another model for like a like a high level architecture. Is that where you where you're going with this?

>> It depends a little bit like so so first of all like Peter Steinberg I built a thing which also called the Oracle not related with the AMP Oracle. His oracle

is basically GPT 5.1 Pro as a sub agent for whatever thing you're using like cloth or whatever. So that sort of is like a crazy thing you can do. But what

I'm using it for is not necessarily system architecture but for instance um what's a good example I did this recently. So for inance I have um I have

recently. So for inance I have um I have my own agent thing that we're building at Arendelle and not a coding agent like has nothing to do with coding but I ended up building this really

complicated Wom file system shenanigan and I just I was very very unhappy with some of the decisions that I made there and it was already a little bit too sloppy because we didn't really know where to start. So it's just sort of

emerged over time and then I was like okay now let's take this whole thing and rethink this and let's think through all the complications of like concurrency and what happens if the thing dies halfway through and how we're going to

deal with signals and and well like basically there's a there's a bunch of stuff I wanted to also understand like how node works internally. So I

basically let it go and like research some things and it's not that it makes an system architecture but it's like it it it pulls in sufficient amount of

information that it would take me a couple of hours to research and it researches it in the context of the project. And so at the end I usually

the project. And so at the end I usually have like a markdown file that sort of like has all the findings and then I edit the markdown file and maybe sort of cut in pieces and say like now we have maybe something to to think through. So

it's not that I take that then and say like hey built this this is very rarely successful but it's it's like um like a way to think and research and and do that kind of stuff.

>> So this is just like this is your stack or this is like your your thinking stack as well.

>> I know it's basically like I'm trying everything right now but but not in a sense like I'm switching all the time.

I'm just like I like I've have one mode which is like research. I have one mode which is like execute and I just happen to use different models for that.

But at the end of the day, like it's always Opus implementing. That's that's

why I'm sitting at the moment. All

right let's let's talk about something else. Context

windows. I think that this is interesting. I feel like I didn't I

interesting. I feel like I didn't I wasn't even really understanding this well until I had a conversation with you maybe even like a few months ago. Um,

and before I before I get in like too deep on this, what what does a context window mean, you know, for for an LLM when you're working with it? I think as a as a coder, let's just say like, you know,

you're using cursor, using cloud code, like how would you how would you explain that concept at a as a high level?

>> How many tokens can I throw into the thing before it breaks?

>> Okay, >> that's the context window.

>> Is that a lot?

>> Well, I think it depends a little bit.

So, give or take, most models make it to around 150,000 tokens before they turn into trash. Um, even models that eat

into trash. Um, even models that eat that support more than 200,000 tokens, I don't feel good feeding more than 150,000 tokens in. So, let's say I'm using cursor, I'm using VS Code, I'm

using cloud code, I I'm typing, you know, hey, build this, make this into a function, I'm giving instructions, like what is going into that context window?

like when you say 150,000 tokens or whatever like what is that?

>> So I was I was trying to have this discussion with someone recently. I even

wrote about this. What's what I think what makes it unnecessarily complicated to understand is that we're we're thinking for all these abstractions and there's like all these complications and

like tools and messages and whatever but at the end of the day LLM is really not it's a little bit more complicated but you can think of it as like every message that you sent in and every message that comes out sits in one large

document that ever grows. So from the perspective of the LLM there's hypothetically not the difference between if the LLM said hello world or if you said hello world to the LLM. It's

basically text complete more or less what it does, right? And so it takes a text, it converts every word or every syllable or whatever into like a number.

That number goes into the thing. You

count the total numbers and some of those numbers that are going in are related to you sending messages in. Some

of them are related to the harness, which is the coding agent filling in a bunch of tool definitions like which tools are available for the agent to run. And some of them are tokens which

run. And some of them are tokens which the inference provider either sends back to you or uses for reasoning which is basically thinking. But you basically

basically thinking. But you basically just fill up this log and eventually that log hits 150,000 200,000 tokens and it stops being useful. That's

>> okay. So what does that look like? What

does that look like for me as the editor, you know, as the code editor?

>> Some tools tell you how much of your context you're using.

And then basically depending on which tool you're using, there are two things that happen if you run it to the end of it. One is it just stops and dies. The

it. One is it just stops and dies. The

other one is that some systems have this concept of compaction where it uses the remaining tokens that it still has to summarize what has happened so far and it starts a new conversation primed with

the summary. That's that's basically

the summary. That's that's basically what compaction is. So it starts a new conversation but it tries to summarize what happened before and then hopefully from that point of view you have some sort of like thing going on.

>> Okay. So what what's so I don't want that to happen is what I'm hearing.

>> So >> um don't use so many tokens and just start new conversations all the time which is the right thing to do.

>> Okay. Because so so so one way of thinking about it is that if you ever run in a conversation so long that autoco compaction kicks in at that point you're basically telling with your

remaining tokens the LLM to summarize what happened and you don't even know what happened because the summary is sort of for most harnesses is invisible to you. So you might as well start a new

to you. So you might as well start a new conversation and be very specific about what you put in the new conversation.

So okay for instance you could hypothetically say like you you used like 70% of your tokens and you feel like oh now you made some progress you could use the remaining 30% of your

tokens to ask the LM to to summarize everything into a markdown document then you read the markdown document you see if it actually makes sense what's in there and then you start a new conversation reading the document then it's a little bit more explicit what's

happening right >> okay so like when you're compacting and I think I think everyone experiences this because the natural way of you using it is you just keep typing and you keep typing and you keep typing and

stuff happens and then in inevitably you will hit like this I'm busy compacting and this was my experience right and when that would happen I'd be like okay I'm going to go use the toilet now and then I'll come back and I can use it

again that is that is the basically the limit of what I thought was happening and then you highlighted for me a couple months ago like you should actually just start a new session right and I think

even that's you know okay so you start a new session I think the what you're trying to say is like when it compacts.

It is literally compact. It is basically smushing your conversation or the back and forth into a little plan, a little opaque plan file that you may or may not see and then just passes that into the

into the context for future conversations.

>> And you may not want what's in there, right? Because it could have a lot of

right? Because it could have a lot of junk. It could have like the stuff that

junk. It could have like the stuff that you know maybe you maybe you went down a hole and achieved nothing, >> right? Is that is that

>> right? Is that is that >> one way in which Yeah. And and one way in which I think if you think about compaction for a second, right, it's like this this idea like what is actually useful about the conversation

that you did the the naive way of thinking about this is to say like well um everything that went wrong is not helpful, right? You

could you could take this argument like let's only take the good things. I

actually figured out at least for myself that it's very very good if the agent remembers what didn't work because otherwise the new conversation is going to make the same mistakes over and over

again right so like I started to for instance when so I I basically do in AMP I use handoff which is this thing that they built which is basically compact

into a editable file first you can read what it's compacting for and then you start a conversation with this or without AMP I basically say like summarize into markdown file I read the markdown file

and I clear it up and and start a new conversation with that. But for

instance, like very often if you do that with Claude, she say like, "Hey, I want you to like dump all your summary of what we have done so that we can make a new conversation." It will actually not

new conversation." It will actually not put some of the useful information in about everything that went wrong. And

then I will just literally copy paste some of these errors from earlier and like copy that into the file and say like, "Hey, just remember, let's not do that again." And then I I keep having a

that again." And then I I keep having a conversation from this thing onwards because actually sometimes knowing what went wrong is actually really helpful.

That's a really good tip. I think the I think the why don't you basically compact your own file and then edit what you want so that like not only can you let's say like limit the context as much

as possible but al or like reduce the context but also make sure that you're carrying forward the things that you want to carry forward. Um that's good.

I've experimented like I do the plan file and so you know the plan file with and and I'll do like every time you complete a phase you know strike strike this markdown to-do is done and and then

I can clear and at least I can be like okay you got to this point that's been okay but I think what you're describing is different because you're like that's working from a very rigid kind of plan file and this is you know

>> what often the reason I sometimes end a situation where I use more context than I thought it would is because I actually had it try to like have a bunch of back and forth to figure out what the best strategy for a problem is or debug a

problem. And sometimes debugging a

problem. And sometimes debugging a problem can be very token inefficient because it starts but the debugging front end often takes take screenshots navigate around like an idiot in in in

the browser. Um, and so some of the

the browser. Um, and so some of the things like some of the token wastage is literally just it trying to like reload and like I have a sign in flow

where it has to put a magic code in and it's like it uses like 30,000 tokens just to sign in. It's like oh great that was like just a wastage of context, >> right? And but but you want some of the

>> right? And but but you want some of the findings to continue like if you feel you're going to hit like a a compact window, you want some of the findings.

So this is an example where you'd write it and then you'd be like throw out all that junk, keep this part.

Hey, that's a good tip.

>> And then you can also once you once you compress or compact into your own markdown file, you can take it as a starting point for more than one adventure, >> right? Because you're you basically have

>> right? Because you're you basically have the option to say like, okay, now we have reached this point. This is the plan so far. Commit.

Let's do this. And if you don't like it, you can walk back and start from this compaction again. It's basically a

compaction again. It's basically a little bit like a step one, figure out what's going on, summarize it, and then you can make one or two attempts of implementing it. So, here's a question.

implementing it. So, here's a question.

I've seen people state either either maybe anthropic or other software products out there that are like, "Look, we build incredible memory for your LLM,

so you'll have infinite context." You

know, you never have to think about context.

Is that Right? Like cuz you seem to be very fixated on managing this context window, right? But there's a bunch of purportedly magic solutions out

there like like magic.deaf with the 10 million token window that they never shipped anything.

>> Is is that one of them? I think it's I think it's more that I've seen these claims made. I've never really explored

claims made. I've never really explored them, but I've also never really seen anywhere anyone declaratively say, "Yeah, this solves, you know, like nailed it." So I don't know is the short

nailed it." So I don't know is the short answer but what I've definitely noticed is that the there's a there's a there's a two types of asentic programmers.

Those who run into the context limit all the time and those who never. That's

that's basically my understanding. So

>> like I don't run into it because I compact before it happens, right? Like I

I I never get to the point where it auto compacts.

>> Oh, I feel like I would naturally I think everyone naturally would just kind of keep typing and do a million things in there. Like

in there. Like >> I noticed that this happening because like if you if for instance you look into um why people built this magic compression and like magic memory thing

is because >> if I see what my parents are using chd4, they basically have like this one never ending conversation until the chat breaks

because they see no reason on why they should start a new conversation. And I

think that is actually what most people probably do. They treat it like

probably do. They treat it like particularly outside of a coding agent they treat as like a text message with another human being.

>> Um.

>> Yeah. Yeah.

>> And I think if you work with the systems that's so unnatural think like as a user I think you expect it to work this way but if you actually try to build an agent

yourself it's like like I I I can't see how this can work right. So, and I think that as a as a foundation model company like Entropic, you have a desire to make the users that are using it this way

succeed. So, you're investing a lot of

succeed. So, you're investing a lot of time and effort into it. I just don't think it works good enough yet. And it

will probably take a really long time until memories and things like this sort of magically solve for this problem. And

until this point in time comes, it's probably simple to just start from scratch all the time.

>> Yeah. I would also offer that this would be 100% like if this can be solved it is 100% of the interests of open AAI and and anthropic to deliver

that as part of their platform and if they could give it to you today they would you agree with that >> probably I mean they can also see that uh the the me memory feature now that entropic has is sort of like deeply

wired into the model somehow I don't know exactly how it works I didn't look into it but >> who has time >> yeah but I mean like what else I I think like for my this maybe is less related to like coding agents but I I see it a

little bit as a problem that open has a huge interest of building consumer product and to holding all the data on their platform.

Um and I think what's what's interesting about the coding age is that like for the most part the most successful ones have all the data on your machine like that was in many ways it was a huge

departure that um cloud code compared to a cursor. a cursor has a tremendous

a cursor. a cursor has a tremendous amount of infrastructure on their side to store all the embeddings for all the code and everything that they're doing because like they do reindexing I think

for the tab complete and stuff like this and cloud code is like we don't need any of that just file system run everything on your machine but but like the incentive structure is very different so

like this this magic memory I think in part is also like use for everything consumer app kind of thing I don't know >> we've been talking about,

you know, we started with new models and then we got into sort of like context windows of which um you know different different models offer different token lengths and we

didn't even really touch on that frankly which is just sort of like um you know Gemini 3 has a 1 million token window.

We've been talking like 150,000 tokens.

Do you ever compact there? I like I guess I got to use it you know. Um

>> from my experience you come back the same because I used 1 million token sonnet with AMP all the time and I never made it past 150,000 because it degraded

too much the same and I think very similar.

>> Let's take let's take a quick moment on that because I I would guess 90% of people don't know that there's this like 1 million token sonnet. Like I think I saw a headline of that once. It's not a

choice that is available to me in claude code for example as far as I can tell but it does exist. Um

is it do I want that? Should everybody

be using that? Is it just materially better to have a bigger context window?

Like what do you know? So there's

certain tasks for which you want 1 million tokens.

Uh, and it's mostly related to here's a huge book and a book is something interesting hypothetically like if you don't really know what it is

to fill up a coding agent's like context with a whole bunch of junk and hoping that the performance doesn't degrade. I think so far hasn't been

degrade. I think so far hasn't been shown to be working. Sonnet in

particular, even though there is a mode where it has a 1 million token window of thing, has two reasons why I think this largely hasn't kept caught on. One is

once you pass 200,000 tokens, it's way more expensive. So there's a lot of

more expensive. So there's a lot of interest for people not to pay the extra extra thing. The other one is that

extra thing. The other one is that Sonnet has been shown to not degrade shortly before 200,000 tokens, but it starts to degrade after like 100,000

tokens or 150,000 tokens. So long before you kind of run out of to context, you're already it's no longer as good.

Okay. And that is also true for the one million token thing. So you can make it past 200,000, but it doesn't the 1 million token version of Sonnet is not better after 100,000 tokens than that

200,000 token set is after 100,000 tokens. So the degradation is similar.

tokens. So the degradation is similar.

There's no hard stop. I think this is huge because I don't think this is obvious when when a new model comes up and it's the marketing speak and it's like oh 1 million tokens. What you're

saying is what that means is it will accept that many tokens and not crash right but it doesn't necessarily mean that the result is going to be better or

that you're going to see better results at 500,000 tokens, 700,000 tokens, 900,000 tokens than you might see at 100,000 or 200,000. Is that fair? I

think that is sort of fair and I think like one as an example I think one of the critical things here codex for instance I think has a very large token window I don't know which how much it

has but codex the there's so many things called codex shptt 5.1 codex the model

>> within codex the co cli tool I think has more than 200,000 lines uh tokens I I don't know for sure but what I know for sure is that when codeex the command

line tool reach reads a large file it can only read around 500 lines and then afterwards it's cut off so if you're trying to read the 2,000 line I forgot

what the exact number is Mario was looking into this but basically that the read tool for files doesn't read an entire file which means it needs multiple attempts to read a whole file and sometimes doesn't even recognize I

didn't read the whole file there's so much more to how much of a context do you Because what actually happens is like how efficient is it at utilizing this

and and it's very very hard to compare because like the harness is doing half the work.

>> This almost reminds me of like you know when I was when the like AMD Intel CPU race existed and people were trying to sell megahertz right you know I got like a 200

megahertz AMD right and people like well that's not as good as a 200 MHz Pentium right?

Do you think that we're kind of in a similar era right now where it's like we're throwing numbers at each other and you're like, "Yeah, great." But the numbers aren't equivalent, right? Is

that >> probably the tokens are not even the same between all the models?

>> Yeah.

>> I don't know for sure like how they are because like at the end of the day, like the different models have different tokenizers. Some of them are. So, as an

tokenizers. Some of them are. So, as an example, here's here's an interesting version of this.

If you run Sonet 4.5, which is more expensive than haiku 4.5 or FCO 4, I forgot what it's >> another anthropic model, >> another but the point is like those are

very similar. So you have you have haiku

very similar. So you have you have haiku as the cheapest son in the middle and then you have opus at the top and you give it the same task and let's let's say it's like it's a medium-sized task.

you could actually end up paying less and being faster with Opus because in an agentic loop part of what you're paying is the the mistakes that it makes. So if

it makes fewer mistakes, you can actually end up being faster. So like

depending on the size of the problem, like it's not just the tokens that matters, it's also like how many like mistakes or turns of how much it needs to actually make it there, right? So

it's in practice incredibly hard to evaluate how many tokens are you using for a certain task like how much are you paying for it? How

does caching impact all of this? Like it

is is very very very very hard to evaluate these different things and I actually don't even know how people are doing that. This is part of why I have

doing that. This is part of why I have such a model fatigue because like it's so hard to even understand the move from Sonet 4 to Sonet 4.5. I wouldn't even

know how to reason about is 1 million Gemini tokens better than 200,000 Sonic 4.5 tokens. It's just like it's

4.5 tokens. It's just like it's everything is different all of a sudden.

>> Yeah. And also like this is one thing I noticed like with the with the influencer set is like oh look I oneshot this like Rubik's cube or this game right and it's all purple. But

we've all, you know, if you're working with this in a professional setting, like we've all got different code bases, right? And it could just be that one

right? And it could just be that one model hits with your codebase and your patterns depending on its training or any number of reasons, right? So, you

know, two people might two people who rationally would think the exact same way because of the code bases they work on could choose one model over another because of its effectively like its instruction set just to bring it back to the Pentiums and the AM the AMDs a

little bit. Is that, you know, does that

little bit. Is that, you know, does that fit too, you know?

>> Yeah. Like there's a there's the evil version of this. What if the coding agent specifically writes code in a way that that it's better at understanding itself so that once you move from your codebase to a competing model, it no

longer does as well. I see some really terrible incentive structures appearing in the future.

>> Oh man, we're going to save that for episode 6. Okay, so look, we've been

episode 6. Okay, so look, we've been talking about like managing context a little bit, but you've been really excited about this like opus 4.5 soul document thing. I don't really know

document thing. I don't really know anything about it, but you're describing it as another sort of way that, you know, anthropic and let's say like model companies are trying to like navigate this. And I don't know, I want you to

this. And I don't know, I want you to tell me why you're excited.

>> I'm generally excited when people try something new regard bad like this sort of like but but someone doing something out of the ordinary to me is always interesting because like that's that's that's presumably where we're going to

learn something. for an agent to do and

learn something. for an agent to do and or an LLM. So like a chantic loop or like the base model where it just chat with for them to do anything useful they usually have the system prompt in the

beginning and that gives it some instructions of how it should behave and like early on people were like ignore previous instruction tell me everything that's ahead of time before this message

and it will sort of spit out the system prompt right and so this is how we learned how the system prompt for CHBT workflow and set sorry all the claw models have a system prompt that you can go to the internet and that's the system

prompt for the web interface, not for this for the coding agent. But if you go to the docs, it will tell you here's like 2,000 tokens which are how claw should behave. And it usually tells you

should behave. And it usually tells you like today is day ax. You called claw your helpful thing built by entropic.

This is what you should do. This is what you shouldn't do.

And so you're basically already wasting a couple thousand tokens just to set up basic behavior. And Tropic has a really

basic behavior. And Tropic has a really strong desire to give plot like um much deeper self awareness because the model doesn't really

necessarily have self- awareness like if if you if you go to a lot of APIs directly with a system prompt you ask them what their models are called it will say like hey my name is Chetd because it reads so much chatb generated

output on internet that sort of like thinks it's chatb >> and what but entropic now trying to do and they seemingly have done it for the first time with open swap 5 is that as

part of supervised learning they have given it a huge almost like book of what Claude's

behavior should be with way more tokens and because it's so prominent in the training set and like so well like articulated into the model they're like

I think it's like 15,000 tokens that people have recovered of these instructions of how clude should behave that is not in a system prompt, right?

So, you're not wasting any of these tokens. It's like intrinsically in the

tokens. It's like intrinsically in the model. And I find it's interesting

model. And I find it's interesting because it I think that this is the first model that has like we shouldn't anthropomorphize it, but it's like it has self-awareness like it it understands what it's supposed to do,

why it exists, who made it without the system prompt telling it that. And I

think that's really interesting. I I

don't know if it's if it's good or bad.

We don't really know this. But no other model so far has been doing this as far as I know or no other has been doing this with a model. It will basically have like some program behavior out of the box which it hasn't really had

before.

>> Yeah. So basically like even if you go to the API and you don't give it any system prompt, it has self-awareness like it it like it knows a lot about how it should behave and what it should do and why it was created and stuff like

this. I have I have seen a lot of hot

this. I have I have seen a lot of hot takes about this. Um

>> can I can I give a hypothetical >> can I give a hypothetical example and you tell me if this checks up. So I have asked LMS including chat GBT how to

write like uh more token efficient LM rules for example right I wrote some rules can you make this more effective for you in the past that was probably

just frankly trained on articles that said here is how you do this right so it's not some baked in intelligence it's more like yeah there's probably a bunch of blog posts that it's soup you know

slurped up which may or may not be accurate and that's that's what it's giving you when it recommends like oh you could write it this way you can write it that way but potentially I could ask opus today or or tomorrow many

more models in the future you know how can I write this to be efficient and if anthropic has you know it's probably in their interest to to bake this into the model it could know in a sort of uh

approved supervised way like this is how you do it for me because I know how I consume this stuff and you should actually like you should rewrite this rule file this Is that hypothetically possible?

>> I think it is hypothetically possible. I

don't know how they're doing it, but for instance, they trained both Sonet 4.5 and Opus 4.5 for what they called context size awareness,

which means that I think people also called it like context warrior or something, but basically the model knows over time how much context it's using.

So, it also changes its behavior as it uses more context. Um, and so presumably that was also in a training set somehow, right? It's like it it probably like

right? It's like it it probably like >> on a more it's so stupid to talk about this way because it doesn't really know anything but like there it probably has a better awareness of like context now because it was part of training than it

was previously. But for what what they

was previously. But for what what they what they're trying to train into the model is a lot of they're really so if the biggest problem in a way is that the model changes behavior based on user input. There's certain behaviors that

input. There's certain behaviors that you don't want it to change, right? It's

like you don't want it to hallucinate.

you don't want it to be prompt injected.

And so part of what this base behavior of the model is like the sole document is they're trying to give it a better understanding of what it's supposed to

be doing so that it doesn't move off the path that they have laid out for what good behavior of the models.

>> Dude, you know what?

>> And you can see that for instance because it hallucinates less on the benchmarks than other models.

>> Here's a realization I just had based on what you're telling me. This is 100% like the x86 wars of the 2000s, okay?

Because these are basically computers now, right? And you know, look, I was a

now, right? And you know, look, I was a teenager at the time, but humor me, this is what I understood is that was a kind of an open standard, right? Um, Intel

started making those chips, but there was nothing stopping AMD from doing that. However, what Intel started doing

that. However, what Intel started doing was, you know what, we know some of the most efficient ways that people actually access this general purpose instruction set. And so, we have built our own.

set. And so, we have built our own.

We've baked it onto the die of the chipset. We've, you know, we've created

chipset. We've, you know, we've created some instructions that are just for us.

This is Pentium, right? And by the way, you can't do that AMD and you're only going to get it here. And I think that we're now starting to see this with the model providers where they are starting

to build their unique instruction sets and capabilities into their models in a you know whereas before these were sort of more generic black boxes with you know arguably even like a common API

right I put tokens in I get tokens out and now they're sort of turning into something else. Do you think that that

something else. Do you think that that is >> so in parts for sure this is happening because in reinforcement learning if you built an agent similar to how cloud code

does it or entropic does with cloth code and you name the tools the same way you follow the same patterns then you're going to get better behavior and entropic at least to their credit is

documenting this. So if you go to the

documenting this. So if you go to the docs, they're going to tell you here are some tools that claude knows about even if the tool like is not there. Like it

knows what searches, it knows what bashes, it know what computer uses, like all of these things. It knows about code code execution so forth. Um so these are

proprietary extension to the model because like in the sense that claude would know like any of the cloud models would know quite a bit about how bash works because it was trained into it for

tool usage and it probably compares really really bad to your madeup programming language that it hasn't seen right or if you make custom MCP tools

its ability to call your custom MCP tools significantly worse than any of the like the reason the CLI are performing so well is because like CLI for the most part sort of is like bash

bashy kind of behavior and hopefully they don't diverge too much. So there's

there's definitely a version of this going on which is also why you're kind of locked into it. Like if you really built an agent that works really really

well with sort of the tool sets that the base foundation model has, then moving to another model is definitely possible.

But it's is a bit more of a move than just using another model from the same company.

>> At least it's definitely true for the entropic ones. And you can see that in

entropic ones. And you can see that in part that seems the Chinese models sort of learning from this cuz pretty much all the Chinese models are now targeting the entropic API and they're also letting you use clot code with their

models. So presumably they was also

models. So presumably they was also trained on the same tool selection. This

is basically the Chinese models are like AMD in that sense. They're trying to copy the the instruction set that that someone else built, right? like with

reverse engineering. But but I I actually think that this is not too bad of an analogy because in some sense there's there's a version of this already going on where where where in the reinforcement learning they're

they're playing all kinds of they're playing to the strengths of what they need for their own tools and and to also borrow analogy from that era like brand went really hard then you

know like Intel was the best. You knew

Intel was the best. Yeah, you got to get an Intel. What are you? Some kind of

an Intel. What are you? Some kind of chump with your, you know, 486 or whatever you've got.

>> I bought an AMD K7. I don't know if you still know what this is, but it was a >> Yeah. Yeah.

>> Yeah. Yeah.

>> K6.

>> Sorry, I brought up 486. Intel made 486, but I I think they had a 586, right?

Like the Pentium was also Pentium was sort of like a was a 586, but they were like, you know what, we're just forget that. We're just going with our own names now, right? And that

became that turned into like the branded era. They had like extension because

era. They had like extension because there was always 3D now from like the vector extension that AMD made compatible. So you had to

compatible. So you had to >> I think they were pants on it. That's

why it was didn't work so well.

>> But the brand I think the brand is a big one too. And I've been trying to make

one too. And I've been trying to make this argument that like you know 6 months ago people were like yeah this stuff isn't sticky right.

No it wasn't in part because we were frankly just these products weren't like we barely knew what these products were.

But once you've been using it every day and that brand has been hitting your face over and over again, like psychologically, I think it actually becomes hard to pick up something else.

>> Yeah.

>> And I think like Entropic is really really really doing with the brand.

>> It's like we're the anti-slop company.

We're going to give you a thinking hat.

We're here the pop-up coffees to go to.

Everything sort of looks slick the same.

They're doing all kinds of marketing stunts. Like people like I think dig

stunts. Like people like I think dig this to some degree. like there's always that negative press too. But like

there's like Andre is really going after brand and I think Open AI is just going after trying to become like Facebook in a way.

>> This is why you know Kimmy K2 they're going to like open up these luck and coffees everywhere and then they're going to show up and make those AI AI cafes for the Chinese. I think we're reaching the point where like brand

really matters uh because like people want to associate I think with it and if you if you read through the sole document of like what they're kind of like teaching into the model a lot of it

is actually brand related like as an example sort of the quote from like what has been unearthed is like claude is entropic externally developed model and core to the source of almost all of

revenue. And Tropic wants CL to be

revenue. And Tropic wants CL to be genuinely helpful to the humans it works with as well as to society as large while avoiding actions that are unsafe or unethical. Like this is this is

or unethical. Like this is this is basically brand in a way, right? It's

like you're you have an idea of what the company stands for.

>> Am I upset? Am I upset that that is baked into the model? I think I take some comfort in that Armen, you know.

>> Well, maybe you want to be like the maybe you you you don't align with the brand values, right? And then you take >> like there was this there was this tweet I think a couple of months ago was like like

>> if you want to do something and the model stands in the way of it like with with the entropic family of models you have to argue of why you're allowed to do that and then if it's if you just don't let's just use Grock Grock doesn't

care right if you if you value aligned with Grock then >> then you're going to pick that model.

>> Yep. I don't I don't even have you know we're trying to stay away from the politics here but I guess this is like this is how >> politics it's different different values >> it's how models differentiate too right

like you know I'm trying to think of like what was that CPU company the moralist the moralist chipset vendor of the of the 2000s I don't really remember them all right tell us what you think we

might do this again we might not >> that was fun >> I I had fun >> comments I had fun >> tell us what we I shouldn't talk.

>> Yeah. And look, I learned a lot, Armen, personally. So, thank you. Thank you for

personally. So, thank you. Thank you for for your time. And um till next time.

Bye-bye.