Most Valuable Skill of 2026: Managing AI Agents

I don't think it's crazy to say that the people who know how to run teams of AI agents are going to be able to outperform everyone else in this AI age we're in. So, how do you actually run

we're in. So, how do you actually run teams of AI agents? What does this mean?

What's the stepbystep way to actually do this? And what are the tools you need to

this? And what are the tools you need to know like Devon AI to actually go and do this? Well, on today's episode, I

this? Well, on today's episode, I brought on Ryan Carson, and he clearly explains all of this for you. even gives

you a tour of his insane setup for how he manages uh his AI agent. So, by the end of this episode, you're going to understand what cloud agents are, how

you can set up automations, how you can create a software factory that creates software 24/7, and you're going to be able to understand it just really clearly because he does such a good job at explaining all this. So, enjoy the

episode. I'll see you at the end, and

episode. I'll see you at the end, and have a creative [music] day.

Ryan Carson back on the pod by the end of this episode. Ryan, what are people going to get out of this?

So, you're going to do be able to do three things really well. You're going

to be able to run cloud agents. You're

going to be able to do automations. Uh,

and you're going to be able to ship faster. So whether you're a founder, a

faster. So whether you're a founder, a soloreneur, a stay a stay-at-home parent, you're just going to become an agent pro by the time we're done.

Okay? Cuz I think a lot of people listening to this might have they've played with agents, cloud agents. Um,

but you're going to help them just get the confidence basically to be world class. You know, I think that's what I'm

class. You know, I think that's what I'm trying to that's what I've been thinking a lot about. It's like okay there's you know a bunch of people who have played with agents but how do we become

worldclass agent operators?

Yeah essentially you are a manager of agents now. So what no matter what you

agents now. So what no matter what you used to do, whether it was a people manager, an IC, you are going to become a manager of agents now and you need to be the best in the world. And it doesn't

matter whether you're a VC, a founder, uh an IC, a mom, a dad, like a college student, you should manage agents and be the best at it. And we're going to show you some tips for that.

Okay. So where do we start?

Okay, so let me give some context, right? So people understand where is

right? So people understand where is this coming from. So, I've been a founder CEO for like 25 years. Um, uh,

I'm working on my fourth company now.

Um, at my, uh, last company, Treehouse, which was acquired, we had, you know, I think up to like 110 full-time employees, taught a million people how to code, you know, so I've done the whole build organization thing and

manage organization thing. Um, and you learn a lot about managing people on that. Um, and now this startup which I'm

that. Um, and now this startup which I'm running it, I am one employee right now.

Um, so we've raised a seed round. We're

scaling really fast. Our revenue

probably is going to 4x this month. Um,

and I'm probably going to be hiring my first employees. Um, we are essentially

first employees. Um, we are essentially an AI divorce agent for divorce firms. So, uh, we we we do the the dirty topic that no one seems to want to want to tackle, which is family law. So, I've

learned a lot about how to scale me, right? Um, and so I'm going to help

right? Um, and so I'm going to help folks understand how do you manage a lot of agents well? Um, how do you check the work? How do you automate things? How do

work? How do you automate things? How do

you work from your phone? Um, you know, I do almost 50% probably more of my work from my phone. Um, and uh, it's pretty wild. So, that's where we're going to

wild. So, that's where we're going to take you. And and and I will say as

take you. And and and I will say as well, like this is going to change every 3 to 6 months. Um, so, uh, we're going to share a lot with you. Get your hands

dirty. Try it. Um, and, uh, the best way

dirty. Try it. Um, and, uh, the best way by to learn is to do. Cool. Yeah, I'm

actually I'm particularly interested in the phone stuff because I actually do zero% of my stuff on the phone.

What? Yeah. Okay, let's [laughter] talk.

So, uh let me start by just showing you well when I'm at my desk, what is the world like? Right. So, right now I'm in

world like? Right. So, right now I'm in my home office. Um I I I I work from Connecticut. Um and I do all my desk

Connecticut. Um and I do all my desk work here and I've got this big 52-in Dell monitor. Um, I used to have like a

Dell monitor. Um, I used to have like a couple monitors and I was like, screw it. I just want a huge monitor. So, uh,

it. I just want a huge monitor. So, uh,

let me share my screen and, uh, all right. So, what I do is I set up eight

right. So, what I do is I set up eight screens on my monitor and, uh, I also, uh, enjoy food, uh, and snacks as as I

code. Um, but the key is to have eight

code. Um, but the key is to have eight things up at once because I have to multitask a lot of agents at once. Um, I

also use a paper system to keep track of some of my to-dos, which is a little bit weird. I use the UG Monk system, which

weird. I use the UG Monk system, which is kind of fun. And I've got a killer Razor mouse. Um, I was getting a lot of

Razor mouse. Um, I was getting a lot of wrist pain, actually. Um, so I decided to go uh vertical mouse on that. Um, I

also uh have a button uh which actually is connected to Whisper Flow. So, I

almost uh use Whisper Flow for all of uh all of my stuff. So, I've got Slack in the top left. That's where I get notifications and things about what my agents are doing or what is happening.

Um that that's the primary thing I use Slack for. Uh and then what I do is I

Slack for. Uh and then what I do is I lay out uh various other screens. So

I've got a one password always set up.

It's very important to keep your keys uh safe, secure, and separated from the agent. Right? So I've got all my prod

agent. Right? So I've got all my prod write keys in one password. I do not give my agents uh prod keys. Um, so

production writing is dangerous, right?

And your agents will do something bad.

Um, and so you need to set up a system where they uh ask you for the key when they need to write to prod and then you go into one password, you copy and paste it into the session and you're both very

clear about what you're doing. So uh

that's one tip. Um, keep your keys secure. And then I use Devon. So uh

secure. And then I use Devon. So uh

Devon is is created by Cognition. I

think it's one of the best software factories in the industry. It's not

cheap um but it's good. Um the things I'm going to talk about today are are not Devon specific. Uh but they do work very well in Devon. And one of the keys

is cloud agents. Um then I usually have two screens set up where I'm testing um my app. It's just a lot of testing. And

my app. It's just a lot of testing. And

then of course I've got X in the top left and top right. And then on the right I've got codeex um which we'll talk a little bit more how I do most my work in in cloud in Devon. Uh, and a lot

of that work, like we talked about, 50% is probably on my phone. Um, and then whenever I do desktop or whenever I'm doing, uh, things locally, I'll do it in codecs. Uh, usually that's in the bottom

codecs. Uh, usually that's in the bottom right. So, that's kind of my when I'm at

right. So, that's kind of my when I'm at my desk, uh, my setup, um, and what I do when I'm here. But the truth is I do a

lot of work, you know, from this my iPhone. This is my beautiful wife. Um,

iPhone. This is my beautiful wife. Um,

uh, it's just I get a lot of stuff done.

You know, I I'll be in the shower and I'm thinking, okay, I need to check on that PR or maybe I want to land that PR, right? You know, grab it, talk to Devon,

right? You know, grab it, talk to Devon, which is just done in a browser on my on my mobile and I get a lot of stuff done that way. So, that's my setup.

that way. So, that's my setup.

So I I keep hearing this that Devon is is almost you know it's the most powerful coding agent and you know can you just for folks who haven't heard of

Devon or or who haven't played with it like why is that what is going on? So,

um, so about 2 years ago, maybe two and a half years ago, Devon came out and they pitched it as this idea of like it's a software engineer, right? And

honestly, it didn't work very well, right? The the models were just not good

right? The the models were just not good enough. And no matter how good the

enough. And no matter how good the harness was, the models just weren't there. Um, and then fast forward to, you

there. Um, and then fast forward to, you know, 2026, um, the models are definitely good enough. And when you have a good harness that is cloud-based,

um, it's really good. So, think of it as you've got your choices, right? So, if

you're using agents to build things, you're probably using codecs, you know, from OpenAI or using Cloud Code from Anthropic or you're using one of the indie, I call them the the indie uh,

agent harnesses. You've got AMP, you've

agent harnesses. You've got AMP, you've got Cursor, which is not independent anymore, but sort of own owned by Elon.

Um, and then you've got cognition which makes Devon. You've got factory uh as

makes Devon. You've got factory uh as another one as well. So you've got these choices, right? And so I thought, okay,

choices, right? And so I thought, okay, I want to scale myself horizontally.

Like I want to be able to run as many agents as I possibly can and I don't want to be working on my laptop and trying to figure out

which code is colliding, you know, which work tree am I on? all a lot of technical stuff that really [clears throat] it's it keeps you from shipping. Um, and you know, I know

shipping. Um, and you know, I know there's a lot of different folks that listen to the show. You might have soloreneurs who are are are newer to coding. They don't have computer science

coding. They don't have computer science degrees. They're not super technical,

degrees. They're not super technical, but they're building things. Then you

may have, you know, software engineers who are multiplying themselves 10 to 100x. They they they have deep

100x. They they they have deep understanding of software and architecture. And then you have founders

architecture. And then you have founders who are coding but are abstracting themselves. and you have VCs who are

themselves. and you have VCs who are coding, you know. Um, but, uh, everybody's going to learn a couple skills. Um, and what's interesting is

skills. Um, and what's interesting is I'll sort of, uh, zoom out for a second.

I think we all thought that engineering and being technical was going away, but actually what's happening is the more you become a better agent manager, um,

actually the more technical you become.

uh and I I think it a good analogy is that you are basically an engineering manager right and so to be a good manager of many agents you do need to

become technical right there's this reality of you have to understand what Postgres is you have to understand a production environment versus a dev environment you have to understand

migrations like but the truth is you'll learn all that by using agents right and so I am way more technical than I have ever been in my whole life. Um, and so I

think the idea is become more and more technical. Therefore, okay, so let's

technical. Therefore, okay, so let's back up. Why Devon? What is it? It's a

back up. Why Devon? What is it? It's a

really good agent. They have uh their cloud environment really nailed down. So

that's the first takeaway I want to talk about today is cloud. Okay. Um, how can you run more agents, get more done? Uh,

and the answer is you work in the cloud, not on your local machine. Now,

for the folks that aren't super technical, I'll sort of explain what does that mean. So, when you're writing code or you're building things, you are editing code, right? That code has to be

edited and then hosted somewhere, right?

Uh and and usually what we used to do is we used to all work on our Macs and that's called local development. And you

have to set up your whole local development environment. You have to

development environment. You have to have your database, you have to have your authentication, you have to have everything local, you have to have your dev server set up. Um, and then what happens if you want to do two things at

once? You have to either use get work

once? You have to either use get work trees, which is pretty technical and hard to understand, or you have to literally make another copy of your code in another directory. And then, so then

to work on two things at once, you have to have two copies of your code, which is hard. And then what about three or

is hard. And then what about three or four? What if you want to do five or 10

four? What if you want to do five or 10 things? It just doesn't work. So what

things? It just doesn't work. So what

Devon does and and I think a lot of good uh independent agent harnesses like cursor is doing this now um I think codeex is starting to do this okay is

they create what's called a VM or a virtual machine in the cloud and it's it's basically a little server and it is your development environment and you

click a button and it spins up and all of a sudden you can code in there and the beautiful thing is if you can you can create infinite numbers of these

things. So I often have, you know, at

things. So I often have, you know, at least five uh uh cloud agents working at once, often 10. And there's no risk that

the code is going to collide. So what it does is it takes away all of the mental overhead of trying to coordinate these things. Like I want to work in this part

things. Like I want to work in this part of the code, but what if it overlaps this part of the code or what if, you know, I want to do that? Have I

synchronized and pulled the latest? like

I don't know. And when I work in the cloud, I literally just open up my browser and I click new session and a new and a new VM spins up and I never ever have to think about it. So I think

if you are working locally, I honestly think you are a caveman. Like I I think you're you are holding yourself back and

you are shipping 10x less than you could be. And it is not smart. No matter if

be. And it is not smart. No matter if you think the cool people work locally and that all the smart ends, it's not true anymore. You work in the cloud.

true anymore. You work in the cloud.

Yeah. I mean, there are a lot of people like on X who are sort of like local maxis who look at people who spin up virtual machines and cloud

agents as sort of like amateur-ish. What

do you say to people like that?

They they are not they are not doing real work.

[laughter] Um, here's the deal, right? So, as soon as we hit PMF with Untangle and we and we started acquiring real customers and having to ship real fixes, real

features, like I had to probably 50x my output, right? Um, and

you just can't do it if you're working locally. So, I think it's it's nice to

locally. So, I think it's it's nice to work locally and sometimes you have to if you're doing heavy front-end stuff.

So, say say you're shipping a brand new feature with a lot of UI, like a lot of new UI, right? Then yes, of course, you're probably going to spin up a local

agent and you're going to do some light front end, you know, probably wireframing. Um, but as quickly as you

wireframing. Um, but as quickly as you can, you want to move that into the cloud and then let a cloud agent take over. Um, so I think this is happening

over. Um, so I think this is happening to a lot of people. I think a lot of the traditional knowledge of how to work is is is out of

date. Um that's where we're at.

date. Um that's where we're at.

Okay. So if you do start using cloud agents and you do 50x output, you know, in some ways that is a little stressful because you have so much stuff going on,

right? And now all of a sudden you have

right? And now all of a sudden you have to manage more. So how do you how do you structure a system so you don't go absolutely crazy?

So it's interesting I think first of all you have to shift your expectation of what your work is and what your work is

now is making high stakes decisions almost all day. So in the past like we all had the luxury of making probably you know at most maybe two to three

highstakes decisions per day right you'd have an important meeting with your key team or with a customer and you'd make a couple important decisions and you would communicate those decisions and then

they would get done over you know two weeks. Um, you know, I think you have to

weeks. Um, you know, I think you have to sort of mentally pace yourself to say probably by lunch you're going to have

made, you know, 10 to 20 high stakes decisions.

Um, and so you have to learn how to do that. So, which is your question, how do

that. So, which is your question, how do you do that? Um, and I think there's a couple simple things, right? Um, so when I'm working with my agents, I actually

have to organize my threads by and I pin the ones that are the most important work we're getting done today, right?

Cuz there's a lot of quick buck bug fixes or, you know, small things you're shipping and you kind of learn to separate two into two buckets buckets.

What are the big big important things I'm getting done today?

Pin those. And and then the other things just let them rip and get back to them when you can. Um, and then set yourself this kind of uh almost timer, right,

where you're like, "Okay, I'm going to go crazy if I just click through my threads like this all day." You just get completely wiped out, right? And so,

it's almost this discipline of saying, "I'm actually, you know, that the phrasing smooth is uh slow is smooth and and smooth is fast."

Yeah. I I think there's a little bit of of that where you realize I can only check on my agents and make highstake decisions probably every 25 minutes.

Um and so what you might do is settle into this cadence where you check on your high stakes threads like every 25 minutes just so you can like literally

mentally rest. Um uh and then the other

mentally rest. Um uh and then the other side is that is just understand that this is the world we live in and it is kind of exhausting. Um uh and and that

is kind of where we're at. And in in order to survive in this new world, you're you're going to work a lot more, not a lot less. And on that note, I'll share my screen because there is

essentially uh a world we live in now where you're going to ship probably, you know, so this is

my PRs, right? And we're looking, these are days, right? And you can see, you know, the average here is is sort of

this 22 to 25 PRs a day, right? And and

sometimes, you know, 40 a day. Um most

of these are being, you know, merged, right? A couple are being closed. And

right? A couple are being closed. And

what's funny is what happened this day.

Um I went uh hiking with my son.

Which day is that? I can't see your cursor.

Oh, it's uh the was the 18th.

18th. Um, and so we went to climb Mount Washington and I literally couldn't access my my phone. Um, but notice immediately the morning of, right? You

know, while he was sleeping, I I shipped what is that, you know, eight PRs.

[laughter] Um, so this is kind of, you know, the world we live in now. Um,

and we could talk about that. Um, but

let's zoom out to the the lessons here.

You've got to go cloud, right? You have

to understand in order to compete in this new world, you're going to be working half the time from your phone.

And it is tiring. Um but but but it's like going to the gym. Okay, I have this new muscle like I never had to exercise before and it turns out this weird back muscle. It turns out like that's the

muscle. It turns out like that's the most important muscle in the world.

[laughter] And if you don't do it, you're not going to win. So the way to do that is then

to win. So the way to do that is then set some sort of cadence for yourself.

Pin your most important threads. Write

it down. So, this, you know, what I'm going to sort of flash up on the screen here is my written to-do list. Um, every

day, you know, I use this hilarious analog system to like to remind myself what are the most important things we're going to ship today because there's going to be 10 probably PRs that we ship

that are fire, you know, putting out fires. Um, and and you can get

fires. Um, and and you can get distracted and exhausted. So, so cloud and then exercise this new weird back muscle which is making high stakes decisions all day. Um,

including on your phone, right? What

you're basically saying is like you're going to because of the velocity of uh

of you know do knowledge work and you're you're just you'll have [sighs] stuff going on and you will need to be there to say yes. know actually this,

you know, giving feedback in real time and you know why your point is why wait until you're in front of your Mac to give feedback.

um it makes no sense if you're trying to hit product market fit and if you're trying to 4x revenue in you know in a month or whatever it is um you know why got to be available

got to be available it and you know I think it just it's easy to think of these things as humans right like you have a team now that is infinitely scalable they're going to do work very fast and they're going to be

blocked by you making important decisions right um and you can't delegate the decisions Yet I think we're still very much even if you use Fable

like this is a trick is is use a Fable thread as kind of the manager and then ask it to spin up children and I do this a lot. So, you know, I'll spin up a

a lot. So, you know, I'll spin up a Fable thread and I'll say, "Okay, um, you know, we want to accomplish these five things. You know, spin up five, you

five things. You know, spin up five, you know, child Devon sessions. Don't use

Fable. I use Fusion for that, which is a cheaper model or it it's cool. It's like

their their new model where they've got um Fable is kind of the parent and they've got a sidekick.

Point is, you're not going to it's not viable to run Fable or any super premium model all the time, but have a a sort of a parent, you know, very intelligent

model run children. You can do these things, but in the end, you're going to have to pay attention. Um, and it's just like if you ran a big team and you were

out to lunch on your boat all the time, you got to get in the office, and the office is your phone. Um, and so get it done. And so, Greg, this is my homework

done. And so, Greg, this is my homework for you. Next time we talk, I want you

for you. Next time we talk, I want you to be doing more than 50% of your work on your phone.

Yeah. I mean, that's why that's why you're here, right? Like, you're a lot of people ask me like, "How do you choose guests?" And and why do you

choose guests?" And and why do you explore topics, certain topics over other topics? And it's it's really it's

other topics? And it's it's really it's for everyone. Like, I love arming people

for everyone. Like, I love arming people with the information, the tactics, the systems, but I'm also everyone. Like,

I'm a part of that, right? Like a part of that is like I know that you're you're doing this and I want to understand why you're doing it. And I

also want to understand like you know one of the takeaways I have is your system for dealing with tasks is not is not too dissimilar to a a very organized

person pre- AI right like the idea of like pinning you know your most important things like it's on one hand it's pretty obvious but on the other hand if no one tells you like

hey you know the way to to do this so you stay sane is via that system you're you it's overwhelming. Right.

Yeah, it is funny. It's like back to the basics, right? It's just in an agent

basics, right? It's just in an agent age. Um, and it turns out like being a

age. Um, and it turns out like being a very very good engineering manager is is probably one of the most valuable skills you can you can teach yourself. And you

know, I'm uh have the honor of sort of mentoring uh my nephew right now and and he's coming to me and asking me how do I win? You know, he's just come out of

win? You know, he's just come out of college soon. How do I win in this new

college soon. How do I win in this new world? Right? And you know, I'm

world? Right? And you know, I'm basically teaching him what we're talking about in the show. It's like,

okay, your job is going to be managing as many agents as you possibly can. Get

good at that, right? Which means, you know, you need to start to adopt these these these behaviors, right? Work in

the cloud, prioritize what you're doing, learn how to make high stakes decisions quickly, get technical. Um, you know, it it is a farce that engineering is going

away. Like if anybody tells you

away. Like if anybody tells you engineers are going away or becoming less technical, they're just not doing the work, right?

Um and it's like being a carpenter. Like

say you had robot carpenters around. If

you were a real carpenter, you'd be a much better manager of those robot carpenters because you understand how to make the cut. You know where to put the

nail. Um and so and so it's important.

nail. Um and so and so it's important.

So, to back up and if you're listening to the show, like the thing I want you to walk away is is cloud agents. Get

your reps in, figure out how that works.

Try it a couple times. Um, do it from your phone and get comfortable, you know, looking at your small phone interface and and learning how to work

that way. Um, so I think that's that's

that way. Um, so I think that's that's thing one. Uh, it's very important. The

thing one. Uh, it's very important. The

second thing I want to dive into is automations.

Okay, so say you're working in the cloud. Well done. Um, you know, you're

cloud. Well done. Um, you know, you're starting to separate yourself from the crowd. The second thing is automations.

crowd. The second thing is automations.

Um, you need to sort of think about things that you would have normally had a meeting with, uh, you know, once a week with your team, uh, where you check on something or daily you check on

something and build an automation. What

does that mean? Um,

so the first thing to do is go back to your agent, start a session, and say, "Okay, I want you to automate this task every x amount of days." And so, one of

them that I do, it's really simple, is that I want to automate the test of signing up for Untangle, um, creating a case, onboarding a client, having that

client go through discovery, you know, basically our user experience. But I

want that to be automated in a browser.

Like this is not rocket science. This is

basic user testing, right? Um but it turns out that good agents can do this.

And so the big unlock is uh set up your agents so that they can make themselves better um uh instead of you trying to

build these systems. Um and so I set that up and then in Devon it's really cool. It's called a playbook. But the

cool. It's called a playbook. But the

idea is it's it it's it's sort of like here's how to do this thing. It's

different than a skill. It's more of a list of things to do, how to do them correctly. Um, and I have a uh it's

correctly. Um, and I have a uh it's called endtoend uh signup test and that runs three times a week. Um, because it is expensive.

a week. Um, because it is expensive.

It's it's probably 60 bucks in tokens.

Um, because it's doing a lot of it's doing a lot of browser testing. Um, and

uh, I think that's sort of a side point I want to explain here is it's it's now vital that your agent can can properly brow browser test. Um, and it really

should work out of the box, right? This

is part of the reason I used Devon is because they've been using cloud agents that can do browser testing for 2 years now. It's so good. It records a video.

now. It's so good. It records a video.

It annotates the video. it it then looks at its own video and it fixes bugs that it sees, right? So you have this agentic loop that is specifically around browser

testing that then is automated three times a week, right? And then what what what do you do with that? And then you need to trigger a triage based off of

that. So say that it's like, oh wow,

that. So say that it's like, oh wow, during this the the third test failed in this automation, I'm going to spin up a Devon session or an agent session to

then fix it. Um, and then you have to figure out where do you know that that failed? How do you know? Did the agent

failed? How do you know? Did the agent just kind of f, you know, silently fail and then fix itself? What about the PR that was created? So you have to start thinking about how do you see these

things when they happen, right? as the

manager of all these agents, how do you know uh what's failing, what's not? And

and that's again where you should talk to your agent and say, "Okay, we've got this end to end test. You know, I understand it runs Monday, Wednesday, Friday. Um I understand it, you know, it

Friday. Um I understand it, you know, it it triggers a child triage session if anything goes wrong. How do I know about it?" And it and then your agent might

it?" And it and then your agent might say, "Well, you know, why don't I post in Slack?" You know, and and you can

in Slack?" You know, and and you can say, "Well, okay, how are you going to do that?" Well, we need to connect the

do that?" Well, we need to connect the MCP. you know, how am I going to know to

MCP. you know, how am I going to know to look in that channel? There's just all this machinery that you need to to to automate and figure out and and work

through um to build that system um of automation so that you are doing less of the work.

So this specific automation is this more of a like bug testing QA automation or

is it more of a um you know UX QA perspective? Th this one is more about um uh finding bugs that

have been introduced to prod that somehow uh we missed in our in our user tests um in our automated test suite.

So that one in particular is like hey we just can't have signups go down and not know about it. Um, but you know, sometimes your your automated test suite just it it doesn't catch everything. Um,

but a good oldfashioned clicking through the site in a browser catches that.

Um, it is not really about the UX. Um, I

think that there's there's no one thing I will say is there's no substitute and we probably won't have a substitute for

a while of you using your own app. there

just isn't a good substitute right now.

So, right now, every every so often, I literally go through our entire app myself. Um, and then I obviously quickly

myself. Um, and then I obviously quickly spin up, you know, sessions to fix bugs I find. But, um, we're we're just so far

I find. But, um, we're we're just so far from agents still. you the the smartest models in the world still lack this sense of obvious

you know uh intelligence where you're like why did you think that was okay?

Well, yeah. I think the bigger question is, you know, if if there's a division of labor between human beings and agents, what are the tasks that you as the human being are going to be doing?

What are the sets of tasks? And what are the sets of tasks that the agents are going to be doing that you're okay with them doing and keeping you in the loop?

So, my question for you around automations is beyond beyond QA, what are some other automations that you know

a founder could be implementing? So um

let me pull up my list for you. So the

kind of automations that I have are uh clearly uh uh QA bug fixine. Um the

other is what I call my production watchdog. Um so we've got a lot of real

watchdog. Um so we've got a lot of real law firms using Untangle. got a lot of real clients, you know, going through discovery and we've got just a huge

amount of activity and and honestly I can't keep track of it uh you know mentally by just looking through logs.

And so what I do is every day at 9:00 a.m. I have what's called a production

a.m. I have what's called a production watchdog automation which what it does is it goes through all of the events in the database um that happened for our

paid customers and it summarizes that in uh a JSON file and then that JSON file is in our admin and then I I you know know every morning I go in and I click

on uh the production watchdog and I read through the summary of what the customers did um and it's so valuable.

Um, so it's sort of like you can imagine, you know, your chief of staff showing up and saying, you know, all right, Greg, here's the important stuff that happened yesterday for our customers, right? Um, and I'm going to

customers, right? Um, and I'm going to roll it up and explain, you know, the things that went well and a couple bugs that we saw. Um, so that's an automation. Um, essentially what

automation. Um, essentially what happened yesterday that was important.

Um, so that one that's a big one.

Yeah, that's honestly a big one. like it's

it's like one of those ones that sounds small but is a big one because you're going you think you know what's going on but you don't really you don't I mean and it's it's so

shocking to me like yeah the amount of times I'm like oh uh and this is the other trick man so use a production watchdog automation pick your most

important customers and then make sure that in that report it's like the customer did X here's a link to view that right and so I actually have a production instance where you click it

and you actually see what the customer was doing and then I'll be like whoa that's weird like and we have this amazing ability that you know over you

know uh you know millions of years of evolution in very quickly picking up things that are off. [laughter]

And so there's a couple times I'm like, "That's weird." And I click into it and

"That's weird." And I click into it and it turns out, oh, the user experience um that the agent built on that wasn't was wasn't quite what I thought it was. Um

and so yeah, production watchdog automation that links out to real UI um is just man, it's a game changer. So

that's a big automation. Um this is a really important a self-improvement loop.

How does this work? Okay, everyone's

talking about self-improvement, right?

Um, so in Untangle, we have an agent called Grace, and she is essentially a parallegal. Um, and she has a lot of

parallegal. Um, and she has a lot of chats with both uh our customers who are attorneys, our customers who are parallegals, and our customers and their customers who are clients that are

getting divorced. Um, so how do I

getting divorced. Um, so how do I actually grade those chats and then improve them uh without managing the fine details? So what I do is I every

fine details? So what I do is I every day I have an automation that looks at these chats and then grades them on a rubric. And so you and again just talk

rubric. And so you and again just talk to your agent about this. You pick the most important part of your app that you want to self-improve and say here's how you agent judge whether this thing that

happens is good or bad. Um and then every day I want this automation to kick off. Look at those things for us.

off. Look at those things for us.

They're conversations that Grace is having. and then I want you to grade

having. and then I want you to grade them. And then if anything is bad or

them. And then if anything is bad or below this score, I want you to kick off a child session and fix it. And it is shocking to me the number of small, you

know, fine UX details that have been picked up by this loop and have been fixed. And honestly, there are things

fixed. And honestly, there are things that I just wouldn't bother doing.

Either I didn't know about them or I'd be like, me, you know, it's a paper cut.

Like, are we really going to, you know, ship a fix to that? But because the agent already identified it, spun up a PR, and it's ready to ship, I just go ship it. [laughter]

ship it. [laughter] And you know, we probably ship I would say probably three of these a day.

Wow.

That's that's huge.

And you know, a lot of a lot of the backlash on on loops is is just that it's expensive. Um, how how how have you

it's expensive. Um, how how how have you with with Devon specifically, has it been like absurdly expensive? Like what

are we talking? No. So he Well, so here's the thing. So last month I spent probably 20 grand in in tokens, uh, which is just too much. Like it's not viable. I I think all of us are in a

viable. I I think all of us are in a place where we we're getting to the spot where it's realizing, okay, for real engineering work per employee, you're looking at probably 5 grand a month.

Like that's probably where we're going to shake out here. And anything more than that, you you really need to figure out model routing. Um and so thankfully Cognition knows this and every

everyone's figuring this out like yeah whether it's cursor cognition I would say factory AMP all of the independent agent labs understand like [laughter] we our customers have a limit

like they're not going to pay infinite token budgets and then you have the this is me ranting a bit but it's like you have cloud code and codeex who are just severely you

know fund they're they're funding tokens and making it seem cheap, but this is not going to work long term. And so what I do is basically I have learned how to

use the right model for the right task.

And so these reinforcement loops, they use what's called SUI 1.7. It's a

fine-tuned model that Cognition has specifically built for coding. It's

super cheap um compared to, you know, Opus 48, you know, GPT56. And so a lot of these loops happen with this uh you know cheaper you know finely tuned model

and we're just going to see more of that. So you're talking I don't know it

that. So you're talking I don't know it you know actually I could probably I could probably look and dig it out but you're t probably talking like five bucks a session on that.

Okay that's not bad.

I mean and if you're not willing to pay 15 bucks a day to improve one of the core you know feature sets of your like what are you doing? Like either it's not a real

doing? Like either it's not a real company or you don't care.

Yeah.

Totally. spend the money like and and then you know figure out the the system uh to reduce your token cost. I will say

this is why I don't understand why anybody would build their engineering on a frontier lab stack. Like I'm just going to call it out like if you are if

your whole engineering uh you know motion is happening inside of cloud code or inside of codeex what are you doing like because they are not incentivized

to make it reasonable for you long term right they're going to lock you in to their models to their process whereas if you use an independent agent lab you know like an AMP like a Devon like a

factory like sort of a cursor but they're kind of weird now those systems are going to optimize for affordable engineering long term. Um, and uh, I

don't think you want to be locked in to anthropic or or open AI, you know, solely.

I mean, to be clear, you can still use their models within some of these products, right? So, you're not

products, right? So, you're not abandoning ship.

No, it's actually so it's it's yes and right. So the beautiful thing is if you

right. So the beautiful thing is if you pay you know an independent agent lab like a factory like a cursor like a cognition like an amp you know etc they're incentivized to figure out how

do I give you the best results for the lowest price. So they'll they'll say,

lowest price. So they'll they'll say, "Okay, we're going to use GPT56 soul on extra high for this type of task, and then we're going to route, you know, to

a uh you know, to a SWE 1.7, you know, and then we're going to have it checked with a Fable." Like, they're going to figure out that hard stuff.

Um, and the other thing I will say is if you're trying to build a software factory for yourself, stop. like it, you know, the reason why you see, you know, Ramp launch, Inspect, which is their

custom in in-house agent, is because they got to a size where they had to build their their software factory in-house. And if you get to a certain

in-house. And if you get to a certain size, you will do that. But now, if you're a oneperson shop and you're just building something that's just you, you can probably get away with this $200 a

month uh, you know, angle on a pick your frontier lab. But as soon as you

frontier lab. But as soon as you graduate into, okay, we're building a real product. We have product market

real product. We have product market fit. We're going to start hiring people.

fit. We're going to start hiring people.

You got to build a software factory. And

at that point, you want to be on an independent agent lab. Like otherwise,

you're going to have to build it yourself, which is stupid, or you're going to pay and be locked into, you know, a walled garden. Um, and you can't do that. The the analogy I use for it

do that. The the analogy I use for it is, you know, working with an independent lab is sort of like a uh like working with like a mortgage broker or travel agent. Like they're going to

make a lot of calls to make sure to [laughter] get you the best price and stuff like that. Um

that's an Yeah, to me I use both. You

know, I I'm in cloud code, but I'm also I use cursor personally. Um, although

Devon I'm interested in I might if people are interested I might just do like a whole breakdown on Devon if people if people want to want to do that.

Devon's so good now. Um, you know I just it's so good. It it's amazing.

Um, but I will say I do use codecs, right? The the the the Mac app is so

right? The the the the Mac app is so beautiful like and there's and they are subsidizing the tokens so much that if you want an agent

on your machine just to do really cool helpful stuff, open browser tabs and create Google Docs and check your email and and do all this information work and and basically get it for free. I mean,

you're paying 200 bucks and you get almost infinite tokens. like awesome,

but don't build your company software factory on that. Like it I just think it's bonkers to do that.

Mhm. We've covered a lot today. Is there

anything else you wanted to uh [laughter] cover?

I feel like I'm just vomiting, you know, all all the stuff in my head that Well, no, I it's important that like this is what I wanted. I wanted you just

to vomit everywhere because like it's sort of like advice that you you you would give to your to your nephew I think um or 22-year-old um

you know a 22-year-old fresh out of college. It's like what are the minds I

college. It's like what are the minds I need to know in terms of managing agents? So is there anything

managing agents? So is there anything else that people need to know? Cloud

agents automations.

Yeah cloud automations like th those are the two that I want people to walk away from. I

guess the third is is really you will be using a software factory like and what I mean is the agents are going to be writing 100% of your code, reviewing 100% of your code, shipping 100% of your

code. Like that's where we're going. Um

code. Like that's where we're going. Um

and so the the more you can get there, the the better. thing one uh thing two I think you know thing three is um it it

is important to build um uh a credible reputation for yourself right and I will say you know the work I put in over you

know the last 20 years on X like it really pays dividends and what I mean by that is if you if you share what you're learning publicly and you generally give

like write very helpful articles on X.

Um, and obviously the ALGO really does reward articles still. So, you should be writing articles on X. Um, you should absolutely be on X, no matter how you

feel about Elon. Um, uh, because sharing your knowledge really does build up credibility, which will pay dividends.

And I don't mean just like getting paid to post. I mean the the the the network

to post. I mean the the the the network and the relationships. I mean, the reason why I'm on the show is because we built a relationship over X. Um, and so I would encourage people take the time

to share what you're learning. It

doesn't have to be super polished. Um,

but get out there and be useful to people. Um, and it will open up doors.

people. Um, and it will open up doors.

Um, so, uh, you know, more than you think. Just get out there and share it.

think. Just get out there and share it.

Um, and, uh, and you'll reap the rewards. Not immediately, but

rewards. Not immediately, but eventually.

I mean, even if you don't know, you can just say, "I don't know." And that's your and that's your strategy, right?

And that's interesting, too.

Amen. And if you don't know, then ask an agent to help you.

Yeah. So,

I mean, my original co-host of this show was Sahel Bloom. Sahil Bloom worked in private equity and during COVID he just

started writing about topics he was interested in like he would be like just finance topics or you know particular individual who was interesting and it

like it it it read as like Wikipedia articles but really focused on Twitter specifically like optimized for Twitter and and that was his thing and he wasn't saying I know everything he was just

saying like I'm learning this thing.

Here's the thing I'm learning. He grew

his audience over a million Twitter followers, wrote a bestselling New York Times bestselling book, started companies on top of it, raised a fund on top of it. So, anything is possible for

sure. Uh on on X.

sure. Uh on on X.

Yep. Just just get out there and do do and learn. Um

and learn. Um uh you know, there there's no try.

There's only do as Yoda says, right? So,

totally. Uh Ryan Carson, I feel smarter and that's that was my goal. Um

and uh speaking of X, you're a wonderful follow. I'll include links where you can

follow. I'll include links where you can follow Ryan across the internet, including X. Um he's a mustf follow, so

including X. Um he's a mustf follow, so please do that. Thank you for coming on, sharing your generous uh thoughts, incredible setup, and uh is there

anything you want to leave people with?

No, thanks Greg. Thanks for having the show. I appreciate you putting out the

show. I appreciate you putting out the constant knowled uh constant knowledge and sharing. Um it really makes the

and sharing. Um it really makes the internet a better place. So happy to be here and thanks for having me.

All right, I'll see you next time.

Take care.