Software engineering with LLMs in 2025: reality check (at LDX3 by LeadDev)

Good morning.

It's uh great to be back in in London. I

I was supposed to be here five years ago, but finally we made it happen.

These days when when I look around, AI is all across the headlines. I just

collected a few of the things when I search for you know what is going on and some of that honestly just kind of triggered me.

So here's this one from Microsoft CEO saying that 30% of all code is written by AI. And at that point, you I was

by AI. And at that point, you I was talking with people like what does that even mean? like is is this big or or

even mean? like is is this big or or anyway it's it's a CEO clearly talking up their their own product right like Microsoft is interested in in in selling then we have a few months ago Antropic

CEO saying all code will be generated in in a year or he also said things like in six to three to six months 90% of all code will be written by AI again is an

AI company founder very much interested in in in this and then we also had Jeff Dean uh an engineer actually a chief scientist at at uh Google saying that AI could be at the level of junior coder in

a year. And again, all these headlines

a year. And again, all these headlines are from executives at large companies.

But on the other hand, when I look at the ground reality, there are some things that don't really match these really positive and really enthusiastic predictions. For example, this is from

predictions. For example, this is from January. This is a software engineer at

January. This is a software engineer at a startup saying that they use this tool called Devon which costs $500 a month autonomous AI agent and added a bug and

it cost them $700 extra dollars because of all the 6 million poss events. So

clearly you know like AI is not we know this by the way right like the bugs will will make it through but this is just a good example of yes it's it's it's not that great. And then there was this

that great. And then there was this Reddit thread that went absolutely viral after Microsoft's build conference. Some

of you are laughing. You read this.

After Microsoft's build conference, Microsoft showed how they released the copilot agents in the .NET codebase. And

Microsoft engineers were really trying too hard to help this agent land a fix in a production and really complex codebase, a .NET codebase, and they failed spectacularly. So, for example,

failed spectacularly. So, for example, the agent would add tests that break uh engineers would prompt them to to break it. And there's a lot of laughing going

it. And there's a lot of laughing going around. Now on on one hand, I do

around. Now on on one hand, I do appreciate that Microsoft was really transparent about this. No other startup has shown their agents do like this. But

again, we see that this thing is just really limited. So there's a big

really limited. So there's a big disconnect. We have the executives

disconnect. We have the executives talking uh about one thing and the other. But then I look back to my

other. But then I look back to my thinking and and writing and and research and in the last just month or or two months, the last couple of deep dives on the pragmatic engineer have all

been related to AI. I just realized on how cursor was built on what VIP coding means for us professional software engineers how these Microsoft tools work or don't work how chat GPC images was

was built and and scaled and I would I just for for this event I just want to pause a little bit and just get a temperature check of what is really happening like there's extremes

here with CEOs there's other extremes where like it doesn't work at all and you know what what is really happening so I happen to talk to a lot of software engineers like this is the the perk of of being and seeing a software engineer.

I mean I I do write a lot about it but I I try to stay close to the ground. So I

just asked them how are you using AI tools at your company and I asked for different types of of kind of companies and categories.

I asked this from a couple of AI dev tools startups who are you know selling this thing. So you would expect that

this thing. So you would expect that they are really all in. I asked some of big tech companies, some AI startups that are not selling AI tools, but they're they're they're building AI tools and some independent software

engineers. So, let's start with the AI

engineers. So, let's start with the AI dev tool startups. First, I talked with the team at Entropic. Um, I just did it over the last week and I asked them, hey, what are you what are you seeing?

Like, again, let's keep in mind, they will be biased necessarily, right? But

this is what they told me. When we gave cloud code access to our engineers, they all started using it every day, which is pretty surprising. Now, this was months

pretty surprising. Now, this was months and months ago. Cloud code was released in public one month ago, but this was internally. But they said they saw a

internally. But they said they saw a really big poll immediately. Cloud code

is a command command line interface.

It's not an IDE. It works in a terminal.

And they also told me that 90% of cloud code the product is written with cloud code, which seems obscenely high. You

would think this is kind of an advertisement, but again, I did talk with engineers, and engineers aren't exactly the ones who make things up.

Now, on traffic also told me something interesting. They launched cloud code

interesting. They launched cloud code less than a month ago. I think May 22nd, so like 3 weeks ago, and they said on day one, they had 40% increase uh in usage. And since the launch in less than

usage. And since the launch in less than a month, there's been 160% increase.

This just means they they're seeing a poll for this product for whatever reason. Now, one more thing that

reason. Now, one more thing that Entrophic is has started actually this thing called MCP, the model context protocol. I won't go into all the

protocol. I won't go into all the details. I I have a deep dive on it and

details. I I have a deep dive on it and you can find a lot of articles but the idea is that you can have MCP clients that can be your IDE or agents and you have a protocol and you can kind of connect things like your database like

GitHub, Google Drive, Puppety or whatever you want to. I actually used it to connect it to my database. So one of my APIs and I can now chat with it. I

say like hey you know how many people have have uh claimed this pro promo code that I have an API for and it kind of creates SQL. It's it's pretty neat. It

creates SQL. It's it's pretty neat. It

is a fun way of an interesting way of doing it. And then trophic was telling

doing it. And then trophic was telling me that they open sourced this protocol in November. In December and February, a

in November. In December and February, a few smaller companies and scaleups adopted it. In March and April, the big

adopted it. In March and April, the big guns, OpenAI, Google, Microsoft all added support. And today, they're

added support. And today, they're estimating there's thousands of MCP servers happening. And we'll we'll see a

servers happening. And we'll we'll see a little bit later on why this is relevant. Now, I also talked with

relevant. Now, I also talked with Windinsurf, another AI uh ID editor. Um,

and I I I was asking their their uh team what they're seeing and they said they see 95% of their code is being used written using wind surf. So either their agent or their passive tabbing. Now I

mean this this sounds awfully high and I'm I'm a bit surprised but again uh this is what they're they're seeing in again don't forget they're going to eat their they're going to be dog fooding these companies. I finally I reached out

these companies. I finally I reached out to cursor and they told me that about 50 40 or 50% they didn't have as exact but they're like h that's kind of roughly what it feels you know they're a bunch

of it it works but a bunch of it doesn't again these are the companies that that want to get to 100% because that's what they're selling right so okay not too surprising that it's it's as high as is

I do appreciate the honesty from cursor by the way so um now on to to big tech here I talk with people anonymously so at Google I talk with about five

different uh engineers and so first thing that we need to know about Google is everything is custom there. They

don't use Kubernetes they open Kubernetes but they have something called Borg. They they don't use uh

called Borg. They they don't use uh GitHub they have their own repository.

They don't use they they have their own critique uh code review tool and so on and their ID is called cider which which has which is an acronym for something integrated development environment and

repository. uh it's a cloud integrated

repository. uh it's a cloud integrated development environment repository. It

used to be a web tool. Today it's a VS code fork and it is integrated across all the Google stacks. All their

internal services are integrated. It

works really really nicely inside of Google. Now engineers told me that AI is

Google. Now engineers told me that AI is just everywhere. LMS have been

just everywhere. LMS have been integrated into Cider the ID the VS code fork that they use the web version called Cider V. They have autocomplete.

They have a chatbased ID. They said it works pretty good. Maybe not as good as let's say cursor, but it's it's it's pretty good. Critique their AI review

pretty good. Critique their AI review tool. It gives you feedback and they

tool. It gives you feedback and they said it's just it's sensible. It it

works. Code search, something that's apparently amazing inside Google. And

again, it has LLM support. You can ask about stuff and it spits out parts of the codebase. And I've heard there's

the codebase. And I've heard there's been a lot of progress. So, a a former Googler who who left Google about 6 months ago said that about a year ago, it it was just weird how all of this was

not really used inside of Google, but now it is. So, things have just evolved pretty quickly. And a current software

pretty quickly. And a current software engineer told me that they think Google internally has this really slow approach where they're taking a cautious approach. They want to get things right

approach. They want to get things right so that engineers stick with it, that they they don't mistrust it. Uh also

Google has a bunch of other tools that again these are coming from engineers.

Notebook LM this is a product we can all use uh uh as well. You can just put docs and chat with them. LM prompt playground which is like open playground but Google apparently built it internally before

open released it. They have this thing called the MoMA search engine a knowledge base using LMS and engineers are using it all the time and a lot more are being built. Now, this is a quote from a Googler who will definitely not

be on on the record with their name, but they say there's an orc specific genai tooling happening everywhere because that's what leadership likes to see. And

honestly, that's how you get more funding these days. Now, you know, if you work in a large organization, you can see this is true, but this is probably also deliberate. Like, this is how tools like notebook LM have been built inside of Google. A team just

funding it and building it. So, that's

Google. And one really interesting thing that really got my attention, this is from a former SR who is really good friends with a bunch of Google S people.

They said, "What I'm hearing from my SR friends at Google is they are prepared for 10 times the lines of code making their way into production. So they're

beefing up their infra, their deployment pipelines, their code review tooling, uh feature flagging, all of these things.

This was really, really interesting.

What is Google seeing that we might not be aware of?" Amazon. I also talked with with engineers here and Amazon is not really known as well for for AI but apparently internally almost all devs

are using this tool called Amazon Q developer pro. Uh it's really good for

developer pro. Uh it's really good for AWS related coding. In fact the Amazon devs that talked to me said they're really surprised that people outside of Amazon don't really know about it. So

apparently if you're doing anything with AWS it's it's really good with the with the context and they just like it. Uh

again six months ago when I talk with people they were not that enthusiastic and a year ago they're like h it doesn't really work that well Q but now it does and engineers also told me they use cloth for everything. This engineer was

telling me how when they have to write a PR fact which is Amazon six pager uh or or or kind of press release they use it a lot for it. Perf season apparently has this engineer did a a lot of it with

with that and just with a lot of writing tasks. Uh Amazon has a relationship with

tasks. Uh Amazon has a relationship with with with Entropics. So they have an internal cloud and one interest the with Amazon the most interesting thing with MCP servers

we we mentioned how entropic came up with MCP servers. Now let me take just a little bit of detour about how Amazon is this massive API company in 2022 based

on this is how Steve Yaggi and former Amazon engineer and and and well-known uh person in the industry summarized what happened there. Jeff Bezos had this big mandate. It went along these lines.

big mandate. It went along these lines.

One, all teams will expose their data and functionality through service interfaces aka APIs. Two, teams must communicate with each other through these interfaces. Three, there will be

these interfaces. Three, there will be no forward or forward interprocess connection allowed. And I think four was

connection allowed. And I think four was something like if you do don't do this, you're fired. Uh but Amazon has done

you're fired. Uh but Amazon has done this and internally this is how AWS was partially b born as well. All all their services that they use internally, they can expose externally because they have all these APIs. They've been doing this

for more than 20 years. And if you have a service with an API, it is trivial to to bolt on an MCP server so your ID or your AI agents can use it. And this is

Amazon. What is this doing? This is I

Amazon. What is this doing? This is I I've never heard this before and I'll talk with this person and you're probably the first one to hear this, but most internal tools and website inside Amazon already have MCP support.

Automation is happening everywhere. So

people were telling me, devs were telling me that they're they're automating ticketing system, emails, internal systems, and devs are loving it. Some some of them are automating a

it. Some some of them are automating a good a huge part of their workflow.

Again, no one's talking about it, but it's happening. So I I wonder if Amazon

it's happening. So I I wonder if Amazon by being API first since actually that that's 2002. Apologies for the typo,

that's 2002. Apologies for the typo, they might be MCP first starting in 2025.

With big out of the way, I I wanted to talk to some some smaller startups that are are have no real pull for the AI dev tools themselves. they do have a pool

tools themselves. they do have a pool for AI and I talk with uh a startup called incident.io Oh, they didn't start as an AI startup. They started as on as as on call platform, but with AI, it's kind of an obvious place to to integrate

and and have resolution all that. So now

they're they're turning it to pretty much AI first and and I talked with Lawrence Jones who will later be doing a talk at at uh uh at LDX3 and he said that our team is massively using AI to

accelerate them and they share tips and tricks in the Slack and he just generous to share a few of these with me. So one

of them is an engineer saying, "Hey, I just used another MCP server for the first time and it works really well for well- definfined tickets." So this engineer realize, "Oh, if you have a really well- definfined ticket, you can pass it to an agent and they can come up

with a first pass." And sometimes it's pretty good and they just share this to the chat saying, "Hey, this works for me. Why don't you try it? See what you

me. Why don't you try it? See what you think?" And there's a lot of chatter and

think?" And there's a lot of chatter and you know, they're sharing all these things. A second example is another

things. A second example is another engineer saying that their new favorite trick is is prompting to ask for options. For example, can you give me

options. For example, can you give me options for writing a code that does this and this that I need to do? What do

you think you're I'm seeing this error?

Can can you give me explanations? How

would you train Zapont? And so on.

And what I really love about this is is inside of the company they're they're experimenting. They're seeing this is it

experimenting. They're seeing this is it works for me. Do you think it works for you? And you can see the you know the

you? And you can see the you know the reaction discussions etc. There's there's a lot more examples but they're they're really coming around to it. And

and Lawrence closed with this. They said

the biggest change has been from cloud code just released again three weeks ago. I just checked yesterday. So this

ago. I just checked yesterday. So this

was this was uh on Sunday and and their entire team are regular users. Again

this is no affiliation with with with any of the vendors but they're they're starting to use startups. Now I also talked with a biotech AI startup who asked not to be named and I I'll tell you why in a second. Uh they do really

cool stuff. They use AI and ML models to

cool stuff. They use AI and ML models to design proteins. They've been founded

design proteins. They've been founded three years ago. Uh they have a team of about 50 to 100 people. They have a lot of automated numerical pipelines built on Kubernetes. They're using Python,

on Kubernetes. They're using Python, Huns, and so on. And an engineer told me this. We've experimented with several

this. We've experimented with several LLMs, but none of it has really stuck.

It's still faster for us to write the correct code than to review the LM code that that will and have fixed all those problems. And even this is even using the latest models, even using like Solid

3.7 or or maybe even Solomon 4. Given

the hyperlms, I think we might just be in a weird niche. And this is why this engine didn't want to give their name to to this. They're like, I I we we don't

to this. They're like, I I we we don't want to be the AI skeptic. But it's

true. There are really, you know, fastmoving startups that are experimenting, but it's it's it's just not working for them. Like it they're trying it. It doesn't work. They move

trying it. It doesn't work. They move

on. Again, they they tried AI code review tools, uh, and they're kind of using it on and off, but it's just it's just not a thing for them. Again, don't

forget they're they're building novel software, right? Like this has never

software, right? Like this has never been built before. So, just just keep that in mind. So having gone through the startups, I I just wanted to turn to a few independent software engineers, people who have been accomplished before

AI. They've done a bunch of cool stuff

AI. They've done a bunch of cool stuff and they love coding. Like it just you you can they love the craft. So first I I turn to Armen Ronacher who is the

creator of the Flask framework at at at uh Python. Uh he was a founding engineer

uh Python. Uh he was a founding engineer at at Sentry. Uh and he just recently left Sentry to to just maybe do do a startup. He's been coding for 17 years.

startup. He's been coding for 17 years.

a really nice coder and he got really excited about AI development recently.

So I he published this article u just uh a few weeks ago saying AI changes everything in and he wrote I I'm quoting the the highlighted text if you would have told me even six months ago that I

prefer being an engineering lead to a virtual programmer intern aka an agent I would have not believed it and so I asked him like what's changed like you love coding like why are you into this

whole agent stuff and he told me a few things first cloud code got really good I don't know if you're seeing a trend here by the way there's zero affiliate iliation here and this is this is not

any sort of advert for anything. He also

said by using alums extensively he got through this hurdle of not accepting it and most importantly he said that the faults of the model hallucination are avoided because the tool just runs itself and sees the results and and it

gets feedback. So I like okay that's

gets feedback. So I like okay that's interesting. Let me talk with Peter

interesting. Let me talk with Peter Seinberger. He is the creator of

Seinberger. He is the creator of PSPDFKit. He is an iOS junkie. He loves

PSPDFKit. He is an iOS junkie. He loves

he is inside iOS internals. He has

strong opinions about the API changes.

He's PSP PDF kit was uh one of the I I think it's still the most popular like kind of PDF related iOS uh tool and he sold his startup I think one or two a year and a half ago or so and he's been

tinkering on his side and he didn't really do much and then again he published an article that caught my mind and and in it he said the spark returns I haven't been this excited astounded and amazed by technology in a very long

time. So I reached out to him and I

time. So I reached out to him and I said, "Hey, hey Pete, what has changed?"

And he told me he feels there's some inflection point where it just works as an iOS junkie. He's who loves Objective C and and SEXA and Swift. He told me that languages and frameworks just

matter less because it's so easy to switch. He's now coding in I don't know

switch. He's now coding in I don't know the TypeScript and other languages. I

don't think he would have touched because of of these tools. And he's

saying that a keepable engineering can just have a lot more output.

and and then he he posted this on on social media actually sent this over to me. He's saying that his all his tech

me. He's saying that his all his tech friends are just and often have trouble going to sleep and it's such a mind-blowing technology. And it's kind

mind-blowing technology. And it's kind of ironic because we exchanged messages with him at 5:00 a.m. when I was already awake for some other reason and he was awake coding. And another engineer uh

awake coding. And another engineer uh said that he's seeing a lot of burntout developers come back into the field to create stuff.

So I I I I I now shout out to Bri Brigita who uh is also doing a talk here at LDX3. She's a distinguished engineer

at LDX3. She's a distinguished engineer at Thoughtworks and she's been very thoughtful about exploring understanding what works and what doesn't in in AI.

She's methodological. I love her article. She she does a bunch of them.

article. She she does a bunch of them.

Uh you should check them out. And I

asked her to her take and she said that she feels that LMS are this tool that we can use it in any abstraction level. And

this is the difference. we can now create low code like assembly highle languages or or you know even even human language if we want to it thinks that this is like a lateral move like it's it's not just a new layer on top of it

it's like across the stack and this is what makes LMS really exciting again this is someone who's been thinking about LMS for quite a while and very accomplished engineer before LMS finally

I turn to Simon Willis he is the creator of Django at independent software engineer and he's been blogging on the side for like 23 years karpathy co-founder of OpenAI posted this just a

few days ago saying he loves his blog and reads almost everything and Simon's blog is known as the LLM blog because he's been tinkering with since every Chad GPT came out on what works and what doesn't again really good writing. So I

asked Simon, how would you summarize the state of Genaii tool? And again, Simon is as independent as can be. Like he has an open source project. He he makes enough from that and from donations of the blog like that's his is his income stream. And this is what Simon told me.

stream. And this is what Simon told me.

He said coding agents actually work. You

can run in a loop do compilers and all that stuff. And the model improvements

that stuff. And the model improvements in the last 6 months have been some sort of tipping point and and now it's becoming useful.

So to sum it up, this is roughly what I've heard. AI dev tool startups do

I've heard. AI dev tool startups do heavy usage, not too surprising. Big

tech is very heavy investment and growing usage. AI startups, you know,

growing usage. AI startups, you know, maybe a hit or miss. Some are using it, some are not. Independent software

engineers, they're a lot more enthusiastic than before. This is

interesting.

But there are still a bunch of questions left. As as I was looking through like

left. As as I was looking through like it it doesn't feel to me like the slam dunk of, oh, you know, the future is here. Not at all. And I'm going to give

here. Not at all. And I'm going to give you four of my questions. Number one,

why is it that founders and CEOs are far more excited than engineers? Now, some

of the engineers that we've seen who are excited like Armen and and Peter, they will probably be founders themselves.

And here is an example of of Zack Lloyd who is the founder of Warp. Uh this is an AI terminal, so kind of an AI dev tool if you will. and he's saying anyone else having a hard time that their most senior engineers are are not really

using AI and the most enthusiastic adopters are the the founder the PM and and this is from an AI tooling company.

It it is an interesting question. I see

this all the time and also if you remember the headlines from CEOs from public CEOs they're super enthusiastic about it. Why is that? I don't know.

about it. Why is that? I don't know.

Number two is how mainstream or niche is AI usage across devs? Hands up if you're use if you're using any AI tools for for coding or software engineering for at

least once a week.

So I'm I'm kind of seeing roughly like 60 70% of of the room go up. And this is data that I've gotten from DX who ran a survey of 38,000

uh devs recently. They're seeing that the median organization has about 50%.

Five out of 10 use it on a weekly basis.

And not daily. This is this is weekly.

And the very top companies have six out of 10. So on on one end I mean this is

of 10. So on on one end I mean this is amazing it given that this technology didn't exist 3 years ago but it's not really the story that I've told you right so most of the stories you've just heard they are all above the median

except for maybe that uh unnamed uh AI biotech startup so just keep that in mind you know the reality is and and maybe there's a selection bias maybe the ones who are using it are more willing

to talk about it number three how much time do we save so you know Peter uh or Pete told me that he thinks output is 10 to 20x more. But then DX did the survey

and they found that on a weekly basis, uh, estimations are maybe 3 to 5 hours, maybe 4 hours. I mean, okay, 4 hours saved is pretty good, but that's not 10x, you know, even on a 40hour work

week. And what do we do with that time?

week. And what do we do with that time?

Like, do do we produce anything more? I

don't know.

Finally, why does it work so much better for individuals and teams? Uh, we we see this all the time. And Laura Tasha from DX told me the same thing. These tools

are great for individual developers but not yet good at the org level. So in

summary, I'm not really surprised to see the CEOs and founders especially for AI related companies to be so enthusiastic.

It's kind of like you know their their financials are the line. Big tech

investing into AI kind of makes sense starts experimenting with AI tools also makes sense. But what makes me pay the

makes sense. But what makes me pay the most attention is the exper experienced engineers who have been around for a long time. They find a lot more success

long time. They find a lot more success and they want to use them more.

My sense is that we are seeing some sort of step change happen in how we build software looking ahead and I reached out to Martin Fowler and asked his take on this on a on on a piece that were uh he

reviewed and this is what he said. These

are his words. He said I think the appearance of elements will change software development to similar assembly similar level to when we went from the assembler to high level programming languages. after the high level

languages. after the high level programming languages the newer ones didn't really add a step change of productivity compared to assembler but he's saying that he thinks that LMS will

give us the same kind of productivity boost and going from assembly to highle languages did except these things are non-deterministic for the first time in

computing and this is a big difference and so I turned to a veteran software engineer who is still alive in coding and is doing it for doing it for 52

years Ken Beck. Uh we we have a long conversation on on on the podcast and Ken told me this really interesting statement that I had a hard time believing. He said, "I'm having more fun

believing. He said, "I'm having more fun programming than I ever had in 52 years." My first question was like,

years." My first question was like, "Ken, is someone telling you to say this or someone like he's like, "No, like he's just doing his side projects and

he's having more fun because he got a bit tired of of learning new technologies again and again and moving and migrating to new frameworks." and he said that the LMS really help him just be really ambitious and he's now

building a small talk server that he's always wanted to do that's going to run a bunch of parallel stuff and and do bunch of virtual computing he's doing a small talk a language server to

integrate in into all these things and I asked Kent how do you compare LMS to all the technologies changes through your lifetime and he said I've seen something like this before in fact a few things

one was microprocessors going from mainframes to to smaller computing which was a huge shift apparently developers had a hard time putting their heads around it. Number two was the internet,

around it. Number two was the internet, which I think we can all agree it's it changed the economy. And then the smartphones, they just changed how you can have live location and people spend a lot more online. And he's comparing it

to to these things. And this is what he closed. He said the whole landscape of

closed. He said the whole landscape of what is cheap and what's expensive has just shifted. Things that we didn't do

just shifted. Things that we didn't do because we assumed we're were going to expensive or hard just got ridiculously cheap. So we need to be trying things.

cheap. So we need to be trying things.

So my takeaway is things are changing and we need to experiment more. I think

we need to do more of what the startups are doing. Try out what works, what

are doing. Try out what works, what doesn't. Understand what is cheap, what

doesn't. Understand what is cheap, what is expensive. And I'm I'm leaving you

is expensive. And I'm I'm leaving you with this message. Thank you very much and see you around the conference.