Why Developers Think They're 20% Faster with AI ... But aren't.
Study reveals the productivity issue destroying developer competence
What if the AI revolution in coding isn’t really making us more productive? What if it just makes us feel that way? We might actually be getting slower, less secure, and more dependent.
That’s exactly what METR’s 2025 randomized controlled trial found. Sixteen talented open-source developers had repositories. Each one averaged over 22,000 GitHub stars. They took 19% longer to complete tasks when using AI tools. They thought they worked 20% faster.
Stanford researchers found that in 4 out of 5 tasks, developers using AI wrote much less secure code. They felt more confident, though, that their code was secure.
We’re not in a productivity revolution. We’re in a competence crisis.
Or is the above study somewhat … BS?
The Rise of “Vibe Coding”
In February 2025, Andrej Karpathy, who co-founded OpenAI and used to be Tesla’s AI director, unveiled “vibe coding.” This means you should fully embrace the vibes, focus on exponentials, and forget about the code itself.” He warned that it’s “not too bad for throwaway weekend projects.” This suggests it may not be suitable for production systems.
The scale is staggering. AI now generates 41% of all code being written, 256 billion lines in 2024 alone. By March 2025, 25% of Y Combinator’s Winter 2025 batch had codebases that were 95% AI-generated.
Over 95% of developers say they use AI-generated code in production. However, only 43% trust its accuracy.
That’s a trust gap you can drive a truck through.
The Hidden Costs of AI-Generated Code
The Security Problem: A study from Stanford found that developers using AI help wrote less secure code in 4 out of 5 tasks. When manually checked, 73% of AI code samples contained vulnerabilities. The most dangerous part? These developers felt more sure that their code was secure. This shows dangerous incompetence.
The Quality Problem: Recent research from Bilkent University showed that ChatGPT generated correct code 65.2% of the time. GitHub Copilot had a 46.3% accuracy rate, while Amazon CodeWhisperer was correct only 31.1% of the time.
GitClear’s 2024 data showed an 8-fold rise in code blocks with five or more duplicated lines. At the same time, refactoring activity fell by almost 40%.
The Debugging Trap: Here’s a key issue for every developer: 67% spend more time fixing AI-generated code than they save by using it.
In comparison, 68% spend more time resolving security vulnerabilities. Senior developers write code 22% faster with Copilot, but junior developers? Only 4% faster. The difference isn’t the tool, it’s the deep technical knowledge required to use it effectively.
The Market Reality
Entry-level developer positions have dropped 60% between 2022 and 2024. Tech internship postings have declined 30% since 2023. Employment for software developers aged 22-25 dropped nearly 20% from its peak. As Stack Overflow points out, “Junior developers have always trained tomorrow’s senior engineers and CTOs. If AI removes this entry-level layer, companies may face a big skills gap in the next decade.””
By 2025, 78% of tech jobs will need AI skills. This is a basic requirement, not a standout feature. CTOs now focus on different priorities. Systems thinking, design skills, and debugging ability are more important than just coding speed.
What AI Can’t Replace
Despite the hype, AI handles only 18% of architecture planning and 24% of API integration work. The remaining 76-82%? That still requires human expertise. 63% of developers feel that AI tools miss key context. This context is vital for grasping codebases, internal architecture, and institutional knowledge.
Simon Willison, co-creator of Django, says he feels “2-5× more productive” with AI. He adds, “The job of a software developer is to deliver code that is proven to work.” AI doesn’t prove anything. It generates possibilities.
Martin Fowler’s classic observation has never been more relevant: “Any fool can write code that a computer can understand. Good programmers write code that humans can understand.”
The Path Forward
In September 2025, Fast Company reported that the “vibe coding hangover” had begun. Senior engineers are facing “development hell” while using AI-generated code.
The data shows a clear winner: developers who gained deep technical knowledge before AI came. They learned to use AI well and now write code 2-5 times faster. Everyone else is debugging.
The METR study developers thought they were 20% faster but were actually 19% slower. The difference between perception and reality? Deep technical knowledge.
AI is an amplifier, it multiplies whatever skill level you bring to the table. Give it a shallow understanding, and you’ll generate shallow code. Give it deep technical knowledge, and you’ll unlock genuine productivity gains.
In the AI age, deep technical knowledge isn’t optional. It’s the only moat you have left.
Cheers,
Eric Roby
Find me online:
LinkedIn / YouTube / Threads
Key Sources
METR 2025 Study - “Early 2025 AI Experienced Open-Source Developer Study” - https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/
Stanford University Security Research (2024) - AI-assisted code security vulnerabilities study - https://ee.stanford.edu/dan-boneh-and-team-find-relying-ai-more-likely-make-your-code-buggier
Bilkent University Code Accuracy Study - ChatGPT, GitHub Copilot, and Amazon CodeWhisperer correctness rates - https://www.augmentcode.com/guides/debugging-ai-generated-code-8-failure-patterns-and-fixes
GitClear Technical Debt Research (2024) - Code duplication and refactoring analysis - https://leaddev.com/technical-direction/how-ai-generated-code-accelerates-technical-debt
Stack Overflow Developer Surveys (2024, 2025) - AI tool usage, trust metrics, and employment data - https://survey.stackoverflow.co/2024/ai and https://survey.stackoverflow.co/2025/ai
Harness State of Software Delivery 2025 Report - Debugging and security vulnerability time analysis - Referenced in https://www.qodo.ai/blog/technical-debt/
Y Combinator W25 Batch Analysis - AI-generated codebases data - https://techcrunch.com/2025/03/06/a-quarter-of-startups-in-ycs-current-cohort-have-codebases-that-are-almost-entirely-ai-generated/
AI Jobs Report 2025 - ICT roles requiring AI skills - https://adsknews.autodesk.com/en/news/ai-jobs-report/
Stanford Employment Study - Software developer employment aged 22-25 decline - Referenced in https://stackoverflow.blog/2025/07/29/developers-remain-willing-but-reluctant-to-use-ai-the-2025-developer-survey-results-are-here/
Expert Quotes:
Andrej Karpathy (OpenAI):
Simon Willison (Django): https://simonwillison.net/2025/Mar/19/vibe-coding/
Martin Fowler: https://www.goodreads.com/quotes/6341736-any-fool-can-write-code-that-a-computer-can-understand
Fast Company (September 2025): https://en.wikipedia.org/wiki/Vibe_coding
Andrej Karpathy@karpathyThere's a new kind of coding I call "vibe coding", where you fully give in to the vibes, embrace exponentials, and forget that the code even exists. It's possible because the LLMs (e.g. Cursor Composer w Sonnet) are getting too good. Also I just talk to Composer with SuperWhisper11:17 PM · Feb 2, 2025 · 5.22M Views1.35K Replies · 3.37K Reposts · 30.7K Likes
Thanks for the article. I read somewhere online, AI is good only if you know what you are doing (I believe macro & micro alike). Seems like it.
The METR numbers are wild but I reckon there's a missing variable nobody talks about. The framework matters. I've seen completely different outcomes depending on whether the AI is working inside an opinionated framework with strong conventions versus an unopinionated one where it has to guess at everything. Covered this angle in depth here: https://reading.sh/laravel-is-the-framework-for-the-agentic-era-76a8f8ec1fb4