Why Code Quality Alone Won't Save Your System
Clean code crashed in production. This is how I learned that beautiful code isn't enough
I watched clean code crash at one-tenth the scale of the legacy code it replaced. I was the mid-level engineer who saw it coming.
I raised concerns three different times. Nobody listened. This is how I learned that beautiful code isn’t enough.
The year was 2018. I was working for an e-commerce company, $50 million ARR. Management brings in Stephen, 8 years of experience, recently gave a talk called “Why Code Quality is Your Competitive Advantage.”
I’m two years in, eager to learn.
First team meeting, Stephen’s on the whiteboard for hours. Clean architecture, repository patterns, all the textbook stuff. I’m thinking this is impressive. This is how you’re supposed to build systems.
But there’s something in how he talks about it. This confidence that borders on dismissiveness. He waves his hand at the legacy system. “This is exactly why we need this migration. Clean code is what matters.”
I was about to watch him optimize for the wrong thing.
Then I Looked at the Numbers
Stephen assigns us to study the legacy PHP system. I open it. One file: 4,847 lines. No classes. SQL concatenation. Magic numbers. Pure spaghetti.
In standup, Stephen says, “I looked at the legacy code. It’s worse than I thought.”
But I’m curious. It’s messy, but it works. So I dig deeper.
And I start finding things. Buried in config files. Scattered through deployment scripts. This whole hidden infrastructure layer.
Three-tier caching. Cache hit rate: 91%.
Forty-seven database indexes on the orders table alone. Partitioning. Three read replicas.
Then I find a custom metrics client. 247 metrics. Tracked across every step of checkout. Error rates. Latency percentiles. Real-time dashboards.
I pull up the Black Friday 2017 data:
85,000 concurrent users
77,000 orders
245ms average latency
99.4% uptime
So I grab coffee with Mike, the legacy team lead. Fifteen years at the company. I show him the numbers.
“Mike, the code’s a mess but these numbers are really good. How?”
He thinks for a second. Takes a sip.
“Code quality’s just one dimension, Eric. Infrastructure. Database design. Observability. Those matter too. Maybe more when you’re under load.”
That still sticks with me.
I Raised Concerns.
First time, in planning:
“Stephen, I looked at their Black Friday metrics. They handled 85,000 concurrent users. Should we look at their caching and observability setup?”
Stephen barely looks up. “That’s just compensating for bad code, Eric. Clean architecture won’t need all that infrastructure. Trust me.“
And I’m thinking... but the data shows it worked.
But he has 8 years. I have 2.
Maybe I’m missing something.
Over the next two months, we build Stephen’s vision. The code IS nice. Clean structure, proper patterns, everything testable.
But infrastructure gets delegated to a DevOps engineer who’s never scaled before.
Our cache hit rate in staging: 23%. Legacy had 91%.
No observability. Basic logs. No custom metrics.
Week before beta. Load test. 1,000 concurrent users. Fifteen minutes.
Latency: 215ms. “Acceptable.”
Second time, I speak up:
“We’re testing 1,000 users. We had 85,000 on Black Friday last year.”
Stephen pauses. Looks at me. Then at the rest of the team.
“Eric, I appreciate you being thorough. But we can’t let perfect be the enemy of good. We’ll monitor closely in beta and optimize based on real data.”
Other engineers nod. It sounds reasonable.
I almost push back. I have the words ready.
But everyone else seems confident.
So I stay quiet.
I was out of chances.
Then It Crashed
Beta launch day. We flip the feature flag. New checkout enabled for 5% of users.
I’m at my desk watching. Three metrics. That’s all we have:
CPU: 15%. Okay.
Memory: 38%. Okay.
Error rate: 0%. Okay.
But I’m thinking about the 244 custom metrics we’re NOT watching.
Connection pool utilization? No idea. Cache hit rate? Can’t see it. Query performance? Blind.
2:33 PM. Customer support message in #incidents: “Getting reports of slow checkout, is something wrong?”
Stephen checks the dashboard. “Probably just perception. Numbers look fine.”
Then it happens.
DatabaseConnectionError: connection pool exhausted
Error rate climbing. Support tickets flooding in.
Behind that dashboard showing CPU and memory, here’s what we couldn’t see: our beautiful repository pattern was lazy-loading everything. N+1 queries cascading. Connections exhausting.
We had to roll back.
The rollback took 47 minutes.
Four Pillars. We Nailed One.
Four pillars to production systems. We nailed code quality. We ignored the other three. Here’s what I learned:
Pillar 1: Code Quality
Our repository pattern was textbook clean code. Single responsibility. Dependency injection. Testable.
But it created N+1 queries. Beautiful abstraction, invisible performance cost.
Clean code is necessary. But it’s not sufficient.
Pillar 2: Infrastructure
Under load, infrastructure beats architecture every time.
Legacy had 91% cache hit rate, three read replicas, proper connection pooling.
We had 23% cache hit and default configs.
Even perfect code needs infrastructure at scale.
Pillar 3: Observability
Legacy had 247 metrics. We had 3.
When things went wrong, we were blind. We could see error rate climbing. We couldn’t see which queries were slow, which endpoints struggled, where the bottlenecks were.
You can’t fix what you can’t see.
Pillar 4: Operational Culture
Confidence isn’t expertise.
I raised concerns three times:
“Trust me.”
“Don’t over-engineer.”
“Can’t let perfect be the enemy of good.”
Each time, discussion shut down. Because Stephen had 8 years and I had 2. His confidence was treated as expertise.
But he’d never scaled a system to 85,000 concurrent users. Neither had I. Neither had anyone on our team except Mike.
And we weren’t listening to Mike.
Culture kills more systems than bad code ever will.
Mike was right. Code quality’s just one dimension.
Clean code is easier to maintain, easier to onboard to, easier to reason about. Those are real benefits.
But clean code on weak infrastructure, with no observability, in a culture that dismisses concerns? That crashes.
Balance matters. Especially under load.
Have you seen this? A team that optimized one dimension and ignored the others?
Or been in my position, seeing problems early but unable to get anyone to listen?
Drop a comment below.
And if you want a deeper look at the 4 pillars framework, what good looks like for each one, let me know. I learned it the hard way.
Cheers friends,
Eric Roby
Find me online:
LinkedIn / YouTube / Threads
If you enjoyed this read, please share with one person or restack it. This is the absolute best compliment I could receive.