Why Your Architecture Is More Complicated Than Your Product

Developers do not over-engineer because they hate simplicity. They do it because hype is persuasive, and CVs reward novelty.

The Siren Song of the Shiny New Kit (The Problem)

I won’t pretend to be above it all; I’ve been there myself. Putting forward your best foot in a meeting, trying to convince the team of a specific technology, as though you had just finished a weekend course in aggressive negotiation, all in favour of you getting more exposure on that fancy new technology of yours.

Though I can say I’ve never done it to favour my own CV, I think it was still done indirectly, as a simple product of industry hype. I think most of us have been impacted by the who’s who of Gartner’s Hype Cycle.

We start with The Innovation Trigger, think of the trinity of over-engineering:

• Kubernetes - The answer to the question nobody asked about their simple monolith.
• Kafka - The message queue for when one really needs to complicate logging.
• GraphQL - The endpoint for the developer who dislikes REST so much they’ve reinvented it, but fancier.

We see adoption of these hyped-up technologies time and again when a standard database and simple REST endpoint could manage the workload. The costs become astronomical.

Next up, we hit the peak of inflated expectations:

Someone in a meeting says they have the answer to all our questions, and before you know it, they’re selling it like the latest Silicon Valley evangelist tries to sell us on more AI products. Before you know it, there’s a rush to implement it everywhere (the Resume Driven Development, or RDD, stage), and the inevitable, crushing realisation that it comes with a crippling operational overhead.

We hit our low in the trough of disillusionment:

The systems built purely for CV-polishing are often the first to enter the ‘trough’ phase, leaving behind a technical mess for the next poor soul, or maintenance engineer, to inherit. This is where we learn more about the costs of the system, but more on that in a bit.

Finally, we enter the slope of enlightenment and reach the plateau of productivity:

We begin to contrast the perceived glamour of these technologies with the quiet, boring competence of older, dependable tools. A simple SELECT * FROM Customers has never asked for a day of my weekend.

In the end, the cost extends beyond simple licensing. It goes well into the realm of inordinate cognitive load, maintenance overhead, and, not to mention, the difficulty of finding suitable talent for an over-engineered system.

As I mentioned, I’m not immune to hype and have been guilty in the past of turning what should have been simple CRUD into unnecessarily convoluted distributed systems that were extended into Elastic Search simply because I really wanted to learn that technology.

A Case of Digital Scrapheap Syndrome

Let me touch on the central paradox of the outlined problem: a system that boasts “three nines of availability” but takes six months to implement a trivial feature like a “sugar button.”

When we talk about the “sugar button”, we are referring to a really simple business request that becomes an epic journey through various queues, containers, and services. We end up with an impressive, diagram-worthy tech stack with sluggish real-world delivery speed and a miserable developer (and likely user) experience.

We need to pose the somewhat cynical question: Is the architecture designed to serve the business’s needs, or merely to serve the developer’s needs for their next job interview?

The Underrated Virtue of the ‘Boring’ Tool

There is a sensible philosophy of choosing the simplest, most robust, and most easily understood tool that genuinely solves the problem. A philosophy my good friend and colleague Bruce Morris often reiterates to me: Pragmatic Architecture.

In this philosophy, well-trodden, ‘boring’ technologies (eg, a mature relational database, a sensible monolithic service) are often the foundation of genuine business productivity. I go back to an older post of mine about delaying complexity, which touches on this very philosophy. Instead of introducing complexity in the hopes that it will simplify your solution, start simple and only move to more complex systems when the need truly arises.

The benefits are simple:

• Faster feature delivery.
• Lower maintenance overhead.
• Less stressed and more productive employees (probably the most crucial of the bunch).

Stop Polishing the CV; Start Delivering Value

Technology moves at such a fast pace that the latest and greatest will be old and dilapidated before long. Let’s forget about new and shiny and focus on demonstrable project success.

True competence is measured by what you ship and maintain with ease, not by the list of fashionable technologies you have merely touched.

If this rings true to you, why not leave a comment with your own RDD horror stories or architectural confessions?