Cloud is key to reinvigorating your IT infrastructure, but you won’t get very far until you stop the behavior that feeds your monolith
AirBNB for Fitness
Imagine that you’ve secured VC funding for a new concept called Muscle Unbound. Silicon Valley is referring to your concept as the AirBNB for fitness — homeowners worldwide can now rent out their personal exercise equipment when it’s not being used. Brilliant!
In preparation for the big summer launch, you’ve started deploying your cloud architecture and finalizing the design of a mobile UX. The entire platform is coming together so quickly that Werner Vogels is constantly chiming you about delivering the keynote presentation at re:Invent — and now he’s leaving messages on your Alexa.
Back in reality, you’re the technical lead for a national gym chain with 1000 locations. While your company is planning to introduce the exact same concept — you’ll need to make it work using existing on-premise systems.
Your company stores all transactional data in an Oracle nightmare, accounting data in PeopleSoft, member logins through a third-party application, your product data arrives lazily through mainframe batches, and there’s a security governance team approving code releases monthly.
Every one of these platforms will be touched as part of your project implementation. Can you hear that sound? That, my friend, is the airy sound of the candles burning on your retirement cake and all hope evaporating — long before version one is ever rolled out the door.
Welcome to the Monolith
This experience is common to anyone who’s worked in a reasonably large company — with the added pleasure of being security-slapped and Docker-blocked to the point where all-day meetings seem productive. After a while, you become obsessed with the idea there’s a better way. You’ve read about large organization like Google, Amazon, Facebook and others shipping code like a start-up, although they have the advantage of employing more engineers than Starbucks has baristas.
While trying to find a better way, you might’ve heard about using micro-services to strangle the monolith. While it’s inspiring, it’s not at all clear how you get there. Even with the cloud’s latest suite of goodies, it’s hard to strangle something you can’t get your hands around — and monolith fights back. Surprisingly, a monolith’s survival usually has more to do with the people than the technology involved.
The behavior and interaction of teams is a big driver of dysfunctional infrastructure design. When something fails, whatever the management group lacks in knowledge they often make up for with loud opinions. Based on how individuals and teams are incentivized, the technical teams will often decide that it’s much safer for their careers to release less often, resist change, and avoid failure at all costs. The dynamic of the two groups results in monolith-buiding.
The mainframe never died. It’s still here.
Oh, how we laugh at those old school companies with their IBM contracts. We picture how they have one special room with tons of air conditioning housing one big computer. We mock how it’s nursed by an army of middle-aged, well-dressed engineers who still use a Casio FX calculator and a pencil. Too smart for a job at the Geek Squad and too scared of heights to become cable installers — they are the sworn protectors of the mainframe.
In reality, we all build mainframes everywhere we go — no matter how small an application starts. The team adds features, bolts on unexpected interfaces, and lasso crap around something that once was nimble. The monolith is a virtual mainframe — it’s an unmovable black box of ordered chaos that always arises out of corporate systems.
You laughed at the actual mainframe programmer. But now you’re policing who gets to interface with your system, planning downtime — and dammit — you’ve got a pencil in the top corner pocket of your pressed white shirt. Richard Matheson would be proud.
When the tech moved from mainframe to client-server, then to n-tier and mobile, the premise was to move the work away from some central source. The process of decentralization itself supposedly breaks apart this rigidity, magic happens, and cue the end credits. But it doesn’t, it hasn’t, and it won’t.
The virtual mainframe of having one central system is now spread across lots of machines. It’s still there — hardening itself with every passing day. How did it survive when we thought we watched it die? How is this happening all over again?
Why does everyone build a monolith?
I’ve pondered this question extensively while pretending to watch The Crown. There are three things I’d like to mention as background in my evolving theory of why monoliths occur:
- Conway’s Law, which says your infrastructure design mirrors your organization’s communication structure. So, pyramidal bureaucratic hierarchical structures produce hierarchical IT systems. A friend who worked at Dell said internally they were organized around enterprise, consumers and students. Hence when you want to buy a laptop on their website, you’re forced to declare yourself one of these first. Conway, by the way, was brilliant and well worth reading.
- Human nature, which likes to simplify complicated things by keeping similar things together and expanding the surface area of entities that seem to work well. Like your house? Add a garage. Like your customer database? Keep adding columns. Attaching onto things that already work seems to the right thing to do but in IT it creates the ball of mud.
- Software development preaches loose-coupling and code reuse but it rarely happens after the first version of something is out of the door. Code gets added on and kluges happen increasingly often until it becomes fragile and untestable. Object-oriented techniques all too frequently decompose into a wicked-slick object model that works primarily because of a God object in the middle. That’s a baby monolith right there.
This pattern repeats itself in businesses over and over. You raise the golden goose, it’s laying eggs, you keep feeding it. Over and over. The architecture always ends up looking the same — a hub-and-spoke diagram with little boxes emanating from the giant monolith in the middle. Here we go again.
But why does it have to stop?
While we’ve been feeding our own private monster, the IT world has gone from mainframe and in-house architecture to open source, cloud-based and disparate solutions, all while the rate of change has accelerated. The fundamental fact is that this monolith will never work with these newer paradigms, and your company will never be able to keep up with customer technical demands.
I’m convinced this is why new start-ups are effectively trouncing the old dinosaurs — it’s not that they have beanbags in the office and pajama days on Fridays.
Why did Lemonade think of AI-centric insurance claims that are processed in 3 seconds and not Geico? Geico has a monolith — managed by a British lizard — which prevents radical change.
Why isn’t a single major traditional retailer beating Amazon? You can rest assured that the heart beating in the middle of Sears, Macy’s and Nordstrom’s is a cold, concrete monolith that will never be delivering the hundreds of features a day that Amazon is shipping.
That’s why. And beanbags.
The 5 Steps to Starving The Monolith
Getting passed this problem requires some rethinking of how things work because we cannot build truly distributed, agile systems this way.
1. Commit to starving the monolith
Don’t kick the can down the road and decide you’ll just add some more technical debt this one time. From now on, the monolith doesn’t get feed. That’s it. I know your Kanban board is growing relentlessly and you want a promotion, but we have to draw the line in the sand today.
Conway (brilliant, remember) also observed in systems design behavior that there’s never enough time to do it right but always enough time to do it over. So let’s just do it right for once.
2. “Two pizza teams”
Amazon is the only large company I’ve known that slayed the monolith violently and directly. And from the Amazonians I know, it sounds like the two-pizza team concept was key.
Let’s steal that. In your project, get the right 8–10 people together and own every single part of your solution. Don’t depend on a dozen other teams and getting prioritization in their queues because dinosaurs will be walking the earth again before your customers get any software. And let’s not wheel out the usual excuses of who’s going to be upset.
3. Build generically
When you’re building the shipping label system for your company, imagine it’s actually a start-up for shipping labels that will have thousands of external users. There is no existing monolith to connect into. You have to build everything you need to support your user base and their wide variety of systems. On your virtual private island of pristine code beaches, only a handful of APIs will connect to these systems of which you know nothing. Make those APIs rock.
If you’re not convinced, think about PayPal — they have a widget embedded on millions of websites and successfully manage payments with no idea about how any of their customer operate technically. Make sure you are always building the consumable widget or service that doesn’t know how its consumers work. Be RESTful and use the standards out in the wild that will help you.
4. Learn to embrace eventual consistency
In science fiction, HAL, the Matrix and the Terminators were all monoliths — they were single systems that knew everything going on real time. But notice how they could never get any upgrades out? The T-800 series was excellent at pursuing the Connors, but in reality Skynet could never have deployed a successful mobile app.
In our new world, our independent systems will be slightly out of sync with each other but that’s okay. Only when we realize that we don’t need to know the exact number of sticks of spaghetti in every retail store can we allow the zen-like feeling to wash over us. We are going to be building lots of small systems with their own independent data stores that don’t always know the score … just yet.
5. Use your cloud superpowers
Working on-premise encourages you to repeat the same behaviors. Move your code to GitHub. Work remotely. Use Slack. Try decomposing into serverless functions. The sheer lack of compatibility between these approaches and monolithic behaviors is the beginning of the revolution. It will feel odd at first — but as you build more and more away from the old, it will start to decay and die.
[Spoilers!] How does this end …
Companies that have monoliths sporadically realize the technical noose is tightening and occasionally launch initiatives that clearly come from business people and not developers. Their 5-year plans to slowly migrate away lose steam after 6 months. The noose tightens. And the plan to document the old system and build the new either duplicates the problem in a different version and newer hardware, or it never gets funded because the consultant’s analysis was so expensive.
Starving the monolith ultimately leads to dozens, hundreds and thousands of other systems, functions and processes that slowly but surely take over. The King is Dead, but there’s no single point when the heart stopped, and no single point when the revolution started — we just stopped feeding the monolith and put it out to pasture.
Thanks for reading! If you like what you read, hit the ❤ button below so that others may find this. You can follow me on Twitter.