After years of consuming a heavy diet of Microsoft Office and Oracle, we’ve been conditioned to think that software improvements are packaged in versions. This atomic ‘all or nothing’ upgrade comes from the days when vendors batched all the updates for a release, scripted the installation procedures, and sent you a CD that contained all the magic.
The user didn’t know whether the latest build contained a single change or 100% new code— it was just ‘the next version’. We also came to expect pain and unpleasantness from this method of upgrades. For most IT shops, upgrades are synonymous with outages, instability and irate calls — not fun.
This approach to upgrades is single-handily the most unhelpful idea to making better applications for users. We must shift this way of thinking about software changes.
I’ve been trumpeting the idea of features over versions for ages — and corporate IT leaders still look at me like I’m crazy. But do you know which version of Google search you’re using? What the current version Amazon.com’s homepage? And what are the odds that you’re using the exact same version of those pages as me?
In the world of cloud applications, nobody — nobody — is talking about versions. Features are all users care about — and with the cloud, we can now deliver without the tyranny of traditional deployment.
James’ Book Listing Service
I’ve been busy again with another unicorn venture. This time I’ve built an incredibly useful Book Listing Service that allows you to add book titles and authors, and then list them. In expectation of its success and VC funding, I’m reserving my new Tesla.
Under the covers, the Book Listing Services looks like this:
As users are prone to do, they immediately provide feedback with a list of new feature requests which they never mentioned before. Tssk, users! And the developers also injected a couple of changes as well — so my backlog is already looking like this:
There are now hundreds of active paying users expecting me to start rolling out these changes — so what are my options? Traditionally, there are only a couple of ways to release changes into a production environment:
- The Big Bang. We take the system offline late at night, run whatever processes are needed to make the changes, and put it back with fingers crossed so hard we’re practically cutting off the circulation.
- Build a duplicate. We buy new hardware, install the latest version, see if it looks okay and then cut everyone across.
As a Product Manager, I want to roll all these features together into version 2 because multiple deployments are painful. But one of my best customers is unhappy and is demanding an immediate bug fix or she’ll stop using the product— you can only add book titles up to 50 characters. What to do!?!
A feature-driven approach to the rescue
Fortunately, our best developer is a master of micro-services, a curator of the cloud, and likes nothing more than using her services sorcery to solve problems like these. She decomposes the back-end design further:
We start mapping features to required changes in the main three components and realize we can do some clever things. It turns out that our two main services are actually Lambda functions and our front-end talks to AWS’ API Manager reach them.
I call our star customer and ask if she’s interested in becoming a beta user — we’re going to make the change today and she will be the only person to receive the feature. If it looks good and doesn’t cause any issues, we’ll then roll it out to everyone.
We will allow a subset of our users to see a different version in one component of the system, while everyone else stays on the old version:
Fast forward a week and this change is promoted to all active users — and I’m now choosing the color and interior finish of my Tesla. I work with my lead developer and very soon we are flowing testers and developers onto different versions of the components so everyone sees a slightly different ‘version’ of the system:
Our developers are now working on a feature that will check the book price using Amazon’s API, but it’s working against production versions of the other services. Needless to say, our QA people are doing back-flips down the hallway.
In this model of our coding factory, the features list— the backlog — is our list of customer orders, and we keep the conveyor belt full of code moving into production in a fairly constant, continuous state. We can easily roll back changes if needed, apply automated testing to the release process, and generally feel good that we are both delivering customer value and not hurting stability. Just so much winning.
The Myth of the Next Version
Although my book listing example is very simple, you can see how we can do things very differently with service oriented architectures and the cloud. By decomposing functionality into smaller and smaller units, we can move versioning down from the overall product level down to the code level.
This approach has positive impacts on customers and product managers:
- You can blend development, testing and production environments together. That’s crazy, right? Well no, not really — look at Gmail Labs features as a example. By throwing beta-level functionality to a subset of users, Google can launch testing-level features into a production space.
- You can A/B test ideas. You see this on webpages all the time (or not, since it should be invisible) where companies test multiple features to see which perform better. You can adopt this approach in any other software to compare how features are used.
- You can release constantly. Looking at the feature board for products I’ve worked on, they’re usually composed of customer-driven features that can often be released one at a time (or in small groups). There’s no need to wait for some “next version” — just keep pushing out the features.
- You can make upgrades to systems that previously were considered untouchable (the ‘too important to fail’ problem of monolithic systems).
- You can change direction more easily, making it much more compatible with agile-minded product management.
Types of upgrade in the cloud
The Books Listing application illustrates a micro-service approach, but there are several other types of upgrade that commonly affect cloud infrastructures.
- Upgrading your instances
If you’re managing your own code on EC2 instances, this isn’t too different from on-premise upgrades and has the same likelihood of success or failure. An upgrade script is running on the instance and if it works — profit! — and if not, we attempt to rollback and hope the rollback state is just like it was before.
Alternatively, and preferably, you have a stateless army of instances. You upgrade one, test it, and if it works as expected create an image to generate new clones. If it doesn’t work, you terminate it — a few pennies poorer but the lights are still on.
Amazon Machine Images (AMIs) are grossly underused among the clients I’ve worked with. You can have an inventory of different OS’s, stacks and applications at different version numbers all stored as APIs, waiting patiently to get spun up into live instances whenever you need them.
2. Managed application upgrades
If you’re on Google’s App Engine or Amazon’s Elastic Beanstalk, these environments create a safe space to do tightly-controlled upgrades with reliable rollbacks and versioning baked in. These are almost always a better way to deploy apps than juggling EC2 instances and really handle a lot of the administrative pain for you. These are fantastic services that developers fall in love with quickly.
3. Complex project upgrades
This is where on-premise IT often shuns the cloud believers — “You could never perform an Exchange/PeopleSoft/Dynamics/[insert horrific software here] upgrade in the cloud!” they proclaim. Well, we can and we do, more quickly and with better results than the equivalent on-premise plate-spinning spectacle.
The reason is we have CloudFormation — which I’ve come to believe is the greatest thing since Python. Even if your upgrade has a mixed of server types, databases, firewalls and security changes, we can script the upgrade in CloudFormation and safely test by spinning up entire cathedrals of virtual hardware with the new version. Once it passes the tests, we just point users to the new stack. It’s a beautiful thing.
The factory that produces nothing
The reason I’m fascinated by versioning is because the process is central to getting new software out the door. As a lifelong Product Manager and entrepreneur, ‘releasing stuff’ is the most important thing a development team can do for me and, by extension, my customers.
In most IT shops, the traditional versioning and upgrade cycle becomes a reason not do things. But I’m more interested in working in a cycle that makes the lack of change the exception, not the rule. I want to ferret out anything that doesn’t change and increment its version number just to make a point. User feature requests never stop — if they do, our product stinks. And we need to keep delivering — if we don’t, our product will stink.
Technology managers often see a contention between change and stability. Their empire is a factory where widgets roll off an assembly line and anything that threatens the widget-rolling is bad. The problem is that just being operational doesn’t mean you’re producing anything of value to the customer.
We are standing up a whole operations where virtual conveyor belts run with nothing on them. But in software, the feature releases are the widgets. We need to turn this thinking on its head — when you release nothing, you produce nothing.
TL;DR — Let’s release features not versions
- Pushing versioning from the overall application level down to the code level can have a major positive impact on agility and stability.
- Using the tools available in the cloud, we have enormous flexibility in implementing a highly controlled and testable approach to versioning.
- Being stable is nothing to celebrate — it’s expected that you can talk and chew gum at the same time.
- Don’t be the factory that produces nothing — your customers want to see a steady stream of improvements and feature releases.
- Versioning isn’t atomic — we can reveal different versions of infrastructure to different audiences and even put testing, development and production environments together if we want.
Get the skills you need for a better career.
Master modern tech skills, get certified, and level up your career. Whether you’re starting out or a seasoned pro, you can learn by doing and advance your career in cloud with ACG.