by Tim Wagner and Shruthi Rao
The world of The Fifth Element
Picture this scene from the movie, The Fifth Element: Leeloo (Milla Jovovich), the lab-generated ‘perfect being’ from the DNA remains of the Fifth Element, has just escaped her body generation tube and made her way to the outside of the building via the air vents. She stares down to see thousands of cars, flying seamlessly as though they are driving on well marked streets. To evade capture, she jumps, crash lands into a taxi driven by Major Korben Dallas (Bruce Willis) and breaks his autopilot. Bruce Willis attempts to navigate the car manually, constantly breaking the rules — the other flying vehicles warn him about his errant behavior and his license points are deducted when he doesn’t comply. Finally the police is automatically dispatched (though of course he evades capture) and his taxi even refuses to start based on his recent driving record.
Building tomorrow: things talking to things without human intervention
For much of history, communication was the sole prerogative of human-to-human interactions. In the last century, however, we’ve increasingly created a world where we can ask machines, rather than people, to help us. Phones that help us reach anyone in the world, automated teller machines to give us money or movie tickets, search engines that scour the globe looking for information of interest…human-to-machine interactions permeate our everyday lives. But to reach the future of flying cars that help keep passengers in their own and in other cars safe, we need a new era of machine-to-machine communication.
In this new era, ‘things’ are autonomous, workflows are automated, and prediction is the name of the game — like predictive supply chains that learn ketchup consumption patterns, helping everyone in the supply chain from tomato farmer, to ketchup manufacturer, to bottling plant, to transportation provider, to warehouse space planners, to grocers and restaurants ensure that they’re ready to meet demand accurately, without stockpiling or waste. In this new world, information moves in every direction: airlines can add flights when more business travelers book tickets, but businesses can also adjust their travel plans when airline prices or availability changes. It’s not just a globally connected world, it’s a global database with AI-assisted workflows running on top, optimizing businesses 24×7.
Much of what we need to build this future we already have: the internet, the cloud, ubiquity of mobile devices, cost effective IoT. But we’re still missing some key pieces. The most important gap is real-time data sharing: Whether they are self driving cars or autonomous supply chains, multiple parties need to be able to share data in real time to make mission critical decisions. This data needs to be trustworthy and tamperproof, secure, respect PII regulations and corporate governance requirements, and most of all must travel with its provenance — the lineage of who owns a piece of information, what can be done with it, how fresh it is, and what happens if the owner changes their mind about who can access it can no longer be divorced from the bits themselves. This new form of data isn’t just shared or owned, it’s mutually managed.
The Fifth Element milieu requires more than just sharing data, however — machines need to be able to propose transactions, consider alternatives, and arrive at consensus automatically. When Korben Dallas’s taxi arrives at an intersection, the traffic light, the other vehicles, the vehicles behind those vehicles — all of them execute a literal choreography in the sky, driven by algorithms, powered by real-time data, and assisted by predictive AI. Of course, in his intention to save his new beau Leeloo, Korben overrides the taxi and ploughs through the intersection, disrupting the fully autonomous flow of traffic.
Once cars, factories, or devices agree on a path or a transaction, both the decision and the resulting actions of all the participants need to be immutably recorded to ensure compliance…or in the case of The Fifth Element, penalize Korben’s bad behavior. The warnings from his Taxi Control Unit, from other vehicles, and the final deduction of his driver points from the licensing system are all critical to ensure the carnage from our smitten hero does not continue to put other passengers in danger.
Today: the world is not enough
To contrast The Fifth Element’s flying cars with today’s reality, consider this the next time you enjoy a blueberry: the shelf life of blueberries decreases rapidly once inside a truck, literally turning into jam under their own weight…and warm temperatures accelerating their decline into mush. A blueberry distributor needs real time data on traffic and weather conditions, real-time location from delivery vehicles and the driver’s mobile device, the IoT data feed from the truck’s refrigeration unit along with the temperature inside the truck to ensure blueberries get to your grocer intact. With this data, automated workflows can run machine learning models, recommending optimal routes to the driver, scheduling stops at distribution centers where the berries can cool off in a blast chiller or even redirect berries beyond saving directly to a jam plant. Drivers who, like Korben Dallas, decide to break the rules (in this case by taking so long to get to a destination that the berries lose their consumer value) can be warned and even penalized…all without human intervention.
How close are we to this outcome? Many of the difficult elements already exist: internet connected temperature sensors for inside a delivery vehicle are inexpensive and work hands free. Truck drivers have always-connected mobile phones that can run powerful apps. AI/ML training is an established technique for enterprise data teams. But unfortunately some of the key pieces are still missing. The most critical of these gaps: dumb pipes. Point-to-point DIY REST APIs can ship bits around, but that information is often inconsistent, out of date, and noisy. Want to know where a logistics or distribution center currently has your stuff? Ask it. Want to know if it changed? Ask it again. IT departments that would never permit in-house databases to have incorrect or missing information routinely accept these limitations when that data is multiparty. And the rub? 80% of business data is now outside a business’s four walls!
Transactional (‘ACID’) data sharing across parties is foundational, but it’s just the lowest level of the future tech stack. Automated code and workflows — smart contracts — need to be able to operate on this tamperproof, shared database that can cross company, cloud, and regional barriers. Multiple parties need to make decisions, often without human intervention. Compliance and results need to be tracked with the same fidelity as the source data that triggered those outcomes. The data needs to know where it came from, enabling the owners of data (whether individuals or corporations) to exercise control over privacy protection, monetization, and security. And most importantly, all this needs to happen without human intervention. That means no spreadsheets or phone calls, but also no operators turning dials: autonomous systems need to also be “babysitting-free” IT stacks in order to become ubiquitous.
If you read this and think “blockchain”, you’re half right: there’s no way to build across companies without a distributed ledger that consistently replicates data across clouds, companies, and geographies. No single cloud (or on prem) provider’s “walled garden” will be big enough to hold all the world’s interconnected companies inside it. No real-time data sharing can succeed with the impoverished semantics of “dumb pipe” APIs. And no individual company can hope to centralize all the world’s identities, workflows, and device management or supply chain data.
If you read this and think “serverless”, you’re also half right: we can’t build a system capable of autonomous workflow if step one is for every company to hire a new team of engineers, teach them Solidity, and have them stand up a massive new fleet of servers to start sharing data. The cloud is foundational to building the future, but it’s the serverless part of the cloud that’s needed — those services that don’t require capacity planning, constant security updating, network managing, extensive monitoring, or other PhD-level distributed zen skills to create reliable, highly available business intelligence and outcomes.
The real-time, autonomous cloud of the future also needs to be cost effective in two ways: staffing that scales sub-linearly with the number of machines and high utilization. Server- and container-based approaches, even those built using modern technologies like Kubernetes, typically have single digit utilizations. That economic reality isn’t plausible at the scale of an interconnected world, to say nothing of the energy and greenhouse gas consequences. Only serverless approaches, with their 100% utilization by design, can ultimately deliver an energy- and cost-efficient solution that spans the Fortune 1000 and beyond.
Unfortunately, the blockchain technologies that exist in the market today are little more than toys. Ethereum’s 15 transactions per second, limitations such as Amazon Managed Blockchain’s “no more than 14 participants’’, and POCs that can take up to a year and $1,000,000 just to run an experiment are a nonstarter for most would-be adopters. Meanwhile, Serverless offerings, such as AWS’s Lambda, while fantastically scalable and multi-tenanted, are also single owner and proprietary — they don’t have any easy way of spanning accounts, let alone other clouds, and lack a built-in data model, turning the most foundational element of any application into a complicated, home-grown DIY: Step 1, build your own multi-cloud, distributed ledger. Then, layer AWS Lambda on top of it and try to solve the cross-cloud compute problem.
Building tomorrow, today
Let’s get back to the sea of flying taxis in the world of Korben and Leeloo. What does it take to create that dynamic consortium and help them reach agreement on who flies where? Assuming that multiple vendors are involved in making taxis, one thing is clear: no single company’s IT stack, no single cloud, no single (centralized) database, and no single API is sufficient. Autonomous things at scale must also inherently be internal IT stack agnostic. The shared data and contracts are powerful and choreographing these outcomes are based on a shared schema and a shared notion of business agreements, all of which have to be machine executable.
In this world, code isn’t the province of a single company, cloud, or database vendor — it exists in replicated form across a collaborative network that spans all of those. Shared data provides flexibility, enabling business relationships to come and go. It also provides fault tolerance, since the system can survive the loss of any given party and even heal itself automatically when that party (or its infrastructure provider) comes back online.
Shared data must also be operationally isolated — one party with millions of mobile devices hooked up to its internal systems can’t be allowed to “brown out” its business partners. Capacity planning decisions, good or bad, in one company’s IT can’t be allowed to compromise the data integrity or internal operations of its partners. Hyperledger Fabric, Ethereum, and other existing blockchain and distributed ledger solutions fail this basic litmus test, with their ability to scale compute, memory, and communication bandwidth limited to a single machine and their network capacity fundamentally dependent on the internal infrastructure scaling decisions of every major participant.
Vendia addresses the key shortcomings of both blockchains and serverless cloud offerings by filling in the “missing pieces” of each: By adding a distributed ledger to serverless, it solves the multi-cloud problem and provides a stateful data layer. By building a distributed ledger using a serverless architecture, Vendia turns blockchain concepts into a truly scalable and distributed solution — one that can “put a supercomputer in every node” through the nearly limitless capacity of serverless infrastructure. And by offering it all in a SaaS delivery system, Vendia frees companies and organizations from the challenges of managing servers, dialing capacity up and down, and dealing with 24×7 ops staffing. Vendia makes the blockchain vision real…and makes serverless architectures more useful.
A leading berry company wants to solve their “mushy blueberries” problem. They want to easily connect with all their partners and customers without worrying about any given company’s existing infrastructure choices: AWS, Azure, Google, or on prem; SAP, Oracle, or DynamoDB; Redshift or Snowflake…regardless of those choices, they want to participate easily in continuously on, globe-spanning autonomous workflows that operate without human intervention. The berry company has no appetite for multi-month, need-a-new-team challenges associated with standing up conventional, server-based blockchains or to build out multi-region, multi-company solutions using serverless approaches, and want to create a multi-party distributed database and smart contract platform directly from a schema (JSON or alternatives) that represents the data model. The need here is a process that is as easy as creating a centralized database table…except that “table” now spans companies, clouds, regions, and tech stacks!
At Vendia, we are helping build tomorrow, today by using the best tenets of Serverless and Distributed Ledger Technologies. The berry company would create a simple JSON schema to model their business workflows, including suppliers, routes, and distribution channels. Vendia turns that into a scalable, distributed, multi-cloud network in under five minutes — speed of deployment and lack of infrastructure to operate is one of the benefits of an all-serverless architecture. The vendor’s partners would also connect to the running application and start sending and receiving transactions…confident that their data is secure and can be restricted to viewing or editing by a subset of the partners. An immutable two-phase commit protocol replicates data with transactional, ACID semantics and cloud native architecture makes it easy to express smart contracts as AWS Lambda functions, connect to mobile and web apps through GraphQL-based APIs, and integrate with existing databases.
Vendia’s nodes are “no ops” — thanks to a SaaS-based delivery model, an independently contracted truck driver or a blueberry farmer — neither of whom possess an IT staff — can have their own, independent nodes. Analytics, ML training, and supply chain optimizations can all now be enacted on a corpus of data that is guaranteed to be consistent, up to date, and trustworthy, greatly simplifying both automated and human workloads. It’s not flying cars yet, but it’s a strong step in the right direction!
Building your own ‘Fifth Element’ solutions
Multi-party, multi-account, and multi-region applications can be daunting to construct, even for experienced teams of distributed engineers. In the past, autonomous workflows and machine-to-machine automation felt like a difficult challenge to surmount. Vendia takes the guesswork and heavy lifting out of these scenarios, making them as easy to create as a conventional web app. Whether you’re creating a cross-cloud IoT solution, building a multi-region AWS web app, implementing a multi-party supply chain, or modeling data ownership for an ML training set, Vendia makes the complicated easy. To get started, visit www.vendia.net, sign up for our developer preview, and start building the future!
Shruthi Rao is the Co-Founder and Chief Business Officer at Vendia — a multi-cloud serverless platform for sharing distributed data in real time. She was the founding member of Blockchain at AWS, where she launched and grew Amazon Managed Blockchain and Amazon Quantum Ledger Database and led Business Development for Serverless at AWS including Lambda and API Gateway. You can find Shruthi on Twitter @shruthialways.
Dr. Tim Wagner is the Co-Founder and CEO at Vendia, he is regarded as the Father of Serverless and creator of AWS Lambda. He was previously the GM of Serverless at AWS and most recently the VP of Engineering at Coinbase. You can find Tim on Twitter @timallenwagner.