The business of Information Technology (IT) has a complexity problem.
As a domain IT requires deeply specialized skills, the language of IT is arcane and business needs require multiple levels of translation and interpretation before being implemented as systems. This conceptual complexity creates a significant barrier between business and IT leadership. But there’s a bigger problem: The most complex issue facing IT today is technology itself.
The IT infrastructure of most reasonably sized businesses is a horrendous cats cradle of interconnection and information flow between multiple aging systems that were never designed to work together and yet the emergence of horizontally integrated – and increasingly globally distributed – business processes demands and depends on the friction free flow of information.
Time-to-Value is perhaps the most important metric in today’s dynamic, competitive service-led economy but against this measure most corporate IT organizations fail miserably. The requirement for IT to keep pace with the speed of business has never been greater but the ugly reality for most CIOs is that the complexity of their IT infrastructures is overwhelming and an increasing percentage of IT budgets is now spent on maintaining and integrating aging legacy systems rather than delivering new value to the business.
No wonder most CEOs are skeptical about the strategic value of IT.
Scarcity and Complexity
Our legacy IT systems are inflexible, difficult to integrate with each other and require incredible effort and resources to transform to meet the needs of modern business strategies and service models. Even when we do successfully integrate multiple systems the cost of operating and maintaining these complex – and fragile – environments is unacceptably high.
The vast majority of business IT systems operating today are the product of what Zysman and I call the “Era of Scarcity” – the period until 2006 where computation, storage, program memory and bandwidth were treated as scarce and often incredibly expensive resources.
As a result of the resource economics of the “Era of Scarcity” the architecture of IT systems focused on shepherding and optimizing the resources used every layer of software in the architecture.
An IT system designed during the “Era of Scarcity” can be likened to a finely crafted watch. Each part is precisely engineered and is highly optimized for its function and each part has a distinct role to play. The parts making up a pocket watch cannot easily be transformed into a wrist watch: So it is with most IT systems today.
Wrist watches and today’s IT infrastructures are examples of tightly coupled systems. Each component of the system is designed and optimized to work only within a unique context. The drive to high degrees of optimization means that parts cannot be shared or re-used in other contexts. It also means that the failure of one part – more often than not – causes a failure of the entire system: Tightly coupled systems are often highly fragile.
The more tightly coupled a system is the more difficult it becomes to add new capabilities and to integrate one system with another. Tightly coupled systems are optimized within themselves but create significant friction when we try to use them in a broader context not anticipated by the original designers.
The overwhelming complexity of most business IT infrastructures arose directly from the historic need to build tightly coupled systems in order optimize the use of scare computing resources. As we now attempt to satisfy the needs of complex global business models the many limitations of this tightly coupled approach are becoming abundantly clear.
That most IT departments are today failing to deliver new capability at the rate demanded by the business is no surprise. The complexity and fragility of systems designed during the “Era of Scarcity” makes them not fit for purpose for the era we are now entering.
Abundance and Opportunity
In 2006 Amazon launched their new “Amazon Web Services” business. Amazon placed a bet on the coming “Era of Abundance” when computing resources would be both cheap and essentially unlimited.
The economics of computing crossed a threshold in 2006. Computation, storage and – increasingly – network bandwidth had become commodities. As the supply of available computational power outstripped demand, prices declined and Amazon saw an opportunity to become the de facto utility provider of this new era.
As was the case during the “Era of Scarcity” the new commodity economics of the “Era of Abundance” is driving a fundamental shift in IT systems architecture. This new approach to architecture is commonly referred to as “Cloud Computing.” [A short guide to cloud computing can be found here]
If a tightly coupled approach to application and systems architecture was the optimal solution during the “Era of Scarcity” then the opposite – loosely coupled – approach is the defining characteristic of abundant cloud computing architectures.
Abundant and cheap computing resources mean that systems today can be designed to be flexible from the start.
Modern cloud based applications are designed to scale elastically as user demand waxes and wanes. New cloud based approaches to data storage keep multiple copies of data in different locations to improve performance and reliability. Multiple layers of management software are used to decouple applications from the underlying infrastructure to improve scalability and reliability.
Well designed cloud computing systems take a “Lego building block” approach to their architecture. The APIs – Application Programming Interfaces – between components are a contract defined using standard and open protocols. These contracts are the glue that define the structure of loosely coupled cloud architectures.
The use of APIs between system components is the key to building loosely coupled systems. This approach enables different systems – designed independently – to be rapidly integrated and re-configured as needed by the business. Individual building blocks can be replaced over time as new technologies and better approaches become available. Systems based on cloud architectures are designed from the start to be both composable and reconfigurable.
The Great Decoupling
The arrival of the “Era of Abundance” and with it the emergence of new cloud computing architectures may very well offer a light at the end of IT’s complexity tunnel.
The ability to now design and construct a new generation of loosely coupled business systems based on cloud computing architectures offers the opportunity for IT departments to deliver flexible, scalable, reliable and agile solutions at a pace demanded by business leaders. The concept of loose coupling may turn out to be solvent that finally dissolves IT’s historic complexity problem.
It’s interesting to note that both mobile computing and BigData are also decoupling mechanisms. Mobile computing decouples information access and physical location. New approaches to Big-Data and analytics decouple the need to understand how data is structured from questions that will be asked over that data in the future. It is the decoupling effect of these technologies that makes them such powerful transformation tools.
Accomplishing the transformation from a complex, fragile and tightly coupled legacy environment to a new generation of agile, scalable, fault-tolerant loosely coupled systems will be the defining challenge for every CIO. As difficult as the transformation might be it is now not one that can be avoided. The career of most CIOs will hinge on the outcome.
The combination of cloud computing architectures, mobile computing and Big-Data analytics represent the “Great Decoupling” of business IT. The flexibility and agility of the systems that result from this will enable a fundamental transformation in the way we do business. The powerful combination of loosely coupled business processes and IT systems will turn complexity from something we struggle to control and overcome into something we embrace and exploit.