Wasteful spending is all around us. Federal, state, and local government, companies of all sizes, and likely even your own household. Come to think of it, why am I paying for that landline again? So, the question begs…when evaluating IT spend, do you contemplate new approaches (like DevOps) to help reduce waste?
Wasteful behavior isn’t usually something that anyone sets out to exhibit. Rather, it is a result of maintaining the status quo. Contracts are negotiated, teams are built, technologies deployed. Then we move on to addressing the next challenge. Over time we standardize and create processes in an attempt to keep costs in check and control waste. Unfortunately, many times we are measuring success against historical data, like industry standards for year-over-year discounting. So we press on, spending on items we believe are required albeit at reduced cost. Success you say? We’ll get back to that.
In 2014 Gene Kim (co-author of The Phoenix Project) asserted that there is roughly $2.6 trillion in opportunity cost directly related to wasted IT spend. Let’s think about that. By continuing down “Status Quo Avenue”, collectively we’re missing out on opportunities equalling 2.6 trillion dollars. That could be expanding into new markets or garnering additional market share, faster feature release cycles, or even increased operational efficiencies. Assuming that this number is reasonably accurate, how does one go about identifying previously overlooked or endemic waste and use this to capture new opportunities? More specifically, how can DevOps be used as a mechanism to accomplish this or catalyze our efforts?
Before we can determine where the waste lies and recapture spend, let’s first take a closer look at opportunity cost itself. The earliest use of the term was in 1914 by an Austrian economist, Fredrich von Wieser. In his book Theorie der gesellschaftlichen Wirtschaft (Theory of Social Economy), von Wieser proposed the Alternative cost theory. The theory resulted from a debate between the English, who believed that cost was of a technical nature and that something needed to be produced from spending. The Austrians believed that cost was a result of demand which in turn determined a level of production, which could be directly attributed to a buyers readiness to pay that cost. Rising from this argument, if a consumer was willing to pay the cost associated with a good, they were willing to forgo another opportunity. Opportunity cost was born.
We can further break down opportunity cost into two categories: Explicit and Implicit.
- Explicit costs are those that an organization incurs when a good or service is paid for directly, removing the ability to spend elsewhere. For example, if a company purchases $100,000.00 in Red Hat Enterprise Linux licensing to obtain support for the Linux kernel, that money cannot be used for another purpose.
- Implicit, or sometimes referred to as implied costs, is an opportunity cost reflected not by cash transactions but the ability to allocate existing resources which are owned to the best alternative use. Applying this to IT, an example implicit cost could be assigning a team to an upgrade of a legacy system rather than supporting the implementation of a new platform.
Having defined explicit and implicit costs we can now start to make headway in determining spend that may not foster new opportunities. We can make outline both opportunity costs and file our spend under each category – providing our first step in determining real monetary output in an attempt to align DevOps with waste reduction.
- Data center (power, insurance, telecommunications, leases)
- Software licensing (OS, middleware, databases)
- Hardware infrastructure (server, storage, network)
- Project assignments (internal IT and business)
- Changes (feature release, upgrades)
- Unplanned work (bug fixes, outages)
With the spend categorized, our next step would be to look at each and see where we can potentially reduce spend. But this, as most companies have found, is very difficult to do or has already been significantly optimized. Furthermore, there are departmental dependencies and relationships that are not easily broken. For example, most organizations cannot remove a data center and expect to continue operations. There are services running in the facility, on hardware, which requires cooling and power. Also, removing the team of individuals which manage the infrastructure would not go over very well, especially when there are failures and no personnel to intervene. Acknowledging this, let’s take a different approach. What if we were to look at the same opportunity costs at a typical organization that has not implemented DevOps? How could we identify potential areas of improvement and optimization – reducing waste and capturing new opportunities?
We know that many organizations have decided to adopt agile development methodologies. By focusing on the principles of agile, including storyboards and tasks and taking iterative steps through sprints additions and changes are quickly made and implemented. This velocity is fantastic, however, what happens when there is no mechanism to build, test, and release at the same velocity? What tools are required? How do team responsibilities change to meet this demand? And most importantly – what is the cost impact?
In keeping with a typical company using agile development, let’s evaluate the next step following code development: Continuous integration (CI). Many shops are using CI today, but many have not optimized for cost, as they applied outdated approaches to the setup and management of the environment. Many companies use dedicated instances or virtual machines to support the elastic demands of build and test. Static configuration and manual instantiation supported by traditional capacity planning is de rigueur. Let’s look at the categorized costs a typical organization has for this single function:
- Underutilized, pre-provisioned server capacity for CI jobs
- Linux OS licensing for execution nodes
- Ops team to provision and maintain execution nodes
Rather than continue running a CI system and pipelines supported by legacy tools and processes, we can reconfigure them using a DevOps approach. First, we can reduce the number of systems that are pre-provisioned to handle workloads. By automating the scaling of execution nodes based on load (i.e., autoscaling groups, or better yet an orchestration platform) we can minimize the number of always-on devices. We treat infrastructure as a utility, only paying for what we use. Taking this one step further considering running CI tooling in containers. By doing so, we increase optimization. The CI tooling will schedule the execution of tasks in ephemeral containers, using version controlled images. If we have ample resources in an orchestration cluster (that runs multiple workloads) we do not need to pay for Linux support licenses on the execution nodes! Our containers appear to be full Linux environments to the CI tooling, which we can elastically scale without the need for additional commercial support for increasing capacity, thereby recapturing some explicit cost.
Notice above I mentioned version control of containers that are running our CI tools. But isn’t this an operations function? Indeed it is, and if you begin using version controlled infrastructure, we are removing the need to develop and maintain complex configuration management systems and scripts to instantiate our environments. Sure, there is some of this which is still required (i.e., setting up the physical or virtual infrastructure) but in general we can greatly simplify this. And since the containers are the same, regardless of what location we decide to run them, we can now spend more time working on other items. If you’ve thought to yourself “this is recouped implicit cost”, you are absolutely right.
Clearly, these are simple examples of how by reviewing our existing workflows and tools through a new lens and applying DevOps principles we can work towards recovering a fair amount of IT waste. By identifying the opportunity cost category, we are able to further refine and optimize our environment – capturing opportunities that may have eluded the business otherwise.
Opportunity cost. (n.d.). In Wikipedia. Retrieved May 10, 2016, from https://en.wikipedia.org/wiki/Opportunity_cost
David R. Henderson. “Opportunity Cost.” The Concise Encyclopedia of Economics.2008. Library of Economics and Liberty. Retrieved May 10, 2016 from the World Wide Web: http://www.econlib.org/library/Enc/OpportunityCost.html