For a while already we have been working with a large enterprise client, helping them to migrate their on-premise workloads to the cloud. Of course, as added value to the process, they are also migrating their legacy development processes to the modern, better, agile DevOps approach. And of course, they have built a modern Continuous Integration/Continuous Delivery (CI/CD) pipeline consisting of Bitbucket, Jenkins, Artifactory, Puppet and some relevant testing frameworks. “It is all great!”, you would say “what is the problem?”.
Because I am on all kind of mailing lists for this client, I noticed recently that my dedicated email inbox started getting more and more emails related to the CI/CD pipeline. Things like unexpected Jenkins build failures, artifacts cannot be downloaded, server outages and so on and so on. You already guessed it – emails that report problems with the CI/CD pipeline and prevent development teams from doing their job.
I don’t want to go into details what exactly went wrong with this client but I will only say that year ago when we designed the pipeline there were few things in the design that never made it into the implementation. The more surprising part for me though is, that if you search on the internet for CI/CD pipelines you will get the exact picture of what our client has in production. The problem is that all the literature about CI/CD is narrowly focused on how the code is delivered to its destination and the operational, security and business side of CI/CD pipeline is completely neglected.
Let’s step back and take a look at how a CI/CD pipeline is currently implemented in the enterprise. Looking at the picture below there are few typical components included in the pipeline:
- Code repository – typically this is some Git flavor
- Build tools like Maven
- Artifacts repository – most of the times this is Nexus or Artifactory
- CI automation or orchestration server – typically Jenkins in the enterprise
- Configuration management and deployment automation tools like Puppet, Chef or Ansible
- Various test automation tools depending on the project requirements and frameworks used
- Internal or external cloud for deploying the code
The above set of CI/CD components is absolutely sufficient to getting the code from the developer’s desktop to the front-end servers and can be completely automated. But those components do not answer few very important questions:
- Are all components of my CI/CD pipeline up and running?
- Are the components performing according to my initial expectations (sometimes documented as SLAs)?
- Who is committing code, scheduling builds and deployments?
- Is the feature quality increasing from one test to another or is it decreasing?
- How much each component cost me?
- What is the overall cost of operating the pipeline? Per day, per week or per month? What about per deployment?
- Which components can be optimized in order to achieve faster time to deployment and lower cost?
Too many questions that none of the typical components listed above can provide a holistic approach to answer. Jenkins may be able to send you a notification if a particular job fails but it will not tell you how much the build costs you. Artifactory may store all your artifacts but will not tell you if you are out of storage or give you the cost of the storage. The test tools can give you individual test reports but rarely build trends based on feature or product.
Hence, in our implementations of a CI/CD pipelines we always include three additional components as shown in the picture below:
- Monitoring and Alerting Component that is used to collect data from each other component of the pipeline. The purpose of this component is to make sure the pipeline is running uninterrupted as well as to collect data used for the business reporting. If there are some anomalies alerts are sent to the affected parties
- Security Component used not only to ensure consistent policies for access but to also provide auditing capabilities if there are requirements like HIPAA, PCI, SOX etc
- Business Dashboarding and Reporting Component used for providing financial and project information to business users and management
The way CI/CD pipelines are currently designed and implemented is yet another proof that we as technologists neglect important aspects of the technologies we design – security, reliability, and business (project and financial) reporting are very important to the CI/CD pipeline users and we should make sure that those are included in the design from get go and not implemented as an afterthought.