Sometimes software teams decide to engineer their way out of CI/CD vendor fees. The impetus for this post came from the author’s recent experience with a company who spent close to a developer-decade on a homegrown, Jenkins-based delivery pipeline in order to avoid the fees and ‘limitations’ of Harness.
If you are comparing the cost of a CI/CD vendor to that of a do-it-yourself project, consider that home grown systems carry the risk being “free like a puppy.” The human capital, dilution of focus and maintenance burden are easy to underestimate and can dwarf the cost of a paid service. In 2022, CI/CD vendors have come a long way so before you embark on your internal delivery platform journey consider what it would take for your team to match the functionality of what’s available.
So what is available to help your team deliver quality software today? In this article, we'll provide a light exploration of the staggeringly expansive CI/CD market. So let's jump into it by starting at the beginning.
What was it like before CI/CD? Twenty years ago it wouldn’t be hard to find a developer who manually scp’d code to production servers as part of their standard deploy process. If you were one of those developers you may have developed a habit of first cp-ing the existing production code and maybe that habit came after self-imposed downtime. You might have eventually scripted the whole process and put that Bash or Perl code in a scripts folder.
What about testing? Back then, humans did that. Unit tests were standard practice for serious software teams but scaffolding an automated integration testing system required a prohibitive amount of engineering. It was easier to fill a Word doc with test plans and throw bodies at the problem. It wasn’t unheard of for QA teams to out number developers!
But the 2000s developer was not exactly living in the dark ages. Kent Beck’s Extreme Programming was gaining popularity and one of its core practices – Continuous Integration (the “CI” in CI/CD in case you arrived here by mistake) – became best practice. But it was a best practice that very few software teams actually achieved until the right tools became available.
Jenkins launched in 2011 as a general-purpose “automation server” and quickly became the CI tool of choice for companies embracing the idea that automated test runs and frequent deploys are a Good Thing. Jenkins executed ‘jobs’ in response to ‘events’ where an event could be a git push and a job could be something like “compile this Java to bytecode and then copy it to a server”. Jenkins’ plugin system yielded a robust ecosystem of third party software packages enabling Jenkins to accomplish all manner of things. Today there are close to 2,000 plugins.
Alas, the richness of the Jenkins plugin universe eventually came to be seen as its downfall. There were too many ways to do a given thing. Each development shop ended up with its own unique combination of cobbled-together plugins and the term “Jenkins PlugIn Hell” was coined. Any sufficiently complex build system required its own build team with dedicated Jenkins experts to manage the sea of plugins and fleets of servers.
Over the next few years, a swath of companies emerged to create today’s CI/CD market including: CircleCI, Codeship (now CloudBees CodeShip), Travis CI, Harness and Octopus Deploy. Another decade on and these products have become mature and feature-rich. With a few lines of a config file (for simple pipelines), you can point one of them to your GitHub repository and they’ll test your commits, PRs and merges. They’ll deploy your code or let you know when something goes wrong. They’ll parallelize jobs in order to get you feedback as soon as possible.
AWS CodeStar is Amazon’s answer to CI/CD. It was launched in 2017 and is ultimately an umbrella term for the combination of Code Commit, Code Build, Code Deploy and Code Pipeline. A brief examination of its components can help us understand the requirements for a modern CI/CD offering.
As you may have guessed, Code Commit is a version control product - à la GitHub - and Code Build is build-as-a-service where you specify how to build artifacts from source code. A build can be a Java compile, a Lambda package or a Docker build command. In all cases “build” boils down to: extract some code, transform it and store it someplace - all in an autoscaling, serverless manner.
As we move down the pipeline from CI to CD, we find Code Deploy which manages the process of getting built images into your development or production systems. This can be done in rolling or blue/green styles. It uses an AppSpec (application specific) file to determine the scripted actions to perform in response to various deployment events. And finally, CodePipeline weaves each of these components into a full system.
Infrastructure as Code appeared on the scene as the CI/CD market was heating up. CloudFormation allowed engineers to replaced the imperative insfrastructure scripts with declarative description of the desired state where they could then rely on software to “make it so”.
An engineer merging a PR to his company’s Terraform repository
While CloudFormation is AWS-only, Terraform supports a similar declarative solution that is cloud agnostic and has better support for modularity and re-use. They are both focused on deploying infrastructure and building artifacts though you still need to compile your Java, package your Lambdas or build your Docker containers. Wait a minute, this is stuff that Jenkins knows how to do! And so, with Jenkins improving version-by-version many companies embraced a hybrid solution: Jenkins to build artifacts and then run CloudFormation or Terraform jobs.
By now it's late 2010's and modern tooling has made it easier than ever to manage Jenkins. All you need to do is monitor and patch the servers, tame plugin sprawl, maintain a few homegrown Python scripts and Lambdas, manage meta-pipelines and update a few YAML files. If, at this point, you decided to stay the course, it might have been because sunk cost fallacy shrouded your judgement and prevented you from paying for one of the wonderful CI/CD services available to you.
A VPE wishing he’d answered that InMail from his Harness rep
So we've made the case that you should have moved off of Jenkins a long time ago but what if you're considering your options today? Before we get there, let’s round out our survey with two of the latest CI/CD offerings: GitHub Actions (publically available in 2019) and Hashicorp Waypoint (announced in 2020).
If you already use GitHub, GitHub Actions has immediate appeal due to how easy it is to hang an automated workflow off your existing repository. But GitHub Actions is a general-purpose workflow engine so for pure CI/CD use cases, users will find it a less polished experience. CircleCI’s UI, for example, is purpose built for CI/CD and provides specific and useful insights into your delivery pipelines.
On the other hand, GitHub Actions seems to be drawing developer attention because of its focus on extensibility and reusability. An advanced delivery pipeline would probably take longer to build out in GitHub Actions than CircleCI and based on our above arguments against Jenkins, you might assume we’d recommend sticking with a previous-generation solution. But well-designed, composable primitives provide a powerful developer experience. Building a complicated structure with legos (GA) is not a comparable experience to attempting the same with string and duct tape (Jenkins). And compared to the existing CI/CD providers, reusability might be the killer feature. At its current trajectory, GitHub Action’s extensibility will soon enable more useful, shared functionality than available on previous-generation CI/CD platforms.
GitHub Actions does cost money, though there doesn't seem to be a note-worthy difference to other CI/CD providers. There is the option of self-hosted runners if you need to do-it-yourself. There are plenty of legitmate regulatory reasons to run workloads yourself but we are skeptical of self-hosting efforts for cost-cutting measures.
The focus of Waypoint is on the CD part of CI/CD. In fact, it has first class integrations with CI providers such as CircleCI. It's a workflow engine that manages three lifecycles stages (build, deploy and release) and attempts to create a unified abstraction across the complexity and variability of deploy targets. The argument against self-hosting Waypoint is not as cut and dry as with older generation solutions. Their docs describe self-hosting is "advanced and is not generally recommended" but if you're managing your own Kubernetes or Nomad cluster, deploying and managing Waypoint is probably not a huge lift compared to older projects such as Argo, GoCD or, you guessed it, Jenkins. Waypoint's abstractions and plugability make it an interesting choice to gain fine-grained control over your delivery lifecycle at a low cost of complexity.
Probably. There is an arms race between several great companies to make testing and deploying your code as easy, fast and cheap as possible. Standing on the shoulders of the talented engineers at these CI/CD providers can help you keep your team’s precious focus on your product. Even popular open source CI/CD projects (e.g. Spinnaker, Onedev) will usually have managed solutions.
Which one of the dozens of CI/CD products should you adopt? You can't go wrong with one of the incumbents. To hack an old phrase: you won't get fired for buying CircleCI. But we can't help but be intrigued by the innovations of the latest generation. It feels like they had the benefit of time to be thoughtful about creating effective abstractions. It's hard to craft the most elegant solution when you're racing to take a new market. Perhaps the incumbents would have taken different approaches in hindsight. Obviously your choice will depend on your stage, risk-tolerance, experience, requirements, priorities, etc. But hopefully our experience has helped you make a more informed decision.
Sym can help mitigate the risk of sensitive deploys with intelligent approvals-as-code. Check out our CircleCI orb for an example of how you can add approvals to your delivery pipeline.
