Chapter 5: How to Set Up CI and CD
Chapter overview
Learning resources
Tools
- CI Server Tools (14)
- Feature Flag Tools (9)
- OpenTofu / Terraform Deployment Tools (5)
- Kubernetes Deployment Tools (2)
- Workflow Engine Tools (2)
Learn how to automate your software development lifecycle using continuous integration (CI) and continuous delivery (CD).
By Jez Humble and David Farley (Addison-Wesley Professional)
Winner of the 2011 Jolt Excellence Award! Getting software released to users is often a painful, risky, and time-consuming process. This groundbreaking new book sets out the principles and technical practices that enable rapid, incremental delivery of high quality, valuable new functionality to users. Through automation of the build, deployment, and testing process, and improved collaboration between developers, testers, and operations, delivery teams can get changes released in a matter of hours—sometimes even minutes–no matter what the size of a project or the complexity of its code base. Jez Humble and David Farley begin by presenting the foundations of a rapid, reliable, low-risk delivery process. Next, they introduce the “deployment pipeline,” an automated process for managing all changes, from check-in to release. Finally, they discuss the “ecosystem” needed to support continuous delivery, from infrastructure, data and configuration management to governance. The authors introduce state-of-the-art techniques, including automated infrastructure management and data migration, and the use of virtualization. For each, they review key issues, identify best practices, and demonstrate how to mitigate risks.
By Michael T. Nygard (Pragmatic Bookshelf)
Whether it's in Java, .NET, or Ruby on Rails, getting your application ready to ship is only half the battle. Did you design your system to survive a sudden rush of visitors from Digg or Slashdot? Or an influx of real world customers from 100 different countries? Are you ready for a world filled with flakey networks, tangled databases, and impatient users? If you're a developer and don't want to be on call for 3AM for the rest of your life, this book will help. In Release It!, Michael T. Nygard shows you how to design and architect your application for the harsh realities it will face. You'll learn how to design your application for maximum uptime, performance, and return on investment. Mike explains that many problems with systems today start with the design.
By Yevgeniy Brikman (Talk)
To go faster in a car, you need not only a powerful engine, but also safety mechanisms like brakes, air bags, and seat belts. This is a talk I did as part of NerdWallet’s NerdTalks Series where I discuss the safety mechanisms that allow you to build software faster.
By Yevgeniy Brikman (Talk)
At Gruntwork, I've had the chance to see the cloud adoption journeys of hundreds of companies, from tiny startups to Fortune 50 giants. I've seen those journeys go well. I've seen those journeys go poorly. In this talk, I discuss a few of the ways cloud adoption can go horribly wrong (massive cost overruns, endless death marches, security disasters), and more importantly, how you can get it right. To help you get it right, we looked at the cloud journeys that were successful and extracted from them the patterns they had in common. We distilled all this experience down into something called the Gruntwork Production Framework, which defines five concrete steps you can follow to adopt the cloud at your own company—and hopefully, to end up with your very own happy cloud deployment.
By Yevgeniy Brikman (Blog post)
By default, OpenTofu and Terraform record information about what infrastructure they created in a state file on your local file system called terraform.tfstate. For personal projects, this works just fine, but for professional projects with a team, you need a way to manage state that supports collaboration, locking, encryption, and multiple environments. A few years ago, I wrote a guide to managing state with Terraform and managing multiple environments with Terraform. Since then, OpenTofu has added several important features that provide new ways to solve these problems. This blog post is a tutorial on how to manage state and environments with OpenTofu in a way that is more secure and more convenient than what you can do with Terraform.
By Martin Fowler (Blog post)
Every developer integrates their work into mainline at least every day.
By Martin Fowler (Blog post)
'If it hurts - do it more often'. Good advice if the amount of pain raises exponentially with the time between actions, such as for integrating software.
By Pete Hodgson (Blog post)
Feature Flags can be categorized into several buckets; manage each appropriately. Smart implementation can help constrain complexity.
By Jez Humble (Talk)
Businesses rely on getting valuable new software into the hands of users as fast as possible, while making sure that they keep their production environments stable. Continuous Delivery is a revolutionary and scalable agile methodology that enables any team, including teams within enterprise IT organizations, to achieve rapid, reliable releases through better collaboration between developers, testers, DBAs and operations, and automation of the build, deploy, test and release process. I'll start by discussing the value of CD to the business, inspired by the lean startup movement. I'll then present the principles and practices involved in continuous delivery, including value stream mapping, the deployment pipeline, acceptance test driven development, zero-downtime releases, and incremental development. I'll cover how CD is enabled by an ecosystem including Devops, cloud computing, agile testing, and continuous deployment. Finally I'll talk about how continuous delivery can co-exist with ITIL and compliance in an enterprise environment.
By Rachel Potvin (Talk)
This talk will outline the scale of Google’s codebase, describe Google’s custom-built monolithic source repository, and discuss the reasons behind choosing this model of source control management. It will include background on the systems and workflows used at Google that make managing and working productively with a large repository feasible, in addition to a discussion of the advantages and trade-offs of this approach.
By Mateusz Machalica, Alex Samylkin, Meredith Porth, Satish Chandra (Blog post)
To develop new product features and updates efficiently, we use a trunk-based development model for changes to our codebase. Once an engineer’s code change has been accepted into the main bra….
By Paul Hammant (Blog post)
Trunk-Based Development (TBD) is where all developers (for a particular deployable unit) commit to one shared branch under source-control. That branch is going to be colloquially known as trunk, perhaps even named “trunk”. Devs may, on their own dev workstations, do some multi-branch development (say with Git), but when they are “done” with a change or a bug fix, it should go back to the shared trunk. It is not “done” if it is not there - watch for that little lie of omission.
By Martin Fowler (Blog post)
Feature flags (toggles) switch features on/off at runtime. They can hide partly built features and for A/B testing.
By Dan McKinley (Talk)
Back when I was at Etsy, I did a presentation internally about the craft of sizing opportunities. I finally got around to writing a public incarnation of that talk.
By OpenGitOps (Standards)
OpenGitOps is a set of open-source standards and best practices.
Easily build, package, release, update, and deploy your project in any language—on GitHub or any external system—without having to run code yourself.
From planning to production, bring teams together in one application. Ship secure code more efficiently to deliver value faster.
Get the best continuous integration and delivery (CI/CD), in our cloud or on your own infrastructure. Start for free and scale as you grow.
Jenkins – an open source automation server which enables developers around the world to reliably build, test, and deploy their software.
All In One CI/CD including everything you need to start exploring Kubernetes. Multi-cluster GitOps, Tekton pipelines, Secrets management, Pull Request ChatOps and Preview Environments.
Try TeamCity - the powerful Continuous Integration and Deployment tool for Developers and DevOps Engineers.
Drone is a self-service Continuous Delivery platform for busy development teams.
GitHub Enterprise Server is a self-hosted version of the GitHub platform. Your business can benefit from increased control and avoid issues associated the public cloud, while your developers can benefit from familiar features and workflows from GitHub.com.
Download, install and maintain your own GitLab instance with various installation packages and downloads for Linux, Kubernetes, Docker, Google Cloud and more.
Cloud Native CI/CD.
Travis CI is the most simple and flexible ci/cd tool available today. Find out how Travis CI can help with continuous integration and continuous delivery.
AWS CodePipeline automates the build, test, and deploy phases of your release process each time a code change occurs.
Continuously build, test, and deploy to any platform and cloud.
A fully managed continuous integration, delivery & deployment platform that lets you run fast, consistent, reliable automated builds. Focus on coding.
Open Source Feature Flagging and A/B Testing Platform.
Flagsmith makes it easy to create and manage features flags. Use our hosted API, deploy to your own private cloud, or run on-premises.
Flagr is a feature flagging, A/B testing and dynamic configuration microservice.
OpenFeature is an open specification that provides a vendor-agnostic, community-driven API for feature flagging that works with your favorite feature flag management tool or in-house solution.
Split is a feature delivery platform that powers feature flag management, software experimentation, and continuous delivery.
Maximize the value of every software feature through automation and feature management.
ConfigCat is a feature flag service for teams with unlimited seats, awesome support, and a reasonable price tag.
Statsig is a modern feature management, experimentation and analytics platform that empowers teams to 10x their product velocity, shipping every feature in a data-driven way.
Faster application deployments with automated safeguards.
The HashiCorp Cloud Platform (HCP) is an enterprise-grade SaaS platform that provides solutions for Infrastructure and Security Lifecycle Management.
Automate the provisioning of Terraform, OpenTofu, Pulumi, CloudFormation, and more. Gain visibility, predictability & governance on your cloud deployments and design the controls to empower your teams with self-service cloud environments.
Learn more about the platform that allows you to design and implement Terraform & OpenTofu to meet and exceed your needs.
Spacelift helps orchestrate your entire infrastructure pipeline (Terraform, OpenTofu, Ansible and more) to deliver secure, cost-effective, and high-performance infrastructure.
Powerful, programmable CI/CD engine that runs your pipelines in containers — pre-push on your local machine and/or post-push in CI.
The Common Workflow Language (CWL) is an open standard for describing analysis workflows and tools in a way that makes them portable and scalable across a variety of software and hardware environments, from workstations to cluster, cloud, and high performance computing (HPC) environments. CWL is designed to meet the needs of data-intensive science, such as Bioinformatics, Medical Imaging, Astronomy, High Energy Physics, and Machine Learning.
