As cloud-based CI/CD pipelines become more popular, organizations are increasingly looking for ways to secure these critical systems. Often the central pillar of automation for DevOps and agile development teams, CI/CD infrastructure can be a very inviting target for attackers. As a result, organizations that depend on cloud-based CI/CD need to be even more diligent in securing their CI/CD workloads.
What steps can an organization take to secure its cloud CI/CD infrastructure? Can managed services help alleviate security concerns, or will they simply introduce new attack vectors?
In recent articles, we’ve surveyed the capabilities of the “Big Three” CSPs (cloud service providers) for specific security features, in articles such as Bot Management in the Cloud, API Security in the Cloud, and DDoS Protection in the Cloud, among others.
Now in this article, we’ll continue this theme, and look at their managed service offerings for CI/CD in the context of security.
Why Is CI/CD Security Important?
Most everyone who has worked in software engineering in the last decade has probably been exposed to the concept of DevOps, and more specifically, continuous integration/continuous delivery (CI/CD) pipelines—a central pillar of automation in most DevOps and agile environments, and critical to efficient software delivery.
Because it’s such a core fixture, it contains a wealth of sensitive information. Also, the infrastructure itself is often empowered to make critical changes within an organization’s cloud infrastructure. So, it’s vitally important to secure this system.
In cloud architecture, maintaining CI/CD security is doubly important since cloud resources are often exposed to the public internet. This can lead to a fragmented security posture if not properly managed, which can make it difficult to maintain compliance and audit trails.
Potential Security Risks in CI/CD Architecture
CI/CD systems are often a complex deployment of various compute resources, APIs, storage, and frontend services. They provide a wealth of functionality for automating deployment and testing, but with that complexity comes several potential security risks.
During the software deployment lifecycle, various artifacts may be generated by workflow steps and automation in CI/CD. Intermediate container images, binaries, compressed archives, and log outputs all may be written to files.
These artifacts can contain a wealth of valuable information for an attacker, up to and including critical credentials and API keys. Therefore, their storage is of paramount interest. Ideally, storage is encrypted at rest, with artifacts expiring after a short, fixed period, and access to them is gated by role-based access control (RBAC).
Like any critical system in software architecture, CI/CD infrastructure should be subject to the same RBAC system as other components. The more users and system identities that have elevated privileges, the more routes an attacker can take to gain elevated access. The principle of least privilege states that systems and users should only have the minimum set of privileges needed to carry out a task.
Good CI/CD solutions enable users to configure their tooling according to least privilege standards.
Cloud applications are often heavily dependent on a variety of sensitive values. SSH keys, API tokens, and cloud credentials are just some of the secrets that require careful protection. Unfortunately, secrets are also vulnerable to compromise simply as a result of carelessness.
Good CI/CD security demands the use of a secrets management solution to provide protection, including pre-commit checks to detect staged credentials.
CI/CD runs on the same infrastructure as traditional software applications. That infrastructure needs to be monitored, scaled, and updated to ensure it maintains a strong security posture. Unpatched infrastructure can lead to a variety of attacks that provide bad actors with root access to this critical platform.
The software supply chain has become a much more frequent vector of attack. Rather than try to directly attack the target system, hackers go after related systems and components, including dependencies and libraries that are part of build processes but are not controlled by engineers that maintain the primary target.
Some CI/CD solutions are starting to integrate supply chain protections as part of their platform.
Cloud CI/CD Services and Security
The following section lays out the major CI/CD services and tooling provided by the major cloud providers: AWS, Google, and Azure. For each provider, we’ll look at its specific offering and how it satisfies the vulnerability concerns from the previous section.
Amazon Web Services
AWS provides AWS Code Commit for managed private Git. One of the stronger features is the advertised out-of-the-box compatibility with a variety of compliance frameworks. There’s also AWS CodeArtifact, a managed artifact repository that’s specifically targeted at language libraries and modules. Code Artifact can provide private package repositories for languages like Python and Node.js; it also acts as a proxy for public package repositories like NPM and PyPI.
AWS CodeDeploy is described as a deployment service, offering the ability to deploy a variety of build artifacts, applications, and files. It can integrate with AWS services, or third-party providers outside of the AWS ecosystem.
For actual CI/CD workflow orchestration, AWS CodePipeline provides configurable, fully managed continuous delivery.
When it comes to storage, CodeCommit and CodeArtifact repositories are automatically encrypted at rest, while data is encrypted in transit. However, CodeDeploy users can make use of Amazon S3 buckets for the storage of artifacts like application revisions, which generally do not enforce stringent security measures (like encryption) by default.
Like most AWS services, AWS IAM is fully integrated with CodeCommit, CodeDeploy, and CodePipeline. You can also build a multi-layer access and authorization system using role-based access control.
Customers that want to take advantage of native AWS Secrets Manager integration will need to make use of CloudFormation templates for infrastructure, or direct API calls.
CodeCommit and CodePipeline are fully managed services that generally don’t require customers to manage the underlying infrastructure. However, CodeDeploy requires additional management overhead, as you need to install and manage build agents on your infrastructure.
For your supply chain, CodePipeline enables integrations with third-party AWS partners like Snyk. Typically, these are paid solutions with additional licensing requirements. Customers that utilize Elastic Container Repositories can take advantage of included image scanning.
Google Cloud Platform
GCP Cloud Build offers a serverless CI/CD platform for a variety of deployment scenarios, including Docker images, Go binaries, GKE clusters, mobile apps, and GitOps with infrastructure as code. Meanwhile, Artifact Registry lets you store development and deployment artifacts. Lastly, Cloud Source Repositories are for customers needing a hosted VCS solution and serves as a managed Git offering.
For storing build artifacts, Artifact Registry provides the ability to configure uploads as part of the larger build and deploy your configuration in Cloud Build. Anything uploaded to a repository is automatically encrypted at rest and can also be encrypted with customer-managed keys. Repositories can additionally be placed inside a network perimeter.
Artifact Registry uses the same role-based system for access and authorization as the rest of Google Cloud, while Cloud Build can work directly with Google Cloud Secrets Manager. Once a project is added as a valid principle in the permissions policy, users will be able to access and store secrets directly from builds.
The serverless nature of Cloud Build is also meant to be interpreted as an operational responsibility directive; users do not need to manage the underlying infrastructure used in Cloud Build. Google provides nodes, including traditional VMs, as managed resources.
Artifact Registry integrates with Container Analysis as well to provide vulnerability scanning for containers and language modules within them when stored in a registry.
Azure DevOps is actually the larger category into which a variety of DevOps, CI/CD, and Agile development tools fit. These tools include those discussed here below.
Lastly, Azure Artifacts is an artifact repository that provides package management and hosting for most of the popular languages.
Azure DevOps mainly relies on Azure managed services. For storage, Azure Blob Storage is used, providing encryption at rest and role-based access control to the underlying storage.
Azure DevOps provides its own RBAC permission system, but the centralization and standardization story is a bit messier. Azure generally expects customers to take advantage of Azure Active Directory to handle identity management on-premises and in the cloud, which can be integrated with Azure DevOps.
Azure also provides a secrets management tool in the form of Azure Key Vault. Similar to GCP services, users can simply issue the correct service permissions, then refer to any secrets needed in a workflow configuration.
In terms of infrastructure, the general use of managed services for Azure DevOps means customers do not have to deal with the overhead of self-managing their own CI/CD systems.
Azure Pipelines implements something called deployment gates as well, which allows you to define arbitrary breakpoints and checkpoints for deployment health, including artifact scanning and code signing.
Securing CI/CD Pipelines in the Cloud Is Essential
CI/CD pipelines provide an incredibly powerful automation platform for deploying software; consequently, they also make an inviting target for attackers. As much as you require powerful security tooling to help secure cloud infrastructure, ease-of-use and configuration are also important in terms of achieving a good security posture.
So what platform should engineering organizations choose? The obvious choice for most is to simply use the CI/CD managed services of their current cloud provider. Most teams will likely have to choose as well between self-managing their infrastructure and integrating with existing third-party solutions.
Cloud Provider Security Services: Should You Rely on Them?
As you can see, the “Big Three” offer a variety of services for protecting CI/CD pipelines in the cloud. Organizations that use them can safeguard their CI/CD infrastructure, as long as they avoid a few potential pitfalls (such as storing artifacts in improperly secured S3 buckets).
However, it’s important to remember the overall limitations of the CSPs’ security services. While some types of security (such as for CI/CD pipelines) are covered well, others are incomplete. Among the latter is perhaps the most important security issue of them all: http/s traffic.
Each of the major CSPs offers web security tools (the most prominent of which are AWS WAF, Cloud Armor, and Azure WAF). Although these services have their uses, they do not provide full web security. We’ve written about this before (e.g., Built-In Cloud Provider Security Tools: Are They Enough?).
For complete protection, consider Reblaze, a cloud native web application security platform. As a comprehensive web security suite, Reblaze includes a next-gen WAF, multi-layer DDoS protection, advanced bot management, and much more. To learn more about how Reblaze can safeguard your organization’s digital journey, you can contact us here.