DevOps Pipeline Security19 NOV 2020 | Femitha Majeed
Many organisations today adopt an agile software development technique by being open and responsive to iterative requirement changes and improvements. To achieve the benefits of improved flexibility, delivery velocity, and efficient deployment, DevOps (Development and Operations) teams have developed certain practices and tools such as automated configuration management and Continuous Integration (CI) & Continuous Delivery/Deployment (CD). These practices enable rapid software changes while maintaining the stability and security of the whole system.
Along with these benefits, they have also exposed a new attack surface and associated set of vulnerabilities. In this blog, I will take a look at the components of a DevOps pipeline, some of the attack scenarios, mitigation steps, and best practices.
By integrating security into DevOps practice, thus turning it into DevSecOps, security becomes one of the core components of the process that continuously ensures and evaluates the security posture of services or applications being delivered from the DevOps pipeline. A DevOps pipeline (Figure 1) is a workflow that consists of several steps, each of which includes a number of actions to be carried out. It is made up of different components: code repositories, artefact builders, image verifiers, deployers, and infrastructure environments. It retrieves portions of the final system from a variety of sources, builds and tests the application, and deploys the final application package to selected infrastructure environments. The components are collectively called a CI/CD tool suite.
Figure 1:Generalised CI/CD Pipeline (Source: References )
Generally, multiple environments such as development, test, and staging environments are used in this process. The application source code is automatically pushed to a successive stage/environment, and tests are triggered once each stage finishes successfully.
Security teams focus on integrating automated/manual dynamic and static code analysis throughout the application life cycle. However, each component/tool/technology used in the pipeline, and their interactions, introduces its own flavours of security issues. If an attacker successfully subverts a pipeline by exploiting these security issues and gains privilege of the entire process, he could inject malicious code or even replace the build image. The developed code would have gone through several rounds of static analysis and manual review, but if, the same “security reviewed code” is not the one that is finally deployed to production, there is no final assurance of security. Therefore, hardening each step of the pipeline, and the tools involved in the workflow is crucial.
There are many CI/CD tools available in the market, both open-source and commercial. Some of the leading tools are shown in Figure 2. A recent survey has revealed that CI/CD tools have more vulnerabilities than those communicated through vulnerability databases and security communities. Recently, GitLab advised its users to apply urgent security fixes to its community and enterprise versions, where it addresses several vulnerabilities including arbitrary file read, path traversal and Server-Side Request Forgery (SSRF) .
Figure 2: CI/CD process and tools
Here, I analyse a few attack scenarios spanning across the CIA (Confidentiality, Integrity, Availability) triad, that could happen in the CI/CD process, and identify remedial measures to reduce the attack surface.
Scenario 1: Malicious code injection into the repository.
Description: Malicious code could be committed directly to the master branch or into a third-party repository, thus inserting backdoors that can try to steal API keys, pivot to private networks, create reverse shells, exfiltrate data, access other repositories on the same server, or steal SSH keys.
Recommendation: Secure access to the repositories and audit access to them.
Scenario 2: Bypass the predefined sequential steps in the pipeline or change the order.
Description: If some of the security checks on the code in the pipeline can be bypassed, and the untested application is deployed directly to production, the final application would not be secure.
Recommendation:Lockdown CI/CD server and configuration files.
Scenario 3: Image is built from malicious sources.
Description: If a backdoored/poisoned image from public repositories is used or code is fetched from malicious third-party repositories, the whole system becomes vulnerable. In research conducted by Kromtech ., they identified images on public repositories that contained shell scripts to install reverse shells and even add authorised SSH keys on the user’s system.
Recommendation: Identify and restrict the input and output of each component.
Scenario 4: Replace the build image with a malicious one.
Description: The final image could be stored at a location that is accessible to the attacker, and could be replaced by a malicious image which would be picked up by the deployer to deploy into production.
Recommendation: Verify the integrity of the deployed image by comparing it with the checksum of the constructed image.
Scenario 5: Modify plain text configuration files.
Description: CI/CD pipelines use configuration files to describe the different stages such as build, test, and deploy and contains instructions for the build process. Environment variables, secret keys, and other parameters are also declared in these files. The pipeline can be compromised if the configuration files were readable and manipulated by the attacker to perform operations that would not be permitted normally.
Recommendation: Ensure the configuration files are encrypted and integrity-protected.
Scenario 6: Leakage of credentials and sensitive information.
Description: An attacker may extract credentials from the pipeline configuration or environment variables and use them to access other sensitive systems.
Recommendation: Use audited, secure secrets managers to store sensitive information.
Scenario 7: Denial of Service (DoS) on the pipeline.
Description: An attacker may cause the pipeline to slow down or crash and thus affecting the application delivery.
Recommendation: All of the above recommendations.
Apart from performing security testing and static analysis of the code, the above scenarios call out for a detailed design level formal verification of the different components of the pipeline. Trustworthiness and access rights of each component should be examined carefully. Regular auditing and updating, access control, host hardening, environment isolations, configuration change, and anomaly logging and monitoring are few measures that should be implemented to ensure that the integrity of the pipeline is not compromised.
Connect with us