Software Development Pipelines Offer Cybercriminals Free-Range Access to Cloud, On-Prem
A Q&A with NCC Groups Viktor Gazdag ahead of a Black Hat USA session on CI/CD pipeline risks reveals a scary, and expanding, campaign vector for software supply chain attacks and RCE.
Continuous integration/continuous development (CI/CD) pipelines may be the most dangerous potential attack surface of the software supply chain, researchers say, as cyberattackers step up their interest in probing for weaknesses.
The attack surface is growing too: CI/CD pipelines are increasingly a fixture within enterprise software development teams, who use them to a build, test, and deploy code using automated processes. But over-permissioning, a lack of network segmentation, and poor secrets and patch management plague their implementation, offering criminals the opportunity to compromise them to freely range between on-premises and cloud environments.
At Black Hat USA on Wednesday, Aug. 10, Iain Smart and Viktor Gazdag of security consultancy NCC Group will take to the stage during
RCE-as-a-Service: Lessons Learned from 5 Years of Real-World CI/CD Pipeline Compromise
, to discuss the raft of successful supply chain attacks theyve carried out in production CI/CD pipelines for virtually every company the firm has tested.
NCC Group has overseen several dozen successful compromises of targets, ranging from small businesses to Fortune 500 companies. In addition to
security bugs
, the researchers say novel abuses of intended functionality in automated pipelines have allowed them to convert pipelines from a simple developer utility into remote code execution (RCE)-as-a-service.
I hope people will give some more love to their CI/CD pipelines and apply all or at least one or two recommendations from our session, Gazdag says. We also hope this will spark more security research on the topic.
Tara Seals, Dark Readings managing editor for news, sat down with Viktor Gazdag, managing security consultant of NCC Group, to find out more.
Tara Seals:
What are some of the more common security weaknesses in CI/CD pipelines, and how can these be abused?
Viktor Gazdag:
We see three common security weaknesses regularly that require more attention:
1) Hardcoded credentials in Version Control System (VCS) or Source Control Management (SCM).
These include shell scripts, login files, hardcoded credentials in configuration files that are stored at the same place as the code (not separately or in secret management apps). We also often find access tokens to different cloud environments (development, production) or certain services within the cloud such as SNS, Database, EC2, etc.
We also still find credentials to access the supporting infrastructure or to the CI/CD pipeline. Once an attacker gets access to the cloud environment, they can enumerate their privileges, look for misconfigurations, or try to elevate their privileges as they are already in the cloud. With access to the CI/CD pipeline, they can see the build history, get access to the artifacts and the secrets that were used (for example, the SAST tool and its reports about vulnerabilities or cloud access tokens) and in worst case scenarios, inject arbitrary code (backdoor, SolarWinds) into the application that will be compiled, or gain complete access to the production environment.
2) Over-permissive roles.
Developers or service accounts often have a role associated with their accounts (or can assume one) that has more permissions than needed to do the job required.
They can access more functions, such as configuring the system or secrets scoped to both production and development environments. They might be able to bypass security controls, such as approval by other developers, or modify the pipeline and remove any SAST tool that would help searching for vulnerabilities.
As pipelines can access production and test deployment environments, if there is no segmentation between them, then they can act as a bridge between environments, even between on-prem and cloud. This will allow an attacker to bypass firewalls or any alerting and freely move between environments that otherwise would not be possible.
3) Lack of audit, monitoring, and alerting.
This is the most neglected area, and 90% of the time we found a lack of monitoring and alerting on any configuration modification or user/role management, even if the auditing was turned on or enabled. The only thing that might be monitored is the successful or unsuccessful job compilation or build.
There are more common security issues, too, such as lack of network segmentation, secret management, and patch management, etc., but these three examples are starting points of attacks, required to reduce the average breach detection time, or are important to limit attack blast radius.
TS:
Do you have any specific real-world examples or concrete scenarios you can point to?
VG:
Some attacks in the news that related to CI/CD or pipeline attacks include:
CCleaner attack
, March 2018
Homebrew, August 2018
Asus ShadowHammer
, March 2019
CircleCI third-party breach, September 2019
SolarWinds
, December 2020
Codecovs bash uploader script, April 2021
TravisCI unauthorized access to secrets, September 2021
TS:
Why are weaknesses in automated pipelines problematic? How would you characterize the risk to companies?
VG:
There can be hundreds of tools used in pipeline steps and because of this, the tremendous knowledge that someone needs to know is huge. In addition, pipelines have network access to multiple environments, and multiple credentials for different tools and environments. Gaining access to pipelines is like getting a free travel pass that lets attackers access any other tool or environment tied to the pipeline.
TS:
What are some of the attack outcomes companies could suffer should an adversary successfully subvert a CI/CD pipeline?
VG:
Attack outcomes can include stealing source code or intellectual data, backdooring an application that is deployed to thousands of customers (like SolarWinds), gaining access to (and freely moving between) multiple environments such as development and production, both on-prem or in the cloud, or both.
TS:
How sophisticated do adversaries need to be to compromise a pipeline?
VG:
What were presenting at Black Hat are not zero-day vulnerabilities (even though I found some vulnerabilities in different tools) or any new techniques. Criminals can attack developers via phishing (session hijack, multifactor authentication bypass, credentials theft) or the CI/CD pipeline directly if its not protected and is Internet-facing.
NCC Group even performed security assessments where we initially tested Web applications. What we found is that CI/CD pipelines are rarely logged and monitored with alerting, other than the software building/compiling job, so criminals dont have to be that careful or sophisticated to compromise a pipeline.
TS:
How common are these types of attacks and how broad of an attack surface do CI/CD pipelines represent?
VG:
There are several examples of real-world attacks in the news, as mentioned. And you can still find, for example,
Jenkins instances with Shodan on the Internet
. With SaaS, criminals can enumerate and try to brute-force passwords to get access as they dont have multifactor authentication enabled by default or IP restrictions, and are Internet-facing.
With remote work, pipelines are even harder to secure as developers want access from anywhere and at any time, and IP restrictions arent necessarily feasible anymore as companies are moving towards zero-trust networking or have changing network locations.
Pipelines usually have network access to multiple environments (which they shouldnt), and have access to multiple credentials for different tools and environments. They can act as a bridge between on-prem and cloud, or production and test systems. This can be a very wide attack surface and attacks can come from multiple places, even those that have nothing to do with the pipeline itself. At Black Hat, were presenting two scenarios where we originally started off with Web application testing.
TS: Why do CI/CD pipelines remain a security blind spot for companies?
VG:
Mostly because of the lack of time, sometimes the lack of people, and in some cases, lack of knowledge. CI/CD pipelines are often created by developers or IT teams with limited time and with a focus on speed and delivery, or developers are just simply overloaded with work.
CI/CD pipelines can be very or extremely complex and can included hundreds of tools, interact with multiple environments and secrets, and be used by multiple people. Some people even created a periodic table representation of the tools that can be used in a pipeline.
If a company allocates time to create a threat model for the pipeline they use and the supporting environments, they will see the connection between environments, boundaries, and secrets, and where the attacks can happen. Creating and continuously updating the threat model should be done, and it takes time.
TS:
What are some best practices to shore up security for pipelines?
VG:
Apply network segmentation, use the least-privilege principle for role creation, limit the scope of a secret in secrets management, apply security updates frequently, verify artifacts, and monitor for and alert on configuration changes.
TS:
Are there any other thoughts you would like to share?
VG:
Although cloud-native or cloud-based CI/CD pipelines are more simple, we still saw the same or similar problems such as over-permissive roles, no segmentation, over-scoped secrets, and lack of alerting. Its important for companies to remember they have security responsibilities in the cloud as well.
Tags:
Software Development Pipelines Offer Cybercriminals Free-Range Access to Cloud, On-Prem