Lessons from XZ Utils: Achieving a More Sustainable Open Source Ecosystem

A month ago at CISA, we held our first Open Source Software Security Summit, bringing together leaders from open source foundations, package repositories, civil society, and industry. As part of this summit, we held a tabletop exercise testing coordination of community response to a hypothetical vulnerability under active exploitation in a widely used open source library. Participants noted that they gained an improved awareness of CISA’s ability to assist in coordinating response efforts between private sector and OSS non-profits, as well as new insights into how their organizations could implement resilient recovery plans. 

Little did we know how soon the lessons from the tabletop would be applicable. The XZ Utils compromise – a multi-year effort by a malicious threat actor to gain the trust of the package’s maintainer and inject a backdoor – highlighted the fragility of key points in the open source ecosystem, the very real and ongoing risks created by maintainer burnout, and the enormous benefits realized through open collaboration as demonstrated by the communities’ response. We are fortunate that the open nature of the wider open source ecosystem allowed a developer to spot this supply chain compromise before it could cause much harm. Next time, we may not be as lucky. 

This compromise highlights a fundamental shift needed: every technology manufacturer that profits from open source software must do their part by being responsible consumers of and sustainable contributors to the open source packages they depend on. In line with our Secure by Design initiative, the burden of security shouldn’t fall on an individual open source maintainer—as it did in this case to near-disastrous effect. Rather, companies consuming open source software must contribute back – either financially or through developer time – to ensure a sustainable ecosystem where open source projects have healthy and diverse maintainer communities that are resilient to burnout. 

Technology manufacturers and system operators that incorporate OSS are responsible for the safety of the systems they build and operate, and should work to ensure – either directly or by supporting maintainers – that a secure by design software development approach is being followed. This includes regular code reviews, eliminating entire classes of vulnerabilities, applying security scanning tools, isolating build environments, having a documented process for responding to vulnerability reports and security incidents, and more.  

At CISA, we’ve been working hand-in-hand with many open source communities to drive a more resilient open source ecosystem so that organizations across the world can continue to reap the countless benefits of open source software. As laid out in our open source roadmap, we’re working on a number of areas, including to build relationships with open source communities, understand open source prevalence, secure the federal government’s use of OSS, and help secure the broader open source ecosystem. 

Specific to the XZ Utils compromise, through CISA’s Joint Cyber Defense Collaborative (JCDC), we are collaborating in real-time with open source community members to better understand the impact. More broadly, we’re continuing our efforts to secure open source, including working with package repositories to scale out security improvements to entire open source ecosystems. We’ve also released the package from our tabletop exercise that any open source community can use to practice and refine their incident response coordination abilities. 

Interested in helping support CISA’s open source security efforts? Drop us a note at [email protected].