The endless nature of updates

While keeping dependencies up to date is important, immediately moving to a new version can introduce risks, including the potential for application instability or breakage due to unforeseen regressions in dependent software. Finding a balance between quality and security can seem like a Sisyphean task, as an ever-growing number of updates are required, especially if you must spend time searching for crucial information about each dependency update.

What is package health?

Package health refers to the overall security and reliability of a particular version of a software package (or library), including:

• Whether it has any known vulnerabilities (and how severe they are)
• How old it is
• Adoption rate (how many people have adopted the particular version out of all available versions)
• Test passing rate (success of moving from your current version to the new version)
• How active the maintainer is 

Let’s break each of those down a little further.

Known vulnerabilities

Nearly all packages have some known vulnerabilities, but some are higher risk than others. If there are known exploits available, that vulnerability immediately becomes higher risk than vulnerabilities with only theoretical risks. Developers should know what new risks they’re subjecting applications to when updating dependencies.

It’s useful to know how many CVEs a particular version of a dependency has as well as how severe they are. If an update removes a few medium and low CVEs but introduces a new critical CVE, it may not be worth it.

Package age

The age of a package is critical for knowing how trustworthy it is. If a package is more than a year old, it likely contains some vulnerabilities that have been addressed in newer versions. While the newest package is not necessarily the best, if a package version is too old, it is probably riskier to use.

Adoption rate

By monitoring the number of users who are actively using a specific version of a package, developers can assess the overall popularity and reach of that version. If the latest version has a low adoption rate, it might indicate that other developers have tried it and rolled it back after it caused issues.

Version success

This might be the most critical metric related to a package’s health. Knowing the percentage of users who have successfully updated from your current version to a specific later version empowers you to make informed decisions about the ease or difficulty of updating without breaking your build. For example, if a version boasts a 90% success rate among adopters, you can be confident that your update will likely go smoothly as well.

Package maintenance

Developers rely on the responsiveness of library maintainers to address vulnerabilities and ensure the ongoing security of their projects. An active, well-maintained package provides peace of mind, while an abandoned or deprecated one raises a red flag, signaling potential risks and the need for alternative solutions.

More information means better decision-making

Comprehensive package health information allows you to make informed decisions, reducing the risk for negative side effects in your applications—namely, creating new vulnerability risks or breaking an application’s usability. The more you know about a package, the better decisions you can make about it.

Just like a doctor needs a complete patient history to prescribe the right medication, developers need comprehensive information about a software package before updating it. Think of it this way: a new drug might promise to cure a disease, but what about the side effects? If a patient is already vulnerable, is it worth the risk when older, proven medications exist?

The same applies to software. While a later version might seem best, it could introduce bugs or break compatibility with existing code, disrupting users. It’s a delicate balance between innovation and stability—just like a doctor weighing the best treatment options for their patient.

Comprehensive package health information helps you make informed decisions, minimizing the risk of unintended consequences. Ultimately, the more you know about a package, the better equipped you are to make the right call.

Mend.io makes updates easier

Balancing security with application stability is crucial. Mend.io offers valuable package health data, sourced from its widely-used dependency update tool Mend Renovate. This data helps developers make informed decisions about updating packages and mitigating vulnerabilities, enabling them to strike the balance between security and stability.  

To update or not to update? That is the question—and one we can help you answer.

This article also provides insights into the challenges of dependency management, as well as a number of best practices and things to consider. It highlights the benefits of regular application updates and how automated dependency management can simplify the process. Lastly, the guide provides strategies and tips for managing dependencies.

What are dependencies in code?

In the past, companies would typically either write their own custom software for business needs or use off-the-shelf software packages. But today, most organizations have software stacks composed of many different third-party or open-source components.

Dependencies in code are the connections among all those different software components. A piece of software is said to be dependent on another piece of software if it invokes it. For example, a company might have a custom point-of-sale solution that uses open source payment processing libraries or receipt-printing libraries. That means the solution is dependent on those open source libraries. 

In short, external code that’s required for a piece of software to operate correctly is a dependency. Dependencies can include libraries, frameworks, modules, APIs, and services. 

Dependency check

The most significant risk for companies is not knowing what they don’t know. That’s where dependency checks come in. 

A dependency check identifies all the components, libraries, or modules that are linked to a specific piece of software to create an inventory and identify out-of-date software components, license issues, compatibility issues, version numbers, and more.

Dependency checking is a key way organizations can reduce security risks and identify known vulnerabilities. Some of the key items being checked include:

• Known vulnerabilities. Dependency checks typically start with an automated scan that checks all the dependencies in a given code set against a database of known vulnerabilities to identify potential issues.
• Outdated software. A dependency scan also checks for outdated code by seeing if new versions of libraries or software components are available.
• Compatibility. If new versions of software components are available, it’s important for a dependency checker to identify whether they are compatible or not, so an organization can manage the risk and the level of effort required to update to a new version.
• License information. Dependency checking can also evaluate the licensing information for any third-party code to avoid possible legal issues and ensure compliance. 

Ideally, the results of a dependency check will generate a detailed report identifying vulnerabilities or issues and suggest recommended actions. Most companies will also want to conduct continuous dependency checks, because new vulnerabilities may be introduced any time software is altered or updated.  

Dependency checks improve application security in several ways. First, vulnerabilities in third-party components can be identified and eliminated before ever making it into the code base, which shrinks the potential threat footprint. Second, eliminating known bugs improves code quality and reliability. And finally, organizations can ensure that all software components comply with licensing requirements.

What is dependency management?

A dependency check allows organizations to identify the connections or dependencies among all the different components in its code base. But that’s just the start of the process. Once an organization knows the initial connections and dependencies, it needs to create an on-going management strategy for them. 

That’s where dependency management comes in. Dependency management is a structured approach to managing all the external software components (including libraries, frameworks, modules, APIs, and services) linked to a specific project or code base. 

Proper dependency management ensures that organizations are using the appropriate versions of software components and ensures on-going consistency and security of the software. 

Essential components of a robust dependency management strategy include: 

• Version control. Identify and manage the specific versions of software components to ensure they’re up to date or using the most recent viable version.
• Package managers. Dependency management is best automated due to both the need to continually rerun dependency checks and the large volume of software components that typically make up today’s solutions. Package managers help by automating the downloading, installing, and updating of dependencies.
• Dependency resolution. Knowing that a component has different versions is just the first step. Knowing which one to use is the next step. Dependency resolution is the process by which a package manager or dependency management process determines which version is best to use based on potential conflicts, compatibility, or specific requirements.
• Transitive dependencies. Software isn’t only dependent on the libraries or code it invokes; it’s also reliant on whatever libraries or code those initial libraries depend on—the transitive dependencies. That’s why it’s essential to manage transitive dependencies to ensure they are compatible and appropriate as well.
• Security management. One big reason to bother with dependencies at all is to manage security and prevent known bugs. Organizations need to regularly monitor and resolve security issues or vulnerabilities in software dependencies and incorporate known patches as appropriate. 

In addition, a dependency management strategy should ensure isolation among the updates for different projects so that updated dependencies on one project don’t cause problems with separate projects. Organizations should also make sure to maintain clear documentation on the dependencies and versions and implement automated testing procedures to ensure compatibility and stability. 

What are the risks of not updating dependencies?

Using third-party components for software development can save organizations considerable money, time, and resources. However, those same components can open the organization to potential risks, especially if they drift out of date. While no one likes to interfere with working software, organizations that let the dependencies in their software code base become out of date are opening themselves up to a variety of non-trivial risks. 

Consider some of the dangers that not updating dependencies can generate:

• Security risks. The most obvious problem with out-of-date dependencies is potential security risks or vulnerabilities. The older a piece of software is, the more likely potential vulnerabilities have been identified or exploited by bad actors. Using such out-of-date components dramatically increases an organization’s security risk profile.
• Compatibility issues. Most software continues to change over time via updates with new features, patches, or bug fixes. The more out-of-date a component is, the greater the risk of compatibility issues. Also, the older the software is, the more risk there is that an update will be problematic or have integration problems when trying to update.
• Malicious updates. Organizations need to remember that not every update is good! Malicious updates are updates that bad actors create to introduce vulnerabilities or bad code that can be exploited. For example, a malicious update may introduce a hidden software back door for data breaches or install malware that provides external actors control over a system.
• Increased application fragility. Organizations that don’t update dependencies risk increasing the fragility of their applications. The longer it has been since updates have been applied, the more difficult they may become, or the more work may be required once they are applied.
• Performance issues. Due to optimization or updates, new components or software may provide better performance characteristics than older software. Organizations that don’t update dependencies will miss out on potential performance updates. 

Decreasing risk in updating dependencies

Application development teams find themselves balancing functional risk—that is, the chance that updating software might break something—with security risk. How do organizations resolve these competing factors and decrease the risks that might come from updating dependencies?

There are two primary ways to decrease both types of risk when updating dependencies: Merge Confidence metrics and blocking malicious software.

Merge confidence from broken dependency releases can help organizations determine how likely a potential update will have a negative impact or “break something.” Merge confidence metrics can take into account statistics such as the frequency of updates with that component, the number of users, the regularity of updates, and how quickly component problems are resolved. Merge confidence can also take into account details from a component’s change log to help understand the specific changes that have occurred. 

Detecting and blocking malicious open-source packages is equally important to prevent the introduction of problematic code when updating dependencies. Putting measures in place or using software tools that can detect and block malicious open source software updates helps prevent potential security breaches that such software might introduce, whether it’s malware, trojan back doors, or other hostile behaviors. 

What are vulnerabilities in dependencies?

Unfortunately, organizations should look for vulnerabilities not only in their own code but also in any dependencies that are called or referenced within that code.

Vulnerabilities in dependencies are security risks or vulnerabilities in third-party libraries, frameworks, or code. Organizations need to be just as diligent in identifying and mitigating vulnerabilities in dependencies as they are in their own code. 

There are several ways that vulnerabilities manifest themselves in dependencies, including:

• Outdated or unmaintained versions. Older versions or unmaintained versions of software components can have known vulnerabilities or risks.
• Malicious packages. Organizations must be careful that third-party components don’t have intentionally harmful code.
• Indirect vulnerabilities. Dependencies can have dependencies, so organizations need to dig deep when analyzing potential vulnerabilities.
• Insecure configurations. Some software packages may come with default configurations that can open an organization to risks unless altered. 

What is technical debt?

What happens when there’s a problem in a software project that’s not resolved? It tends to grow bigger over time. As a result, it can become more challenging to solve because of changes in the environment or lack of skills availability. 

That’s technical debt in a nutshell. Like financial debt that increases over time due to additional interest charges, technical debt is a technical problem that’s not addressed immediately and is left to linger. As a result, it typically grows larger and harder to solve, leading to increased maintenance costs.

There are several causes of technical debt, including quick fixes, lack of expertise, poor design decisions, changing requirements, use of outdated technologies, and even lack of documentation. Moreover, organizations can accrue technical debt intentionally or unintentionally. Intentional technical debt comes from taking shortcuts or making quick fixes to meet a deadline without regard for the future impact. Unintentional technical debt accumulates accidentally due to a lack of expertise, knowledge, or complexities within a software environment. 

Unfortunately, technical debt isn’t just an IT problem, as accumulating technical debt can hurt an organization in a number of ways. increased maintenance costs, decreased productivity, slower development, and higher failure/defect rates are just a few examples. In response, organizations need to recognize the impact that technical debt can have on their development process and business outcomes and plan to minimize it. 

What are challenges in dependency management?

One important way to reduce technical debt is through dependency management, which can shrink security risks and enable more rapid and agile change.

But as helpful as dependency management can be, there are potential challenges that organizations must be aware of, including:

• Conflicting dependencies. Sometimes, more than one software package needs to use the same dependency, but each requires a different version. The versions may not always be compatible, so by solving the dependency for one piece of software, you could risk breaking the compatibility of another.
• Versioning issues. As software evolves over time and new versions are released, it can be difficult for an organization to keep track of the different versions of a component needed for different projects. For example, versioning issues can arise when teams lose track of the most up-to-date versions and continue to use out-of-date versions, or when different projects require different versions of the same component.
• Dependency license issues. Open-source license management issues can also arise as a challenge in dependency management. As components are referenced or updated, software developers need to ensure that the license terms and conditions of the dependency are up to date and work well with the overall project.
• Managing dependencies across multiple environments. Most organizations have more than one software project, so managing dependencies across multiple environments can quickly become a significant challenge. It’s important to recognize that dependencies can behave differently when used in different environments and when they are combined with other components. Like cogs, they have to fit together properly; otherwise, the whole mechanism doesn’t work. Mismanaged dependencies can, therefore, disrupt or disable entire projects.  

What are best practices for dependency management?

Dependency management requires organizations to keep track of many different software components across many projects and versions. It’s a complex task. 

That’s why it’s critical for organizations to establish some best practices for dependency management so they can ensure it is implemented correctly to streamline development, reduce security and performance risks, and ensure software consistency. Good dependency management is critical for ensuring the security, stability, and performance of software. 

Since there is so much pre-written code and third-party software to choose from, a few best practices for dependency management include the following items:

• Compatibility. A good place to start is to find out whether the dependency works on the platform you’re using. If not, discard it.
• Licensing. Verify that a dependency’s licensing terms and conditions are up-to-date and consistent with your project needs.
• Maintenance and versioning. Check to make sure the dependency is updated regularly and properly; otherwise, avoid it.
• Breadth of use. Look for dependencies that are currently in wide usage. 
• Quality. Look at the source code—is it clean and readable? Is the documentation solid? Does the dependency come with its own unit tests?
• Ease of integration. Investigate how easy it is to integrate the dependency into your current workflow and how easy it will be for developers to start using it.
• Size. Smaller, more focused dependencies are better than larger ones since they are less likely to bloat and slow down your code base.
• Community support. Evaluate how active the community support is for a dependency.
• Speed and efficiency. Consider how responsive the solution is and if you can manage dependencies in real time. 

What is semantic versioning?

Managing dependencies effectively requires a solid strategy for managing different software projects and components. 

That’s where semantic versioning comes in. Semantic versioning is an approach for defining and communicating the meaning of software changes within a specific release. It provides a visible way to note the hierarchy of software releases so developers can easily understand how they may relate to underlying code changes.

A typical semantic versioning format comprises three parts: a major version, a minor version, and a patch version. 

The major version is incremented or advanced when there are significant changes within a software project, such as when an API changes or there are changes that are not backward compatible.

The minor version is advanced when new features are added but don’t necessarily impact existing functionality. 

Lastly, the patch version is incremented when bug fixes are made. It’s typically used for minor changes that don’t affect functionality. 

Semantic versioning has several benefits, including consistent communication among developers, enabling automated updates and compatibility checks, providing a clear understanding of the extent of a version change, and facilitating compatibility management by communicating the extent of change. 

Semantic versioning is a simple way to keep track of changes and indicate what kind of change happened and at what point in development. It can also help developers gauge update risk based on whether a change is major, minor, or patch. Most importantly, semantic versioning helps developers avoid dependency hell by making it easier to resolve version conflicts and know what versions are acceptable to use.

Regular application updates and maintenance

Regular updates and maintenance to applications and dependencies give organizations the best chance to avoid security risks. Attackers are always looking for vulnerabilities and ways to exploit them, so if an organization has an older version of a popular component, there’s a good chance the bad actors have found ways to exploit it. Updates can counter these efforts and reduce overall risk by addressing flaws or weaknesses before they cause trouble. 

In addition, keeping dependencies current can provide increased stability and potential performance improvements while reducing technical debt. Updated dependencies may also deliver new features or provide more robust or security applications.

Critical benefits of regularly updating application dependencies include:

• Security improvements
• Bug fixes
• Performance or feature improvements
• Increased compatibility 
• Reduced technical debt
• Improved compliance

One consideration when updating dependencies is reachability. Reachability is the determination of whether all the dependencies and transitive dependencies are identifiable, accessible, available for use, and can be successfully resolved. 

However, any application update, even those made to fix dependency problems or bugs, can introduce new problems. 

Some of the potential risks of updating an application’s dependencies include: 

• Breaking an application
• Introducing new bugs or security risks
• Performance problems
• Introducing dependency conflicts
• Integration issues
• Increased maintenance overhead

While the risk of breaking an application during dependency updates exists, those potential risks can be minimized with the processes and automation in place. Regular scanning for vulnerabilities is necessary to ensure that every dependency used is secure and updated. 

Automating dependency management

Given the number of dependencies and the volume of changes occurring, most organizations cannot effectively manage dependencies without automation. Automating dependency management boosts efficiency while also improving the security and maintainability of software projects.

The first step in automating dependency management is selecting a dependency management tool that typically integrates with your version management system and other components in your software environment. The next steps are configuring your CI/CD pipelines, automating tests and scans to run during dependency updates, and establishing a process for reviewing and merging automated dependency pull updates. 

Automating dependency management instead of manually managing it provides significant benefits, including consistency, security, efficiency, and compliance.

Strategies for managing dependencies

Since managing dependencies can be a complex process, it’s best to have a strategy to ensure consistency and security. Here are three different types of dependency management strategies to consider:

• Centralized management. A centralized approach to dependency management focuses on keeping all the software components in one centralized repository. This would typically include all libraries, components, versions of third-party dependencies, and APIs. The benefit of this approach is having one place to consult and one source of “the truth.” This centralized approach can reduce the risk of dependency conflict or confusion. A consideration, especially for larger companies, is whether this approach will work for situations with vast numbers of dependencies or projects since the large size could lead to performance problems.
• Decentralized management. A second approach is to go in the opposite way and empower teams to manage dependencies by themselves rather than having them all in a single corporate repository. This can theoretically be simpler since each team is managing its own dependencies, which can result in faster performance when finding, identifying, and fixing problems. However, it also reduces the overall visibility and control of the dependencies since there is no centralized resource. It may also lead to different versions of the same third-party libraries across different teams, possibly resulting in conflicting versions and dependency confusion.
• Hybrid approach. A hybrid approach combines centralized and decentralized repositories. To do this, an organization sets up a single centralized repository that keeps shared libraries, third-party dependencies, and APIs together for use by all teams. Teams would then extend this with their repositories for code specific to their needs and not shared by anyone else. The approach will reduce dependency confusion while accelerating the ability to find project-specific code.

Tips for managing dependencies

Understanding which dependencies are vulnerable to security threats and which updates won’t break your code is complex, especially with so many direct and transitive dependencies. To help manage the process, consider starting with the following best practices for dependency management:

Identify trusted sources of dependency information. Identify trusted sources of dependency and version information to avoid including malicious dependencies or introducing risky code into your project.acks.

Prioritize. Organizations should focus their attention on the highest risks, so it’s essential to recognize that some dependencies are more important than others—and those are the ones that should be analyzed first. This may include understanding which potential open source vulnerabilities are being accessed by your code and which aren’t, so you can prioritize the updates.

Automate. Bug fixes and software updates can be time-critical, so it’s essential to put automated processes in place for dependency management in order to quickly and easily update new versions when appropriate.

Create and communicate policies. Policies let development and security teams know how to handle threats in third-party components, how to update them efficiently, and what priority is placed on continuous dependency management.

Have a consistent approach. Dependency management isn’t a one-off project. It should happen continuously so that potential vulnerabilities are eliminated before they become real vulnerabilities.

What is dependency management?

Let’s first briefly breakdown what dependencies are. 

Dependencies are the relationships between software components that rely on each other to work. You have direct dependencies, where a software component directly calls another, and indirect dependencies, which describes the relationship between software component A and component C that is not called directly by A but instead by a direct dependency of A, component B.

Sounds fairly straightforward, but of course component A could rely on components B, D, E, F, and G, and component B could rely on C and G, and component G can only work with one particular version of component F, and…you get the idea. These relationships can get very complicated. Moreover, these components are usually projects managed by other developers, meaning that there is no control over how their newer versions will be built or function.

Dependency management involves selecting, identifying, or defining all of these different relationships and resolving the conflicts that arise between them. Some minimal amount of dependency management will be done every time a new component is added to a project and every time an existing component is updated.

Some specific dependency management tasks include:

• Identifying all external components that a project relies on
• Choosing appropriate components
• Defining components in a configuration file
• Ensuring the correct versions of components are retrieved
• Recognizing constraints in compatibility between particular versions of components
• Resolving compatibility conflicts between components
• Keeping dependencies up to date

For a deeper look at dependency management, check out this blog.

Obviously, some theoretical static software project that’s working just fine has no need for dependency management once it’s built. But in reality, nearly all projects will be expanded upon at some point, or at least updated for security purposes.

Which leads us to dependency updates.

Dependency updates

Software will need to be updated. It’s a simple fact of life and is the main reason solid dependency management throughout a project’s entire lifecycle is so necessary. 

But updating isn’t always easy. As new versions of each component come out, perhaps to add new functionality or security patches, the relationships between multiple other components can make the updating of one component cause another component to fail to work. 

And perhaps updating that other component would fix the first problem but create a new problem with an entirely different component! This is referred to as dependency hell, and even fairly small projects can find themselves in the depths of it.

So how are dependency updates managed?

Managing dependency updates

While small projects can conceivably get away with manual updates, larger projects almost certainly cannot. The absolute worst way to deal with managing updates is to not update at all. Don’t do that. That means you are choosing to take on unnecessary technical debt that you really don’t need. (You can read more about dependency management and technical debt here.)

Here are some ways to manage dependency updates without getting sent straight to dependency hell:

• Update early. With new projects, set yourself to update frequently—even if not strictly necessary to the health of your project—so components don’t end up many versions behind by the time an update is no longer optional.
• Update automatically. Use an automated dependency updating tool (we like our own open source tool, Mend Renovate) to group and pull updates on a schedule that works for you.
• Start small on big projects. If you’ve got a project that’s seriously behind in updates, check out the tips in this blog for getting going.

tl;dr

Updating dependencies is just one aspect of dependency management, albeit a very important one. Dependency management also includes keeping track of which components go into your projects, their versions, and their reliance on other components as well as resolving conflicts between components.

Why you should keep your dependencies up to date

Most of us are usually happy to have the latest updates to our components because it means we also have the latest features, but there’s more to gain from regular dependency updating than new features. Keeping packages up-to-date means improved development velocity in both the near and long term. Over time, outdated dependencies accrue technical debt that only gets harder to reduce as time goes on. There are security concerns to consider as well. If a critical vulnerability is found in a package that you’re many versions behind on, and only fixed with the latest version which has conflicts with the rest of your codebase, you may have a lot of work (and time) ahead of you before that vulnerability can be patched.

Why dependency management gets ignored

If updating dependencies is such a great thing, why does it sometimes live permanently on the backburner? Sometimes the pace of development is so great that devs simply don’t find the time for it and in the chaos it can be difficult to explain to a boss why you’re “fixing” something that’s functioning just fine (fine for now, anyway). The longer it goes, the harder an update can be to make, which then in turn makes putting it off even longer the easiest choice, if it’s still a choice that’s possible to make.

OK I’m convinced. How do I get started?

Here are some tips for motivated developers who want to make regular dependency updates as painless as possible:

1. Start small. Getting started is the hardest part–it only gets easier, promise–so start small enough that you can handle the initial load entirely on your own.
2. Spread the good news to everyone else. Your teammates will likely notice and appreciate your initial effort and embrace the new way of doing things. Other teams will see the value as well. Have a talk with management so they know what you’re doing and why it’s important.
3. Automate and build into your pipeline. There are free tools available through both GitHub and BitBucket that automatically locate outdated dependencies and create pull requests for updates. Some tools, like Mend Renovate Community, have features that group pull requests by type and rate the likelihood a given update will merge without any problems, so you can spend less time poring over release notes.
4. Start new projects with dependency updating already in mind. If you’re starting something fresh, put the effort in to get the tools and structure in place to keep dependencies up to date from the get-go. Who knows, maybe in seven years your app will be the “old app” that’s still so up to date no one is afraid to touch it.

Hopefully you find these tips helpful. We have found that the minor pain of introducing regular dependency updates is more than worth the gains of reduced technical debt, new features, and increased security, and we think you will, too.

Until next time, stay up to date, friends!

In this article, you’ll learn how this backdoor infiltrated the software, the potential consequences of an attack, and how to identify and patch vulnerable systems.

This article is part of a series of articles about malicious packages.

What is XZ-Utils?

XZ-Utils is a collection of tools for the XZ compression format that have high compression ratios and fast decompression. The XZ format utilizes the LZMA2 compression algorithm, an improved version of LZMA that provides better compression and improved flexibility in the compression process. The XZ-Utils package typically includes several command-line tools for compressing and decompressing files. One of the most commonly used is XZ itself.

What happened

On Friday, March 29th, a developer named Andres Freund issued a security advisory regarding a backdoor he found in the upstream XZ/liblzma 5.6.0 and 5.6.1 versions that led to SSH server compromise. After he noticed some odd symptoms around liblzma, such as SSH login taking up too much CPU, and debugging with valgrind, he figured out that the XZ tarballs have been backdoored.

Backdoor mechanism

The backdoor was not directly inserted into the source code of liblzma that is visible in version control systems or utilized by XZ directly. Instead, it was hidden within binary test files in the XZ compressed format. These files appeared benign and were theoretically part of the library’s test suite.

The attackers used a sophisticated method that split the backdoor into parts, which were then concealed within two XZ compressed files. These files were disguised as ordinary test files, evading detection from casual inspection or automated tools that scan for malicious patterns.

Those crafted test files can be found here:

tests/files/bad-3-corrupt_lzma2.xz (cf44e4b7f5dfdbf8c78aef377c10f71e274f63c0, 74b138d2a6529f2c07729d7c77b1725a8e8b16f1)

tests/files/good-large_compressed.lzma (cf44e4b7f5dfdbf8c78aef377c10f71e274f63c0, 74b138d2a6529f2c07729d7c77b1725a8e8b16f1)

Figure 1. Different version of build-to-host.m4 file in the released tarballs

Execution flow

Upon decompression and execution, these files collaboratively manipulated the build process of liblzma. The process involved extracting and executing obfuscated script code, leading to the injection of malicious code into the build output.

This manipulation effectively appended malicious data to the build process without raising suspicion, as it mimicked legitimate testing adjustments.

The deobfuscated code can be found here

Malicious code injection

The attackers introduced a new object related to the CRC64 algorithm, claiming it was an improvement. This object was, in fact, a trojan that, when included in the build process, embedded the final backdoor into the liblzma library.

The essence of the backdoor was to intercept function calls related to CRC32 and CRC64 resolution and replace them with malicious variants that could execute arbitrary code under certain conditions, likely tied to specific crafted inputs.

Linker manipulation and RSA decryption hook

The backdoor installed an “audit hook” into the dynamic linker of Linux, a critical component that resolves library dependencies at runtime. By hooking into this mechanism, the backdoor could alter the behavior of the linker to intercept and modify the resolution of symbols, particularly those involved in RSA public key decryption.

This manipulation meant that during SSH key authentication, the backdoor could substitute the legitimate RSA decryption function with its own, allowing for unauthorized access if the input matched a certain condition likely known only to the attacker.

Stealth and implications

The backdoor’s sophistication lay in its ability to hide within the normal build process and manipulate low-level system components undetected. Targeting the dynamic linker and encryption routines posed a severe threat to system security, potentially allowing attackers to bypass authentication mechanisms.

How to look for the XZ package with Mend Container

With the Mend Container solution, you can effortlessly scan individual images or integrate your container registry to scan your entire registry thoroughly. Additionally, you can leverage our in-house container reachability analysis to check if the vulnerability is reachable. Our updated scanner will enhance your knowledge with the most updated data regarding this vulnerability.

Vulnerable Linux distros and their fix versions

False positive announcement

Due to a conflict between the Debian advisory and the official announcement, customers are expected to detect false positive alerts on older versions than 5.6.0. We recommend either upgrading to the matched fixed version listed above or downgrading to the latest uncompromised version, which is 5.4.6.

*April 1 update. It was confirmed that Fedora 40 is not affected by the backdoor. However, users should still downgrade to a 5.4 build to be safe.

A landscape of malicious packages

The critical backdoor found in XZ Utils serves as a stark reminder of the evolving threats within the software supply chain. While this specific vulnerability involved a backdoored library, malicious actors employ various techniques like license tampering, credential theft and many more. 

In conclusion, the XZ Utils backdoor vulnerability (CVE-2024-3094) underscores the critical need for vigilance in software supply chain security. By staying informed about these evolving threats, implementing robust security practices, and patching vulnerable systems promptly, organizations can significantly reduce their risk of falling victim to malicious attacks.

For many years Renovate has been provided free of charge by Mend.io, with support available from the open-source community. All free versions of Renovate are limited to running against one repository at a time. While this is often suitable for most small to medium development environments, when companies have thousands of repositories developers can find themselves waiting a long time for Renovate to run against their code. As such, large enterprises have been looking for better performance and more reliable support options.

Now, Mend Renovate Enterprise solves the problem, offering dedicated support and unlimited scaling of server resources. Finally, organizations have the ability to process large numbers of repositories in a short amount of time, while taking comfort in the peace-of-mind provided by dedicated Mend.io support. The Enterprise Edition also comes with premium features that are not available in the free versions.

Features and capabilities

Renovate Enterprise and Renovate Community are both self-hosted applications that run Renovate against your organization’s code repositories. Renovate Enterprise comes with unlimited processing capabilities, advanced features, and dedicated support from Mend.io. Let’s look at the main features in detail.

Runs Renovate

First and foremost, Renovate Enterprise runs Renovate — the world’s premium dependency management software, which automatically creates pull requests to update your code to use the latest versions of external dependencies. Renovate Enterprise ships with the latest stable version of the open source Renovate CLI, thus ensuring that new Renovate features are constantly being delivered to Renovate Enterprise.

Self-hosted application

Renovate Enterprise is a self-hosted application, meaning organizations can keep everything on their own premises and have full control over the hosted servers. Self-hosting Renovate is essential for organizations that have their code repositories behind multiple layers of internal security.

Job scheduler and webhooks

At the core of Renovate Enterprise is the job scheduler and priority job queue, which automates the process of running Renovate against each of the onboarded repositories. Combined with this, webhooks will trigger a Renovate job when important activities occur on the repository – like updating a package file or merging a PR.

Enterprise-only features

While the above features come standard with the free Renovate Community, the following features are available only with Renovate Enterprise.

Dedicated support

Renovate Enterprise comes with a support package from Mend.io, so there is always someone there to help you get up and running, maximize the benefits of Renovate, and easily troubleshoot any issues that might arise.

Unlimited scalability

While the free versions of Renovate are limited to processing one repository at a time, Renovate Enterprise unlocks unlimited server scalability. This means an organization can scale out their own server resources to deliver Renovate updates in a highly responsive manner, allowing them to balance their own cost vs performance needs.

Premium features

The Enterprise version of Renovate comes with additional features that are not available in the free versions, including Smart Merge Control, which is a feature that allows developers to customize the way PRs are delivered based on how likely an update is to pass the automated tests. As an example, a repo might be configured to suppress bad updates, group highly trusted updates, or even auto-merge updates of very high confidence. This can help developers quickly process updates with limited testing and with less failures, providing significant productivity gains.

Benefits

• Automatic dependency updates. The sophisticated job scheduler and priority job queue ensure that Renovate runs continuously against all your repositories, even while you sleep. Updates are automatically delivered to your door with pull requests ready to merge.
• Highly-responsive interactivity. With unlimited processing power, Renovate Enterprise can rapidly respond to all developer interactions on the repository.
• Reduced technical debt. Keeping dependencies updated is the most effective way to reduce security vulnerabilities in an application. Also, updates usually include bug fixes or performance improvements, thus improving the overall quality of your application code.
• Improved productivity. Utilizing Renovate with advanced features like Smart Merge Control can drastically reduce developer effort relating to dependency maintenance.
• Trusted insights. Only Mend Renovate has the knowledge of millions of successful merges to provide insights into the adoption, passing rates, and merge success of updates being applied. This information helps you make decisions with improved confidence about which updates to accept into your application.

The importance of dependency management

Keeping dependencies up to date is one of the most effective ways to reduce technical debt and avoid software vulnerabilities. Dependency management is now a vital part of any proactive application security strategy and a key component of a best-in-class development function. That’s why so many of the world’s most prominent corporations and organizations rely on Mend Renovate to manage their application dependencies.

Now, with Mend Renovate Enterprise, there is a commercially supported version of Renovate built with the power to handle enterprise-grade needs!

I’m not morbid but it’s a natural time to think about how things wind down and finally breathe their last breath. Nothing lasts forever. Not trees. Not animals. Not people. Not cars. Not houses. Not software. Especially not software.

People who actually make applications definitely know this. But instead of showing respect to our apps and letting them die a planned and peaceful death, we let our products turn first into Frankenstein’s monster with mismatched parts sewn crudely together, then finally into a zombie with fully rotting parts that fly apart at the smallest bump.

In this blog, let’s look at how and why you should retire some software gracefully, before it transforms into something scary.

Stopping zombies: Why you should let some software go

Here’s the classic graphic of the software development lifecycle (SDLC). There’s no obvious place where death comes in.

If you don’t want a zombie product, it needs to come in right at stage 1: planning. You have to plan on how you will replace all of the pieces, and you need to think about when it’ll become too complex. If you don’t decide ahead of time that you are going to budget and plan for building a new house every 100 years, what you end up with is a cursed 200-year-old mansion that’s falling down, a danger to anything it touches, and that anyone can walk right into and steal your stuff. In software, we don’t get 100 years (more like five) but the result is the same.

Here’s the SDLC in practice on a large time scale, or at least what we wish would happen: you spend a lot of time and money on the build, and then you try to maintain the plateau indefinitely to live happily and profitably ever after.

So you reiterate and replace piece by piece, but meanwhile quality (and security) goes by the wayside. You don’t plan for deprecation and getting rid of your product, you just focus on maintaining it. Here’s what actually happens: eventually maintaining that zombie will cost your entire revenue stream with no money leftover to rebuild with. When you’re spending all of your resources maintaining a product, it’s difficult to keep it secure or functional, let alone to iterate and make it better.

This is make or break stuff. Many software startups fail in the first year or two. There’s a second huge cliff between eight and ten years. This makes sense.

For the majority of startups the first couple of years are focused on making ends meet, growing fast, and building quickly with what they can get cheap. If they don’t then slow down enough to plan for the retirement and rebuild of their product, they’ll end up with a product that’s costly to maintain, impossible to secure, and too complex to keep functional. If that’s happening across all of their applications, that company will fry by that ten year mark. 

Why you need to think ahead to evade the monsters

Anyone who doesn’t know when their product will expire isn’t thinking very far ahead. Humans generally aren’t great at long term planning and those in charge of software companies are no exception.

One source of short term thinking comes from high developer turnover. With the average developer only staying at a company 1-2 years, the longevity of a product is seen as someone else’s problem, and it very likely will be. The decision to plan ahead and avoid zombie products can’t be left up to developers. Companies need to have the right long-term perspective and use that to tell employees how to build products that can be retired and rebuilt without chaos.

Why too many companies don’t think ahead

So what stops companies from having that perspective? Well, it’s a painful pitch to make. “Hey, I know the thing we have is working and making us money but in a year I’m going to have to replace pretty much the whole thing. We better spend some money now and rebuild the thing we already have.” It’s one of the toughest business decisions to make in application development. It’s not going to make any money; all it does is cut down future costs. Should we spend $2 million now to not spend $10 million in three years? That’s a long time frame for many companies.

But the alternative to making that tough decision is bleak. It’s easy to be penny-wise and pound-foolish. I’ve killed a lot of products in my career. I’ve retired product lines that were still profitable for the company because they were too much of a pain. They had too many quality issues and couldn’t be secured. If someone had had a little bit more foresight and made some fundamental changes to these products three or four years prior, before I got to them, I would have been able to hold onto them longer. But it was too late.

Why is the threat worse now?

Five to ten years ago you could maybe get away with keeping products around longer. Today, applications are increasingly dependent on each other and the application development supply chain is far more complex. It’s become much harder to find things that fit older software; replacing old parts is a pain in the rear that never works right. And you just can’t properly secure old software. You can try to plug up the holes as you find them but more will pop up and you’ll find some holes are simply out of reach.

I’ve tried to keep my analogies to a Halloween theme so I’m sorry to go out with one about cars. If you try to keep a car going for 30 years by replacing each part piece by piece, you’re not going to end up with a better car. You’re going to end up with a crappy car with terrible gas mileage that can barely get you where you need to go without breaking down, and can take advantage of few if any advancements in efficiency or safety.

Embrace change and refresh to keep the zombies at bay

Because of its dependency and complexity, the problems with approaching software this way is worse than any real world analogy. The road that you drive on doesn’t change every five years but the platforms, networks, infrastructure, and so on that your software rides on do. 

You have to know things in software are always going to change. You have to plan for that change or at least recognize that the change is going to happen. There’s going to be better and cheaper ways of doing things that you’re going to want to take advantage of. So let’s embrace death in the SDLC. Plan for a peaceful death of your software now or be haunted by it later.

Are you set up to manage your dependencies efficiently and avoid zombie software from affecting your codebase?

Some sources of technical debt, including architecture, design, and documentation deficiencies, can only be addressed through a manual process of reviewing and refactoring. But there is a source of technical debt that has an automated solution: outdated dependencies. With such a large percentage of modern codebases being third-party software, aging dependencies can add crippling amounts of technical debt to your projects. And, over time, once you get too far into that debt, it can be nearly impossible to climb out without major remediation, making the addition of new features extremely difficult. It’s easy to update from version 1 to version 1.1, and even version 1 to version 3 or 4 is usually possible. But version 1 to version 20? Not so simple.

Old code is risky code 

Older code is a large target simply because there’s more of it and exploits have already been published. Most vulnerabilities that have been found in older versions of open source software have already been fixed in newer versions, which is great, but only if you’re on the newer version. Naturally, our focus skews towards security debt issues, but that’s only one perspective. From a non-security perspective, everything is harder: your quality, your reliability, and the basics of what you’re making become increasingly difficult if you don’t stay up to date.

Like the occasional slice of pizza or blowing off a workout, a little dependency-related technical debt usually won’t do much harm. But the longer you go without eating healthy and exercising, the worse things get. When your technical debt gets too big and your dependency health too far gone, you can be looking at the software equivalent of massive heart failure or even death. Once you’ve reached that point, it can be difficult if not impossible (not to mention costly) to reverse course, but it’s not all that difficult to avoid it altogether simply by updating on a regular basis. Of course, you have to actually make sure it gets done. It’s not very difficult to put off or altogether avoid updating, especially when it’s a manual process. As with preventive health for your body, preventive health for your software is always more likely to be achieved when it’s automatic and built into your processes and daily routine.

Dependency management is the health routine that keeps your code up to date

Developers are typically happy to have the latest versions of open source packages, if, and it’s a big if, those versions won’t break their code. Most updates are minor and unlikely to cause any problems, but you still have to remember to get them. Consumers rarely have to deal with problems like apps not working because they don’t have the right version of iOS or Android; our phones update automatically. Developers building new things into existence don’t have that same luxury.

What they do now have at their disposal is pretty powerful, though. Dependency management tools, like Renovate, allow developers to keep dependencies up to date easily by automating, scheduling, and grouping pull requests. An open source tool with a large number of users, Renovate also compiles crowdsourced information to give developers a merge confidence score so they can determine at a glance the likelihood that an update will break their code without having to comb through the release notes.

Dependency management and technical debt at enterprise scale

Making dependency health easy and painless for every developer makes a huge impact across an organization and across time. In the long term, you can avoid a lot of technical debt without much extra thought, time, or effort. It’s true that in many ways we’re talking about avoidance of debt rather than true reduction of existing debt. But even if you haven’t touched your code in five years, there are still some updates that will work and it’s worth using a dependency management tool like Renovate to figure out which ones they are. There’s an ancient saying (translated from FORTRAN, probably) that goes, “The best time to update your dependencies was five years ago. The second best time is today.”

So there’s no time to waste and no point in delaying. Dependency management can do a fantastic job of helping you tackle some older technical debt but its real value is in all of the technical debt it helps you avoid. If you’re not already requiring your developers to keep up with updates, it’s time to start. Your future self will thank you.

Why has the landscape changed in the software supply chain?

First, more software and applications are being developed than ever before, with ever more complex chains of components and dependencies that have led to a rise in vulnerabilities and an increased threat of malicious packages. To demonstrate this, the Mend Open Source Risk Report recorded a 33 percent growth in the number of open source software vulnerabilities added to the Mend.io vulnerability database in 2022, which outstripped the estimated 25 percent growth in the amount of open source software available. And Mend Supply Chain Defender showed a steady quarterly increase in the number of malicious packages published in 2022, with a significant jump in Q3, which increased 79 percent from Q2.

In turn, this has caught the attention of governments across the world from, the U.S. and the EU, to as far afield as Australia and New Zealand, leading to the second big trend, which is the introduction of governmental cybersecurity strategies and a recognition that regulation of the software supply chain is necessary to protect security on national, economic, public and private levels.

Why is dependency management important for securing the software supply chain?

Simply put, you can only protect your software and applications, identify issues, and then fix them if you know the following:

• What components and dependencies are in your software and applications
• Whether they are up-to-date and properly patched
• Whether open source third-party components and dependencies are compliant with the terms and conditions of their use
• Whether these components and dependencies are behaving as expected or are operating unusually
• If any others are being adversely affected
• What components and dependencies should and shouldn’t be there

Dependency management gives you the visibility you need to answer these questions and the capability to address any issues that you detect as a result. It involves identifying all dependencies used in your software, regularly monitoring and updating dependencies to ensure they are not vulnerable to known security issues, and maintaining a list of approved and unapproved dependencies.

How will escalating regulation affect the role of application dependency management?

More regulation means more accountability. More accountability requires increased visibility into the supply chain, comprehensive disclosure of components, dependencies, and any flaws and malicious packages that could threaten the stability and security of the software. So, the ability to catalog every component and dependency, identify threats and remediate them through a process of dependency management, becomes necessary.

With more stringent regulation and governance likely to be implemented, the transparency of software supply chain constituents becomes essential. Spearheaded by the likes of the U.S. Cybersecurity and Infrastructure Security Agency (CISA) and the Five Eyes Alliance, government cybersecurity strategies will require any company working with federal, public, national, and international organizations to disclose all the components and dependencies in their software, applications and code bases. These strategies shift the responsibility to vendors, to both infuse security into their platforms and to declare any threats and vulnerabilities that they encounter. What has until now been voluntary best practice will become pre-requisites, and what begins with large public contracts will inevitably spread to the private sector, as vendors to federal and public bodies will demand that their suppliers in turn undertake the same level of due diligence.

What tools do you need to manage application dependencies effectively?

Effective dependency management involves the implementation of a prophylactic strategy that should include a combination of identification, detection, prioritization, updates and remediation.

Identification lies at the core of the regulations and strategies proposed by governmental agencies. It is the key to cataloging and knowing what components and dependencies exist within your software and applications, and it’s the foundation for accountability and transparency. You achieve this by using a software bill of materials (SBOM) and making it available to all your customers. The SBOM will not only provide comprehensive visibility into your code base, but it will improve your ability to detect anomalies, flaws, and vulnerabilities in your code which could be a threat. As a result, using an SBOM can reduce your time to respond and remediate any issues. As such, we anticipate that the SBOM will become a prerequisite tool for effective dependency management.

Having identified and detected any vulnerabilities, you need to update and patch your software regularly, frequently and in a timely manner, to ensure that it’s resistant to newly arising issues and attacks. First, you’ll need to employ a tool that prioritizes which vulnerabilities are most critical and should be addressed first. Prioritization optimizes the updating and remediation process, ensuring that you address dependencies that are most important to your code base, so that you don’t waste time on those that are unimportant or redundant.

Then you can move on to actively updating your dependencies, and you can do this most effectively with a tool that automatically applies updates when they are identified. Additionally, you can apply automated remediation to vulnerabilities that need to be fixed, using software composition analysis (SCA) for open source dependencies and static application security testing (SAST) for custom code. Automation slashes mean time to remediation, thereby making your dependency management as efficient as possible. Couple these tactics with a tool that prevents malicious open source packages from entering your codebase and the capability to apply governance and compliance policies, will redouble your security and your dependency management.

Ideally, all of these capabilities should be rolled into one comprehensive platform that brings all these processes together in a security package that operates seamlessly with your development workflow, maximizing security without sacrificing development speed or efficiency. 

That’s how you can ensure that dependency management plays the most effective role in securing your code base and meeting the requirements of increased due diligence that comes with more regulation and governance of the software supply chain.

Another thing cybercriminals get is automation. Attacks are up and their rise is expected to continue, in no small part due to the fact that attackers are using automation to scale their criminal enterprises. With automated tools they can locate more victims faster, launch more attacks at once, and reach further along the supply chain than ever before. 

Well that’s just great. What’s to be done about it? The obvious answer is that software vendors must use defensive automation of their own.

The automation revolution in cyberattacks

Automation plays a significant role in the modern business of cyberattacks. Malicious actors use automation to their advantage to lower their costs, streamline their organizations, lend speed and scale to their attacks, and increase their chances of success. 

Here are just a few ways adversaries are using automation:

• Malware and botnets – The creation and management of botnets, networks of compromised computers which can be remotely controlled to perform various malicious activities including the distribution of malware, is largely done through automation.

• Phishing campaigns – Automated phishing kits are readily available for purchase on the dark web. These kits allow adversaries to easily create convincing phishing websites complete with fake login forms which can then be distributed, automatically of course, to a large number of potential victims.

• Credential stuffing attacks – Attackers use automated tools to test large lists of stolen or leaked username-password combinations against various websites and online services to gain unauthorized access. Let this be yet another reminder to not use the same passwords across multiple websites, systems, and services.

• Exploiting vulnerabilities – If you’re not scanning your code and infrastructure for security weaknesses, well you’ll never guess who is. Attackers employ vulnerability scanning tools to automatically identify security weaknesses in target systems, particularly outdated software and misconfigurations.

The race for automation in application security

Even with a generous budget for a large, well trained security team, manual testing cannot reach the scale, completeness, speed, or consistency of coverage that modern automated security tools can. Although manual testing certainly has its merits, it has many limitations that could have severe consequences for your organization.

And let’s face it, most security teams are overworked and understaffed. Here’s a formula that should help: automation plus a shift-left approach to application security equals shift smart. With the right tools, you can put some of the work of securing your code onto your developers and earlier into your software development lifecycle (SDLC) without slowing down releases.

Here are some automated application security testing tools you should be considering.

• Dependency updating tools – Chances are your software relies on a lot of open source components that are constantly releasing updates, many having to do with security. A good dependency health tool like Renovate enables automatic dependency updating by creating pull requests for your team to review and accept or reject. Free as both a GitHub app or self-hosted, Renovate also includes a Merge Confidence score to make decision making even faster and easier.

• Software composition analysis (SCA) – About those open source components… just because you have the latest version, doesn’t mean there aren’t still security concerns. SCA tools scan your open source code, including containers and other artifacts, and give you feedback on vulnerabilities. Advanced SCA tools (like Mend SCA) also give developers automatic alerts about vulnerable open source packages before a pull request is made and the component enters the system.

• Static application security testing (SAST) – For your bespoke code, SAST tools are used in both the coding and testing stages of the SDLC to analyze your code for coding errors and policy or regulation noncompliance that may indicate security concerns. We like Mend SAST, but we would – not just because we made it, but because it features automated remediation which writes the code to fix the security flaw for you.

Security everywhere all the time

Of course it’s not only automation that keeps attackers at bay. You also need to develop a good security culture in your organization that includes training and awareness programs, encourage responsible coding practices, and shift security left (and everywhere) so that it’s part of the fiber of your day to day activities.

But embracing automation in the SDLC with automated code review and continuous security testing must be on your menu if you want to have a robust and mature security program at your organization.

The bad guys are using automation to wreak havoc across the web. It’s time for the good guys to catch up and use automation to stop them.