One significant container security challenge stems from the common use of publicly available open source images. Developers typically pull container images with known vulnerabilities, or do not update container images fast enough to remediate CVEs, leading to significant security backlog. Software companies that build and operate containers spend thousands of hours each year, if not more, on vulnerability management. However, identifying, triaging, and patching CVEs found in your container images is a lengthy, cumbersome process that introduces a lot of toil and friction for organizations that need to move fast and ship new products and services to customers.

‍Conducting robust and accurate CVE triage requires dedicated staff and a significant time investment, translating into higher operational costs and potential delays in addressing critical vulnerabilities. Teams often become inundated with these tasks, which distracts them from more important tasks like product and business innovation.

‍Reachability: A new security paradigm 

Reachability has become a must-have feature for application security scanners because it helps separate exploitable risks from false positives. Reachability analysis takes vulnerability prioritization to the next level. It identifies relationships between software functions, so you can understand open source library vulnerabilities in their real-world context. By linking dependencies, you can trace forward through application code to see if an attacker could potentially access a given software flaw. Reachability can answer questions like:

• Is my code actually invoking this library with vulnerable code within it?

Reachability analysis lets you smoke out those vulnerability scanner findings for issues that pose minimal risk in your product or network. It also enables you to evaluate the safety of library upgrades and the elimination of unnecessary dependencies.

Effective reachability for container scanning¹

Before analyzing the needed reachability technology for container scanning, it’s helpful to differentiate between software composition analysis (SCA) and container scanning because vendors use the term “reachability” for both types of scanning. These are distinct but commonly confused tools. 

SCA

SCA is a tool that analyzes open source code (from other repositories, which are most commonly public but can also be internal dependencies) that have been imported by a developer and used in applications. The purpose of an SCA tool is to create a Software Bill of Materials (SBOM), surface potential vulnerabilities and risks with those OSS packages, and provide remediation recommendations.

Container scanning

Container vulnerabilities affect libraries that are installed onto the operating system of an image. Similar to an SCA tool, the container scanner looks for vulnerabilities in those imported images. However, given the nature of containers, which includes the operating system (OS) layer with the application code, the amount and the severity of the discovered vulnerabilities is significantly higher. 

For example, let’s take a look at Redis, a popular open source key-value store that functions as a data structure server. Scanning Redis source code (https://github.com/redis/redis) reveals the following results:

Scanning Redis’ official container image (https://hub.docker.com/_/redis) reveals:

Examining the results reveals that almost all CVEs are coming with the Debian OS (version 12.5) and that without this layer, the accumulated CVE results are minimal (7 CVEs).

Similar results can be seen with other popular open-source container images like Postgres. When scanning its source code (https://github.com/postgres/postgres) you can’t detect any known vulnerabilities. However when scanning its docker-hub official image (https://github.com/docker-library/postgres) build on Debian OS, you find more: 

Here, too, all CVEs come with the OS level.

But it’s not just Debian OS that adds multiple vulnerabilities to container images. Let’s take a look at another popular CNCF project called Buildpack using Ubuntu OS. Scanning its source code (https://github.com/paketo-buildpacks/go) reveals the following results:

Scanning its official container image (https://hub.docker.com/r/paketobuildpacks/build) shows the following:

Adequate reachability for containers²

Let’s examine the different reachability technologies to assess which one has the best results for container-security scanning. 

Function-level reachability (SCA)

An SCA tool with source-code reachability analysis scans your code to check if the vulnerable library is being used in a way that can be exploited by the detected CVE. It can trace entire application contexts to verify if the vulnerable circumstances can happen outside of simple code execution flow. With function-level reachability, if CVE exists in the code but can’t be exploited, then the CVE is a false positive that you don’t need to send for remediation.

There are some inherent challenges with this methodology, because multiple CVEs don’t contain the function-level data needed for the reachability analyzers, and some CVEs require the general application context to validate if an exploit is possible. Nevertheless, reachability in the SCA context provides a great solution for application source-code analysis.

In container images, the vulnerable libraries are installed onto the operating system. SCA reachability doesn’t cover the OS layer and consequently does not identify the vast majority of CVEs reported for container images.

Consequently, it’s not suitable for container security. 

Package used in image (Container)

A common method to detect reachability for container image vulnerabilities utilizes runtime methods. Equipped by technologies like eBPF, modern runtime security tools extract the loaded code packages when the container is running; with runtime assessment, users can identify whether the packages (not functions) are being used/loaded and thus prioritize the CVEs of these packages. 

Examining the previous Redis container example deployed in AWS EKS cluster, we see the following results:

A first glance shows great results: between 7-15 CVEs are in use while the container is running, no need to fix the other 265 CVEs.

However, a deeper look into the result reveals fluctuations in the number of reachable CVEs, which is based on the actual usage of the container while it’s running. This method is actually a runtime snapshot of the loaded packages, which might change based on the container usage. Hence, there’s a potential for CVEs to be missed with the “in use” methodology that captures current usage status. 

There are also some practical aspects that might limit the “in use” methodology. A common argument is that when the container is deployed, it’s too late to identify which code parts need to be changed/modified due to code vulnerabilities. Additionally, the cost implications of addressing security issues in a running application, rather than earlier in the development cycle, cannot be overstated. Remediation at this stage often entails significant operational disruptions and resource allocations, as fixes may require rolling back production environments, comprehensive system audits, and extensive testing to ensure that patches do not negatively impact functionality or introduce new vulnerabilities.

Runtime prediction—Container images reachability

To bridge the gap between function-level reachability and package-in-use for running images, one may use a static reachability analysis for container images. This method introduces an algorithm that analyzes the compiled code and configuration files in the container’s file system and indicates which of the packages can be used at runtime, and which are considered as “dead code”. In contrast to runtime solutions, this analysis can be used in the classical shift-left (pre-production) scanning points of containers images (e.g., CI or image registries) or in Git repos (with adequate build option). It requires no additional setup, and it is much faster and more efficient than capturing data from running environments in production, a process which usually takes a few days to get accurate.

There’s also an often-overlooked value in this case. Because SCA and container teams are sometimes separated, AppSec managers may find themselves in between SDLC stages, just to figure out way too late (in production) or too early (code without the OS context) about security risks and their actual reachability status. Looking back at the Redis example with runtime prediction analysis, we get an 84% reduction before deploying anything to our running environment.

Conclusion

Reachability has become a must-have feature for open-source vulnerability detection. It identifies the exploitable risks, separating them from false positive alerts. Reachability analysis allows efficient prioritization by identifying relationships between code components and understanding library vulnerabilities in their real-world context. 

In container images, reachability-based prioritization is crucial, as the amount of detected CVEs is typically skyrocketing. However the classical reachability used in SCA isn’t adequate for container security usage, it requires a reachability analysis that addresses the OS level CVEs. 

¹The following security scanning results were done by Mend’s SCA and Mend’s Container solution but were compared to other commercially available and open-source scanners that reveal similar (but not identical) results.

²The reference for this section is from the remarkable research work by James Berthoty (5 types of reachability analysis: https://www.endorlabs.com/learn/5-types-of-reachability-analysis-and-which-is-right-for-you).

Most applications deployed through Kubernetes require access to databases, services, and other resources located externally. The easiest way to manage the login information necessary to access those resources is using Kubernetes secrets. Secrets help organize and distribute sensitive information across a cluster.

What are secrets?

A Secret is an object that contains a small amount of sensitive data such as a password, a token, or a key. Such information might otherwise be put in a Pod specification or in a container image. Using a Secret in Kubernetes means that you don’t need to include confidential data in your application code.

Because Secrets can be created independently of the Pods that use them, there is less risk of the Secret (and its data) being exposed when creating, viewing, and editing Pods. Kubernetes, and applications that run in your cluster, can also take additional precautions with Secrets, such as avoiding writing sensitive data to nonvolatile storage.

Secrets are similar to ConfigMaps but are specifically intended to hold confidential data.

Related: Kubernetes Pod Security Policy Best Practices

Why are Kubernetes secrets important?

In a distributed computing environment it is important that containerized applications remain ephemeral and do not share their resources with other pods. This is especially true in relation to PKI and other external confidential resources that pods need to access. For this reason, applications need a way to query their authentication methods externally without being held in the application itself.

Kubernetes offers a solution to this that follows the path of least privilege. Kubernetes Secrets act as separate objects that can be queried by the application Pod to provide credentials to the application for access to external resources. Secrets can only be accessed by Pods if they are explicitly part of a mounted volume or at the time when the Kubelet is pulling the image to be used for the Pod.

Learn More: Kubernetes Security: Best Practices and Tools

How does Kubernetes leverage secrets?

The Kubernetes API provides various built-in secret types for a variety of use cases found in the wild. When you create a secret, you can declare its type by leveraging the `type` field of the Secret resource, or an equivalent `kubectl` command line flag. The Secret type is used for programmatic interaction with the Secret data.

Updating and rotating secrets

Regularly updating and rotating secrets is a crucial security practice. While you can update a secret using kubectl edit, this approach is not recommended because it can be error-prone and can potentially lead to unintended consequences.

When you edit a secret with kubectl edit, you are modifying the existing secret in place. This means that any existing references to the secret (such as environment variables or volume mounts) will continue to use the old secret data until the application is restarted or the pod is deleted and recreated.

If you need to rotate a secret, you must update any references to the old secret to use the new secret instead!

That’s why past a certain size, it becomes useful to either implement app-specific mechanisms that reload configuration at runtime or deploy a sidecar container that monitors for changes and restarts the main container when necessary.

Limitations of Kubernetes secrets

While Kubernetes Secrets provide a convenient way to manage sensitive data, they have some limitations:

• Limited encryption. By default, secrets are stored unencrypted in etcd. Kubernetes does support encryption, but the encrypting key needs separate management.
• Limited rotation. Kubernetes secrets are designed to be immutable, which means that they cannot be modified or versioned once they are created. This, as we have seen, makes it difficult to rotate secrets. They can’t be audited either.
• Limited access control. While Kubernetes provides RBAC (Role-Based Access Control) to control access to secrets, it is still possible for unauthorized users to gain access to secrets if they can compromise the cluster or the underlying infrastructure.
• They may not be suitable for large-scale or highly regulated environments, where more advanced secret management solutions might be necessary.

Despite these limitations, Kubernetes Secrets remain a handy tool for managing secrets when you don’t need to scale immediately your cluster.

Kubernetes external secrets

Kubernetes External Secrets offer an alternative approach to managing secrets in Kubernetes by integrating with external secret management solutions. This allows you to maintain sensitive data outside of your Kubernetes cluster while still providing seamless access to applications running within the cluster.

How does it work?

Kubernetes External Secrets are custom resources that act as a bridge between your Kubernetes cluster and external secret management systems.

Instead of storing secrets directly in Kubernetes, External Secrets fetch and synchronize secrets from external systems, making them available as native Kubernetes Secrets. This ensures that your applications can access sensitive data without any code changes while benefiting from the security features provided by external secret management solutions.

Integrating with external secret management solutions

Kubernetes External Secrets can integrate with a variety of external secret management solutions, such as HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, and Google Cloud Secret Manager.

To integrate External Secrets with your chosen secret management system, you need to deploy the corresponding External Secrets controller in your cluster and configure it to communicate with the external system.

For example, to integrate with HashiCorp Vault, you would deploy the Kubernetes External Secrets controller for Vault and configure it with the necessary Vault authentication and connection details.

Creating and using external secrets in Kubernetes

To create an External Secret, you need to define a custom resource in a YAML file, specifying the reference to the secret stored in the external system:

apiVersion: external-secrets.io/v1alpha1
kind: ExternalSecret
metadata:
  name: my-external-secret
spec:
  backendType: vault
  data:
    - secretKey: username
      remoteRef:
        key: secret/data/my-secret
        property: username
    - secretKey: password
      remoteRef:
        key: secret/data/my-secret
        property: password

Apply the YAML file using kubectl apply -f my-external-secret.yaml

The External Secrets controller will fetch the secret data from the external system and create a native Kubernetes Secret with the same name. This generated secret can be used by your applications in the same way as regular Kubernetes Secrets.

Advantages of Kubernetes external secrets

Using Kubernetes External Secrets offers several benefits:

• Enhanced security by leveraging the features of external secret management solutions, such as encryption, access control, and auditing.
• Reduced risk of exposing sensitive data within the Kubernetes cluster.
• Simplified secret management for organizations already using external secret management systems.
• Centralized secret management across multiple Kubernetes clusters and other platforms.

By integrating Kubernetes External Secrets with your chosen secret management solution, you can achieve a higher level of security and control over your sensitive data while maintaining compatibility with your existing Kubernetes applications.

Best practices for managing secrets in Kubernetes

To ensure the security and integrity of your sensitive data, it is crucial to follow best practices for secret management in Kubernetes. Below are some of the most important practices to keep your secrets secure and maintain a robust Kubernetes environment.

Role-based access control (RBAC)

RBAC is essential for managing secrets securely, as it enables you to control which users and components can create, read, update, or delete secrets. By implementing fine-grained access control, you can minimize the risk of unauthorized access and potential data breaches.

To implement RBAC for secrets management, you should create roles and role bindings that define the allowed actions on secrets for each user or group. For example, you can create a role that allows read-only access to secrets within a specific namespace and bind it to a specific user or group:

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: my-namespace
  name: secret-reader
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "watch", "list"]

Encrypting secrets at rest and in transit

Encrypting secrets is crucial for protecting sensitive data from unauthorized access, both when stored in etcd (at rest) and when transmitted within the cluster (in transit).

Kubernetes provides native encryption options, such as enabling etcd encryption to protect secrets at rest and using TLS for securing communications within the cluster. Ensure these options are configured and enabled to maintain the confidentiality of your secrets.

In addition to Kubernetes native encryption options, you can also integrate third-party encryption solutions, such as HashiCorp Vault or cloud-based key management services, to further enhance the security of your secrets.

Secret rotation and expiration

Regularly rotating secrets is an essential security practice that minimizes the risk of unauthorized access and potential data breaches.

Strategies for secret rotation include manual updates using kubectl or automated rotation using custom controllers or third-party secret management solutions.

Automating secret rotation can be achieved using Kubernetes operators, external secret management systems, or custom scripts that periodically update secrets based on a predefined schedule or events.

Auditing and monitoring

Auditing and monitoring are crucial for maintaining the security and integrity of your secrets, as they enable you to track and analyze secret access, usage, and modifications and detect potential security incidents.

Several tools can be used for auditing and monitoring secrets, such as Kubernetes audit logs, Prometheus, Grafana, and commercial solutions such as Mend.io that provide detection against hard-coded secrets.

Configure alerts and notifications to proactively notify administrators of potential security incidents or irregular secret access patterns, enabling timely investigation and response to potential threats.

Final thoughts

In this blog post, we have discussed the importance of secrets management in Kubernetes and explored various methods and best practices for securely managing sensitive data in your applications. Kubernetes Secrets and Kubernetes External Secrets provide powerful tools and methodologies for managing secrets effectively, ensuring the security and integrity of your sensitive information.

Managing secrets securely in Kubernetes is critical for maintaining the confidentiality and integrity of your sensitive data. By following best practices and methodologies, you can create a robust and secure Kubernetes environment, safeguarding your applications and data from potential threats.