How to Create, Encode, Encrypt, and Monitor Kubernetes Secrets

This guide to Kubernetes Secrets will focus on a critical yet often underestimated security aspect: secrets and non-human identity management. We will cover how Kubernetes secrets are created, encrypted, and used and how important it is to monitor them.

What are Kubernetes Secrets?

Kubernetes Secrets are crucial in managing sensitive information within containerized environments. They serve as secure objects designed to store and manage confidential data such as login credentials, API keys, tokens, and certificates. The primary purpose of Kubernetes secrets is to reduce the risk of exposing sensitive information during application deployment and runtime in Kubernetes clusters. To that end, they are kept separate from the application code and Pod specifications. 

They act as an intermediary but play a vital role in non-human identity management for entities within the cluster since for applications to authenticate and access protected resources, any sensitive data doesn’t have to be embedded directly anywhere. 

The etcd component in Kubernetes acts as a vault for confidential information. This architecture allows teams to manage sensitive data separately from the containerized workloads that rely on it. Applications can then access these Secrets through volume mounts or environment variables, providing flexibility in how sensitive data is consumed. This mechanism ensures that only authorized Pods can access the specific secrets they need to function.

What are the different types of Kubernetes Secrets?

The various types of Kubernetes Secrets have their specific use cases:

Opaque Secrets

Opaque secrets are the most versatile type used for arbitrary key-value pairs of sensitive data. They are ideal for storing passwords, API keys, and other custom-sensitive information.

Eg:
apiVersion: v1
kind: Secret
type: Opaque data:
API_KEY: bXlzZWNyZXRrZXk=

TLS Secrets

Specifically designed to store TLS certificates and private keys, these are commonly used with Ingress resources to secure communication.

Eg:

apiVersion: v1
kind: Secret
type: kubernetes.io/tls
data:
tls.crt: base64encodedcert
tls.key: base64encodedkey

Docker Registry Secrets

These are used to store authentication credentials for private container registries and enable pods to pull images from protected repositories.

Eg:
kubectl create secret docker-registry regcred --docker-server=<your-registry-server> --docker-username=<your-name> --docker-password=<your-pword>

Basic Authentication Secrets

Stores credentials for HTTP basic authentication. While similar to Opaque secrets, this type provides clearer intent and structure.

Eg:
apiVersion: v1
kind: Secret
type: kubernetes.io/basic-auth
stringData:
username: admin
password: t0p-Secret

SSH Authentication Secrets

Holds SSH private keys, facilitating secure connections to external resources or for Git operations within pods.

Eg:
kubectl create secret generic ssh-key-secret --from-file=ssh-privatekey=/path/to/.ssh/id_rsa

Service Account Token Secrets

Although less commonly created manually, these secrets store tokens for service account authentication within the cluster.

Eg:

apiVersion: v1
kind: Secret
type: kubernetes.io/service-account-token
metadata:
name: my-service-account-token
annotations:
kubernetes.io/service-account.name: my-service-account

This creates a long-lived token for the "my-service-account" service account.

Bootstrap Token Secrets

Used during the node bootstrap process, these secrets store tokens for authenticating new nodes joining the cluster.

Eg:

apiVersion: v1
kind: Secret
type: bootstrap.kubernetes.io/token
metadata:
name: bootstrap-token-abcdef
namespace: kube-system
stringData:
token-id: abcdef
token-secret: 0123456789abcdef
usage-bootstrap-authentication: "true"

This creates a bootstrap token with ID “abcdef” that can be used for node authentication during cluster bootstrapping.

Each Secret type serves a distinct purpose, allowing Kubernetes to handle various authentication and security scenarios efficiently. By choosing the appropriate type, you ensure that your sensitive data is stored and accessed to align with its intended use and security requirements.

How to create and encode Kubernetes Secrets?

Kubernetes offers several methods for creating and encoding secrets, each with its own advantages. This section explores three primary approaches: using kubectl, YAML manifests, and Kustomize.

Using kubectl

kubectl provides a command-line interface for creating secrets directly in your Kubernetes cluster and supports two main methods:

• You can create secrets with literal values by specifying key-value pairs directly in the command line. It’s particularly useful for quick, one-off secret creation or when integrating secret creation into scripts or CI/CD pipelines.

Eg.

kubectl create secret generic my-literals-secret \
--from-literal=username=admin \
--from-literal=password=supersecret

This command creates a secret named ‘my-literals-secret’ with two key-value pairs. Kubernetes automatically encodes these values.

• Using kubectl create secret from file:

Eg.

echo -n 'admin' > ./username.txt
echo -n 'supersecret' > ./password.txt

kubectl create secret generic my-file-secret \
--from-file=./username.txt \
--from-file=./password.txt

Here, we create a secret named ‘my-file-secret’ using the contents of ‘username.txt’ and ‘password.txt’. The filenames become the keys to the secret.

This approach is beneficial when dealing with larger secrets, and certificates, or when you prefer to keep sensitive data in separate files for better organization and security.

In both cases, kubectl automatically handles Base64 encoding of the secret values, simplifying the process for users.

Using YAML manifests

For a more declarative approach, you can define secrets in YAML manifests. This method aligns well with GitOps practices and allows version control of your secret definitions (though not the secret values themselves).

Eg.

apiVersion: v1
kind: Secret
metadata:
name: my-yaml-secret
type: Opaque
data:
username: YWRtaW4=
password: c3VwZXJzZWNyZXQ=

In this example, ‘YWRtaW4=’ is the base64 encoding of “admin”, and ‘c3VwZXJzZWNyZXQ=’ is the encoding of “supersecret”. You can create this secret using ‘kubectl apply -f secret.yaml’. 

When using YAML manifests, you must manually Base64 encode Kubernetes Secrets values before including them in the manifest. While this adds an extra step, it provides more control over the encoding process and allows for easier integration with existing configuration management systems.

Using Kustomize

Kustomize, Kubernetes’ built-in configuration customization tool, offers a powerful way to generate secrets. It allows you to define secrets as part of your overall Kubernetes resource configurations, promoting a more holistic approach to application deployment.

Eg.

# kustomization.yaml
secretGenerator:
- name: my-kustomize-secret
literals:
- username=admin
- password=supersecret

Apply this configuration with:

kubectl apply -k .

Kustomize can generate secrets from literals or files, similar to kubectl while also handling  Base64 encoding automatically and reducing the manual steps required. Additionally, Kustomize adds a unique suffix to secret names, ensuring that updates to secret contents trigger a rollout independent resources.

How to edit a Kubernetes Secret?

To edit an existing Kubernetes Secret, you can use the ‘kubectl edit’ command. This opens the Secret manifest in your default text editor:

kubectl edit secret my-secret

You can modify the base64-encoded values in the editor in the data field. For example:

apiVersion: v1
kind: Secret
metadata:
name: my-secret
type: Opaque
data:
username: bmV3LXVzZXJuYW1l  # base64 encoded "new-username"
password: bmV3LXBhc3N3b3Jk  # base64 encoded "new-password"

Save and exit the editor to apply the changes.

How to use Kubernetes Secrets?

Once you’ve created Kubernetes Secrets, you need to make them available to your applications. Let’s explore three common methods: mounting as volumes, using as environment variables, and employing imagePullSecrets.

Mounting Secrets as volumes

Mounting Secrets as volumes allows your containers to access the secret data as files. Here’s an example:

apiVersion: v1
kind: Pod
metadata:
name: secret-vol-pod
spec:
containers:
- name: app-container
image: busybox
command: ["/bin/sh", "-c", "cat /etc/secrets/username && cat /etc/secrets/password"]
volumeMounts:
    - name: secret-volume
mountPath: /etc/secrets
readOnly: true
volumes:
- name: secret-volume
secret:
secretName: my-yaml-secret

In this example, the ‘my-yaml-secret’ is mounted as a volume at ‘/etc/secrets’. The container can then read the secret data as files.

Using Secrets as environment variables

You can also expose Secrets as environmental variables within your containers:

apiVersion: v1
kind: Pod
metadata:
name: secret-env-pod
spec:
containers:
- name: app-container
image: busybox
command: ["/bin/sh", "-c", "echo $SECRET_USERNAME && echo $SECRET_PASSWORD"]
env:
- name: SECRET_USERNAME
valueFrom:
secretKeyRef:
name: my-yaml-secret
key: username
- name: SECRET_PASSWORD
valueFrom:
secretKeyRef:
name: my-yaml-secret
key: password

This method injects the secret values directly into the container’s environment.

Using imagePullSecrets

imagePullSecrets are used to authenticate with private container registries:

apiVersion: v1
kind: Pod
metadata:
name: private-image-pod
spec:
containers:
- name: private-app
image: private-registry.example.com/my-app:v1
imagePullSecrets:
- name: registry-secret

Here, ‘registry-secret’ (created separately) contains the credentials for the private registry. Kubernetes uses these credentials to pull the private image.

Each method of using Secrets has its use cases. Volumes are great for accessing multiple secrets, environment variables offer simplicity, and imagePullSecrets are essential for working with private registries. 

How to monitor and audit secrets?

Kubernetes provides built-in auditing capabilities that allow you to track API server requests, including those related to secrets. By enabling and configuring audit logging, you can capture detailed information about secret-related operations, such as creation, modification, and access attempts. Imagine how invaluable that data must be for forensic analysis and security investigations.

While Kubernetes’s native auditing is powerful, specialized tools like Entro can enhance your secrets monitoring capabilities. Entro offers a comprehensive solution for managing and monitoring secrets in Kubernetes environments with features such as:

• Centralized secrets and non-human identity management across multiple clusters
• Real-time monitoring of secrets access and usage
• Automated secrets rotation and expiration
• Detailed audit logs and compliance reporting
• Integration with existing security and monitoring tools

Best practices for Kubernetes secrets management

Let’s explore some key best practices for Kubernetes secrets management:

1. Avoid hardcoding secrets

Hardcoding secrets is perhaps the single worst practice in your cybersecurity endeavors.

Entro provides advanced capabilities in this area, including the ability to detect hardcoded secrets not only in your application code but also in GitHub repositories and Slack channels. This comprehensive scanning helps identify and eliminate potential security vulnerabilities.

2. Rotate secrets regularly

Rotating secrets regularly is essential to mitigate the risk of compromised credentials. Implement a systematic approach to secret rotation, ensuring that all secrets are updated periodically.

As a secrets and non-human identity management solution, Entro can significantly enhance your secret rotation strategy since it leverages secrets metadata to alert system admins about secrets that need rotation, ensuring timely updates and reducing the risk of using outdated credentials.

3. Implement encryption

It’s worth noting that while Kubernetes Secrets encryption offers improved security over hardcoding credentials but because they are not encrypted, they are not a complete solution. In fact, going Hashicorp vault vs kubernetes secrets for secrets storage — the vault will win every time.

Even Base64 encoding is easily reversible and should not be considered encryption. So, always treat encoded secrets as sensitive data and implement proper access controls and encryption at rest (in the cluster’s persistent storage) and in transit (use TLS across all the communication channels transmitting secrets data).

5. Use namespaces for isolation

Leverage Kubernetes namespaces to isolate secrets and limit their accessibility across different parts of your application.

6. Regularly audit secret usage

Conduct regular audits of secret usage to ensure that only authorized entities access secrets and access patterns align with expected behavior.

Entro’s monitoring and auditing features play a crucial role here. It provides detailed insights into secret access patterns, helping you identify anomalies or potential security breaches. With Entro, you can set up automated alerts for suspicious activities and generate comprehensive audit reports, making maintaining compliance and security standards easier.

Key practices like implementing least privilege access, regular secret rotation, and continuous auditing form the foundation of a secure Kubernetes environment. However, secrets and non-human identity management isn’t a one-time task — it’s an ongoing process that requires vigilance and adaptation. As your Kubernetes infrastructure expands and becomes more complex, your security strategies must evolve in tandem. 

Explore how Entro can enhance your security posture and streamline your operations. Click here for a demo!