GitHub put the world's biggest code repository at risk - Secrets need better security

On Feb 28, 2023 GitHub announced an exciting new feature – secret scanning for public repositories, now free for all. The very next month, on March 24, they reported that they’ve discovered an “inadvertent publishing of private information” which left their RSA SSH host keys exposed in a public repository. Yay, for secret scanning. Nay, for GitHub’s security protocols. This is a big deal because it affects all GitHub repositories, and GitHub being the world’s biggest source code repository makes everyone sit up and take notice. But the bigger questions to ask are what does this teach us about secrets management? If this can happen to GitHub, can it happen to your organization? How can you prepare for something like this, or even better, how can you prevent it? Let’s explore these questions in this post.

What is an RSA SSH key?

RSA is a cryptographic algorithm that ensures secure access to data in a system. It is also used for signing and generating a pair of SSH keys – one public, and one private key. The keys are used to authorize access. The public key can be shared over the networks to other cloud services or networks, but the private key should never be shared, and should be stored with utmost security locally.

Protecting RSA SSH keys

Many times private SSH keys are stored in “convenient” locations for the R&D, such as local files in a non-secured storage, as parameters, collaborative platforms or even in repositories. This is not secure. Instead, best practice is to store private keys in managed, fully monitored and access-enforced locations, such as Hardware Security Module (HSM), Key Management Services (KMS), or even Secret Vaults. Thus the keys can be managed and rotated in a better, convenient way to allow better security enforcement.

Accidents happen, even with the biggest of companies

These best practices are hard to follow in the real world. Especially at a large organization like GitHub with numerous teams and people that access these keys on a daily basis, it’s hard to setup and implement policies that involve HSM’s and ensure everyone follows these processes. Despite the best efforts and detailed processes put in place by a CISO, it can be a humble SysAdmin who ‘inadvertently’ exposes a secret.

“Even GitHub, the home of all developers, is not immune to #secrets #sprawl!” – Ziad Ghalleb

It is unclear if this leak was the result of an external breach or an accidental share by an employee. What’s at stake is the security of all GitHub accounts – and this is a big scare for all GitHub users. Attackers could have used man-in-the-middle (MITM) attacks to intercept communication and gain access to confidential data of users.

Such incidents cause users to lose trust in organizations. Alexandre Blanc comments that “Due to its constant and continuous state of leak, cloud housekeeping is a pure nightmare, endless and costly, trying to hide evidence, deceive and manipulate.” He has a point about cloud housekeeping being a nightmare. A lot of these issues come down to just normal human behavior – they’re unavoidable. But accepting that humans will falter at some point is a good starting point to consider how to respond when this inevitably happens.

What you can do to secure secrets

Organizations deal with numerous secrets today, and the number is only multiplying as cloud estates grow in size and complexity. There are many ways to secure these secrets.

Inventory all your secrets

Peter Drucker said, “If you can’t measure it, you can’t manage it.” This holds true for secrets. The first step to managing your secrets is to know exactly how many secrets exist across your organization’s technology stack.

This is easier said than done. Today, there isn’t a solution available in the market that can give you the exact number of secrets across your organization without missing a single secret. As a result, security teams don’t know how many secrets they have and where they are. This is the first step to protecting your secrets – have a list of all your secrets that you need to protect.

Use a vault

You can use a vault to store secrets in a secure way. Here, the way you manage access to these secrets in the vault really matters. While some think that a vault solution is enough to secure secrets, this is far from the truth.

In reality, A vault can end up being just another place in the cloud to store secrets, and it can be compromised itself. There are a lot of different vault solutions available in the market. Most organizations use at least 5 vaults at any given time. Often, different teams and projects have their own vault. This makes vaults really hard to maintain and secure. These vaults act merely as secrets databases, and like databases they don’t really protect the secrets stored within them. It’s a common myth that a vault alone is enough to secure your secrets in the cloud.

Auto-rotate secrets

Rotating secrets is a critical measure for preventing secret exposure, however, in many R&D teams, it is frequently a complex process requiring manual intervention. This is because secrets are often not centrally managed or adequately integrated into a cycle that enables automatic rotation, secure storage, and authorized access for relevant users and clients.

Scan for exposed secrets

You can use secret scanners to scan for exposed secrets across code repositories, configuration files, environment variables, cloud services, Slack, wikis, Jira tickets, logs, and more. That’s how GitHub found out about this exposed secret. Some scanners scan code before it is deployed as it passes through the supply chain, and others scan code that has been deployed into production. However, finding an exposed secret is only half the job done. Knowing what kind of secret it is, and what to do about it is even more important.

All these practices when done together help. Think of them as basic hygiene for secrets – they’re good to do. They can build on one another and greatly bolster secrets management. Yet, there is still a missing piece when it comes to managing secrets.

The missing piece in secrets management

What’s missing is an end-to-end secrets management tool that can tell you the context of every secret – how important it is, what resources it can access, who has access to it, when it was last used, when it was last updated, and more. This context is essential for RSA key management.

You see, secrets have a secret life of their own – from the time they are created, the many paths they traverse within a network, how they are accessed by many users, and finally when they are revoked. Secure storage, scanning, and auto-rotating are different parts of this entire lifecycle of a secret, but they don’t give you the whole picture.

What modern cloud-native organizations need is a secrets management solution that watches over secrets end-to-end, and takes action on them automatically in real-time. It should be able to understand policies, priorities, and access levels. It should easily spot abnormal and suspicious behavior and report it. It should automatically revoke exposed secrets and replace them with new ones. It should constantly monitor them to prevent abuse, misuse and potential attack threats. The world needs a better secrets management solution.

There have been great advances in cloud-native technologies, and the technology stack has become more advanced and powerful. Unfortunately, secrets management hasn’t evolved at the same pace. It’s time for a new breed of secrets management tooling – one that really gets secrets in a cloud-native world. Stay tuned for more on this topic. We have a lot to say about it, and some exciting updates coming your way.