The Human Error Behind Hacking Kubernetes

by | Mar 23, 2020 | Araido News

As IT infrastructure has evolved with high speed internet and cloud native technologies, methodologies such as Agile and DevOps have become mainstream and enabled development teams to build more rapidly, requiring the cloud native estate to scale in a more flexible way and permit applications to build and enhance performance while maintaining the ability to develop software as quickly as possible. 

As such, containers, like Docker, and container orchestration, like Kubernetes, evolved.

Container orchestration

A container differs from a virtual machine in that it does not contain an abstraction of a physical machine; instead, it only virtualizes the operating system which makes it extremely light weight and cost effective.

Unfortunately, if there is more than one element, or service, simple containerization is not sufficient and container orchestration is required, most popularly in the form of Kubernetes. Unsurprisingly, the more services there are to operate, the more complex it becomes to orchestrate them in a secure manner.

 

Default Kubernetes Security

Using a container orchestrater, such as Kubernetes, can be a powerful way to increase security posture by reducing blast radius and ensuring that particular elements, or pods, are fully partitioned and cannot be touched by any other process within the cluster. By default, Kubernetes is enabled with Role Based Access Control, Namespaces, and TLS. Users can implement network security policies and general policies to restrict read and write access, as with more traditional systems, and can even “taint” nodes and pods so that only other tainted nodes and pods may touch them.

Leaving Kubernetes in an  unadultered state

Where Kubernetes increases its potential hacking risk for many organizations, however, is in the implementation of these measures. When dealing with potentially hundreds of micro-services, the possible omission of a single policy or RBAC permission could potentially leave an API fully open to the public without any need for authentication.

Tesla, a major player and consumer in cloud native technology, has a well equipped, knowledgable fleet of developers which it can afford to pay a generous living wage and has still fallen prey to a container specific attack vectors.  An omission of an authentication requirement resulted in hackers stealing both AWS and Azure credentials to successfully mine for cryptocurrency.

 So, it is entirely possible to deploy Kubernetes and ensure functionality in its ‘unadulterated’ state.

 

Astronomically higher bills

The orchestration will work reasonably well and while no glaring security concerns may be visible and the configurations may be sufficient for parties deploying software into development accounts. These default rules give little to no protection should the Kubernetes database, etcd, be breached or if a crypto-jacker is able to enter through an unprotected API and autoscale the pods in the cluster. With pods set to autoscale, without appropriate limitations should they reach maximum memory capacity, and without proper logging, clusters may be grown as large as the attacker may wish. Many organizations are not aware that they may be under attack until they receive a monthly bill which is astronomically higher than anticipated.

 

Shodan Playbook of exploits

While most do not consider Kubernetes Internet of Things, many do not realize that etcd is listed on Shodan, the IoT search engine on port 2379. A quick search on Shodan indicates that there are 2500 publicly available etcd endpoints that are running on port 2379 with further details pertaining to the tech stack and known vulnerabilities relating to that tech stack. As most security proponents would advise, relying solely on security by obscurity is unwise. However, if consumers are not conscious of the mitigations required to keep clusters, at a minimum, private and require adequate authentication, not only are malicious actors able to unabatedly attack critical parts of the technical infrastructure, but tools such as Shodan essentially give them a playbook of exploits while offering attackers an open door to do so.

 

Port Scanning

Shodan is not the only concern regarding exposed information. Kubernetes listens to rather distinguishable ports. As a result, port scanning is extremely efficient and easy. Information such as the database, application type, and ingress controller can all be easily identified using a simple port scan. In the event one permission is misconfigured in the wrong place, access through an automated scan in the hands of a script kiddy could easily succeed.

 

Kubelet Access

In many cases there is concern of a network breach where the node resides. In this particular case, while some RBAC rules could limit the potential damage and blast radius, access to the nodes on the network infers the attacker gaining access to the core, or kubelet, of a cluster . By gaining control of a kubelet, the attacker has gained unfettered access to the Kubernetes application and likely all of its contents through a back door and thus likely remaining undetected as the breach came from inside the network. It is possible, in some cases, to modify deployments and increase memory limits, again introducing a risk of not only a data breach, but of cryptojacking as well.

 

Forensics

Another challenge that organizations must be aware of is the challenges of forensics with ephemeral artifacts. Containers were not intended to run for extended periods of time. Instead, they are meant to run until a change is made, at which time they are destroyed and a new container with a new image replaces the old. While this is extremely efficient in regard to ensuring homogeneity in environments, there are also negative consequences for backup and restore as well as forensics. In the event that the kubelet is “owned” and the goal of the attack is to cause the most harm possible, a few simple commands would completely eradicate an entire cluster without any backup or the ability to conduct any forensics. As such, it is imperative to examine external tooling and options before moving to a containerized infrastructure.

Conclusion

While Kubernetes comes with some extremely high risks, a vibrant, open source community also works to ensure a high quality, customizable solution that can manage any workload. The security options are robust, and while they may take a fair amount of configuration, when done correctly, the payoff is undeniable. When considering containerization for large scale rollout over an organization, it is important to think about the best way to maintain such a platform without requiring all development teams to have a subset of container security experts within the team itself. A pre-hardened, managed container platform with adequate controls and RBAC specifications should be provided to teams in order to permit them to leverage the power of containerized solutions, without unnecessarily raising the risk level of the organization without their knowledge.

Sarah Polan is a DevSecOps consultant at Araido. She wrote this article as part of her curriculum Master of Engineering – Cybersecurity Policy and Compliance at The George Washington University.

Resources