Crypto-mining hackers hit Kubernetes clusters

New campaign abused Kubeflow dashboards to install malicious containers

Bitcoin cryptocurrency mining

Security researchers have warned of hackers' continued attacks against Kubernetes clusters running Kubeflow machine learning (ML) instances by installing malicious containers that mine cryptocurrencies, such as Monero and Ethereum.

According to Microsoft Senior Security Researcher Yossi Weizman, the attacks began at the end of last month as he and his team discovered a spike in TensorFlow machine learning pod deployments. An investigation of the entry point of the pods revealed deployment aimed to mine cryptocurrency.

"The burst of deployments on the various clusters was simultaneous. This indicates that the attackers scanned those clusters in advance and maintained a list of potential targets, which were later attacked at the same time," said Weizman.

The hackers used two images in the attack. The first was the latest version of TensorFlow (tensorflow/tensorflow:latest), and the second was the latest version with GPU support (tensorflow/tensorflow:latest-gpu).

The images were legitimate but ran malicious crypto-mining code. The attackers abused the access to the Kubeflow centralized dashboard to create a new pipeline. Kubeflow Pipelines is a platform for deploying ML pipelines based on Argo Workflow. These dashboards were exposed to the internet instead of being only open to local access.

Hackers deployed at least two pods on each cluster: one for CPU mining, and the other for GPU mining. Both containers used open-source miners from GitHub: Ethminer in the case of the GPU container and XMRIG in the CPU one.

Related Resource

A guide to enterprise detection and response providers

The 12 providers that matter most and how they stack up

Forrester enterprise detection WPDownload now

The malicious pods all had the same pattern of name; "sequential-pipeline-{random pattern}." 

Weizman said that as part of the attack, hackers deployed a reconnaissance container that queries information about the environment, such as GPU and CPU information, as preparation for the mining activity. This also ran from a TensorFlow container.

"The attack is still active, and new Kubernetes clusters that run Kubeflow get compromised," Weizman added.

The campaign is similar to one staged in April last year. This also abused Kubernetes clusters in a crypto-mining campaign. However, instead of using Kubeflow Pipelines to deploy ML pipelines, it used a Jupyter notebook server. This campaign was the first that Microsoft observed targeting Kubeflow environments.

Weizman said that organizations running Kubeflow should ensure that the centralized dashboard isn't insecurely exposed to the internet. If Kubeflow should be exposed to the internet, make sure you use authentication. Administrators should also search containers that run TensorFlow images and inspect the entry point of those containers.

Featured Resources

Consumer choice and the payment experience

A software provider's guide to getting, growing, and keeping customers

Download now

Prevent fraud and phishing attacks with DMARC

How to use domain-based message authentication, reporting, and conformance for email security

Download now

Business in the new economy landscape

How we coped with 2020 and looking ahead to a brighter 2021

Download now

How to increase cyber resilience within your organisation

Cyber resilience for dummies

Download now

Recommended

ProtectedBy.AI’s CodeLock blocks malware at source code level
software as a service (SaaS)

ProtectedBy.AI’s CodeLock blocks malware at source code level

9 Jun 2021
Nigerian cyber criminals target Texas unemployment system
cyber security

Nigerian cyber criminals target Texas unemployment system

27 May 2021
CISOs aren’t leading by example when it comes to cyber security
cyber security

CISOs aren’t leading by example when it comes to cyber security

24 May 2021
Hackers use open source Microsoft dev platform to deliver trojans
Security

Hackers use open source Microsoft dev platform to deliver trojans

14 May 2021

Most Popular

Best paying tech jobs of 2021
Careers & training

Best paying tech jobs of 2021

7 Jun 2021
Ten-year-old iOS 4 recreated as an iPhone app
iOS

Ten-year-old iOS 4 recreated as an iPhone app

10 Jun 2021
Mythic launches power-sipping AI chip
Hardware

Mythic launches power-sipping AI chip

8 Jun 2021