IT Pro is supported by its audience. When you purchase through links on our site, we may earn an affiliate commission. Learn more

Crypto-mining hackers hit Kubernetes clusters

New campaign abused Kubeflow dashboards to install malicious containers

Bitcoin cryptocurrency mining

Security researchers have warned of hackers' continued attacks against Kubernetes clusters running Kubeflow machine learning (ML) instances by installing malicious containers that mine cryptocurrencies, such as Monero and Ethereum.

According to Microsoft Senior Security Researcher Yossi Weizman, the attacks began at the end of last month as he and his team discovered a spike in TensorFlow machine learning pod deployments. An investigation of the entry point of the pods revealed deployment aimed to mine cryptocurrency.

"The burst of deployments on the various clusters was simultaneous. This indicates that the attackers scanned those clusters in advance and maintained a list of potential targets, which were later attacked at the same time," said Weizman.

The hackers used two images in the attack. The first was the latest version of TensorFlow (tensorflow/tensorflow:latest), and the second was the latest version with GPU support (tensorflow/tensorflow:latest-gpu).

The images were legitimate but ran malicious crypto-mining code. The attackers abused the access to the Kubeflow centralized dashboard to create a new pipeline. Kubeflow Pipelines is a platform for deploying ML pipelines based on Argo Workflow. These dashboards were exposed to the internet instead of being only open to local access.

Hackers deployed at least two pods on each cluster: one for CPU mining, and the other for GPU mining. Both containers used open-source miners from GitHub: Ethminer in the case of the GPU container and XMRIG in the CPU one.

Related Resource

A guide to enterprise detection and response providers

The 12 providers that matter most and how they stack up

Forrester enterprise detection WPDownload now

The malicious pods all had the same pattern of name; "sequential-pipeline-{random pattern}." 

Weizman said that as part of the attack, hackers deployed a reconnaissance container that queries information about the environment, such as GPU and CPU information, as preparation for the mining activity. This also ran from a TensorFlow container.

"The attack is still active, and new Kubernetes clusters that run Kubeflow get compromised," Weizman added.

The campaign is similar to one staged in April last year. This also abused Kubernetes clusters in a crypto-mining campaign. However, instead of using Kubeflow Pipelines to deploy ML pipelines, it used a Jupyter notebook server. This campaign was the first that Microsoft observed targeting Kubeflow environments.

Weizman said that organizations running Kubeflow should ensure that the centralized dashboard isn't insecurely exposed to the internet. If Kubeflow should be exposed to the internet, make sure you use authentication. Administrators should also search containers that run TensorFlow images and inspect the entry point of those containers.

Featured Resources

ZTNA vs on-premises VPN

How ZTNA wins the network security game

Free Download

The global use of collaboration solutions in hybrid working environments

How companies manage security risks

Free Download

How to build a cyber-resilient business ready to innovate and thrive

Outperform your peers in your successful business outcomes

Free Download

Accelerating your IT transformation

How Cloudflare is innovating for CIOs to start 2023

Watch now

Recommended

GTA V vulnerability exposes PC users to partial remote code execution attacks
vulnerability

GTA V vulnerability exposes PC users to partial remote code execution attacks

23 Jan 2023
MSI to release securer BIOS settings after critical flaw discovered
vulnerability

MSI to release securer BIOS settings after critical flaw discovered

20 Jan 2023
China-backed hackers take down Amnesty International Canada for three weeks
Security

China-backed hackers take down Amnesty International Canada for three weeks

7 Dec 2022
'CryWiper' trojan disguises as ransomware, says Kaspersky
malware

'CryWiper' trojan disguises as ransomware, says Kaspersky

2 Dec 2022

Most Popular

Tech pioneers call for six-month pause of "out-of-control" AI development
artificial intelligence (AI)

Tech pioneers call for six-month pause of "out-of-control" AI development

29 Mar 2023
3CX CEO confirms supply chain malware attack
malware

3CX CEO confirms supply chain malware attack

30 Mar 2023
Getting the best value from your remote support software
Advertisement Feature

Getting the best value from your remote support software

13 Mar 2023