From SETI to CERN: What you need to know about cluster computing and how it can help your enterprise

Remember SETI@Home? The Institute for the Search for Extraterrestrial Intelligence was the custodian of gigabytes of radio telescope data (a lot back in 1999) collected from observatories around the world, and programmers hit upon a novel solution.

Rather than one huge central computer crunching all the data looking for signals from alien civilizations, they created an application users could download to their own system.

It was essentially a screensaver, kicking in after a few minutes of inactivity, and it downloaded a small chunk of information to analyze. The result was then sent back to its UC Berkeley home base, giving you a cool visualization of the data synthesis process.

At its peak, SETI@Home had around two million active users. It might be the one of the most recognizable uses of cluster computing, where a large number of nodes – often comprising low-powered or inexpensive hardware – form a network to perform data analysis tasks, little pieces of the larger dataset sent to each one.

Cluster computing is a form of high-performance computing (HPC), used to achieve the best arrangement of architectures and methodologies to scale efficiently. It can be done on individual laptops across the world or even on individual nodes throughout a single data center.

The secret sauce is in how they're put together.

How cluster computing works

Cluster computing nodes are usually regular old computers or chips containing memory, processing, storage, etc. What makes it a cluster computing environment is the middleware that analyzes and parcels out work, making sure it's done with the maximum efficiency around how many nodes there are, their processor capability, response speed etc.

Another critical factor is fault tolerance – given enough processing nodes, it doesn't matter a great deal if some go offline.

It's widely used in various domains and areas today. Scientific simulations, analysis of big data, and financial modeling are all popular cluster computing applications, and until we have a supercomputer with limitless processing and storage power, the breakneck pace of workloads needed for AI means it's fast becoming another frequent customer.

A cluster computing environment is different from a supercomputer because the latter are often designed around a tightly integrated architecture, built to enable high-powered workloads. They often have purpose-built processors like the Cray, AMD EPYC, or IBM Xeon that are built for low latency and high bandwidth.

Most cluster computing builds consist of commodity or off the shelf hardware – often no more than the processors and networking protocols that power consumer laptops or servers. They're more cost effective and more scalable to add than upgrading the intricate internal data pipelines of a supercomputer.

Similarly, a supercomputer will also have a single operating system purpose built for intensive data transport and processing. Cluster computing nodes might run on any and all possible operating systems – it's all coordinated by the cluster middleware that knows how to best communicate with them all.

Cluster computing: Give and take

Cluster computing offers a lot of advantages over other dispersed or distributed computing systems. If you have heavy data analysis needs, it's often a good first step before investing in supercomputing or grid computing.

First, it's cost effective. Because cluster computing is likely made up of the same hardware you're already buying for general data processing, it's cheaper to build – and add to. Supercomputers work on specialized and proprietary (read: expensive) components.

As your data needs scale, cluster computing is as easy as adding more nodes, so the second advantage is scalability. New processing power, memory or storage can be added to the network with a minimum of disruption because the software that integrates it is an integral part of the control environment. When you're working in cloud computing or other architectures where your needs can go up or down quickly, cluster computing can be ideal.

Also inherent in the comparative cheapness of adding nodes is built-in redundancy. If one node fails or isn't up to the task, the cluster computing software (increasingly powered by AI) shunts the process elsewhere, keeping things on track and on schedule no matter how many nodes go down or are cut off.

Even though it's often much cheaper to buy or deploy than the alternatives, that often makes cluster computing more reliable than far more complex and expensive supercomputer builds.

Cluster computing also means the system takes your large chunk of data, chops it up and sends it to every node to do their work concurrently, making it one big parallel processing system. Even with the fastest supercomputer, data has to be transported and processed one bit at a time, so with a big enough cluster, your data can be worked and returned faster than anything else around.

But cluster computing isn't for everyone. It comes with more complex management needs as you add nodes that weren't necessarily designed to be networked together. That means your middleware has to be fairly failsafe.

Even then, not all applications play nice with multi-node environments, and they might need considerable reprogramming or replacing to take maximum advantage of what cluster computing can do.

It's also very dependent on network speed and availability, which can be variable the larger or wider the network (like the public internet).

Cluster computing in action

AI Research Resource (AIRR)

The UK's AI Research Resource is a cluster computing program built to run AI and machine learning research. It provides computing environments for academia and industry users working on AI projects, with an infrastructure built specifically for deep learning, large dataset simulations and analytics including the Isambard-AI and Dawn supercomputers.

Google Compute

Google uses cluster computing to power search, Gmail and its cloud computing services. Optimised for high availability and sky-high levels of parallel processing, its services enable real-time search indexing and AI model training.

NASA Pleiades

This NASA program is a computing cluster designed to run simulations of space mission and exploration, analyse engineering problems in vehicles and machinery and model the climate. Housed at California's Ames Research Center, it's one of the world's most powerful clusters.

CERN Worldwide LHC Computing Grid

CERN, famous for HTML and the Large Hadron Collider, is home to a specialised system that organises and crunches data from the LHC. Spread over multiple locations, it's unlocking the secrets of the universe by computing all the particle physics that comes out of the world's biggest science lab.