Why hardware matters when it comes to moving from AI fiction to AI fact

Software has a big influence on the rapid adoption of AI models that the world is currently seeing. Indeed, cutting-edge generative AI models and AI agents are capable of processing natural language and multimodal inputs in ways that older automation software simply couldn’t.

But good software cannot function without good hardware, and this is doubly true when it comes to AI. The latest generative AI models require significant computational power for both training and inference, as well as the right networking and energy infrastructure to support a vast data center buildout.

Put simply, any business that wants to turn its AI ambitions into reality needs to ensure it’s investing in the right hardware, either up front.

This is reflected in the baked-in spending on AI hardware that major research firms expect to see in the coming years.

This is reflected in the baked-in spending on AI hardware that major research firms expect to see in the coming years.

Analyst firm Gartner predicts worldwide AI spending will hit $1.5 trillion in 2025, with almost a third of this (32.24%) earmarked for AI-optimized servers and semiconductors for AI processing.

This number may soon be considered small. McKinsey & Company research predicts that to meet AI demand, businesses will need to invest $5.2 trillion into data centers by 2030. Much of this investment (60%) will be targeted at companies that design and produce data center hardware.

“Technology developers and designers that invest in scalable, future-ready technologies supported by clear demand visibility could gain a competitive edge in AI computing,” the report states.

AI PCs are another major consideration, with these devices bringing small AI models to the edge to directly assist workers in their day-to-day tasks, boost productivity, and unlock AI-specific capabilities such as real-time protection from cyber scams.

Gartner says AI PCs will account for 31% of the market by the end of 2025 and are likely to become the most commonly used PCs by 2029. This is yet more up-front investment that IT decision makers will have to make in the interests of their workforce, but choosing the right hardware now will help companies to make the most of AI advancements over the next few years.

The challenge ahead for businesses is to establish and make use of the right hardware for their AI ambitions without compromising on scalability and performance.

The power of AI hardware

First and foremost, AI hardware must be able to offer serious computational capacity. This is most true for the training phase, in which large language models (LLMs) and other models are trained using trillions of tokens of data – but this is largely undertaken by AI developers and hyperscalers, rather than the average enterprise.

Inference, the act of actually processing inputs using LLMs and AI models in real-time, is less compute-intensive than training but still requires the best hardware to efficiently run the complex algorithmic computations inherent to AI.

This hardware is primarily realized in the form of graphics processing units (GPUs), which are prized for their powerful parallel processing capabilities and have graduated from being considered largely the stuff of gaming PCs. Leading-edge GPUs can be found in all dedicated AI data centers, as well as sites earmarked for traditional high-performance computing (HPC) workloads such as the LUMI supercomputer.

At their most powerful, data center GPUs are the brawn behind AI deployment. For example, AMD Instinct MI350 Series GPUs are used for the most demanding AI models. These flagship chips pack 256 compute units and 288GB HMB3E memory, with a peak memory bandwidth of 8 Tbits/sec to meet the needs of compute-hungry AI workloads.

That said, the CPU hasn’t gone the way of the dodo and continues to function as the ‘brain’ of the modern AI data center. CPUs and GPUs, such as those found in the AMD lineup, play a tandem role in the data center, orchestrating the workflow and leaving the GPU to handle the most intensive processing tasks.

Last, but by no means least, neural processing units (NPUs) have become a pivotal chip for AI PCs, as the dedicated hardware for background AI tasks and more efficient processing of AI features in software.

Essential investment for AI success

AI PCs are fast becoming a necessary part of business AI hardware investment strategies; as the aforementioned Gartner projections indicate, AI PCs are expected to become the norm over the next four years. With more AI features added to operating systems such as Windows 11 all the time, investment in AI PCs is becoming a non-negotiable route to future-proofing for enterprises, who otherwise risk being locked out of integrated features from which their competitors benefit.

Thankfully, there’s evidence that the up-front cost of purchasing AI PCs could pay off for businesses relatively quickly. In its AMD-sponsored report Planning for the Future: The Importance of Equipping Your Workforce with AI PCs, IDC states that AI PC investment is likely to pay itself off over the lifecycle of these devices, as businesses reap the savings of reduced reliance on cloud AI.

Networking chips and infrastructure, such as data processing units (DPUs), network switches, and even efficient cabling, can be an overlooked component of hardware for AI. But networking is absolutely critical to the smooth operation of any data center. When data is split up across multiple GPUs – as it is in a high-performance data center that may be handling many hundreds of thousands of input tokens per second – it’s vital that it travels as fast and consistently as possible.

The benefits of strong networking can be striking, even if a business has previously deployed powerful GPUs and CPUs. For example, when Oracle adopted AMD’s Pensando DPU for its cloud infrastructure, it tracked a 5x performance improvement as a direct result.

Investing in the right AI hardware now also gives businesses a competitive advantage for the future. Whether leaders opt for direct investment in local hardware for running specialized models, or the more traditional choice of renting hardware in a dedicated server to meet training and inference requirements, having a solid investment plan now allows any business to ensure it’s got the access to the hardware that will be paramount to future AI success.