What’s needed in computing infrastructure for the age of AI

The explosive growth of AI workloads – from training to inference and data processing – is forcing IT leaders to confront two competing imperatives: delivering increased performance while maintaining control over costs and energy consumption. This reality is shifting infrastructure discussions beyond raw compute power to focus on performance per watt (PPW), compute density, and total cost of ownership (TCO).

Hyperscaler cloud providers like Amazon Web Services (AWS), Google Cloud, and Microsoft Azure recognize the need for a modern, efficient foundation for the future of cloud compute and are turning to Arm. In fact, Arm estimates that nearly 50% of the compute shipped to top hyperscalers this year will be based on Arm.

The cloud's elasticity allows enterprises to scale resources up or down based on demand, while enterprise-grade security controls do not require in-house expertise. This approach also eliminates the need for capital expenditure on hardware, as companies can purchase computing power as a service, aligning costs with actual usage.

"Performance per watt isn’t just a measure of efficiency - it's a result of smarter system design. The Arm architecture enables hyperscalers to fine-tune compute, memory, and acceleration for every workload. That system-level elasticity is what defines efficiency in the AI era," said Bhumik Patel, director of software ecosystem, Arm.

The value of Arm

Arm’s Neoverse platform, offering 64-bit Arm CPU cores in various compute subsystem configurations, is designed to power next-generation cloud environments, supporting a range of cloud-native workloads. Arm Neoverse combines advanced compute density, performance, and efficiency to meet the demands of modern infrastructure.

Some highlights to note:

• Arm-based cloud instances deliver, on average, 49.2 percent lower cost per core compared to x86, according to data from Liftr.
• Recently announced Google Axion N4A instances deliver up to 2x better price/performance and 80% better performance-per-watt than comparable current-generation x86-based VMs.
• Arm Neoverse-based Microsoft Cobalt 100 processors and ONNX Runtime optimizations have expanded AI inference performance for large language models (LLMs), delivering up to 1.9 times higher performance and 2.8 times better price/performance than x86-based instances.

In fact, according to Signal65 benchmarking results, Arm Neoverse-powered AWS Graviton4 chips are outpacing comparable x86 offerings from AMD and Intel on overall performance and price/performance across four key enterprise workloads: LLM inference, ML training and inference, database operations, and web serving.

In terms of performance, Signal65 benchmarking also revealed that Arm-based instances delivered up to 168% higher performance for AI workloads and up to 93% higher performance for general compute workloads.

This performance advantage also translated into significant cost-efficiency gains. For example, in Llama 3.1 8B inferencing, Arm-based instances achieved up to 220% better price/performance, and in Redis database testing, up to 149% better prices/performance compared to x86-based instances

The enterprise migration

Software innovation is thriving on Arm platforms as developers leverage general-purpose chipsets to optimize applications for efficiency and performance. Digital music service Spotify, for instance, migrated its cloud infrastructure to Google Axion, which is based on Arm Neoverse cores, in an effort to boost performance, reduce costs, and improve sustainability. The effort focused on services, including Java applications, machine learning pipelines, and data analytics.

Spotify's migration resulted in a 40% savings in compute costs and a 2.5x boost in performance. The effort also enabled engineers to spend more time improving data quality, rather than on operational tasks and issues.

In a similar move, Pinterest enhanced the efficiency and scalability of its recommendation engine by using CPUs based on the Arm Neoverse platform. This trend is also evident among major data platform providers, with companies such as Oracle transitioning their services to Arm-based infrastructure — a process now accelerated by Arm Cloud Migration, driving faster adoption.

A path to Arm for IT pros

For IT pros considering the shift to Arm-based infrastructure, the following step-by-step approach is recommended:

• Assessment phase: Identify workloads best suited to Arm (scale-out services, cloud-native, AI inference, databases, analytics, etc).
• Pilot deployments: Start with non-critical workloads in Arm-based cloud instances. Check the performance of AWS Graviton, Google Axion, and Microsoft Cobalt-powered VMs to measure real-world savings.
• Tools and partners: Leverage migration tools and frameworks from Arm and cloud providers, independent software vendors, and specialist service providers for code refactoring and workload optimization.
• Scale up: Move critical applications once benchmarks confirm TCO advantages, ensuring compatibility with enterprise standards and compliance frameworks.

Conclusion

For decision-makers at cloud-forward enterprises and software-as-a-service companies, the question is no longer if, but when, to migrate to Arm. Hyperscaler cloud providers are deploying, software vendors are optimizing, and benchmarks are proving both efficiency and performance of Arm-based cloud infrastructure. IT pros who opt to migrate can lower costs and future-proof their infrastructure for AI at scale.