Amazon cracks down on 'CPU waste' among engineers as agentic AI crunch intensifies — CPU demand makes low-utilization EC2 instances a hot commodity [Updated]

Amazon Web Services is cracking down on internal use of EC2 instances among its engineers. In May, the company reportedly met with engineers and told them to reduce CPU waste to ensure AWS has enough CPU capacity to meet customer demand, The Information reports. The message comes as demand for CPUs in the data center has hit a fever pitch, with the traditional eight-to-one or four-to-one ratio of GPUs to CPUs moving closer to parity.

EC2 instances make up a large chunk of the modern internet, and they're used in private deployments, as well. Traditionally, AWS engineers have been able to spin up their own instances for development, leveraging the relatively low CPU utilization required for web infrastructure to use more virtual machines. Now, engineers say they're waiting days to get access when they previously could get access within hours. One engineer told The Information that they've never had to wait this long for an instance, even after several years of working at Amazon.

Amazon deploys several different types of CPUs in EC2 instances, including AMD and Intel options and its relatively new Graviton5 chip. Graviton5 is Amazon's most powerful CPU to date, and it uses an Arm-based architecture along the lines of Nvidia's Vera CPU and Arm's own AGI.

The increased demand for CPU comes on the back of AI agents, a new paradigm in productivity that even companies as large as Amazon are struggling to reckon with. Last month, for instance, a coding agent blew through $1.8 million in token costs at Amazon, surpassing a development budget by 860%.

Much of the AI infrastructure currently in place is designed around inference, a workload that's accelerated by GPUs. With the four-to-one ratio, the CPU served as a way to keep the GPUs fed, and nothing more. However, agentic workloads are much more complex. They often involve tool calls that run on the CPU, as well as more complex orchestration of inference on GPUs. This is what has brought CPUs center stage in the agentic era. Intel, AMD, and others have echoed what we've heard from memory and storage companies over the past several months: the demand is so high for CPUs that most companies will take whatever they can get.

The major players are capitalizing on that demand. AMD just recently unveiled its portfolio of Zen 6 'Venice' CPUs for the data center, marking the first time AMD has launched a new architecture in the data center before the client market in decades. Nvidia has also pivoted its messaging away from accelerators and toward its new Vera CPU, vying to stake its claim in an expanding market of agentic AI infrastructure.

Although Amazon is wrestling with CPU capacity across its internal engineers and external customers, The Information reports that shortages are largely a problem for spot instances. A consultant told the outlet that contracted capacity hasn't experienced any shortages.

Following publication, an Amazon Web Services spokesperson reached out with the following statement:

"Demand for AWS services, including EC2, is incredibly strong and growing. Even with this heavy demand, we continue to satisfy the overwhelming majority of compute needs for both our internal and external customers. We work closely with internal teams to meet their compute needs while ensuring they use EC2 resources as efficiently as possible, such as reclaiming idle instances, right-sizing, and scaling up and down as needs change – just as we’ve always done. These efficiencies help manage capacity for internal and external customers alike, and any suggestions that these long-standing efforts reflect new capacity constraints is simply wrong.

This premise is sensationalized. Frugality is in our DNA since Day 1. We have always encouraged our teams to operate efficiently. We also share best practices for how customers can optimize resources to external customers. This isn’t a new directive, and encouraging efficient use of resources isn’t unique to Amazon."

We pressed Amazon on if engineers had been given deadlines to reduce their compute usage, as The Information originally reported. An AWS spokesperson responded: "This narrative on EC2 is inaccurate. As part of our normal business operations, and backed by our leadership principle of frugality, we are always driving efficiency across our resources to ensure teams are optimizing capacity."

Follow Tom's Hardware on Google News, or add us as a preferred source, to get our latest news, analysis, & reviews in your feeds.