GPU instances are taking a larger share of cloud enterprise spending and now are 14% of compute costs compared to 10% a year ago, according to a Datadog report analyzing AWS customer usage.
The report highlights how enterprises are experimenting with training and inference for large language models. A report from Flexera also highlighted how enterprises were experimenting with AI workloads. Datadog said:
"GPU-based EC2 instance types generally cost more than instances that don’t use GPUs. But the most widely used type—the G4dn, used by 74 percent of GPU adopters—is also the least expensive. This suggests that many customers are experimenting with AI, applying the G4dn to their early efforts in adaptive AI, machine learning (ML) inference, and small-scale training. We expect that as these organizations expand their AI activities and move them into production, they will be spending a larger proportion of their cloud compute budget on GPU."
That increased spending is good for Nvidia as well as AWS customers using the cloud vendor's Trainium and Inferentia chips. The focus on GPU instances may also benefit AMD, which is rolling out its new accelerators.
Arm appears to be another downstream winner as cloud workloads are GPU based. Arm-based CPUs are also popular on AWS as enterprises leverage Graviton2 processors. Arm-based instances only account for 18% of EC2 compute costs, but that's double from a year ago.
Arm's data center takeover: A lumpy revolution | Arm launches compute subsystems optimized for AI for edge devices | Nvidia outlines roadmap including Rubin GPU platform, new Arm-based CPU Vera
Datadog noted:
"Arm-based instances still account for only a minority of EC2 compute spending, but the increase we’ve seen over the last year has been steady and sustained. This looks to us as if organizations are beginning to update their applications and take advantage of more efficient processors to slow the growth of their compute spend overall."
Overall, enterprises are mixing various compute instances and containers to optimize costs, but companies aren't adopting the latest technologies. Datadog found that 83% of organizations are still using previous-generation EC2 instance types.
More on genAI dynamics:
- AI infrastructure is the new innovation hotbed with smartphone-like release cadence
- Don't forget the non-technical, human costs to generative AI projects
- GenAI boom eludes enterprise software...for now
- The real reason Windows AI PCs will be interesting
- Copilot, genAI agent implementations are about to get complicated
- Generative AI spending will move beyond the IT budget
- Enterprises Must Now Cultivate a Capable and Diverse AI Model Garden
- Secrets to a Successful AI Strategy
- Return on Transformation Investments (RTI)
- Financial services firms see genAI use cases leading to efficiency boom
