CxOs are being barraged with constant change where AI time frames are compressed to days before there's a new development. The breakneck pace can freeze enterprise technology buyers since they can't spend on every new development, need to show returns and don’t want AI tech debt.
At AWS Summit New York, the focus was putting the fundamental approaches in place to give enterprises the structure to adopt AI agents.
Angie Ruan, CTO Capital Access Platforms division at the Nasdaq, summed up the current AI situation. "Technology used to operate over a decade. If you weren't upgrading something in five years you were behind. Later it became 18 months. Last year it was six months. Today my mindset is you have five days before you don't know what's going on and you're behind," said Ruan. "I've have never seen a pace as fast as AI."
Ruan added that there's a balancing act. "Stay calm, be strategic and be agile so you can be ready to pivot and take very practical delivery steps," she said.
Practical real-world returns for AI projects--generative, agentic and everything in between--was a recurring theme at AWS Summit New York. AWS rolled out a bevy of updates and features including Amazon Bedrock Agent Core, customizable Nova models and lots of talk about frameworks for reliability, security, observability and agility.
- AWS launches Bedrock Agent Core, custom Nova models
- Intuit starts to scale AI agents via AWS
- AWS launches Kiro, an IDE powered by AI agents
In the end, Swami Sivasubramanian, AWS VP of Agentic AI, used his keynote to return to enterprise fundamentals. Sivasubramanian's talk in New York had a lot to do with the balance of innovation and foundational approaches that can change models and underlying technologies. To AWS, a strong foundation and approach enable and accelerate innovation rather than constrain it.
For enterprises, a focus on fundamentals was long overdue. Do you really expect a business to swap out an LLM every time there’s a latest and greatest model that scores 0.06% better on math, coding or reasoning?
“AI agents are a tectonic change. They are a shift in how software is deployed and operated, and how software interacts with the world. Making that possible involves building on the foundations of today. In fact, in a world with agents, the foundation has become more important than ever. Things like security, right sizing, access, control and permissions and data foundations enable the right data to be used at the right time with the infrastructure that offers the right price performance,” said Sivasubramanian.
It's a message that enterprises were receptive to. “We're realistic about what AI can and cannot do. This isn't the silver bullet, but that's true of all AI systems. The value comes when you pair it with strong engineering practices,” said Matt Dimich, VP Platform Enablement at Thomson Reuters.
AWS is setting itself up for AI agent production systems where stability matters and models for most use cases are good enough to last a while. As agentic AI becomes more enterprise ready, basics such as identity, authentication and stability matter.
Constellation Research analyst Holger Mueller said:
“Amazon is making its step into the agent platform business with Agent Core. The good news for Amazon and its customers is that the traditional small, atomic services approach that comes from the AWS DNA, may be exactly the right thing for enterprises to build their first agents. AWS is enabling AI agents in a modular, individual and use case driven way - picking from Agent Core what they need. Adoption in the next few months will be interesting to watch. On the infrastructure side, the S3 vectors announcement is huge, as it makes digital assets stored in S3 available for AI.”
The AI ROI mismatch
Rohit Prasad, SVP and Head Scientist for AGI at Amazon, said enterprises have been struggling with an expectations gap between AI deployments and real returns.
"As exciting as AI is today, ultimately the real world is the real benchmark," said Prasad. "You hear about these models that come out every day. If you're an enterprise CIO you're thinking about the practical applications. How do I make real world applications happen at scale?"
Prasad said the focus on AGI, a topic that borders on obsession in the AI industry, is often a misdirected. "I want to level set on AGI. I think the whole conversation about who gets to AGI first or whether you can get to it is meaningless," said Prasad, noting that Amazon is chasing AGI and building out a full layer stack. "I don't think there will be a switch when we are AGI. Let's focus on whether we can make AI useful in real life. And can we make the complex simple?
AWS announced the ability to customize its Nova models for enterprise use cases. AWS will provide optimization recipes, model distillation and customization to balance cost and performance.
Prasad noted that every enterprise needs to think about AI returns in terms of workflows and processes. "It comes down to measurement. You can only improve on things you measure. Look at the success criteria for every workflow."
He added that metrics can't be stationary because your organization constantly changes. "Just go with very open eyes that in lot of applications at scale, what you measure, what you on a daily basis, also needs to evolve over certain time period," said Prasad.
In terms of AI agent value, measurement will be critical. Prasad said:
"The bar to evaluate the agent should be the same as the bar that is used to evaluate a human from a perspective of safety reliability. I think it's the same thing you want in a reliable human being. I want you to be reliable, which means it's a function of accuracy and consistency and robustness to the environment. If you want to be safe, you should have the values that you want your brand to be about, what your values to be humanity and the society is. So AI agents should be held to the same bar."
Measuring AI value
Erin Kraemer, Senior Principal Technical Product Manager at AWS Agentic AI, said that AI has the potential to fundamentally change how value is delivered.
The problem? Most companies don't properly measure AI impact. "One of the missteps that I'm seeing is how we're measuring AI impact right now and how we're talking about it," said Kraemer. "I'm not sure we're doing it the right way. Organizations that figure out how to thoughtfully apply AI to meaningful problems and measure success, are the ones that are going to adapt quickly and position themselves for the future."
Amazon's approach is to focus on controlled inputs and continual improvement to solve problems whether it's scaling infrastructure, managing product catalogs or admin tasks.
Key takeaways:
Focus on business outcome metrics over volume when it comes to AI, she said. Too much conversation about AI volume revolves around volume-based metrics, especially when it comes to code.
Kraemer said business outcomes trump volume. "I'm going to argue that, rather than volume, value should ultimately be our metric of success. So in my mind, volume, it's an output focus, and it's not even probably the right outcome."
Indeed, Kraemer said the stat that irks her is the commonplace 30% of code is written by AI. "The 30% number. I hate this number so very much. It's a fundamentally flawed number. It tells us very little about what's going on, our systems, our customers," said Kraemer.
Focus on the bottlenecks. She said enterprises need to see AI through business outcomes. Specifically, AWS looks to AI to address bottlenecks in processes. "If bottlenecks tend to be around human reasoning, there's a reasonably good chance that AI is a well-placed solution to that," said Kraemer.
Specifically, human bottlenecks have been an issue for Amazon throughout its history. She said:
"We love automation a lot. We like streamlined processes. We have some pretty massive, complex systems to handle those processes, but for a lot of our work, where we ultimately get stuck is in humans. Human reasoning capability has persistently been our bottleneck. It's not the worst bottleneck to have, but whether it's software upgrades, cleaning up catalog content defects in our shipping network, we either had to build very complicated and sometimes fragile systems, or we literally could not build systems that could scale through bottlenecks. What we're seeing with AI is a technology that's starting to blow by some of these bottlenecks."
Problem-specific metrics demonstrate real value. For code-related AI, Kraemer asked: "Are we fixing defects faster? Are we improving the security posture? Are we able to build things to delay our customers at a rate that we were never able to do before."
Amazon is looking at AI through a customer experience too. Here's a look at specific metrics AWS is using to gauge AI returns.
Software development:
- Defect resolution speed.
- Development velocity.
- Infrastructure cost savings. AWS saved "roughly $260 million in AI-assisted Java upgrades," said Kraemer.
- Developer time savings. AWS saved an estimated 4,500 developer years of effort on Java upgrades.
Customer experience:
- Catalog quality improvements.
- Contact per order decreases.
- Customer satisfaction.
Knowledge work:
- Time saved using AI to answer more than 1 million internal developer questions.
- Research time and data to decision time.
Amazon's approach to AI internally
A panel representing technology leaders from various Amazon units--Amazon Ads, Alexa, technology infrastructure and other areas--talked about AI being integrated into their products and metrics for success.
Here are a few examples:
- Amazon Connect uses genAI to enhance customer engagement and automation with data context as well as entity resolution.
- AI is generating images and video for Amazon Ads and its AI services.
- Amazon Business is using AI to automate business verification, improve accuracy and reduce manual review time. Search relevance and bulk buying reviews are also designed to improve procurement experience for Amazon Business customers.
- AWS Marketplace is using AI for seller onboarding and funding approvals and offering a comparison engine for product insights.
- Alexa is getting a rebuild for more natural interaction and agentic AI actions.
The metrics for these projects revolve around cost, friction elimination and customer experience. As you deploy these key performance indicators and metrics, keep an experiment-based mindset focusing on customer needs and iterate.
Lak Palani, Senior Manager, Product Management Tech at Amazon Business said:
"My recommendation is straightforward. Don't use AI just for the sake of using it. Find the right business cases where AI will really add value. Start small, measure results and remember it's an iterative process. Then you can scale success. Stay super focused on the business value and customer experience."
There’s a method to AWS' meat-and-potatoes focus on agentic AI and fundamentals: Enterprise adoption of AI agents will trail the technology advances and vendor marketing speak. AWS is meeting customers where they are right now.
