Amazon is counting on in-house AI chips to fuel its next phase of profitable growth. A recent internal document shows the cloud giant is still playing catch-up to Nvidia’s dominant GPUs.
AI startup Cohere found that Amazon’s Trainium 1 and 2 chips were “underperforming” Nvidia’s H100 GPUs, according to an internal “confidential” Amazon document from July, obtained by Business Insider. Cohere reported that access to Trainium 2 was “extremely limited” and plagued by frequent service disruptions, the document also noted.
The “performance challenges” with Cohere were still under investigation by Amazon and its chip group Annapurna Labs, but progress on these issues was “limited,” the official document stated.
Stability AI, a well-known startup that generates AI images, had similar concerns. It concluded that Amazon’s Trainium 2 chips underperformed Nvidia’s H100 GPUs on latency, making them “less competitive” in terms of speed and cost, the document also warned.
An essential ingredient
Amazon’s homegrown Trainium chips are an essential part of the company’s effort to compete in the AI-cloud race. The early success of Amazon Web Services, especially its profitability, was based on designing its own data-center chips, rather than paying Intel for these pricy components. In the new generative-AI era, Amazon is trying to avoid paying for expensive Nvidia GPUs, while still providing cloud customers with powerful AI services.
If some AWS customers don’t want Trainium, and insist that AWS run their AI cloud workloads using Nvidia gear, that could undermine Amazon’s future cloud profits because it will be stuck paying more for GPUs.
The customer complaints highlighted internally by Amazon reveal the steep challenge it faces in matching Nvidia’s performance and getting profitable AI workloads running on AWS. This also underscores AWS’s ongoing challenges among startup customers, a segment that has long been its core market.
Feedback and Trainium 3
An Amazon spokesperson said the company is “grateful” for customer feedback that helps make its chips “even better and more widely used.”
The Cohere case is “not current,” the spokesperson added, while noting that Trainium and its other in-house AI chip, Inferentia, “have achieved great results” with customers including Ricoh, Datadog, and Metagenomi.
“We’re very pleased with the growth and adoption we’re seeing for Trainium 2, which at this stage is primarily used by a small number of very large customers like Anthropic,” this spokesperson wrote in an email to Business Insider.
Related stories
Business Insider tells the innovative stories you want to know
Business Insider tells the innovative stories you want to know
AWS claims its in-house AI chips offer 30% to 40% better price performance than the current generation of GPUs. The company has incredible chip-design talent and is working on new generations of these crucial components.
“We expect to accommodate more customers starting with Trainium 3, previewing later this year,” the spokesperson said. “We’ll accomplish that as we always do by listening to our customers and remaining vocally self-critical with each other as we continue innovating to give more customers access to chips with the best possible price and performance.”
Last week, Amazon CEO Andy Jassy said during the company’s earnings call that Trainium 2 chips are “fully subscribed” and are now a “multibillion-dollar” business. Spokespeople for Cohere and Stability AI didn’t respond to requests for comment.
Long-standing complaints
Other AWS customers have also complained about Amazon’s AI chips.
According to the July document, a startup called Typhoon found Nvidia’s older A100 GPUs to be as much as three times more cost-efficient than AWS’s Inferentia 2 chips for certain workloads.
Similarly, a research group called AI Singapore determined that AWS’s G6 servers, equipped with Nvidia GPUs, offered better cost performance than Inferentia 2 across multiple use cases. (Inferentia chips are used for running AI models, a process known as inference, while Trainium chips focus on training models).
Last year, Amazon cloud clients also cited “challenges adopting” its custom AI chips, creating “friction points” and contributing to low usage, Business Insider previously reported.
These struggles are reflected in market share. Nvidia dominates AI chips with more than 78% of the market. That’s followed by Google and AMD, each with just over 4%, according to research firm Omdia. AWS’s chips ranked sixth, with 2% of the market.
Amazon
No Trainium for OpenAI
A new $38 billion partnership between AWS and OpenAI illustrates Amazon’s challenges here. The deal involves AI cloud servers that only contain Nvidia GPUs, with no mention of Trainium processors.
The lack of Trainium in the OpenAI deal “could be viewed as disappointing,” Mizuho analysts wrote after the deal was announced this week.
Perhaps even more damning, these analysts wrote that it was “logical” for OpenAI to start with Nvidia GPUs.
Nvidia’s chips not only deliver superior performance but also come with a widely adopted platform, CUDA, that many developers already know and use. That familiarity is especially valuable when teams are building large, high-risk AI projects, where reliability and existing expertise can make a crucial difference.
In the July document, Amazon employees noted that technical limitations and other comparative issues between its custom AI chips and Nvidia’s GPUs have become “critical blockers” for customers thinking about switching to AWS chips.
Bank of America analysts were cautious about Tranium’s progress last month. In a note published in late October, they warned that investors have been “skeptical” about Trainium’s capabilities, and it was “unclear” whether strong demand will materialize “outside of Anthropic.”
Anthropic and Project Rainier
Trainium’s most high-profile customer is Anthropic, the AI startup behind the powerful Claude models. AWS recently rolled out Project Rainier, a giant data center project that includes a cluster of half a million Trainium chips that will be exclusively used to train Anthropic’s next-generation AI model. Anthropic is expected to deploy more than 1 million Trainium 2 chips by the end of the year.
Anthropic is one of the world’s leading AI labs, regularly challenging OpenAI with state of the art models. If Anthropic can make Trainium chips work, that would be a huge boost to Amazon’s efforts here. But the jury is still out on this.
Some investors were caught off guard last month by Anthropic’s move to broaden its partnership with Google, which provides its own in-house AI chips known as TPUs. Amazon’s stock slipped following this news, although Anthropic emphasized that it will continue using Trainium.
Anthropic has also publicly acknowledged the complexity of using multiple chip architectures, detailing related outages in a September blog post.
Amazon’s spokesperson told Business Insider that Anthropic is continuing to expand its use of Trainium chips and emphasized the company’s commitment to offering customers a range of hardware options across its cloud services.
During an earnings call with analysts last week, Jassy underscored AWS’s focus on offering “multiple chip options.” The goal is not to replace Nvidia, but to give customers more choice, a strategy AWS follows in other parts of cloud computing, he said.
“In the history of AWS, it’s never just one player that over a long period of time has the entire market segment and then it can satisfy everybody’s needs on every dimension,” Jassy said.
Amazon shares surged the next trading day, after it reported AWS revenue grew 20% to $33 billion last quarter, the fastest pace of growth since 2022. That’s still slower growth than rivals such as Microsoft and Google Cloud, on a percentage basis.
Have a tip? Contact this reporter via email at ekim@businessinsider.com or Signal, Telegram, or WhatsApp at 650-942-3061. Use a personal email address, a nonwork WiFi network, and a nonwork device; here’s our guide to sharing information securely.
