In Silicon Valley, where bold technical bets abound, few bets look bolder than trying to break the grip of Nvidia’s CUDA, a software stack that’s quietly become the operating system of the AI boom.
That’s what Modular, a startup founded by software gurus from Apple and Google, is trying to do.
Cofounders Chris Lattner and Tim Davis have spent decades building the software plumbing that sits beneath the modern tech industry. Lattner is famous for creating Apple’s Swift programming language. He also built the software underpinning Google’s TPU AI chips, with Modular cofounder Tim Davis.
They’re now aiming that expertise at CUDA itself. The attempt borders on madness, but it’s the kind of audacious project that could transform the AI industry.
“It’s seen by a lot of people as somewhat crazy,” said Kylan Gibbs, CEO of startup Inworld AI and a former product manager at Google DeepMind. “That’s where Chris has the advantage: He’s smart enough to actually know how to do it, and somewhat crazy enough to set out to do it.”
CUDA entrenched. Competition fragmented.
CUDA began life almost 20 years ago as a way to make graphics chips programmable. Today, it has grown into a multilayered software ecosystem — language, libraries, compilers, inference systems — that most AI companies rely on.
That success comes at a cost: Most of the industry is now optimized around a single vendor’s hardware. CUDA binds AI workloads to Nvidia GPUs. That is great for Nvidia, but deeply limiting for everyone else.
On the surface, there seems to be a ton of competition: AMD sells GPUs. Google has TPUs. Amazon created Trainium AI chips, and a host of startups are building similar hardware.
The problem is that each chip comes with its own software stack optimized just for that component. That means an endless reinvention of the wheel. Most of the time, it’s simpler to just stick with CUDA — and Nvidia’s GPUs.
And yet, AI developers crave portability: Being able to use any combination of GPUs from multiple providers without juggling different software stacks.
“Nobody is building portable stuff, because why would anyone work on software for more than one chip when the chip projects themselves are doing the software?” Lattner, Modular’s CEO, told me in an interview.
Nvidia could extend CUDA to run well on rival AI chips. But doing so would undermine Nvidia’s greatest moat: the closed-loop bond between its software and its chips. “Obviously, they don’t want portability,” he said.
Paradox = opportunity
Chloe Jackman Photography/Modular
For Lattner, this paradox presents a big opportunity.
“We realized there’s nobody in the industry that’s actually incentivized to do this. It’s very expensive, very hard,” he said. “And at the same time, everybody wants it.”
That’s what inspired Lattner and Davis to leave Google and start Modular in 2022, the year ChatGPT took the world by storm.
Since then, Modular has raised $380 million from investors including Greylock, General Catalyst, and GV, Google’s venture capital arm. The latest financing in September valued the startup at $1.6 billion. Modular isn’t the only effort to break the CUDA lock-in. There has been ZLUDA, an open-source project that was funded by AMD, and more recently, the startup Spectral Compute, which has raised $6 million.
Lattner has used some of this money to hire talented programmers from Google, Apple, and other tech companies. They spent three years working in relative obscurity to create the building blocks of a new AI software stack.
The new AI software stack
The foundation starts with a brand-new programming language, called Mojo, that offers deep controls for making AI chips run as efficiently as possible.
Modular designed this to work similarly to Python, a popular and easy-to-use programming language. But Mojo also has the speed and power of other, more complex languages, such as C++, that are essential for AI development. Mojo also works well with PyTorch, an open-source framework that is often used when building AI models and applications.
I first heard about Modular earlier this year when interviewing Carles Gelada, a former OpenAI researcher. “There are several interesting projects to create GPU-agnostic frameworks and platforms, and challenge CUDA,” he said at the time. “Mojo is the most interesting one.”
MAX is the next major layer of Modular’s new software stack. It powers AI inference, which is how models are run. This part of the system works with Nvidia GPUs, AMD GPUs, and similar chips from Apple. Modular hopes to add more AI chips in the future.
On top of that is another layer called Mammoth, which helps AI developers manage GPU clusters.
In late September, Modular announced that it got top performance out of Nvidia’s new Blackwell B200 GPUs and AMD’s latest MI355X GPUs — crucially on the same software platform.
Lattner said Modular got these AMD GPUs to perform roughly 50% better than when these chips run on AMD’s own software.
More importantly, the ability to run different GPUs on the same software stack now supports the tantalizing opportunity to compare Nvidia’s offerings with rival AI chips on a more level playing field.
“The obvious question is: can MI355X compete with Blackwell?” Modular wrote in a blog announcing the results. “Early signs point to yes.”
A customer test
Inworld
Gibbs, the CEO of Inworld AI, has been putting Modular’s software through its paces in the real world.
Inworld builds high-speed, real-time conversational AI technology that supports offerings from big companies, including Disney, NBCUniversal, and Niantic Labs.
Earlier this year, when the startup designed a new text-to-speech AI model and got early access to Nvidia B200 GPUs, they issued Modular a challenge: Cut our costs by 60% and reduce our latency by 40% and we’ll work with you.
“Within about four weeks, we were able to get this incredible performance,” said Gibbs, who signed a partnership deal with Modular soon after. “I’ve bet with my wallet.”
While Inworld was mostly lured by Modular’s performance gains on Nvidia’s latest GPUs, Gibbs likes the flexibility of using different AI chips more easily in the future, if needed.
“The promise is that we’d be able to move to new hardware,” he said. “Let’s say AMD takes off, let’s say TPUs take off for Google, or there could be other new hardware that comes online. So it’s nice to have that optionality.”
‘Nvidia doesn’t have to die’
In fact, Google’s TPUs are suddenly having a moment. The internet giant released a new AI model called Gemini 3 to rave reviews recently. That was trained and run using TPUs, and some other AI companies have signed deals to use these chips instead of, or alongside of, Nvidia GPUs.
That’s put Nvidia on the defensive. A project like Modular, with its promise of portability across different AI hardware, adds to this pressure.
“Nvidia could kill this in a day,” said Gibbs of the Modular project. “Nvidia could basically say, ‘okay, we don’t really care that you run just on Nvidia hardware. Here’s a CUDA option that runs on AMD GPUs as well.’ It’d be a bit crazy for them to do that, but it’s something they could do and that would of course be somewhat bad.”
For all Lattner’s critique of the industry, he says Modular is not trying to kill Nvidia. In fact, he argues that Nvidia will continue to thrive, even if Modular succeeds spectacularly.
“We’re trying to build something like Android, but for AI hardware,” he told me, referring to Google’s mobile operating system that powers most of the world’s smartphones.
Despite billions of people using Android devices, this success didn’t kill iOS, Apple’s mobile operating system. iPhones still rule in the US, for example.
Lattner thinks something similar will happen in AI as Modular’s software makes other hardware more competitive, giving developers more freedom, and chipping away at the industry’s single-vendor monoculture.
“So Nvidia doesn’t have to die, but we do want more competition. We do want more innovation,” he said. “I think that’s good for the world.”
Sign up for BI’s Tech Memo newsletter here. Reach out to me via email at abarr@businessinsider.com.
