Modular Raises $100M to Make AI Inference More Efficient

Lower cost and lower latency are key growth drivers

Modular has announced a new funding round of $100 million to fuel its products that promise to reduce the cost and latency of operating generative AI models. The investment was “led by General Catalyst and filled by existing investors GV (Google Ventures), SV Angel, Greylock, and Factory,” according to the announcement.

Modular announced a $30 million seed funding in June 2022, and The Information reported in August 2023 that the company was seeking new funding at a $600 million valuation. No valuation information was shared as part of the funding announcement.

From Training to Inference

Generative AI through 2022 and even for much of this year has focused on establishing foundation model performance. This meant that model training took center stage, and inference (running models to support production use cases) was a supporting cast member. However, that is changing quickly.

The foundation model creators, software providers, and enterprises are shifting to a focus on the production deployment of generative AI applications in order to capture the promised benefits. Modular was designed to facilitate that shift to production.

Solving the Next Problem

Sometimes, companies deliver the right product at the right time. A WIRED story recently talked about AI computing scarcity and referenced a comment by a Modular cofounder:

Everywhere, engineering terms like “optimization” and “smaller model size” are in vogue as companies try to cut their GPU needs, and investors this year have bet hundreds of millions of dollars on startups whose software helps companies make do with the GPUs they’ve got. One of those startups, Modular, has received inquiries from over 30,000 potential customers since launching in May, according to its cofounder and president, Tim Davis.

…

“We live in a capacity-constrained world where we have to use creativity to wedge things together, mix things together, and balance things out,” says Ben Van Roo, CEO of AI-based business writing aid Yurts.

Modular expanded on these comments in its funding announcement:

In the four months since our product keynote, the Modular community has grown to more than 120K+ developers (including tens of thousands of enterprises) who are excited to utilize our AI infrastructure stack. We have built and delivered the world's fastest unified AI engine that provides full compatibility with major AI frameworks – such as TensorFlow and PyTorch – and delivers unparalleled performance and cost savings on today’s CPUs, with support for GPUs coming in the Fall. We also built Mojo 🔥, a programming language for all AI developers that is 35,000x faster than Python.

To put that in perspective, Modular received 30,000 inquiries before delivering a product for GPU-based inference. There are 120k developers using it for CPU-based AI application deployment because optimizing for cost savings and latency is a high priority. This appears to be an even higher priority for GPU-based generative AI solutions.

Follow the Tools!

Modular cofounders Chris Lattner and Tim Davis have a lot of experience scaling AI-based solutions. Their work at Google, Apple, and Tesla is noteworthy, but their timing might be even better.

Generative AI’s shift from an era of research and development to production and commercialization has led to a new focus on tools, frameworks, and other capabilities. Modular may be able to help mitigate the scarcity problem for GPU processing when its new product arrives and can help today for CPU-based workloads. Beyond this, every application team deploying AI models can benefit from tools that simplify the process and create architectural flexibility.

Weights & Biases, Hugging Face, CalypsoAI, and other recent funding rounds are all about enabling the adoption of generative AI by removing cost, risk, or other barriers. Modular is the continuation of a trend that will grow throughout the rest of 2023 and into 2024.