Positioning itself as leaner and faster at inference workloads, Groq is hoping to outcompete Nvidia at its own game.
(Image credit: Getty Images / Bloomberg)
American AI hardware and software firm, Groq (not to be confused with Elon Musk's AI venture, Grok), has announced it's establishing its first data center in Europe as part of its efforts to compete in the rapidly expanding AI industry in the EU market, as per CNBC. It's looking to capture a sizeable portion of the inference market, leveraging its efficient Language Processing Unit (LPU), application-specific integrated circuit (ASIC) chips to offer fast, efficient inference that it claims will outcompete the GPU-driven alternatives.
"We decided about four weeks ago to build a data center in Helsinki, and we're actually unloading racks into it right now," Groq CEO Jonathan Ross said in his interview with CNBC. "We expect to be serving traffic to it by the end of this week. That's built fast, and it's a very different proposition than what you see in the rest of the market."
It's that speed and efficiency of hardware and operation that Ross believes will give Groq an edge in a market that's currently dominated by Nvidia. While the established graphics card manufacturer has cornered the market when it comes to the hardware to train AI, Ross believes that Groq is well-positioned to take over the day-to-day running of those algorithms by powering the inference calculations that allow them to function so effectively.
"Inference tends to be a higher volume, but lower margin business," he explained, suggesting Groq was happy to take that on. He even suggested Nvidia's shareholders would be happy because it would help propel the industry forward, benefiting Nvidia in the long term.
However, he pulled no punches in going after Nvidia as competition for that inference business. Although complimentary of the power and impressive capabilities of Nvidia GPUs, Ross suggested that those general-purpose chips weren't designed with AI in mind. Groq's LPUs are.
They're ASICs, which are specifically designed for AI inference calculations. They use on-chip memory to reduce the latency of tasks, and the chips are designed to handle the linear algebraic calculations that large language AI models require.
Ross claims that this makes its LPUs both fast and efficient, using around a third of the power of traditional GPU designs to produce the same output. Ross isn't just banking on the speed of his company's hardware, though; he's banking on their speed of responsiveness.
"[Nvidia CEO] Jensen said at GTC, if you want to get GPUs in two years, you need to put your [Purchase Orders] in now [...] That's just a ridiculous requirement," he said. "For us, it's about six months. Which is a fourth of the time, and that totally changes your ability to make predictions about what you need."
"Nvidia can only build as many GPUs as it's looking to build this year, because it uses very exotic components like HBM [...] We don't use any of that, and so we're not as supply limited, and that's really important for inference."
Backed by major investments from Samsung, and Cisco, Groq's new data center is being built in partnership with American data center firm, Equinix. That will help it get up and running faster and provide opportunities for further expansion. Nvidia has its own long list of AI partners in building what CEO Jensen Huang called "AI factories," all over the world, promising hundreds of billions of dollars of investment.
It feels a little like companies are taking sides, and that may be equally important from a software front as much as it is for the hardware. Nvidia has dominated much of the professional graphics card space for some time due to its CUDA software stack, which has become part of many early AI development toolkits built around its CUDA-X platform, in addition to the sheer size and scale of the overall business.
But even with Nvidia's gargantuan position in the market, Groq appears confident it can compete. With its LPUs using a much simpler design than Nvidia GPUs, and leveraging a generic compiler, Groq claims its hardware and software can be leaner, faster, and just as, if not more, capable.
Fierce competition
(Image credit: Moreh)
Ross claimed Groq was ready for its own smaller, upstart competition too. Suggesting that even as hot as AI inference industry is, Groq should come out ahead of startups beginning to build their alternative offerings.
"You'll get lots of startups popping up, but building an AI chip is expensive. You're going to (be) spending anywhere from a quarter to half a billion dollars to get that thing to market, and you can't fund everyone to do that." Ross continued.
When the interviewer pressed about staff retention, highlighting the problems OpenAI had recently faced with Meta poaching staff with large signing bonuses, Ross appeared as unconcerned as Sam Altman (despite losing researchers to Meta's efforts).
"I think we've had an easier time finding and retaining talent, because we're a little adjacent to the AI research space [...] That said, it is a hot industry, and there is a lot of pull for the best talent. In our case, I think a lot of people view us as having a very high growth trajectory to be successful, and they'd like to be on that path, so people join us very much for the equity and growth potential."
If this data center can truly start effectively serving customers within weeks of decision makers launching the endeavour, it's unlikely to be the last. However, the power and speed of adoption for Groq will remain crucial to the company's future success. Nvidia made $35.6 billion from the last quarter in data center hardware alone, while AMD made $3.7 billion. If Groq wants to be a meaningful competitor in the space, the company needs to appeal to customers and companies looking to build their own ASIC solutions for AI workloads, which is a very specific subsection of the market.
Groq would also require an answer to how scalable their hardware is too, as potential hyperscalers looking for an ASIC solution will be watching the company closely. This is in tandem with potential competition from other, more established ASIC businesses, such as Broadcom, Marvell, and Mediatek. If Groq should prove itself worthy, then it might become a part of the same cohort of indirect Nvidia competitors, looking for a slice of the extremely hot AI pie.
Follow Tom's Hardware on Google News to get our up-to-date news, analysis, and reviews in your feeds. Make sure to click the Follow button.
Jon Martindale is a contributing writer for Tom's Hardware. For the past 20 years, he's been writing about PC components, emerging technologies, and the latest software advances. His deep and broad journalistic experience gives him unique insights into the most exciting technology trends of today and tomorrow.
