Elon Musk’s xAI allegedly uses 'illegal' generators to power Colossus supercomputer facility

Four banks of xAI's HGX H100 server racks, holding eight servers each.

(Image credit: ServeTheHome)

Last year it turned out that Elon Musk’s xAI had to install additional ‘portable’ generators near its facility adjacent to Memphis, Tennessee, to power the Colossus supercomputer with over 100,000 Nvidia H100 GPUs as local power grid could not support the load. The Southern Environmental Law Center contends the generators are "illegal," yet they can keep running, reports The Guardian.

xAI’s Colossus supercomputer consumes about 150 MW of power when equipped with 100,000 H100 GPUs. More when it is upgraded to 200,000 processors. However, last July the site near Memphis, Tennessee, could only provide 8 MW of power to the facility, so Musk’s company had to install gas fueled turbines to power the colossal supercomputer.

However, the company had only sought permits for 15 units, but aerial images now show double that number in operation. Apparently, 20 gas generators are now running without the required approvals, raising alarm from environmental groups and nearby residents, according to The Guardian who refer to KeShaun Pearson, director of the advocacy group Memphis Community Against Pollution.
Pearson stated to the Shelby count board of commissioners (Memphis, Tennessee) that xAI is using 35 methane gas burning turbines while only having a permit for 15.

The company seems to be using a legal loophole that allows turbines to run without permits if they are not fixed in one location for more than 364 days. In January, xAI filed for approval of 15 turbines, but the other 20 were not included and still have no legal clearance. However, they can keep running for months before xAI gets formal approval, if local authorities actually approve them.

The Southern Environmental Law Center revealed images and calculations showing that the turbines can produce 420 MW, an amount sufficient to power a city. According to senior attorney Amanda Garcia, xAI has essentially created an unregulated power facility in a residential area, posing significant health and environmental risks.

The law center stated in a letter that these generators are a major pollution source and breach federal air quality rules, including emissions of hazardous and cancer-causing substances. They demanded that the local health agency issue an emergency halt to the operations and fine the company $25,000 for every day it continues to run them without proper authorization.

That's a rather laughable fine, frankly. The 100,000 H100 GPUs in the xAI Colossus would cost about $2.5 billion on their own — never mind the rest of the data center infrastructure and hardware. $25,000 per day would amount to just $9.1 million per year. Providing 150MW of electricity, 24/7, on the other hand, even at a price of $0.05 per kWh (we're not sure what xAI pays to run the portable generators) would be about $180,000 per day.

Small wonder that "a company representative scheduled to address the county board did not attend the meeting," according to The Guardian.

Anton Shilov is a contributing writer at Tom’s Hardware. Over the past couple of decades, he has covered everything from CPUs and GPUs to supercomputers and from modern process technologies and latest fab tools to high-tech industry trends.

24 Comments Comment from the forums

drea.drechsler

So the generators are illegal... except they're legal? Sounds like a non-story until the 365 day period expires.

I also really think it's a weak argument that the generators are any more an environmental concern than the generators the local utilities would use to generate power. Especially since these generators use methane (according to the article). Unlike the LNG (liquefied natural gas) typically used by utilities to fuel their generators methane can be readily derived from carbon-neutral sources (land fill and sewage treatment waste gases, for instance) leaving the only other emission of significance H2O, or water.

Rather pointedly, the story misses it's mark if it doesn't follow the trail to find out where the methane is sourced, and how it's produced. What is interesting is that Space X reportedly plans to produce the methane its launchers use in quantity using something similar to the Sebatier process from atmospheric CO2. So even if they currently source their methane from coal or LNG, that they are using a methane burning generator (instead of more common LNG) suggests a plan to pivot to carbon neutral sources for its production in the future.

This could have been a much more interesting story, instead it looks more like another effort to burn Elon Musk with half-truths and misinformation.
Reply
Pumapaw

Global warming has been disproved. Climate change is what this planet does and now we are tending to be cooling. We need more nuclear reactors for dependable power. Also hydrogen power would be a good thing. Waste product of burning it is water.
Reply
Notton

If the punishment for a crime is a fine, then the law only exists for poor people.
Reply
King_V

Pumapaw said:
Global warming has been disproved. Climate change is what this planet does and now we are tending to be cooling. We need more nuclear reactors for dependable power. Also hydrogen power would be a good thing. Waste product of burning it is water.
Might need to put a /s at the end there. People might think you're serious.
Reply
nrdwka

gaz turbines are pretty environment-friendly compared to fuel/charcoal based 🤔
Reply
Pumapaw

King_V said:
Might need to put a /s at the end there. People might think you're serious.
The byproduct of burning hydrogen gas is pure water. This makes hydrogen a clean fuel option, as it does not produce harmful emissions like carbon dioxide. Look it up.
Reply
EzzyB

drea.drechsler said:
So the generators are illegal... except they're legal? Sounds like a non-story until the 365 day period expires.

I also really think it's a weak argument that the generators are any more an environmental concern than the generators the local utilities would use to generate power. Especially since these generators use methane (according to the article). Unlike the LNG (liquefied natural gas) typically used by utilities to fuel their generators methane can be readily derived from carbon-neutral sources (land fill and sewage treatment waste gases, for instance) leaving the only other emission of significance H2O, or water.

Rather pointedly, the story misses it's mark if it doesn't follow the trail to find out where the methane is sourced, and how it's produced. What is interesting is that Space X reportedly plans to produce the methane its launchers use in quantity using something similar to the Sebatier process from atmospheric CO2. So even if they currently source their methane from coal or LNG, that they are using a methane burning generator (instead of LNG directly) suggests a plan to pivot to carbon neutral sources for its production in the future.

This could have been a much more interesting story, instead it looks more like just another back-door attempt to burn Elon Musk with half-truths and misinformation.
I think you're missing the point. The utilities are regulated. There are laws in place controlling them and they are a known quantity.

This is more like someone came in a built a good-sized power plant in your neighborhood without regard to environmental concerns. You wouldn't stand for it, at all. 420MW is absolutely no joke. As the article says you can power a city with that kind of power.

This is the kind of stupidity that AI is causing. The "winner" of this AI "war" that we're seeing, if there is one, gets the prize. Musk, famously, wants an AI that's "non-woke", Alibaba wants one that's answers are "properly socialist." Are you following? An AI is going to reflect the politics and views of it's creator so the winner gets to control the narrative. These huge companies and individuals like Musk believe that AI will become a standard way to get information and they want to be the one that provides the answers that slant towards their views. To them, that's worth literally anything.
Reply
Notton

Pumapaw said:
The byproduct of burning hydrogen gas is pure water. This makes hydrogen a clean fuel option, as it does not produce harmful emissions like carbon dioxide. Look it up.
and how do you get said hydrogen?
Reply
drea.drechsler

EzzyB said:
I think you're missing the point. The utilities are regulated. There are laws in place controlling them and they are a known quantity.

This is more like someone came in a built a good-sized power plant in your neighborhood without regard to environmental concerns. You wouldn't stand for it, at all. 420MW is absolutely no joke. As the article says you can power a city with that kind of power.

This is the kind of stupidity that AI is causing. The "winner" of this AI "war" that we're seeing, if there is one, gets the prize. Musk, famously, wants an AI that's "non-woke", Alibaba wants one that's answers are "properly socialist." Are you following? An AI is going to reflect the politics and views of it's creator so the winner gets to control the narrative. These huge companies and individuals like Musk believe that AI will become a standard way to get information and they want to be the one that provides the answers that slant towards their views. To them, that's worth literally anything.
Plenty of good reasons to be concerned with whatever it is we might get from "AI". But that's not really what this was about. If the authors wanted to extend this article to include questions surrounding whether it's good for society to be expending these sorts of resources on questionable benefits of AI, that would have been great. But it didn't.

The facility has a period of time they can use this, the law was constructed to allow for it. And there are good, common sense reasons, for it to. What would have been helpful is to contact the facility and get some comments to questions like "what are your plans to get permitting straightened out".

I have a 300MW generating plant about 2 miles away from my house. It's a couple of LNG fired gas turbine generators nestled comfortably in forest clearing. It's what's called a "surge generator" station, for peak power demand periods, usually in summer time when everybody's using their AC at full blast (a large, fairly dense, suburban area). They are very effectively muffled, we can't tell whether it's running or not from our house. I can drive by it from the road and hear it, about same sound level there as freeway traffic from 2 miles away which I hear clearly in mornings during rush hours.
Reply
SomeoneElse23

King_V said:
Might need to put a /s at the end there. People might think you're serious.
He is serious. And he's correct.

That's why the manipulative slogan changed from "global warming" to "climate change".

There's always climate change so they can always use it to push their agenda.
Reply

Show more comments

Stay On the Cutting Edge: Get the Tom's Hardware Newsletter