People seem all too ready to anthropomorphize LLMs and AI chatbots like ChatGPT, Claude, and Gemini. Some humans even admit to ‘relationships’ with one or more of the various examples of machine intelligence. To illustrate how flawed this instinct could be, a Microsoft AI researcher built a tiny neural network inside Age of Empires II using goats, grass, and bridges. Adrian de Wynter shared his work in a paper dubbed If LLMs have human-like attributes, then so does Age of Empires II. The Microsoft researcher, based at the University of York, also talked to 404 Media recently about how he likes to turn absurdism up to 11 to make a point.

In the research paper, De Wynter doesn’t make the argument that LLMs do or do not actually have generalized anthropomorphic attributes. Instead, he illustrates that the AoEII goats can also power the kinds of models that lay behind today's most popular chatbots. That hammers home the argument that “in no case is a machine’s activity to be interpreted in terms of higher cognitive processes, if it can be fairly interpreted in terms of processes which stand lower in the scale of cognitive evolution and development.”

De Wynter also raises the well-known concept of confirmation bias. Those looking for human traits in tech like chatbots will tend to find them, he proposes. However, the big contrast between the absurdist goat example and the commercial LLM chatbot is the way people interact with them, the interface that makes the likes of Claude ‘conversation friendly.’ De Wynter’s research indicates that anthropomizing LLMs is a common trend in computer science papers. From 337 such papers De Wynter looked at, published in the last two years, he says that 57% assumed that LLMs could have human-like traits. This basic assumption could color the research, testing, and, of course, conclusions of these papers.

So, how did the Microsoft AI researcher build the goaty AoEII LLM? Well, he didn’t quite go as far as developing a full-blown LLM. Instead, De Wynter thought it sufficient to use AoEII’s scenario editor to build a working NAND gate, with a 1-bit perceptron, where the goats act as bits. This crude perceptron and the circuit to train it in-game are enough to demonstrate that the simplest building block of a modern neural network could be made this way. And if you think it is absurd that AoEII goats can embody consciousness, then it should be equally absurd to regard any of the well-known chatbots as anything more.

Companies behind the AI boom aren’t discouraging people from anthropomorphizing their wares. In many ways, they might benefit from these human perceptions. Chatbots they deploy are trained with natural language and use techniques to mimic the shape and tone of natural conversation. This makes it easy for users to project personality, emotion, or even consciousness onto them. Top AI company execs have leaned into the perception of their customers, publicly entertaining the idea that their systems could or might be exhibiting signs of consciousness. In his 404 Media interview, De Wynter also highlighted research indicating that people buy more products when they can empathize with them, and that includes AI/chatbot/LLM subscriptions.

A computer vision and robotics expert has created and trained what he boasts is “the ultimate mosquito killer.” Steven Cheng shared details of his high-tech bug zapper project on social media. Key innovations here include the use of computer vision and deep learning technologies to detect and lock onto mosquitoes so the laser ‘artillery cannon’ could do its work.

For scanning the environment, Cheng concluded that a DSLR paired with a high-magnification zoom lens was the best option. This was also used in the training stage to build up a large dataset of mosquito images. A side effect of ‘welcoming’ mosquitoes in for photographs at this stage of the project was “countless mosquito bites all over my body,” recalled the intrepid technologist.

With the image database built and annotated, Cheng moved on to leveraging deep learning techniques. This task “really put my graphics card through its paces,” he commented. However, the detection performance of the resulting model was “quite good” by the end of this process.

A laser source was tuned to “instantly turn mosquitoes into roasted ones.” Then, the equipment was mounted on a high-precision industrial rotary stage / gimbal to complete the ‘ultimate mosquito killer’ apparatus.

Simulations were run, and Cheng decided to add a wide-angle camera to the setup. The purpose of the second camera with a wider view was to detect humans and flammable materials in the house. Logic was implemented where, if there was any overlap between humans or flammable materials and the laser target, no power would be fed to the cannon.

Overall, Cheng was pleased with the results and says all the mosquitos in his residence were “successfully eliminated” after a night’s effort.

Rival Indiegogo product design ships in June

If the above story sounds familiar, that’s because it isn’t the first laser-toting mosquito killer for the home that we’ve reported on. Last year we highlighted the Photonmatrix Indiegogo project, which sought funding for an all-in-one portable laser-driven mosquito-killing machine costing as little as $498.

The Photonmatrix was claimed to leverage a LiDAR scanner combined with a galvanometer-directed laser to seek and destroy mosquitoes at a rate of up to 30 pests per second. However, its detection method doesn’t sound as impressive as the machine-learning tech used by Cheng’s new device.

Hopefully we will see consumer reviews of the Photonmatrix soon, though, as it is due to ship very shortly. Backers will receive devices starting from “June 2026,” according to the latest information.

A team at Tohoku University and Future University Hakodate in Japan trained cultured rat cortical neurons to autonomously generate complex temporal signals using a real-time machine learning framework, according to a study published March 12 in the journal Proceedings of the National Academy of Sciences. The researchers integrated the living neurons with high-density microelectrode arrays and microfluidic devices, creating a closed-loop reservoir computing system that learned to produce periodic and chaotic waveforms without any external input.

The system recorded spike trains from the neurons across a 26,400-electrode array with a 17.5-micrometer pitch, filtered them into continuous signals, and decoded an output through a linear readout layer. That output was then fed back to the neurons as electrical stimulation, completing a feedback loop that cycled roughly every 333 milliseconds. The readout weights were optimized in real time using an algorithm called FORCE (First-Order Reduced and Controlled Error) learning, which continuously adjusted the decoder to minimize the error between the network's output and a target waveform.

The enabling technology, per the researchers, was the use of PDMS microfluidic films to constrain how the neurons connected. Without physical constraints, cultured neurons form dense, highly synchronized networks that fire in lockstep, and these homogeneous networks failed to learn any of the target signals.

Instead, the researchers confined neuronal cell bodies to 128 square wells, each roughly 100x100 micrometers, with each well holding an average of 14.6 neurons. The wells were linked by microchannels in two configurations: a lattice design with uniform nearest-neighbor connections, and a hierarchical design with sparser, multi-scale connections.

Both patterned configurations dramatically reduced pairwise neural correlations compared to unpatterned cultures (0.11 and 0.12 versus 0.45, respectively), increasing the dimensionality of the network's dynamics. Lattice networks consistently outperformed hierarchical ones across all target waveforms, likely because their denser intermodular connections produced higher firing rates that gave the linear decoder more signal to work with.

Tests showed rat brain neurons are 'novel computational resources'

Using the lattice and hierarchical networks, the system learned to generate sine waves with periods of 4, 10, and 30 seconds, as well as triangle and square waves, and the same culture preparation could be retrained to oscillate at different frequencies. The researchers also demonstrated that the system could approximate a Lorenz attractor, a three-dimensional chaotic trajectory, with pairwise correlations above 0.8 between predicted and target signals across all dimensions during the learning phase.

"This work shows that living neuronal networks are not only biologically meaningful systems but may also serve as novel computational resources," said Hideaki Yamamoto, a professor at Tohoku University's Research Institute of Electrical Communication, in a press release published on the institution’s website.

Performance degraded after training was halted and the system ran autonomously, with mean squared error increasing in 99% of trials. The feedback loop's roughly 330-millisecond latency also limited the system's ability to track fast-changing or sharp-edged waveforms. The researchers noted that reducing this delay through specialized hardware or alternative filtering could expand the range of learnable targets, with future applications potentially extending to brain-machine interfaces and neuroprosthetic devices.

On December 5, 1995, IBM took the wraps off its Deep Blue prototype, a supercomputer designed to beat the world’s greatest chess players. IBM would manage to achieve its goal two years later, after a host of software and hardware revisions. In 1997, Deep Blue famously triumphed over an at-his-peak chess Grandmaster Garry Kasparov, during a rematch in New York City. The win was a turning point for IBM, who was increasingly characterized as a has-been, with a dire share price to match. It was also a cornerstone in the company's approach to computing, pivoting from mere chunks of hardware to ‘thinking systems.’

Interestingly, Deep Blue originated from work on a chess chip, which started a decade earlier at Carnegie Mellon University. That hardware research project was dubbed Deep Thought, which will tickle The Hitchhiker’s Guide to the Galaxy fans.

The first Deep Blue prototype revealed consisted of “an IBM RS/6000 workstation with 14 chess search engines as slave processors,” says the Chess Programming Wiki. According to the source, the collective chess-power this first Deep Blue incarnation had at its disposal was enough to analyze “between 3 and 5 million positions per second.”

1995: Losing versus a game running on a Pentium 90

Millions of moves per second sounds like a lot of brute force to crack a chess nut. However, it wasn’t yet enough to beat the best human players, nor even the best chess gaming computer programs of the era.

At its first outing at WCCC in 1995, Deep Blue would lose a decisive match against computer chess program Fritz running on a Pentium 90 PC. The loss showed IBM’s brute force technique couldn’t easily roll over Fritz’s well-curated opening book of game moves, positional heuristics, and chess knowledge.

Chess Grandmaster Kasparov faced Deep Blue for the first time in 1996. This time it showed it could do somewhat better – the IBM machine won Game 1 against the reigning world champ. After its promising start, the tables turned, though, with Kasparov triumphing 4-2.

1997: IBM brings moar brute force

IBM exacted its revenge in 1997, with Deep Blue victorious in a marathon 6-game rematch. Chess Programming says that the win was marginal, at 3½-2½ in favor of the machine. Some sources say that Kasparov accused IBM of cheating and demanded a rematch – an invitation that wasn’t accepted by Deep Blue, once it had grasped the headlines. IBM quotes the chess champ as grudgingly admitting, “I have to pay tribute, the computer is far stronger than anybody expected.”

The 1997 Deep Blue build was quite a significantly beefed-up build compared to the prototype specs we sketched out earlier. In this version of Deep Blue, the 1997 chess challenger was built using 30 workstation nodes of PowerPC processors controlling 16 chess chips each. IBM’s blog says this breakthrough design could “evaluate 200 million chess positions per second, achieving a processing speed of 11.38 billion floating-point operations per second. [FLOPS]”

Looking back at Deep Blue from the AI era

In 1997, we perhaps saw the first real signs of machines being able to rival the power of human thought and intuition – admittedly using a very different technique for success.

Fast-forward to the AI era, and as we approach the end of 2025, we have to comment on the fact that the AI-LLM industry is still pretty bad at chess. As the rapid pace of AI data center buildout and adoption continues, Deep Blue is a prescient reminder of where it all came from. We can only hope that there will be some RAM left over for the rest of us.

As part of its Computex 2025 announcements, AMD has given gamers a sneak peek at the company's major update for its FidelityFX Super Resolution (FSR) technology. Dubbed FSR Redstone, the upcoming installment will bring many new features to match rival Nvidia's Deep Learning Super Sampling (DLSS).

FSR 4, the latest iteration of the technology, debuted a few months ago with the launch of the Radeon RX 9700 series, specifically the Radeon RX 9070 and Radeon RX 9070 XT.

While the initial batch of supported titles was limited to around 30 games, AMD expects to double that number by June 5, aligning with the Radeon RX 9060 XT launch. Nonetheless, AMD is already plotting ahead and preparing FSR Redstone as the next substantial upgrade for FSR.

Although AMD did not provide specific details, the chipmaker emphasized three features to be included in FSR Redstone: neural radiance caching, machine learning ray regeneration, and machine learning frame generation. Some of these features might sound familiar, as they are already part of the Nvidia DLSS suite.

AMD states that neural radiance caching effectively learns how light reflects within a scene. The objective is to predict and store indirect lighting assets in a cache, which can subsequently be used to generate heaps of other rays. Logically, this helps accelerate path tracing.

IBM has introduced the z17, its newest mainframe system, designed for mission-critical business transactions with advanced security capabilities enhanced with AI. The system is based on the Telum II processor that offers both 70% higher general-purpose performance over its predecessor as well as 50% improved AI capabilities. For those who need even higher AI performance, IBM offers to install additional Spyre accelerators. 

IBM's Telum II processor is the heart of the company's new z17 mainframe. The Telum II CPU features eight advanced cores operating at 5.5 GHz, featuring enhanced branch prediction, store writeback, and address translation. The chip is equipped with 36MB of L2 cache, a 40% increase compared to the earlier version. It offers support for virtual L3 and L4 cache levels, expanding available cache to 360 MB and 2.88 GB, respectively. Additionally, Telum II integrates a data processing unit (DPU) to accelerate transactional workloads, which the company says increases overall system responsiveness. The chip is manufactured using Samsung’s 5HPP fabrication process and contains 43 billion transistors. 

However, the Telum II does not only boast enhanced performance. A central element of this processor is its upgraded AI unit, which delivers four times the compute capability of the previous generation, reaching 24 trillion operations per second with INT8 data precision. Perhaps, 24 TOPS wasn't very impressive. However, the NPU is designed for mission-critical time-sensitive application that supports ensemble AI methods (traditional machine learning with a large-language model) to detect suspicious activities and fraud attempts. 

It should be noted that every AI unit within a processor drawer can accept tasks from any of the CPU cores. This ensures even distribution of processing demands and enables the full use of the available 192 trillion operations per second per drawer when all accelerators are active. 

IBM understands that some workloads will require more AI performance. Hence, alongside its Telum II, IBM unveiled the Spyre AI accelerator card with a PCIe interface. This 26-billion transistor processor packs 32 AI cores and features an architecture that closely resembles that of the AI accelerator architecture found in Telum II and, therefore, can be used to dynamically expand AI capabilities and performance of z17 drawers. 

"The industry is quickly learning that AI will only be as valuable as the infrastructure it runs on," said Ross Mauri, general manager of IBM Z and LinuxONE, IBM. "With z17, we are bringing AI to the core of the enterprise with the software, processing power, and storage to make AI operational quickly. Additionally, organizations can put their vast, untapped stores of enterprise data to work with AI in a secured, cost-effective way." 

To support AI workloads at the system level, IBM intends to introduce its z/OS 3.2 in Q3 2025, an updated version of its mainframe operating system. The new OS is designed to work with hardware acceleration and supports NoSQL and hybrid cloud data. 

Traditionally, for IBM's z mainframes, the new z17 features robust security capabilities, including a new tool called IBM Vault, originally developed by HashiCorp to handle credentials, keys, and tokens across hybrid environments. 

The system also includes hardware-level support for data classification and anomaly detection using the inference capabilities of the Telum II CPU. 

As for storage, IBM's z17 will use the company's IBM DS8000 Gen10 system, which is designed to support high-speed transactions, availability, and scalability for mission-critical operations. 

The IBM z17 will be available starting June 18, 2025, with the Spyre Accelerator arriving later in the year.

A robotics and machine learning engineer has developed a command-line interface tool that monitors power use from a smart plug and then tunes system performance based on electricity pricing. The simple program, called WattWise, came about when Naveen built a dual-socket EPYC workstation with plans to add four GPUs. It's a power-intensive setup, so he wanted a way to monitor its power consumption using a Kasa smart plug. The enthusiast has released the monitoring portion of the project to the public now, but the portion that manages clocks and power will be released later.

Unfortunately, the Kasa Smart app and the Home Assistant dashboard was inconvenient and couldn't do everything he desired. He already had a terminal window running monitoring tools like htop, nvtop, and nload, and decided to take matters into his own hands rather than dealing with yet another app.

Naveen built a terminal-based UI that shows power consumption data through Home Assistant and the TP-Link integration. The app monitors real-time power use, showing wattage and current, as well as providing historical consumption charts. More importantly, it is designed to automatically throttle CPU and GPU performance.

Naveen’s power provider uses Time-of-Use (ToU) pricing, so using a lot of power during peak hours can cost significantly more. The workstation can draw as much as 1400 watts at full load, but by reducing the CPU frequency from 3.7 GHz to 1.5 GHz, he's able to reduce consumption by about 225 watts. (No mention is made of GPU throttling, which could potentially allow for even higher power savings with a quad-GPU setup.)

Results will vary based on the hardware being used, naturally, and servers can pull far more power than a typical desktop — even one designed and used for gaming.

WattWise optimizes the system’s clock speed based on the current system load, power consumption as reported by the smart plug, and the time — with the latter factoring in peak pricing. From there, it uses a Proportional-Integral (PI) controller to manage the power and adapts system parameters based on the three variables.

At the moment, the app only supports one smart plug at a time and only works with the Kasa brand. However, Naveem says there are plans to add support for multiple plugs, more smart plug brands, integration with other power management tools, and other features. The app in its current form is a pretty simple tool, but sometimes simple is all you need to solve a problem.

Naveen made WattWise open source under the MIT license, and you can download it directly from GitHub. If you’re interested, you can leave feedback and contributions, or you can fork it and adapt it for other systems. Note that the current version only contains the dashboard, not the actual power optimizer, which still needs further work.

Tech giant IBM secured a patent from the U.S. Patent and Trademark Office on its technology for transporting microparticles using a 4D-printed smart material. According to the patent, these smart materials can use shape-memory alloys or polymers that respond to external forces like temperature, light, magnetism, or electrical currents.

After being deformed, the smart materials return to their original shape, allowing the researchers to induce movement in them and use them to transport minute-sized particles that would be difficult or impossible to transport using traditional delivery methods.

The user must initially set the delivery path and its environmental conditions and note the item's size, shape, weight, and composition to be delivered. Once completed, the machine learning algorithm applies the proper stimulus to move the material. This could be heat or light that causes one part or the other of the 4D material to respond, generating an action that results in an equal but opposite reaction.

Aside from following the user's initial path, IBM’s machine language monitors the 4D-printed smart material for any deviations or blockages. It will resolve the situation, allowing the operation to proceed with little human intervention. All external stimuli are removed when it reaches its destination, allowing the smart material to deliver its payload.

This design allows for the delivery of microparticles between 1 and 100 microns in diameter. Furthermore, its different control methods mean it can travel through various media, making it useful for medical and industrial applications. For example, doctors and medical technologists could use this technique to deliver drugs to specific cells via the blood or the gastrointestinal tract. It could also be used for miniature electronics manufacturing and perhaps introduce a new semiconductor manufacturing method.

4D printing builds upon 3D printing technology, wherein the filament used for printing reacts to external stimuli. Researchers can then use this to generate movement, much like how a single-celled organism can move by using chemical reactions within its cell membrane.

AMD, Cisco, Nokia, and Jio Platforms unveiled plans to develop the Open Telecom AI Platform at Mobile World Congress 2025. The initiative is designed to enhance telecom platforms with AI-driven automation, security, and efficiency, offering a scalable model for service providers worldwide.

Under the terms of the agreement, AMD will provide high-performance computing solutions, including EPYC CPUs, Instinct GPUs, DPUs, and adaptive computing technologies. Cisco will contribute networking, security, and AI analytics solutions, including Cisco Agile Services Networking, AI Defense, Splunk Analytics, and Data Center Networking. Nokia will bring expertise in wireless and fixed broadband, core networks, IP, and optical transport. Finally, Jio Platforms Limited (JPL) will be the platform's lead organizer and first adopter. It will also provide global telecom operators' initial deployment and reference model.

For AMD, getting into a potentially successful telco platform is a big deal. Its arch-rival, Intel, has a major lead with telecom projects, having invested massive amounts of money in 5G and other telecom technologies.

"AMD is proud to collaborate with Jio Platforms Limited, Cisco, and Nokia to power the next generation of AI-driven telecom infrastructure," said Lisa Su, chair and CEO of AMD. "By leveraging our broad portfolio of high-performance CPUs, GPUs, and adaptive computing solutions, service providers will be able to create more secure, efficient, and scalable networks. Together we can bring the transformational benefits of AI to both operators and users and enable innovative services that will shape the future of communications and connectivity."

The platform will function as a multi-layer intelligence system, integrating AI at every level of the telecom infrastructure. It will incorporate various AI approaches, including autonomous AI agents, large and specialized small language models, and traditional machine learning techniques to ensure adaptable and intelligent network management. Open APIs will be a key platform component, enabling seamless integration with existing telecom infrastructure and optimizing network functions for enhanced efficiency.

A primary goal of the project is to improve network security while cutting operational costs. By embedding AI into network management, the system will create self-regulating telecom environments capable of identifying risks, adjusting operations dynamically, and delivering a more secure and reliable service.

"Nokia possesses trusted technology leadership in multiple domains, including RAN, Core, fixed broadband, IP and optical transport. We are delighted to bring this broad expertise to the table in service of today's important announcement," said Pekka Lundmark, President and CEO at Nokia. "The Telecom AI Platform will help Jio to optimize and monetize their network investments through enhanced performance, security, operational efficiency, automation and greatly improved customer experience, all via the immense power of artificial intelligence. I am proud that Nokia is contributing to this work."

Australian biotech company Cortical Labs has introduced what it claims to be "the world’s first code deployable biological computer," which combines human brain cells with traditional silicon-based computing. The system, known as CL1, was presented at the Mobile World Congress in Barcelona and is being explored for its potential applications in artificial intelligence (AI) and machine learning.

The CL1 consists of a silicon chip with lab-grown human neurons cultivated on its surface. These neurons are capable of responding to electrical signals, forming networks that process information similarly to a biological brain. The system is designed to allow two-way communication, where electrical impulses stimulate the neurons, and their responses are recorded and analyzed. To maintain the viability of the neurons, the CL1 is equipped with a life-support system that regulates temperature, gas exchange, and other necessary conditions.

A notable aspect of the CL1 is its ability to learn and adapt to tasks. Previous research has demonstrated that neuron-based systems can be trained to perform basic functions, such as playing simple video games. Cortical Labs' work suggests that integrating biological elements into computing could improve efficiency in tasks that traditional AI struggles with, such as pattern recognition and decision-making in unpredictable environments.

Cortical Labs says that the first CL1 computers will be available for shipment to customers in June, with each unit priced at approximately $35,000.

The use of human neurons in computing raises questions about the future of AI development. Biological computers like the CL1 could provide advantages over conventional AI models, particularly in terms of learning efficiency and energy consumption. The adaptability of neurons could lead to improvements in robotics, automation, and complex data analysis.

However, the scalability of this technology remains uncertain. Producing and maintaining neuron-based systems is significantly more complex than manufacturing traditional processors, and ensuring long-term stability poses additional challenges.

Ethical concerns also arise from the use of human-derived brain cells in technology. While the neurons used in the CL1 are lab-grown and lack consciousness, further advancements in the field may require guidelines to address moral and regulatory issues. The prospect of integrating living cells with computational hardware prompts discussions about the boundaries of artificial intelligence and human-like cognition.

Argonne National Laboratory this week said that its Aurora supercomputer is now fully operational and is available to the scientific community. The machine, which was announced in 2015 and faced massive delays, now offers over 1 FP64 ExaFLOPS performance for simulation as well as 11.6 mixed precision ExaFLOPS for artificial intelligence and machine learning. 

"We are ecstatic to officially deploy Aurora for open scientific research," said Michael Papka, director of the Argonne Leadership Computing Facility (ALCF), a DOE Office of Science user facility. "Early users have given us a glimpse of Aurora's vast potential. We are eager to see how the broader scientific community will use the system to transform their research." 

The availability of the Aurora supercomputer for open scientific research may be considered a formal acceptance of the system by ARNL, which marks an important milestone for the troubled machine. Initially planned for 2018, Aurora missed this target due to Intel's decision to discontinue its Xeon Phi processors. After the machine was re-architected, the project faced further setbacks due to Intel's 7nm process technology delay, pushing the completion date to 2021 and then again to 2023. 

Even after the hardware was installed in June 2023, it took several months for the system to be fully operational and achieve exascale performance, which it finally reached in May 2024. Yet, for well over half a year, the system was only available to select researchers.

While Aurora is not the most powerful supercomputer for simulations, as its FP64 performance barely exceeds one ExaFLOPS, it is the most powerful system for AI as it can achieve 11.6 mixed precision ExaFLOPS according to the HPL-MxP benchmark. 

"A big target for Aurora is training large language models for science," said Rick Stevens, Argonne associate laboratory director for Computing, Environment and Life Sciences. "With the AuroraGPT project, for example, we are building a science-oriented foundation model that can distill knowledge across many domains, from biology to chemistry. One of the goals with Aurora is to enable researchers to create new AI tools that help them make progress as fast as they can think — not just as fast as their computations." 

Some of the first research projects using Aurora are detailed simulations of intricate systems, such as the human circulatory system, nuclear reactors, and supernova explosions. The machine's overwhelming performance is also instrumental in processing data from major research centers, such as Argonne's Advanced Photon Source (APS) and CERN's Large Hadron Collider. 

"The projects running on Aurora represent some of the most ambitious and innovative science happening today," said Katherine Riley, ALCF director of science.
"From modeling extremely complex physical systems to processing huge amounts of data, Aurora will accelerate discoveries that deepen our understanding of the world around us." 

On the hardware side, Aurora clearly impresses. The supercomputer comprises 166 racks, each holding 64 blades, for a total of 10,624 blades. Each blade contains two Xeon Max processors with 64 GB of HBM2E memory onboard and six Intel Data Center Max 'Ponte Vecchio' GPUs, all cooled by a specialized liquid-cooling system. 

In total, Aurora has 21,248 CPUs with over 1.1 million high-performance x86 cores, 19.9 PB of DDR5 memory, and 1.36 PB of HBM2E memory attached to the CPUs. It also features 63,744 GPUs optimized for AI and HPC equipped with 8.16 PB of HBM2E memory. Aurora uses 1,024 nodes with solid-state drives for storage, offering 220 PB of total capacity and 31 TB/s of bandwidth. The machine relies on HPE's Shasta supercomputer architecture with Slingshot interconnects.

Speaking in an interview with Digital Foundry at CES 2025, Nvidia VP of Applied Deep Learning Research Bryan Catanzaro hinted that Nvidia may end up bringing DLSS 4 to Nvidia GPUs as old as the RTX 30 Series, which otherwise do not support Nvidia's own Frame Generation technologies. Instead, RTX 30 Series GPUs are currently forced to rely on games with an AMD FSR 3 Frame Generation solution — which isn't always usable alongside DLSS, though fortunately it usually is. We note that external solutions like Lossless Scaling, which also allows for Multi-Frame Generation like DLSS 4 will, though of course continue to work without AI acceleration.

In the full interview, the specific question asked of Catanzaro on Frame Gen for older Nvidia GPUs by Digital Foundry's Alex Battaglia was "Now that it (DLSS 4) is running entirely on Tensor cores, obviously it's a more intensive task as a result of that, to some degree... what's keeping it, then, from running on RTX 3000?"

Catanzaro replied, "I think this is primarily a question of optimization and engineering and then the ultimate user experience. We're launching this Frame Generation, the best Multi-Frame Generation technology with the 50 Series, and we'll be able to see what we squeeze out of older hardware in the future."

While this is hardly a conclusive statement, the logic leading to the question and the open-endedness of the answer does lead to some optimism. And, of course, Nvidia RTX 30 Series users can always opt for AMD FSR 3 Frame Generation wherever it's available, with Lossless Scaling's recent updates further improving its support for global Multi-Frame Generation.

Aside from this question, which could allude to some nice future boons for older hardware, Battaglia also asks several other questions in the full interview that dive deeper into the technical details of DLSS 4.

Highlights include Catanzaro claiming that DLSS 4 Frame Generation's frame pacing should be much more evenly paced than DLSS 3 Frame Generation, claiming a decrease in frame pacing variability by "a factor of 5 or 10." Basically just meaning that Frame Gen should now look far smoother. Though for now, Frame Generation still doesn't include V-Sync, and when asked, Catanzaro essentially said that the demand would need to be there for Nvidia to have the incentive to get it figured out.

Nvidia has a lot to say about artificial intelligence and machine learning. It's not all data center hardware and software either, though Nvidia's various supercomputers are doing most of the heavy lifting for training new AI models. At its Editors' Day earlier this month, Nvidia briefed us on a host of upcoming technologies and hardware, including the Blackwell RTX architecture, neural rendering, the GeForce RTX 50-series Founders Edition cards, Blackwell for professionals and creators, and Blackwell benchmarking. There were two sessions devoted to generative AI and Nvidia's RTX AI PC ecosystem, which we'll discuss here.

Generative AI came to the forefront with the rise of tools like Stable Diffusion and ChatGPT over the past couple of years. Nvidia has been working on AI tools for a while now that are designed to change the way games and NPCs behave and the way we interact with them. We've heard about ACE (Avatar Cloud Engine) for a while now, and it continues to improve. With Blackwell and upcoming games, Nvidia has partnered with various game developers and publishers to leverage ACE and related technologies. The results range from interesting to pretty bad, so we'll just let these videos speak for themselves and provide more analysis below. 

One of the key issues, as with so many things related to generative AI, is getting the desired results. Live demos of PUBG Ally had Krafton representatives talking to the AI player, who would respond verbally as expected. "Go find me a rifle and bring it to me." "Okay, I'll go do that..." At this point, the AI NPC would seemingly do nothing of the sort. It seemed caught in a loop and still looks far away from being ready for public consumption. But these things change fast, so perhaps it's really only a few months away from being great — who knows? (We also question how having AI NPCs in a multiplayer game will work out, but that's a different subject.)

Other use cases demonstrated include a raid boss in MIR5 that will supposedly learn from past encounters and adapt over time, requiring different tactics to repeatedly defeat the boss. The high-level concept sounds a bit like the Omnidroid from the Incredibles, learning and becoming more powerful over time, though the raid boss won't gain new abilities so it shouldn't become invincible — because where's the fun in that?

Zoopunk allows the player to repaint and decorate a spaceship by interacting with an AI. Again, the live demo was lacking, as there were lengthy pauses before a response, with a simple prompt like "Please paint my ship purple," resulting in a 20 to 30-second sequence that felt entirely unnecessary.

Fundamentally, the problem isn't just about using ACE and AI to create NPCs for games; it's about making those NPCs actually useful, interesting, and fun. These are games, after all, and if we're only adding voice interactions that aren't actually meaningful, what's the point? We're still waiting to see a demo of a game where the AI NPCs make for a better end result than traditional game development, but we're sure there are bean counters looking for ways to cut costs.

The other session, RTX AI PCs, was related to the ACE stuff with more of a focus on all the various tools Nvidia has created rather than on actual game demos. There are lots of new RTX NIMs — Nvidia Inference Microservices — coming, with blueprints (code samples, basically) of how they can be used.

One of the more interesting examples was a blueprint for converting a PDF into an AI-voiced podcast. The tool had several components for extracting text, analyzing images, analyzing tables (all stored in a PDF), and then creating a script for a podcast. The result still sounds very much like an AI reading a script, but you can fully edit the resulting text, and you could even do the voiceover yourself, which would bring some humanity into the equation.

You can see the full slide deck below, and developers interested in these tools will want to get in touch with Nvidia to register for and use the various APIs.

Microsoft is advancing its DirectX API to support neural rendering, signaling a transformative shift in graphics rendering by incorporating AI and machine learning. This development, highlighted in a recent blog post, is designed to enhance visual quality and efficiency in gaming and other graphics-intensive applications.

Neural rendering makes use of machine learning models to generate or enhance visual elements such as textures, lighting, and image upscaling. By offloading complex rendering tasks to AI, this approach improves both performance and visual fidelity while reducing the computational burden on traditional rendering pipelines. Technologies like Nvidia’s DLSS and AMD’s FSR have already demonstrated the potential of AI-enhanced rendering. Microsoft’s initiative seeks to provide a standardized, open framework for such capabilities within the widely used DirectX API.

A key feature of the forthcoming DirectX update is Cooperative Vector Support. This feature will enhance AI workloads for real-time rendering by optimizing matrix-vector operations crucial for AI tasks like training, fine-tuning, and inferencing. This feature allows AI tasks to run in different shader stages, enabling efficient execution of neural networks, such as in a pixel shader, without monopolizing the GPU. By integrating neural graphics into DirectX applications, it provides access to AI-accelerator hardware across platforms, empowering developers to create more immersive experiences.

Microsoft has confirmed that Cooperative vectors will leverage Tensor Cores in Nvidia's new RTX 50-series GPUs to enable neural shaders, enhancing game asset visualization, optimizing geometry for improved path tracing, and supporting tools for creating photorealistic game characters.

Microsoft’s High-Level Shading Language (HLSL) team is said to be working closely with major GPU manufacturers, including AMD, Intel, Nvidia, and Qualcomm, to ensure these new capabilities are optimized for a wide range of hardware architectures.

By embedding neural rendering capabilities into DirectX, Microsoft could broaden the adoption of AI-driven graphics across multiple platforms. Potential applications range from enhanced real-time ray tracing to adaptive resolution scaling for high-definition displays. While proprietary AI rendering technologies have been limited to specific ecosystems, Microsoft’s open approach could democratize access, fostering greater innovation and competition.

Though the updates are still in development and lack a definitive release date, they highlight the increasing role of AI in shaping the future of graphics technology.

Current known lithium reserves are under tremendous pressure due to soaring demand driven by the widespread use of lithium-ion batteries. A recent study by the USGS estimates that there may be approximately 5.1 to 19 million tons of lithium in Southern Arkansas, with the latter representing almost 136% of the United States' current lithium resource estimate.

Delving into the details, this lithium concentration is projected to be in the Smackover Formation brines of Arkansas. The Smackover Formation is a vast and permeable limestone aquifer—a formation of porous limestone rock that can carry water—dating back to the Jurassic geologic era. Importantly, this region is full of mineral-rich brine or salt water. As you may have guessed, these estimates refer not to lithium rocks but to lithium dissolved in brine.

The research team used fresh lithium concentration data in tandem with historical samples to train a machine-learning model that plotted a lithium concentration map, even in areas that didn't have lithium samples. A combination of this model and water testing led the group to the estimate that Arkansas may be housing up to 19 million tons of lithium. Extraction rates of lithium from brine are still a large variable, so the final output will be less than these initial figures; "We have not estimated what is technically recoverable based on newer methods to extract lithium from brines,” said Katherine Knierim, a Hydrologist.

On a more industrial scale, the US relies on foreign countries for roughly 25% of its lithium imports. Researchers estimate that there is enough dissolved lithium in Arkansas to ensure the US no longer needs to depend on imports. In 2022, over 5000 tons of lithium was brought to the surface based on these calculations as a byproduct of the oil, gas, and Bromine industries.

If extraction of even five million tons (lower bound) is possible, this is over nine times what electric vehicles require by 2030. If, hypothetically, 100% of lithium was extracted, the amount of lithium obtained would be over 30 times the global demand in 2030.

Work is currently underway on lithium extraction in this region. ExxonMobil, an energy company, acquired drilling rights in Arkansas in 2023. The company has set ambitious targets to begin extraction by 2027 and power a million electric vehicles by 2030.

The Smackover Formation is emerging as a strategically important location for the US. Otherwise overlooked byproducts of mineral extraction are now proving to be the future's goldmine: lithium.

AMD has expanded support for machine learning (ML) development on its RDNA 3 GPUs with Radeon Software for Linux 24.10.3 and ROCm 6.1.3. This enables developers using frameworks like PyTorch, ONNX Runtime, or TensorFlow to perform ML tasks on a local workstation featuring Radeon RX 7000 and Radeon Pro W7000-series GPUs and avoid the usage of expensive cloud-based solutions.   

Normally, AMD only supported AI/ML frameworks on Instinct accelerators based on the CDNA architecture for AI and HPC applications, but recently it changed its mind and enabled AI acceleration on its RDNA 3-based GPUs. To make it easier for developers to use Radeon graphics processors for AI development, starting from ROCm 6.1.3 and Radeon Software for Linux 24.10.3 software, ROCm documentation now features a dedicated ROCm on Radeon section to simplify installation and reveal compatibility details. 

With ROCm 6.1.3, users can now use these GPUs for advanced AI tasks using frameworks like PyTorch, TensorFlow, and ONNX Runtime. Other highlights of the ROCm 6.1.3 are enhanced inference capabilities for ONNX Runtime, enabling optimized processing with INT8 data format using MIGraphX. 

AMD's ROCm 6.1.3 software stack turns a desktop setup into a cost-effective alternative to servers or cloud-based ML platforms, bringing parallel computing capabilities directly to local machines. Also, since AMD's ROCm is compatible with AMD's CDNA architecture, it makes it easy for developers to transition their desktop-developed applications to datacenter environments without changing frameworks. 

While it is evident that AMD is doing a lot to make its Radeon GPUs more attractive to AI developers, there are still a few things it could add. AMD's ROCm 6.1.3 was released in June. So far, AMD has released its ROCm 6.2 for Instinct APUs and accelerators, which has more features and capabilities, such as broader FP8 support. Perhaps more importantly, ROCm 6.2 brings vLLM, which addresses significant inferencing challenges, such as efficient multi-GPU computation, reduced memory usage, and minimized computational bottlenecks, according to AMD. As a result, AMD's Instinct accelerators and processors still have numerous advantages over consumer GPUs, particularly when it comes to the usage of such GPUs in the datacenter.

ZLUDA, an open-source CUDA translation layer, has lived two quite vivid lives with Intel and then AMD GPUs. It was nearly killed in August when AMD asked to take down the code developed using its funds. However, as its developer, Andrzej Janik, secured funding from a mysterious sponsor, ZLUDA now has a third life. This time around, the focus of ZLUDA will be to run AI/ML software designed for CUDA GPUs on processors from other vendors using a translation layer, reports Phoronix.

ZLUDA was originally designed to run creative professional CUDA-based applications on Intel and then AMD GPUs, while the upcoming iteration of ZLUDA shifts focus to accommodate AI and machine learning workloads. Also, the emphasis is now not just on Intel or AMD. Instead, it offers multiple GPU vendor support, making ZLUDA applicable across different GPU architectures. Nonetheless, for the time being, most development efforts are concentrated on AMD GPUs, particularly RDNA1 and newer architectures. Support is being built around AMD’s ROCm 6.1+ compute stack, laying the foundation for broader, multi-architecture compatibility in the future.

Andrzej Janik is currently working to make AI/ML frameworks like PyTorch, TensorFlow, and Llama.cpp function seamlessly using CUDA on non-Nvidia GPUs using his translation layer, according to Phoronix, who spoke to the developer. Janik predicts it will take about a year to develop the new ZLUDA code to a stable state where it can effectively handle AI/ML workloads across multiple GPUs. Contributions from the open-source community will be welcomed as the project evolves. So, ZLUDA will remain open source, or at least it looks so today.

Although ZLUDA now has a financial backer, the sponsor has chosen to remain anonymous for now. We can only speculate who the sponsor is because they need to run AI workloads at scale and opted for multi-GPU vendor support. Also, we presume it is big enough not to be afraid of getting into a conflict over running CUDA software through a translation layer, which Nvidia does not endorse these days. Yet, the developer says that this ‘stealth’ sponsor is expected to be revealed later, providing more insight into the direction and future support of ZLUDA.

Researchers in Scotland hope to develop a set of artificial intelligence (AI) tools that can predict the risk of dementia in their patients. To do so, they will compare up to 1.6 million CT and MRI scans with linked public health records to find patterns that could help doctors better predict someone’s risk of developing dementia. 

At the University of Edinburgh and the University of Dundee, a team of scientists working as part of a global research effort called NEURii is preparing to collect the data. The CT and MRI scans have been collected from patients in Scotland over more than a decade. Using AI and machine learning, the team hopes to develop a suite of tools that radiologists can use as a standard reference when examining new scans. 

“Should we establish a successful proof of concept, we will have a suite of software tools that are smoothly and unobtrusively integrated with routine radiology operations that assist clinical decision-making and flag the risk of dementia as early as possible,” said Professor Emanuele Trucco, an expert in AI and medical imaging at the University of Dundee, as quoted by The Guardian. 

The efforts could also be used to accelerate the development of treatments for dementia. According to Professor Will Whiteley from Edinburgh’s Centre for Clinical Brain Sciences, the project co-leader, making better use of brain scans could “lead to better understanding of dementia and potentially earlier diagnosis of its causes.” This will, he said, according to The Guardian, “make development of new treatments easier.”

AI is already used to help with other medical conditions. It’s been proven useful in listening for signs of heart disease when paired with a stethoscope. Other recent projects have used AI to help people with vision impairment better understand and navigate the world around them.

Dementia is a growing concern globally. Current studies suggest more than 55 million people already suffer from dementia globally. Researchers believe the number of cases of dementia will nearly triple to 153 million by 2050. Health and social services costs related to sufferers of dementia already exceed $1 trillion (£780 billion) each year, according to some estimates.

The NEURii research project also includes as partners global pharmaceutical company Eisai, Bill Gates’ personal service company Gates Ventures, Health Data Research UK (HDR UK), and medical research not-for-profit LifeArc.

If the research is approved by National Health Service (NHS) Scotland, the team will store its data in the Scottish National Safe Haven, a secure platform commissioned by NHS Scotland for such uses. 

Last week, the Department of Energy (DOE) issued a request for proposals (RFP) to develop a new supercomputer named Discovery. This supercomputer will replace the current known fastest supercomputer in the world, Frontier, at Oak Ridge National Laboratory. Discovery aims to surpass Frontier's performance, offering three to five times more computational throughput (e.g., 8.5 ExaFLOPS) by 2027 or early 2028.  

ORNL mentions advanced AI, machine learning, improved energy efficiency, and comprehensive system modeling among the workloads that will run on the Discovery supercomputer. Unlike previous RFPs, this one does not specify an exact performance increase but only says that the new supercomputer has to be three to five times more powerful than its predecessor. 

Discovery's computational power will support scientific research in various fields, including AI, climate change, drug discovery, nuclear security, and green energy solutions. Researchers will be able to leverage Discovery’s advanced computational power and capabilities in modeling, simulation, high-performance data analysis, and AI to achieve significant breakthroughs in both scientific and industrial fields. Like Frontier, scholars globally will have the chance to compete for computing time on Discovery to address major scientific challenges. 

"Discovery will enable the scientific community to model real-world situations at new levels of detail. It will help us study challenging problems we cannot easily explore with experiment, observation or theory alone," said Georgia Tourassi, ORNL associate laboratory director of computing and computational sciences. "Using Discovery, scientists will improve the safety and efficiency of nuclear power plants and aerospace engineering, pushing the boundaries of what’s possible in sustainable power generation and aviation. They will accelerate the development of new drugs and advanced materials. They will even gain deeper insights in global climate dynamics to inform critical decisions for our collective future." 

Proposals for Discovery are due by August 30, 2024.  

The ORNL has a history of deploying the world's fastest supercomputers. For example, Jaguar, Titan, and Summit led the world's Top 500 list in different years. Frontier is the world's No. 1 supercomputer today. In fact, over the past decade, the facility has increased its computational power 500-fold while only quadrupling energy consumption.

A fundamental limitation of modern artificial intelligence and neural networks is that they aren't good at spatial mapping or navigation without an existing map. However, TechXplore reports that a combination of a predictive coding algorithm and Minecraft gameplay successfully "taught" a neural network how to create spatial maps and subsequently use those spatial maps to predict the following frames of video, yielding a mean-squared error of 0.094% between the predicted image and the final image.

The project demonstrates genuine spatial awareness of AI, which still isn't seen in the impossible architecture and other strange glitches that come with things like OpenAI's Sora.

These findings come from a paper published in the Nature Machine Intelligence journal on Nature.com, automated construction of cognitive maps with visual predictive coding, from James Gornet & Matt Thomson of the California Institute of Technology (aka Caltech). The paper, released to the public just yesterday, details exactly how this was achieved in exhaustive detail and even shares the code on GitHub and Zenodo.

One of the two researchers who worked on the project, Matt Thomson, spoke to TechXplore and provided a few noteworthy quotes about the process and what led them to undertake it.

Per Matt Thomson, "There's this sense that even state-of-the-art AI models are still not truly intelligent. They don't problem-solve like we do; they can't prove unproven math results or generate new ideas. We think it's because they can't navigate in conceptual space; solving complex problems is like moving through a space of concepts, like navigating. AIs are doing more like like rote memorization— you give it an input, and it gives you a response. But it's not able to synthesize disparate ideas."

James Gornet, the graduate student who led the project, encouraged the use of Minecraft and studied neuroscience, machine learning, math, statistics, and biology under the Department of Computational and Neural Systems (CNS) at Caltech. He did not provide a quote about the process, but Thomson says that CNS is uniquely suited for James's work and that "we're hoping to learn about the brain in turn," not just advance AI.

Google may be considering rebooting its much-maligned Google Flu Trends (GFT) service, which died of embarrassment in 2015. First launched in 2008, GFT overestimated, underestimated, and failed to predict several major flu-related events during its seven-year existence. However, Google researchers recently published a paper outlining a new and improved flu rate prediction model that uses modern artificial intelligence (AI) methodology.

To put it mildly, the original GFT service wasn’t a roaring success. Google’s own AI summation (image above) of GFT highlights several studies that found it inaccurate and thus unusable. It failed to predict the 2009 spring pandemic and also “consistently overestimated the relative incidence of flu” in both 2011 and 2013.

GFT launched in 2008 based upon quite a simple and logical premise – people search Google for flu symptoms when they get ill, and the trending flu symptom searches across regions could be used to pre-warn health agencies that a wave of flu infections was likely so precautions/preparations could be implemented. Thus, GFT relied on a kind of Collective Intelligence (CI), which was shoved into a linear model and would be tweaked across the lifetime of the service, but to little worthwhile effect. Hence, Google killed off GFT estimates in August 2015.

The new Google research paper outlines two key techniques that it is hoped will get better results from analyzing and modeling the huge swathes of user data that Google is privy to. These are outlined as follows:

1. we introduce SLaM Compression, a way to quantify search terms using pre-trained language models and create a representation of search data that has low dimensionality, is memory efficient, and effectively acts as a summary of search, and
2. we present CoSMo, a Constrained Search Model for estimating real-world events using only search data. We demonstrate the efficacy of our contributions by estimating with high accuracy U.S. automobile sales and U.S. flu rates using only Google Search data.

You might have heard of similar tech before, as SLaM (Search Language Model Compression) is used for machine learning tasks and has been especially useful in automotive AI. Meanwhile, CoSMo is a new language model (LM) approach that uses around 512 dimensions to predict real-world events.

We must note that Google also seemed quite confident in its science/methods when it originally launched GFT back in 2008. However, this time, we have new AI-related science and even tighter correlations between what the new model would have predicted and what actually happened in history.

Google seems to have found its new approach to be successful, noting, "We also introduce CoSMo, a constrained search model, which has inductive biases that greatly improve the accuracy of our models built on search data. For estimating the flu rates, we show our simple approach is on par or better than the existing complex ensemble methods. [...] Finally, we demonstrate that our models, despite being highly non-linear neural networks, offer interpretability that explains what terms are related to the variables of interest."

Whether the heralded new flu rate modeling research leads to Google resurrecting GFT remains to be seen. However, it demonstrates that Google is still interested in perfecting its protection tech, which could be applied to a wide range of potential uses that would eventually make the company yet more money. 

Optical interconnects are crucial for next-generation AI and HPC data centers, but adding them to CPUs and GPUs is complicated, and much of the infrastructure for mass adoption is still lacking. To that end, Intel introduced its first fully integrated optical input/output (I/O) chiplet at the Optical Fiber Communication Conference (OFC) 2024. The optical compute interconnect (OCI) chiplet can be attached to CPUs and GPUs to enable high bandwidth, low power consumption, and extended-reach I/O connectivity.

Intel's OCI chiplet is one of the industry's first fully integrated optical I/O solutions for co-packaging with compute processors. This chiplet supports 64 PCIe 5.0 channels, each transmitting at 32 GT/s in both directions, totaling 4 Tbps, over distances of up to 100 meters using fiber optics. The chiplet uses dense wavelength division multiplexing (DWDM) wavelengths and consumes only five pico-Joules per bit, significantly more energy-efficient than pluggable optical transceiver modules, which consume about 15 pico-Joules per bit, according to Intel.

This device is crucial for next-generation data centers and AI/HPC applications. It will enable high-performance connections for CPU and GPU clusters, coherent memory expansion, and resource disaggregation. These features will be handy for operating supercomputers for large-scale AI models and machine learning tasks that require tremendous data bandwidth.

Traditional electrical I/O systems, which rely on copper traces for connectivity, offer high bandwidth density and low power but are limited to short distances of about one meter. In contrast, Intel's optical I/O chiplet can transmit data over much longer distances with higher efficiency and lower power consumption.

The current optical I/O chiplet is largely a prototype, and Intel is collaborating with select customers to further develop and integrate this device with next-generation systems-on-chips (SoCs) and system-in-packages (SiPs).

"The ever-increasing movement of data from server to server is straining the capabilities of today’s data center infrastructure, and current solutions are rapidly approaching the practical limits of electrical I/O performance," said Thomas Liljeberg, senior director of product Management and Strategy, Integrated Photonics Solutions (IPS) Group. "However, Intel’s groundbreaking achievement empowers customers to seamlessly integrate co-packaged silicon photonics interconnect solutions into next-generation compute systems. Our OCI chiplet boosts bandwidth, reduces power consumption and increases reach, enabling ML workload acceleration that promises to revolutionize high-performance AI infrastructure."

Intel's silicon photonics initiative is backed by over 25 years of research and extensive deployment in data centers. The company's hybrid laser-on-wafer technology and direct integration approach offer high reliability and cost efficiency, which Intel says sets it apart from competitors.

So far, Intel has shipped over 8 million photonic integrated circuits (PICs) with more than 32 million integrated on-chip lasers. These PICs were integrated into pluggable transceiver modules and used in large data centers at major cloud providers for 100, 200, and 400 Gbps applications. Next-generation PICs, supporting 200G per lane, are being developed for 800 Gbps and 1.6 Tbps applications.

Samsung's board of directors is looking to put serious coin into GPUs this year, though specifics on the spending are yet unknown. Business Korea reports on the "GPU Investment Proposal" first disclosed in Samsung Electronics’ corporate governance report. The investment is likely going towards GPU-based research to improve fabrication efficiency rather than a new Samsung GPU product.

Only just revealed to the public, Samsung's GPU investment proposal was ratified by Samsung execs in a March board meeting. Major players including Han Jong-hee, head of Device eXperience (DX) and co-CEO of Samsung, sit on the management committee, as well as the presidents of the Mobile Experience and Memory Business divisions. This marks the first time since 2012 that Samsung has invested heavily in GPU research.

The scope of the GPU investment is completely unknown, as is Samsung's intent. It is most likely that Samsung's goals in investing in GPUs are to finance GPU-based research into improving its fabrication and manufacturing processes. High-end GPUs are necessary for AI- or machine learning-based research, and AI-based tools such as digital twins and Nvidia's cuLitho software library are powerful ways to improve semiconductor manufacturing efficiency. 

A digital twin is a true-to-reality virtual representation of a real-world physical system, exactly modeling real behaviors and actions in, for instance, a semiconductor fab. This digital twin allows fabs to test new manufacturing methods and processes digitally without disrupting real-life workflow. cuLitho is an Nvidia creation, a software library that speeds up computational lithography, a critical bottleneck in the semiconductor manufacturing workflow, from a weeks-long process to a five-day task. Major players like TSMC have already begun using these resources — powered by Nvidia server hardware and software — in production, and Samsung would be wise to jump on these innovations before it is outpaced. 

Samsung also completed the construction of its new high-performance computing (HPC) center at its Hwaseong campus in April of this year. As the HPC center pushes to full operation, Samsung continues to fill it with leading-edge servers and network hardware for semiconductor production and research. The GPU Investment Proposal may be funding for loading up the HPC center with top-end graphics cards, either for the above-mentioned research tools or simpler GPU-based compute power.

It is deeply unlikely that Samsung is planning its entry into the consumer graphics card market. The last GPU-related project to come out of Samsung was its Exynos smartphone SoCs with graphics based on AMD's RDNA architecture; Exynos chips have only been seen in European releases of some Galaxy phones. Exynos chips have repeatedly failed to impress, especially on the graphics front, with rumors that AMD may part ways with the Exynos project flying since 2023. Samsung would also stand to gain very little from entering a consumer graphics market which now has three major players, split between Nvidia, AMD, and Intel.

Raspberry Pi clocks have been around for a while but the maker community sure seems to have a way of making them increasingly more complex. Today we have an example of such a project—in fact, it's so complex that it's hardly functional. Maker and developer Max Björverud is using a Raspberry Pi to power his clocks that use AI to estimate the current time based on smell and sound.

The clocks work independently from one another, each using a particular input device (one being a microphone, the other being a multichannel gas sensor). The clocks are programmed to improve over time, but in real-time you get an estimation of the time based on the data it's accumulated so far. The clocks output to matrix panels with rows to indicate the hour, minute, and second.

The bottom matrix panel is programmed to output shapes based on recent metrics from the data it collects. As far as accuracy goes, Björverud insisted in the original project thread that uncertainty is at the heart of the purpose behind the project. No matter how correct, the clock will always insist upon the time. Björverud claims his project helps raise awareness of this aspect of artificial intelligence.

Each clock is using a Raspberry Pi, but they aren't using the same model. The ear is using a Pi 3B+ while the nose is running on a Raspberry Pi 4 B. The ear clock is using a microphone to listen for audio output while the nose clock relies on a multichannel gas sensor. As it collects more information over time, the data can be honed in to be more accurate.

The artificial intelligence side of the project is aided by an open-source Python library called Scikit-learn. This free library has plenty of tools adapted for machine learning projects. The matrix panel graphics are operated using an open-source C++ tool called openFrameworks.

If you want to get a closer look at this Raspberry Pi project in action, you can check out the original thread shared to Reddit or see the demo video on YouTube. If you want to get to know the mastermind behind these crazy cool clocks, you can check out our interview with him over at our Raspberry Pi-themed podcast, The Pi Cast.

A recent attempt to fully map a mere cubic millimeter of a human brain took up 1.4 petabytes of storage just in pictures of the specimen. A collaborative effort between Harvard researchers and Google AI experts took the deepest dive yet into neural mapping with the recent full imaging and mapping of the brain sample, making puzzling discoveries and utilizing incredible technology. We did the back-of-napkin math on what ramping up this experiment to the entire brain would cost, and the scale is impossibly large — 1.6 zettabytes of storage costing $50 billion and spanning 140 acres, making it the largest data center on the planet.

The study is full of mind-numbing stats. To image a human brain, the researchers needed to utilize Google's machine learning tech, shaving estimated years off of the project. Scientists first sliced the sample into 5000 wafers orders of magnitude thinner than human hair. Electron microscope images were taken of each slice, which were recombined to count around 50,000 cells and 150 million synapses, the connection points where neurons meet and interact. To recombine these images and map the fibers and cells accurately, Google's AI imaging tech had to be used, digitally working out the routes of the gray matter. 

The synthesized images revealed many exciting secrets about the brain that were previously totally unknown — some cell clusters grew in mirror images of one another, one neuron was found with 5,000+ connection points to other neurons, and some axons (signal-carrying ends of nerves) had become tightly coiled into yarn ball shapes for totally unknown reasons. These and other discoveries made in the course of research excited their scientists beyond reason. Jeff Lichtman, a Harvard professor said to The Guardian on the project, "We found many things in this dataset that are not in the textbooks. We don’t understand those things, but I can tell you they suggest there’s a chasm between what we already know and what we need to know."

For context on the size of the brain sample and the data collected from it, we need to get into mind-numbingly colossal numbers. The cubic millimeter of brain matter is only one-millionth of the size of an adult human brain, and yet the imaging scans and full map of its intricacies comprises 1.4 petabytes, or 1.4 million gigabytes. If someone were to utilize the Google/Harvard approach to mapping an entire human brain today, the scans would fill up 1.6 zettabytes of storage. 

Taking these logistics further, storing 1.6 zettabytes on the cheapest consumer hard drives (assuming $0.03 per GB) would cost a cool $48 billion, and that's without any redundancy. The $48 billion price tag does not factor in the cost of server hardware to put the drives in, networking, cooling, power, and a roof to put over this prospective data center. The roof in question will also have to be massive; assuming full server racks holding 1.8 PB, the array of racks needed to store the full imaging of a human brain would cover over 140 acres if smushed together as tightly as possible. This footprint alone, without any infrastructure, would make Google the owner of one of the top 10 largest data centers in the world, even approaching (if not reaching) the scale of Microsoft and OpenAI's planned $100 billion AI data center. 

All of this is to say the human brain is an impossibly dense and very smart piece of art, and the act of mapping it would be both impossibly expensive (we didn't even begin to guess how long it would take) and likely foolish. Just because mapping is done does not mean that scientists would know what to do with the maps, as just the one-millionth piece of the brain we have mapped will pose questions for researchers for likely years to come. Thankfully, we apparently don't need to know everything about the brain to start meddling with it; in case you missed it, Elon Musk's Neuralink has begun rolling out to very early adopters. And if you want more on Google's efforts in the AI space, OpenAI is not playing very nice with them today.

Apple yesterday announced the latest iteration of its own silicon with the M4-powered iPad Pros. The company claims that the CPU is 50% percent faster than the M2 on the 12.9-inch iPad Pro (6th-generation), sporting two more cores (4 performance + 6 efficiency) over the last generation’s eight cores (4 performance + 4 efficiency). Although we cannot yet confirm these claims, the M4’s Neural Engine was tested today on Geekbench, with machine learning (ML) benchmarks posted under iPad 16,3. As ever with leaks and unverified benchmarks, take the news with a pinch of salt.

The results do not explicitly say iPad Pro (13-inch), but the specifications noted on the benchmark match that of the higher-end M4-powered iPad: a 10-core ARM processor with 16GB RAM. We also compared it with the 12.9-inch iPad Pro (6th generation)’s machine learning test results to see how it would stack up against, and the results show an incremental improvement over the last gen’s ML performance.

The M4 iPad Pro hit a Geekbench ML 0.6.0 score of 9234, around 22.9% better than the M2-powered iPad Pro’s 7511. We also compared the M4 iPad Pro’s ML performance with an M3 14-inch MacBook Pro, where its gains were a little more modest at 10.4% (8365 vs. 9234). However, the M3 MacBook Pro might have a slight advantage over the M4 iPad Pro owing to its larger chassis and battery.

The original M1 Apple silicon was a true generational leap over Intel’s x86 processors. However, the subsequent M2, M3, and M4 chips are just building on the M1’s architecture instead of offering massive gains. Nevertheless, the M4 focuses heavily on AI processing, with its 16-core neural engine supporting up to 38 trillion operations per second (TOPS). In comparison, the previous-gen M3 also has a 16-core neural engine. Apple's M3 was rated for 18 TOPS at FP16 precision, but the M4 is rated for 38 TOPS with INT8. That means, if equalized to INT8 precision, we're looking at a 5% improvement in TOPS for the M4 over the M3.  

However, the May 7 presentation mainly focused on hardware — the new iPad Air, iPad Pro, and the Apple Pencil Pro. We’ve only received a few snapshots of what AI features the M4 silicon will bring us, mostly on its use with Final Cut Pro for iPad 2 and Logic Pro for iPad 2. Aside from that, there wasn’t much mention of AI; we expect that to happen during WWDC 2024 instead.

Apple still has the upper hand in mobile computing, especially as Windows laptops still lag behind in terms of efficiency. But Qualcomm isn’t far behind, with the Snapdragon X Elite and X Plus chips beating AMD and Intel’s top laptop processors. Assuming Windows 11 (or Windows 12) can take proper advantage of these new Arm processors, we’re entering a new golden age of powerful yet highly efficient laptops, and Apple has to step up its game to remain in the lead.

As much as we love seeing the Raspberry Pi used for fun gaming projects and entertainment purposes, it's really cool to see it used in something like the medical industry. Today we've got a really unique project to share with a lot of helpful potential. Maker and developer Shebin Jose Jacob is using a Raspberry Pi 5 to drive his AI-powered stethoscope.

The  standard stethoscope is a device to help you hear sounds within the body by focusing the acoustic pressure waves to the listener's ears via tubes. You might be wondering how AI comes into play. According to Jacob, the stethoscope uses machine learning to evaluate irregularities in heartbeats. If an abnormal rhythm is detected, the AI system can trigger an alert.

Jacob goes on to explain that this process is called auscultation. Using models trained with complex algorithms, AI can recognize healthy patterns from potential areas of concern. This makes it possible to use the Raspberry Pi stethoscope in a professional setting with not just raw data but a head start on troubleshooting.

Hardware-wise, the system was first designed using a Raspberry Pi 5. Because this is the latest model, it works wonders when it comes to performance. However, Jacob wanted to make the unit mobile so the final build uses a Raspberry Pi Zero 2 W. It's connected to an omnidirectional microphone that stands in where the bell and chest piece are on a regular stethoscope. A 400mAh LiPo battery keeps things portable.

The AI system was built using VIAM which lets you create smart devices from scratch with lots of room for customization. It's working alongside TensorFlow—more specifically TensorFlow Lite on the Raspberry Pi Zero 2 W.

If you want to see this Raspberry Pi project in action, you can check it out over at Hackster. There you'll also find a complete build guide and breakdown of its construction.

An engineer has shared his journey in “building a GPU from scratch with no prior experience.” As with his prior project of designing a CPU from scratch, Adam Majmudar took just two weeks to complete this cerebral feat. In a Twitter/X thread Majmudar takes us through the process, step-by-step, and admits GPU designing was a much harder task than expected. To be clear, the current conclusion of the project is a chip layout in Verilog which was finally passed through OpenLane EDA software to verify it. However, the GPU is going to be submitted for tapeout via Tiny Tapeout 7 so is destined to become a physical chip in the coming months.

Above you can see the flow of tasks Majmudar worked through to design his GPU. Yet, as a ‘from scratch’ project, a lot of study and thought was required even before the first step was tentatively taken. Last time we highlighted the engineer’s concerns that GPUs would be a relatively difficult field of study, due to the dominance of proprietary tech, as that prediction came true.

Through several iterations of the above architecture, Majmudar decided to focus on general-purpose parallel computing (GPGPUs) capabilities. Thus he adjusted his Instruction Set Architecture (ISA), which features just 11 instructions, to achieve this goal. Next up, the engineer wrote two matrix math kernels to run on his GPU. These matrix addition and multiplication kernels would demonstrate the key functionality of the GPU and provide evidence of its useful application in graphics and machine learning tasks.

It had been relatively easy for the engineer so far, but building his GPU in Verilog presented “many issues.” Advice from the (in)famous George Hotz helped Majmudar move past one of his first (and second) hurdles regarding memory and a warp-scheduler implementation. A third rewrite of his code did the trick though, fixing compute core execution scheduling.

Some more unspecified redesigns later and the proof of the pudding, a video showing the matrix addition kernel running and validating, was shared in the Tweet thread.

Lastly, the completed Verilog design was passed through OpenLane EDA, targeting the Skywater 130nm process node (for Tiny Tapeout). Again some issues needed to be ironed out. In particular, Majmudar explains that some Design Rule Checks (DRCs) failed and necessitated rework.

After the two-week effort, the engineer enjoyed playing with a cool 3D visualization of his GPU design. That will have to suffice until TT7 returns silicon to participants. Of course, the work isn't going to rank among the best graphics cards. If you want to read more about this homemade GPU check out the entertaining social media thread and / or investigate the dedicated Tiny-GPU GitHub page.

Exercise isn't always easy, let alone building a routine. But what if you could make the process a little bit easier to chew? That's exactly what maker and developer Sir Walter Richardson has done with his Robo DeMo 1000 robot. This Raspberry Pi-powered robotic assistant joins you while you run and uses AI to determine when you or passing runners need a little extra encouragement.

So far, the robot is designed to locomote using wheels rather than legs. It utilizes a few different areas of interest including text to speech, image recognition, and machine learning to help operate. Everything compiles into a system designed to evaluate those who pass by determining whether or not they need cheering on.

This system uses a combination of factors to evaluate the status of passing runners. A Logitech C920 webcam enables Robo DeMo 1000 to collect images in real time so it can scan for things like how fast runners are traveling or what they're wearing. If Robo DeMo 1000 thinks they need a boost, it uses text to speech to shout at the runner to motivate them. In a twist, however, any runner who looks a bit too cocky will get teased with a demotivational phrase.

The main board powering the operation is a Raspberry Pi 5. It's designed to work alongside a Seeed Studio Grove 16-channel PWM PCA9685 driver and a DBH-12V motor controller to drive the servo motor for the wheels. An audio amplifier board allows for audio output and an Xbox controller lets you steer Robo DeMo 1000 around.

According to Richardson, Robo DeMo 1000 uses a handful of tools ranging from custom Python code to building a smart device system with Viam. To flesh out the text-to-speech function, Richardson is using Evenlabs. Early designs of the body have been shared, Richardson confirms that it was designed using Fusion 360.

If you want to get a closer look at this Raspberry Pi project, you can see it in action over at YouTube. Follow Richardson over at Hackster for additional information as well as future updates to Robo DeMo 1000. Until then, you'll just have to convince yourself to go out and hit the trail.

Researchers have created an ultra-low power 2D transistor to mimic the collision-avoidance neurons of a locust in their autonomous robots. Scientists from the Indian Institute of Technology Bombay and King's College London collaborated on the study to explore low-power solutions for autonomous robots and vehicles, which are growing in prominence. 

Autonomous driving and motion have long been a holy grail for machine learning and AI developers and researchers, and collision avoidance is the key to making the tech feasible in the real world. To this end, the IITB and King's College students set out with the goal of creating a collision solution on extremely low power.

In studying collision avoidance, scientists discovered a collision-detecting neuron in locusts. Called LGMD (lobula giant movement detector), this neuron spikes when large objects come near the locust, helping the insect avoid danger. This neuron was able to be duplicated by scientists with incredibly thin two-dimensional transistors, which also produce spikes analogous to the locust neuron and for a similar energy cost: less than 100 picojoules (for context, running a 100 W incandescent light bulb for one second costs 100 joules of energy). The thin and cheap transistor was also fully functional, being able to be reprogrammed to look for different types of movement and successfully avoid obstacles with high degrees of accuracy.

A 2D transistor is an impossible dream for large-scale chip manufacturers, as when transistors become smaller, they also become more energy-efficient. Of course, the transistor used in the IITB study is very simple, spiking when movement is detected within a range and nothing more. But the authors have a vision for where this two-dimensional tech can go after this study.

These super-efficient transistors could help greatly with the energy cost of the often-inefficient AI technologies we have available today. Professor Bipin Rajendran, at King's College London and co-author of the study, writes “We demonstrated that this spiking neuron circuit can be used for obstacle detection. However, the circuit can be used in other neuromorphic (systems mimicking the human brain) applications based on analog or mixed signal technology that require a low-energy spiking neuron.”

If you're curious about more details and the scientists behind the study, you can check out the study here. We've also written much about AI moving itself around places recently. Check out our piece about ChatGPT trying to play Red Dead Redemption 2, or perhaps about how China has used an Nvidia chip to hypersonic weapons for better autonomous flight.

Intel released a new update for XeSS, version 1.3, which offers significant top to bottom changes and improvements. It boasts higher performance, increased image fidelity, new scaling factors, and new modes of operation for Intel's best graphics cards. The updates come from continued training of the models, and apply to both the XMX (Xe Matrix eXtensions) and DP4a (INT8) modes of operation, with up to 28% higher performance on Arc desktop GPUs.

The key improvements in XeSS 1.3 revolve around its updated AI models for the upscaler. Through additional training and optimizations of its deep learning algorithm, Intel has been able to squeeze out more performance while simultaneously improving image quality in areas where XeSS 1.2 (and prior versions) was weak. Specifically, XeSS 1.3 provides better anti-aliasing, less ghosting and more temporal stability than its predecessors.

Intel XeSS is now included in over 100 games, though it's important to note that there's no indication on that page as to what version of XeSS is in use. The original XeSS 1.0 was generally lacking in image quality, particularly when running in DP4a mode (i.e. on non-Intel Arc GPUs). 1.1 and 1.2 offered quite a few improvements, though many games still only use 1.0. As for 1.3, nothing officially supports it yet, but we'll likely see at least a few games add support in the next month or two.

Intel shared a video comparison between XeSS 1.3 and 1.2 running side by side in Like a Dragon: Ishin!, which we've embedded above. Intel’s XeSS 1.3 implementation provides noticeably better image reconstruction on the bamboo curtain in the back, which emits horrible flickering on the older XeSS 1.2 model. XeSS 1.3 rectifies most of the flickering, making the image far more stable, though there's still some slight flickering present, particularly as the scene starts to move.

Intel's internal testing of XeSS 1.3 shows it provides up to a 28% improvement in framerate, using a desktop Arc A750. Intel also shows performance on a laptop equipped with a Core Ultra 7 155H and and integrated Arc Graphics GPU. Testing was done with eight modern games: Hitman 3, Cyberpunk 2077, Hogwarts Legacy, The Witcher 3 (Next-Gen Update), Call of Duty: Modern Warfare 3, Diablo IV, and Ghostrunner 2. All testing was done at 1440p Ultra with RT on with the Arc A750, and 1080P Medium with the Ultra 7 155H, using Performance mode upscaling (half the vertical and horizontal resolution, or 4X overall pixel upscaling).

While Diablo IV saw the largest performance boost, it's something of an outlier. Overall, framerates improved by 10% average on the Arc A750, ranging from just 3.2% in 5% in Ghostrunner 2 to Diablo IV's 28%. Most of the games show single digit percentage improvements. Note also that these are internal test versions of the games that Intel created, and may or may not reflect upcoming support for XeSS 1.3.

On the Core Ultra 7 155H with Arc integrated graphics, performance improved by 8% on average. The gains were somewhat more consistent here, ranging from 5% in Hitman 3 to as much as 12% in Hogwarts Legacy 12%.

The performance improvements appear to chiefly come via a change in the scaling factors. Every mode uses more upscaling than before, depending on the AI training to make up the difference in quality. Note that the new Ultra Quality Plus uses the same 1.3X scaling as the previous Ultra Quality mode, and there's a 0.2X–0.3X increase in the scaling factors (0.4X–0.9X more pixels generated rather than rendered) across all presets.

XeSS 1.3 also introduces three new presets: Native AA, Ultra Quality Plus, and Ultra Performance. Native AA and Ultra Performance bring Intel up to parity with FSR 2/3 and DLSS 2/3 options, while the Ultra Quality Plus option is for those who want maximum image quality with only a small amount of upscaling.

As with FSR3 and DLAA, the Native Anti-Aliasing preset runs at native resolution and is designed purely to improve image quality via anti-aliasing and sharpening without any upscaling. Ultra Performance mode is mostly for the highest display resolutions and matches DLSS and FSR with a 3.0X ratio — in other words, that would mean upscaling 1280x720 to 3840x2160, or alternative if you're after performance at a lower target resolution, upscaling 640x360 rendered content to 1920x1080. Based on what we've seen with FSR and DLSS, we wouldn't expect great image quality with the Ultra Performance mode.

On top of this, XeSS 1.3 now supports dynamic resolution scaling, just like DLSS and FSR. This allows developers and gamers to target a specific frame rate and then let XeSS change its render resolution to compensate.

Compared to AMD and Nvidia, Intel's new presets end up rendering fewer pixels. We're not entirely sure what to think about that, as we haven't had a chance to test XeSS 1.3 ourselves, but it certainly stinks of marketing. We were happier with Intel's prior XeSS scaling factors and names, as we could then let the image quality speak for itself while knowing that AMD, Intel, and Nvidia GPUs were all rendering similar numbers of pixels at each level. Intel's competitors can also improve performance via a similar tactic, but we'd rather they didn't.

Still, it's good to see Intel continuing to innovate with XeSS to make it more competitive with DLSS, as it is the only other image upscaler that currently utilizes deep learning technology. These improvements should put XeSS ahead of FSR 2/3 in terms of image quality as well, though AMD FSR 3.1 will be coming soon, and there are hints that AMD could be exploring deep learning and AI techniques for a future revision (FSR 4.0 perhaps). Nvidia also just released DLSS 3.7, though it's less forthcoming about the underlying changes.

All the changes with XeSS 1.3 apply to both XMX and DP4a modes, though we would expect XMX will bring the best results overall. It's optimized specifically for Intel’s XMX AI cores, where DP4a mode is used on Intel integrated graphics (including the new Arc iGPUs, which lack XMX cores) and GPUs from other vendors. It will be interesting to see how XeSS 1.3 performance and image quality compare to FSR 3 when running on AMD hardware.

Hopefully, we’ll get a chance to test all of these latest upscaling iterations soon, once we find a title that implements all three updates — but that might not be for a while. Microsoft is also working on a way to standardize upscaling integration into games on PC, through DirectSR, and there's also Nvidia's Streamline SDK that further simplifies upscaling in games today. Who knows, with these additions, we might see a game with all three of the latest upscaling solutions sooner than later. 

Yesterday, IBM released a blog post detailing its technology for AI-enhanced protection against malware including ransomware on its SSDs, the fourth generation of IBM's FlashCore Module (FCM) technology. As detailed by them, the latest revision of FCM (FCM4) now supports artificial intelligence, but applied for the purpose of detecting and responding to cybersecurity threats as they arise.

Previous generations of FCM are already capable of scanning all incoming data without impacting performance, but lack the enhanced features of AI. FCM4 monitors statistics for every single I/O operation, and uses machine learning to detect threats like ransomware in under a minute.

This approach from IBM joins the likes of other self-protecting SSD storage. For example those made by Cigent and Phison and other, more performance-intensive hardware protection methods. 

Focusing on IBM's solutions, though, let's talk about more than just threat detection. By measuring data parameters like compressibility, randomness and entropy, the IBM Storage Insights software can alert users to an anomaly. The FCM4 technology gathers real-time IO data which machine learning models use to determine a threat.  By integrating FCM with IBM's Storage Defender Software, IBM can leverage AI detection and data recovery operations on both the software side and the hardware side.

Of course, making the most of these technologies and their assorted backup, restore, and protection features is currently limited to high-end applications. While day-to-day SSDs may one day see protection like this, IBM's FCM technology and corresponding software is targeted squarely at enterprise and professional users, particularly ones needing to deal with high-sensitivity or confidential information.

By dramatically increasing the speed at which ransomware and other malicious activities can be detected, removed, and repaired from storage, IBM has shown that machine learning AI actually can be the ideal choice for some workloads. 

While the ethics and morals of "generative AI" in art, music and literature are quite rightly being debated, the use of AI in this application means better security for enterprise users. Even the most seasoned IT security pros would be hard-pressed to detect and start reversing a ransomware attack within a single minute, but a job like that might actually be perfect for these ever-evolving machine learning models.

Intel's Arc Alchemist GPUs can run large language models like Llama 2, thanks to the company's PyTorch extension, as demoed in a recent blog post. The Intel PyTorch Extension, which works on both Windows and Linux, allows LLMs to take advantage of the FP16 performance on Arc GPUs. However, given that Intel says you'll need 14GB of VRAM to use Llama 2 on Intel hardware, it means you'll probably want an Arc A770 16GB card.

PyTorch is an open-source framework, developed by Meta, for machine learning that can then be used to work on LLMs. While this software works out of the box, it's not coded by default to take full advantage of every piece of hardware, which is why Intel has its PyTorch extension. This software is designed to take advantage of the XMX cores inside Arc GPUs, and saw its first release in January 2023. Similarly, AMD and Nvidia both have optimizations for PyTorch for optimization purposes.

In its blog post, Intel demonstrates the performance capabilities of the Arc A770 16GB in Llama 2 using the latest update to Intel's PyTorch extension, which came out in December and specifically optimized FP16 performance. FP16, or half-precision floating point data, exchanges precision for performance, which is often a good tradeoff for AI workloads.

The demo shows Llama 2 and the dialogue-focused Llama 2-Chat LLMs, asking questions like "can deep learning have such generalization ability like humans do?" In response, the LLM was surprisingly humble and said deep learning wasn't on the same level as human intelligence. However, in order to run LLMs like Llama 2 with FP16 precision, you'll need 14GB of VRAM according to Intel, and we also didn't get any numbers on how quickly it responded to inputs and queries.

While this demo only showcases FP16 performance, Arc Alchemist also has BF16, INT8, INT4, and INT2 capabilities. Of these other data formats, BF16, is of particular note, as it's often considered to be even better for AI workloads thanks to its wider numerical range, which is on par with FP32 at eight bits while FP16 just has five. Optimizing BF16 performance could be high up on Intel's list for its next PyTorch extension update.

Australia’s National Supercomputing and Quantum Computing Innovation Hub is set to use Nvidia Grace Hopper Superchips to push the boundaries of quantum computing. In a news release sent to Tom’s Hardware, Nvidia says that the Pawsey Supercomputing Research Centre in Perth will deploy eight Nvidia Grace Hopper Superchip nodes to power the open-source CUDA Quantum computing platform. It is expected that the new supercomputer will be able to deliver up to 10x higher processing performance than the center has access to now.

The stated purpose of the Grace Hopper Superchip nodes in Pawsey is for researchers at the center to run powerful simulation tools and hopefully make breakthroughs in fields like algorithm discovery, device design, quantum machine learning, chemistry simulations, image processing for radio, astronomy, financial analysis, bioinformatics, and more. It is also hoped to advance scientific exploration in Australia and the world.

The Nvidia Grace Hopper Superchip’s Grace CPU and Hopper GPU architectures are central to the above aspirations and the Nvidia cuQuantum software development kit. This powerful hardware and software melding forms the green team’s open-source hybrid quantum computing platform, known more succinctly as the CUDA Quantum platform.

At Pawsey, eight Grace Hopper Superchip nodes based on the Nvidia MGX modular architecture will be deployed, according to the press release we received. It explains that “GH200 Superchips eliminates the need for a traditional CPU-to-GPU PCIe connection by combining an Arm-based Nvidia Grace CPU with an Nvidia H100 Tensor Core GPU in the same package, using Nvidia NVLink-C2C chip interconnects MGX modular architecture.” A significant benefit of the new interconnects is that the bandwidth between the GPU and CPU is seven times greater than the latest PCIe technology. Moreover, the researchers in Australia are looking forward to a ten-fold increase in application performance when processing data sets measured in terabytes.

We asked Nvidia for some more technical details about the Superchip nodes at Pawsey. It turns out that each node will be using 'just' a single GH200 with Grace CPU and a H100 96GB of HBM3. Thus, the new installation at Pawsey Supercomputing Research Centre in Perth will feature eight nodes each with one GH200 for a total of 8x GH200 (8x Grace CPU and 8x H100 96GB GPU).

One of the other major appealing features of the Nvidia CUDA Quantum platform is that it offers a hybrid solution bridging the worlds of quantum and classical computing. Nvidia claims it is a first-of-its-kind and “enables dynamic workflows across disparate system architectures.” Researchers can use this platform to integrate and program quantum processing units (QPUs), GPUs, and CPUs in one system. It is also, of course, GPU-accelerated for scalability and performance.

The installation of the new Nvidia Grace Hopper Superchip nodes at Pawsey isn’t purely for advancing knowledge or solving some esoteric scientific problems. The Australian government also reckons investments like this make good business sense. According to Australia’s national science agency, the domestic market opportunity offered by quantum computing is set to be worth $2.5 billion per annum. Additionally, it is estimated that quantum advances could create 10,000 new Australian jobs by 2040.

AMD's has released ROCm 6.0.2, its open-source stack platform, along with Linux driver 23.40.2 expanding support for desktop client GPUs and adding ONNX Runtime support. This update broadens the accessibility of artificial intelligence (AI) development tools to a wider range of users and improves the flexibility of AI model development and inference.

Starting from the ROCm 5.7, AMD's software platform for GPU-accelerated computing already gained support for the Radeon RX 7900 XT, Radeon RX 7900 XTX, and Radeon Pro W7900 boards, which are among the best graphics cards available today. Now, ROCm 6.0.2 expands this support to the Radeon Pro W7800 and the Radeon RX 7900 GRE. This expands the number of developers who could use AMD's ROCm 6.0 platform for their AI development efforts on their desktops. The Linux driver is compatible with 64-bit Ubuntu 22.04.3 HWE.

Another notable addition in the ROCm 6.0 platform is support for ONNX Runtime (Open Neural Network Exchange Runtime). This feature enables conversion of AI models between various machine learning frameworks, which greatly enhances the interoperability and flexibility of AI development. As a result, users can now perform inference tasks on a wider range of source data directly on local AMD Radeon hardware, streamlining the workflow for AI model deployment.

The update also adds INT8 data type to MIGraphX, AMD's proprietary graph inference engine, expanding the range of supported data types (which currently includes FP32 and FP16) to enhance flexibility and performance. The inclusion of INT8 support is particularly beneficial for applications that require lower precision but higher throughput.

Finally, AMD's ROCm 6.0 brings mixed precision capabilities with FP32/FP16 to PyTorch machine learning training workflows, which enables faster model development and iteration.

"These are exciting times for anyone deciding to start working on AI," wrote David Diederichs, Product Marketing Manager of Pro Software at AMD, in a blog post. "ROCm for AMD Radeon desktop GPUs is a great solution for AI engineers, ML researchers and enthusiasts alike and no longer remains exclusive to those with large budgets. AMD is determined to keep broadening hardware support and adding more capabilities to our Machine Learning Development solution stack over time."

The developer preview of DirectML DirectX12 extension and Windows' machine learning API has added support for Intel Core Ultra CPUs and their NPU AI hardware as of the release of DirectML 1.13.1 and ONNX Runtime 1.17. DirectML's general purpose is to provide a low-level API and abstraction layer for machine learning accelerators (Direct "Machine Learning"), whether discrete GPUs or integrated NPUs. Thus, adding support for Intel's Core Ultra NPUs was pretty much just a matter of time.

According to the original Windows developer blog post and the related Intel blog post it links to, this release was developed with Samsung's assistance and open source models. According to Hwang-Yoon Shim, VP and Head of New Computing H/W R&D Group at Samsung, "Windows DirectML is one of the most efficient ways for Samsung's developers to make those (efficient machine learning) experiences for Windows."

While this initial release of NPU support for Intel Core Ultra CPUs is a landmark for DirectML, it does come with many compromises as a developer-first release. Limitations listed on the original page dictate that only some machine learning models are supported by DirectML and that only Intel Core Ultra NPUs will be supported for this particular release. According to Microsoft, machine learning models excluded were excluded due to stability or accuracy issues, including issues like not running at all.

Improved support for the Intel Core Ultra NPUs on Windows should help push the image of so-called "AI PCs" forward, which pretty much describes regular PCs with onboard AI acceleration hardware of some kind. While GPUs have, of course, been doing AI acceleration since the initial release of Nvidia RTX in 2018, the latest CPUs have only just started, so it'll take a little time before all the involved software catches up.

As Intel Core Ultra CPUs continue to roll out with the release of Intel's Meteor Lake architecture, support for Core Ultra's onboard AI hardware should continue to improve. Meteor Lake has more to offer than just improved AI performance, though— general power efficiency and especially iGPU performance have also improved on the new architecture compared to Intel's last few generations.

Elon Musk, CEO of Tesla, said his company was procuring artificial intelligence (AI) processors worth billions of dollars for its artificial intelligence training workloads. Nvidia's AI GPUs are on the top of Musk's list, but they are followed by the company's own Dojo processors and hardware from AMD, which already supplies Tesla system-on-chips for infotainment.

"A Dojo Supercomputer [is worth] $500 [million], while a large sum of money, [it] is only equivalent to a 100,000-unit H100 system from Nvidia," Elon Musk, chief executive of Twitter, said in an X post.

A data center full of Tesla's Dojos costing $500 million is an impressive development, considering they will be used for computer vision, video recognition processing, and machine learning. But the company is spending more than $500 million on Nvidia's hardware, presumably H100 and H200 GPUs now and then Blackwell-based B100 late this year. It is noteworthy that Teals only procured 15 thousand H100 GPUs last year, according to Omdia, so apparently, the company is accelerating its Nvidia-powered efforts.

"Tesla will spend more than that on Nvidia hardware this year," Musk wrote. "The table stakes for being competitive in AI are at least several billion dollars per year at this point."

What is intriguing is that he answered positively (albeit laconically) when asked whether Tesla was procuring AI hardware from AMD. At present, AMD has multiple offerings for AI, including its Instinct MI200, Instinct MI250, and Instinct MI300X accelerators, as well as the Instinct MI300A accelerated processing unit which combines Zen 4 x86 general-purpose cores as well as CDNA 3-based clusters (or rather chipsets) for computing.

According to AMD's reported performance numbers, the Instinct MI300X beats Nvidia's H100 80GB in AI and HPC performance. Meanwhile, the H100 80GB is currently widely used by significant hyperscalers such as Google, Meta (formerly Facebook), and Microsoft. Workloads run on H100 will be scaled out to other H100s, so AMD's MI300X does not precisely have to compete against H100, at least for existing customers and workloads. Yet again, performance numbers demonstrated by AMD could indicate that the Instinct MI300X will likely be a strong rival for Nvidia's upcoming H200 141GB GPU.

The latest update for Intel's LLVM compiler adds support for Xe-HPCVG graphics devices and Ponte Vecchio VG, a variant of Ponte Vecchio (via Coelacanth's Dream). The new VG variant of both the Xe-HPC architecture and Ponte Vecchio comes with disabled DPAS instructions (Dot Product Accumulate Systolic), crucial for AI workloads and normally accelerated by Xe Matrix Extension (XMX) units.

In many ways, Xe-HPCVG is unchanged from Xe-HPC: It still has 128KB of shared local memory, double-precision FP64 computing, and half-precision BF16 support. However, the key difference is the explicit lack of DPAS instructions, which are all AI-accelerated versions of half-precision instructions like BF16, FP16, and INT8. These instructions are critical for good AI and machine learning performance.

That a version of Ponte Vecchio comes without DPAS instructions is definitely strange, as the chip is largely intended for AI workloads. The lack of DPAS instructions implies that there are no functioning XMX units on this VG variant of Ponte Vecchio. It's hard to imagine Intel would make a new version of Ponte Vecchio specifically to remove one of its primary features, so Intel may just be repurposing Ponte Vecchio chips with defective XMX units for use in an upcoming GPU.

It's also unclear what the 'VG' means, though it could be for something like the 'SDV' in Ponte Vecchio SDV (which stands for Software Development Vehicle). Unfortunately, the update gives us very little to go on since it's mostly just the removal of specific support for DPAS instructions and not much else.

If Intel is preparing to launch an XMX-less version of Ponte Vecchio, we're not sure about its intended market. XMX units, like Nvidia's Tensor cores, have proven to be very important for AI workloads. Without those, Ponte Vecchio VG is left with FP64 and platform-related features. 52 TFLOPs of FP64 performance is still impressive and beats out Nvidia's H100 at 26 TFLOPs, but it's well behind AMD's MI300X at 81.7 TFLOPs.

Ponte Vecchio represents a massive investment in chip stacking and EMIB for Intel. The full solution consists of 47 different 'tiles' (chips) manufactured on various process nodes from Intel and TSMC. The XMX instructions are an integral part of the compute tiles, so it seems unlikely for defects to only affect the XMX portions of the chips. More likely, this is an intentional decision on Intel's part to disable that feature, but to what end remains unknown. We'll learn more in the future, when or if these VG parts come to market.

Stable Diffusion Introduction

Stable Diffusion and other AI-based image generation tools like Dall-E and Midjourney are some of the most popular uses of deep learning right now. Using trained networks to create images, videos, and text has become not just a theoretical possibility but is now a reality. While more advanced tools like ChatGPT can require large server installations with lots of hardware for training, running an already-trained network for inference can be done on your PC, using its graphics card. How fast are consumer GPUs for doing AI inference using Stable Diffusion? That's what we're here to investigate.

We've benchmarked Stable Diffusion, a popular AI image generator, on the 45 of the latest Nvidia, AMD, and Intel GPUs to see how they stack up. We've been poking at Stable Diffusion for over a year now, and while earlier iterations were more difficult to get running — never mind running well — things have improved substantially. Not all AI projects have received the same level of effort as Stable Diffusion, but this should at least provide a fairly insightful look at what the various GPU architectures can manage with AI workloads given proper tuning and effort.

The easiest way to get Stable Diffusion running is via the Automatic1111 webui project. Except, that's not the full story. Getting things to run on Nvidia GPUs is as simple as downloading, extracting, and running the contents of a single Zip file. But there are still additional steps required to extract improved performance, using the latest TensorRT extensions. Instructions are at that link, and we've previous tested Stable Diffusion TensorRT performance against the base model without tuning if you want to see how things have improved over time. Now we're adding results from all the RTX GPUs, from the RTX 2060 all the way up to the RTX 4090, using the TensorRT optimizations.

For AMD and Intel GPUs, there are forks of the A1111 webui available that focus on DirectML and OpenVINO, respectively. We used these webui OpenVINO instructions to get Arc GPUs running, and these webui DirectML instructions for AMD GPUs. Our understanding, incidentally, is that all three companies have worked with the community in order to tune and improve performance and features.

Whether you're using an AMD, Intel, or Nvidia GPU, there will be a few hurdles to jump in order to get things running optimally. If you have issues with the instructions in any of the linked repositories, drop us a note in the comments and we'll do our best to help out. Once you have the basic steps down, however, it's not too difficult to fire up the webui and start generating images. Note that extra functionality (i.e. upscaling) is separate from the base text to image code and would require additional modifications and tuning to extract better performance, so that wasn't part of our testing.

Additional details are lower down the page, for those that want them. But if you're just here for the benchmarks, let's get started.

Stable Diffusion 512x512 Performance

This shouldn't be a particularly shocking result. Nvidia has been pushing AI technology via Tensor cores since the Volta V100 back in late 2017. The RTX series added the feature in 2018, with refinements and performance improvements each generation (see below for more details on the theoretical performance). With the latest tuning in place, the RTX 4090 ripped through 512x512 Stable Diffusion image generation at a rate of more than one image per second — 75 per minute.

AMD's fastest GPU, the RX 7900 XTX, only managed about a third of that performance level with 26 images per minute. Even more alarming, perhaps, is how poorly the RX 6000-series GPUs performed. The RX 6950 XT output 6.6 images per minute, well behind even the RX 7600. Clearly, AMD's AI Matrix accelerators in RDNA 3 have helped improve throughput in this particular workload.

Intel's current fastest GPU, the Arc A770 16GB, managed 15.4 images per minute. Keep in mind that the hardware has theoretical performance that's quite a bit higher than the RTX 2080 Ti (if we're looking at XMX FP16 throughput compared to Tensor FP16 throughput): 157.3 TFLOPS versus 107.6 TFLOPS. It looks like the Arc GPUs are thus only managing less than half of their theoretical performance, which is why benchmarks are the most important gauge of real-world performance.

While there are differences between the various GPUs and architecture, performance largely scales proportionally with theoretical compute. The RTX 4090 was 46% faster than the RTX 4080 in our testing, while in theory it offers 69% more compute performance. Likewise, the 4080 beat the 4070 Ti by 24%, and it has 22% more compute.

The newer architectures aren't necessarily performing substantially faster. The 4080 beat the 3090 Ti by 10%, while offering potentially 20% more compute. But the 3090 Ti also has more raw memory bandwidth (1008 GB/s compared to the 4080's 717 GB/s), and that's certainly a factor. The old Turing generation held up as well, with the newer RTX 4070 beating the RTX 2080 Ti by just 12%, with theoretically 8% more compute.

Stable Diffusion 768x768 Performance

Kicking the resolution up to 768x768, Stable Diffusion likes to have quite a bit more VRAM in order to run well. Memory bandwidth also becomes more important, at least at the lower end of the spectrum.

The relative positioning of the various Nvidia GPUs doesn't shift too much, and AMD's RX 7000-series gains some ground with the RX 7800 XT and above, while the RX 7600 dropped a bit. The 7600 was 36% slower than the 7700 XT at 512x512, but dropped to being 44% slower at 768x768.

The previous generation AMD GPUs had an even tougher time. The RX 6950 XT didn't even manage two images per minute, and the 8GB RX 6650 XT, 6600 XT, and 6600 all failed to render even a single image. That's a bit odd, as the RX 7600 still worked okay with only 8GB of memory, but some other architectural difference was at play.

Intel's Arc GPUs also lost ground at the higher resolution, or if you prefer, the Nvidia GPUs — particularly the fastest models — put some additional distance between themselves and the competition. The 4090 for example was 4.9X faster than the Arc A770 16GB at 512x512 images, and that increased to a 6.4X lead with 768x768 images.

We haven't tested SDXL, yet, mostly because the memory demands and getting it running properly tend to be even higher than 768x768 image generation. TensorRT support is also missing for Nvidia GPUs, and most likely we'd see quite a few GPUs struggle with SDXL. It's something we plan to investigate in the future, however, as the results are generally preferable to SD1.5 and SD2.1 for higher resolution outputs.

For now, we know that performance will be lower than our 768x768 results. As an example of what to expect, the RTX 4090 doing 1024x1024 images (still using SD1.5), managed just 13.4 images per minute. That's less than half the speed of 768x768 image generation, which makes sense as the 1024x1024 images have 78% more pixels and the time required seems to scale somewhat faster than the resolution increase.

Picking a Stable Diffusion Model

Deciding which version of Stable Generation to run is a factor in testing. Currently, you can find v1.4, v1.5, v2.0, and v2.1 models from Hugging Face, along with the newer SDXL. The earlier 1.x versions were mostly trained on 512x512 images, while 2.x included more training data for up to 768x768 images. SDXL targets 768x768 to 1024x1024 images. As noted above, higher resolutions also require more VRAM. Different versions of Stable Diffusion can also generate radically different results from the same prompt, due to differences in the training data.

If you try to generate a higher resolution image than the training data, you can end up with "fun" results like the multi-headed, multi-limbed, multi-eyed, or multi-whatever examples shown above. You can try to work around these via various upscaling tools, but if you're thinking about just generating a bunch of 4K images to use as your Windows desktop wallpaper, be aware that it's not as straightforward as you'd probably want it to be. (Our prompt for the above was "Keanu Reeves portrait photo of old warrior chief, tribal panther make up, blue on red, side profile, looking away, serious eyes, 50mm portrait photography, hard rim lighting photography" — taken from this page if you're wondering.)

It's also important to note that not every GPU has received equal treatment from the various projects, but the core architectures are also a big factor. Nvidia has had Tensor cores in all of its RTX GPUs, and our understanding is that the current TensorRT code only uses FP16 calculations, without sparsity. That explains why the scaling from 20-series to 30-series to 40-series GPUs (Turing, Ampere, and Ada Lovelace architectures) mostly correlates with the baseline Tensor FP16 rates.

As shown above, performance on AMD GPUs using the latest webui software has improved throughput quite a bit on RX 7000-series GPUs, while for RX 6000-series GPUs you may have better luck with using Nod.ai's Shark version — and note that AMD has recently acquired Nod.ai. Throughput with SD2.1 in particular was faster with the RDNA 2 GPUs, but then the results were also different from SD1.5 and thus can't be directly compared. Nod.ai doesn't have "sharkify" tuning if you use SD1.5 models either, which resulted in lower performance with our apples to apples testing.

Test Setup: Batch Sizes

The above gallery shows some additional Stable Diffusion sample images, after generating them at a resolution of 768x768 and then using SwinIR_4X upscaling (under the "Extras" tab), followed by cropping and resizing. Hopefully we can all agree that these results look a lot better than the mangled Keanu Reeves attempts from above.

For testing, we followed the same procedures for all GPUs. We generated a total of 24 distinct 512x512 and 24 distinct 768x768 images, using the same prompt of "messy room" — short, sweet, and to the point. Doing 24 images per run gave us plenty of flexibility, since we could do batches of 3x8 (three batches of eight concurrent images), 4x6, 6x4, 8x3, 12x2, or 24x1, depending on the GPU.

We did our best to optimize for throughput, which means running batch sizes larger than one in many cases. Sometimes, the limiting factor in how many images should be generated concurrently is VRAM capacity, but compute (and cache) also appear to factor in. As an example, the RTX 4060 Ti 16GB did best with 6x4 batches, just like the 8GB model, while the 4070 did best with 4x6 batches.

For 512x512 image generation, many of Nvidia's GPUs did best generating three batches of eight images each (the maximum batch size is eight), though we did find that 4x6 or 6x4 worked slightly better on some of the GPUs. AMD's RX 7000-series GPUs all liked 3x8 batches, while the RX 6000-series did best with 6x4 on Navi 21, 8x3 on Navi 22, and 12x2 on Navi 23. Intel's Arc GPUs all worked well doing 6x4, except the A380 which used 12x2.

For 768x768 images, memory and compute requirements are much higher. Most of the Nvidia RTX GPUs worked best with 6x4 batches, or 8x3 in a few instances. (Note that even the RTX 2060 with 6GB of VRAM was still best with 6x4 batches.) AMD's RX 7000-series again liked 3x8 for most of the GPUs, though the RX 7600 needed to drop the batch size and ran 6x4. The RX 6000-series only worked at 24x1, doing single images at a time (otherwise we'd get garbled output), and the 8GB RX 66xx cards all failed to render anything at the higher target output — you'd need to opt for Nod.ai and a different model on those GPUs.

Test Setup

Our test PC for Stable Diffusion consisted of a Core i9-12900K, 32GB of DDR4-3600 memory, and a 2TB SSD. We tested 45 different GPUs in total — everything that has ray tracing hardware, basically, which also tended to imply sufficient performance to handle Stable Diffusion. It's possible to use even older GPUs, though performance can drop quite a bit if the GPU doesn't have native FP16 support. Nvidia's GTX class cards were very slow in our limited testing.

In order to eliminate the initial compilation time, we first generated a single batch for each GPU with the desired settings. Actually, we'd use this step to determine the optimal configuration for batch size. Once we settled on the batch size, we ran four iterations generating 24 images each, discarded the slowest result, and averaged the time taken from the other three runs. We then used this to calculate the number of images per minute that each GPU could generate.

Our chosen prompt was, again, "messy room." We used the Euler Ancestral sampling method, 50 steps (iterations), with a CFG scale of 7. Because all of the GPUs were running the same version 1.5 model from Stable Diffusion, the resulting images were generally comparable in content. We noticed previously that SD2.1 tended to often generate "messy rooms" that weren't actually messy, and were sometimes cartoony. SD1.5 also seems to be preferred by many Stable Diffusion users as the later 2.1 models removed many desirable traits from the training data.

The above gallery shows an example output at 768x768 for AMD, Intel, and Nvidia. Rest assured, all of the images appeared to be relatively similar in complexity and content — though I won't say I looked carefully at every one of the thousands of images that were generated! For reference, the AMD GPUs resulted in around 2,500 total images, Nvidia GPUs added another 4,000+ images, with Intel only needing about 1,000 images. All of the same style messy room.

Comparing Theoretical GPU Performance

While the above testing looks at actual performance using Stable Diffusion, we feel it's also worth a quick look at the theoretical GPU performance. There are two aspects to consider: First is the GPU shader compute, and second is the potential compute using hardware designed to accelerate AI workloads — Nvidia Tensor cores, AMD AI Accelerators, and Intel XMX cores, as appropriate. Not all GPUs have additional hardware, which means they will use GPU shaders. Let's start there.

For FP16 compute using GPU shaders, Nvidia's Ampere and Ada Lovelace architectures run FP16 at the same speed as FP32 — the assumption is that FP16 can and should be coded to use the Tensor cores. AMD and Intel GPUs in contrast have double performance on half-precision FP16 shader calculations compared to FP32, and that applies to Turing GPUs as well.

This leads to some potentially interesting behavior. The RTX 2080 Ti for example has 26.9 TFLOPS of FP16 GPU shader compute, which nearly matches the RTX 3080's 29.8 TFLOPS and would clearly put it ahead of the RTX 3070 Ti's 21.8 TFLOPS. AMD's RX 7000-series GPUs would also end up being much more competitive if everything were restricted to GPU shaders.

Clearly, this look at FP16 compute doesn't match our actual performance much at all. That's because optimized Stable Diffusion implementations will opt for the highest throughput possible, which doesn't come from GPU shaders on modern architectures. That brings us to the Tensor, Matrix, and AI cores on the various GPUs.

Nvidia's Tensor cores clearly pack a punch, except as noted before, Stable Diffusion doesn't appear to leverage sparsity with the TensorRT code. (It doesn't use FP8 either, which could potentially double compute rates as well.) That means, for the most applicable look at how the GPUs stack up, you should pay attention to the first chart for Nvidia GPUs, which omits sparsity, rather than the second chart that includes sparsity — also note that the non-TensorRT code does appear to leverage sparsity.

It's interesting to see how the above chart showing theoretical compute lines up with the Stable Diffusion charts. The short summary is that a lot of the Nvidia GPUs land about where you'd expect, as do the AMD 7000-series parts. But the Intel Arc GPUs all seem to get about half the expected performance — note that my numbers use the boost clock of 2.4 GHz rather than the lower 2.0GHz "Game Clock" (which is a worst-case scenario that rarely comes into play, in my experience).

The RX 6000-series GPUs likewise underperform, likely because doing FP16 calculations via shaders is less efficient than doing the same calculations via RDNA 3's WMMA instructions. Otherwise, the RX 6950 XT and RX 6900 XT should at least manage to surpass the RX 7600, and that didn't happen in our testing. (Again, performance on the RDNA 2 GPUs tends to be better using Nod.ai's project, if you're using one of those GPUs and want to improve your image throughput.)

What's not clear is just how much room remains for further optimizations with Stable Diffusion. Looking just at the raw compute, we'd think that Intel can further improve the throughput of its GPUs, and we also have to wonder if there's a reason Nvidia's 30- and 40-series GPUs aren't leveraging their sparsity feature with TensorRT. Or maybe they are and it just doesn't help that much? (I did ask Nvidia engineers about this at one point and was told it's not currently used, but these things are still a bit murky.)

Stable Diffusion, and other text to image generators, are currently one of the most developed and researched areas of AI that are still readily accessible to consumer level hardware. We've looked at some other areas of AI as well, like speech recognition using Whisper and chatbot text generation, but so far neither of those seem to be as optimized or used as Stable Diffusion. If you have any suggestions for other AI workloads we should test, particularly workloads that will work on AMD and Intel as well as Nvidia GPUs, let us know in the comments.

Although many companies are developing accelerators for artificial intelligence (AI) workloads, Nvidia's CUDA platform is currently unrivaled regarding AI support. As a result, demand for Nvidia-based AI infrastructure is high. To address it, Amazon Web Services and Nvidia entered a strategic partnership under which AWS will offer Nvidia-based infrastructure for generative AI. The two companies will partner on several key projects.

"Today, we offer the widest range of Nvidia GPU solutions for workloads including graphics, gaming, high performance computing, machine learning, and now, generative AI," said Adam Selipsky, CEO at AWS. "We continue to innovate with Nvidia to make AWS the best place to run GPUs, combining next-gen Nvidia Grace Hopper Superchips with AWS's EFA powerful networking, EC2 UltraClusters' hyper-scale clustering, and Nitro's advanced virtualization capabilities." 

Project Ceiba is a cornerstone of this collaboration, aiming to create the world's fastest GPU-powered AI supercomputer hosted by AWS and available exclusively for Nvidia. This ambitious project will integrate 16,384 Nvidia GH200 Superchips (using the GH200 NVL32 solution packing 32 GH200 GPUs with 19.5 TB of unified memory) that are set to offer a staggering 65 'AI ExaFLOPS' of processing power. This supercomputer is for Nvidia's generative AI research and development projects. 

The Nvidia DGX Cloud hosted on AWS is another major component of the partnership. This AI-training-as-a-service platform is the first commercially available instance to incorporate the GH200 NVL32 machine with 19.5 TB of unified memory. The platform provides developers with the largest shared memory available in a single instance, significantly accelerating the training process for advanced generative AI and large language models, potentially exceeding 1 trillion parameters.

In addition, AWS will be the first to offer a cloud-based AI supercomputer based on Nvidia's GH200 Grace Hopper Superchips. This unique configuration will connect 32 Grace Hopper Superchips per instance using NVLink. It will scale up to thousands of GH200 Superchips (and 4.5 TB HBM3e memory) connected with Amazon's EFA networking and supported by advanced virtualization (AWS Nitro System) and hyper-scale clustering (Amazon EC2 UltraClusters).

The collaboration will also introduce new Nvidia-powered Amazon EC2 instances. The instances will feature H200 Tensor Core GPUs with up to 141 GB of HBM3e memory for large-scale generative AI and high-performance computing (HPC) workloads. Additionally, G6 and G6e instances, equipped with NvidiaL4 and L40S GPUs, respectively, are designed for a wide array of applications ranging from AI fine-tuning to 3D workflow development and leverage Nvidia Omniverse for creating AI-enabled 3D applications.

Finally, the collaboration will introduce Nvidia's advanced software to speed up generative AI development on AWS. This includes the NeMo LLM framework and NeMo Retriever for creating chatbots and summarization tools and BioNeMo for accelerating drug discovery processes. 

"Generative AI is transforming cloud workloads and putting accelerated computing at the foundation of diverse content generation," said Jensen Huang, founder and CEO of Nvidia. "Driven by a common mission to deliver cost-effective state-of-the-art generative AI to every customer, Nvidia and AWS are collaborating across the entire computing stack, spanning AI infrastructure, acceleration libraries, foundation models, to generative AI services."

A technical presentation from Chinese tech company ByteDance — which is best known for creating TikTok — detailed how it used AI and machine learning to make the Linux kernel run better on any hardware (via TechSpot). ByteDance believes that in the future computer engineers will likely have to lean on AI for kernel optimization. And with the gains touted in the presentation, those claims might be right.

ByteDance gave its presentation at the Linux Plumbers Conference on Nov. 14 — and though you might think the developer of TikTok is out of place here, you'd be wrong. The presentation, delivered by ByteDance engineer Cong Wang, was heavily detailed both technically and academically (it was made for computer engineers, after all).

The general gist of the presentation: ByteDance used AI to make the Linux kernel (the core of the operating system) much more efficient and performant across all kinds of hardware. We've recently seen AI help make GPU drivers more efficient, but doing the same thing in the kernel of an OS is a significant step up as a technical feat.

That this AI-powered solution worked universally is a big deal, as hardware-specific optimizations are often required to achieve good performance — and that can be challenging for developers because there are so many possible combinations of components.

The presentation detailed how AI optimizations were able to reduce memory usage by 30% — and that was using existing Linux tools, just more efficiently. Network latency was also improved by up to 12% with AI that has prior knowledge (which wouldn't be hard to obtain on a computer used regularly).

ByteDance concluded that AI-assisted kernel optimization could also help balance CPU usage, use cache more effectively, and even detect malware. At the same time, it also acknowledged that machine learning and AI wasn't a silver bullet: real human engineers will apparently not be replaced by computers any time soon for coding kernels.

Intel's latest Arc Alchemist drivers feature a performance boost of 2.7X in AI image generator Stable Diffusion. Although some of that boost was thanks to good old-fashioned optimization, which the Intel driver team is well known for, most of the uplift was thanks to Microsoft Olive. Microsoft's machine learning optimization toolchain doubled Arc GPU performance in Stable Diffusion on its own, and it's not even the first time we've seen it happen in GPU drivers.

Microsoft Olive is a piece of software that basically takes an AI or machine learning model and then finds all the ways different hardware can accelerate it. While this isn't impossible without Olive, the AI-focused toolchain makes it so much easier. There is a large amount of AI hardware on the market made by different companies, and being able to skip a big step in the optimization process is really handy.

Intel Arc Alchemist GPUs are equipped with Xe Matrix Extensions (or XMX) cores, which are essentially Intel's version of Nvidia's Tensor cores. These cores can accelerate AI workloads like Stable Diffusion. However, previous drivers didn't fully take advantage of the XMX cores, and Olive on its own boosted performance by two times when applied to Intel's existing software.

The remaining performance uplift was accomplished by Intel optimizing the model that Microsoft Olive created, and the end result was a 2.7X boost in Stable Diffusion.

This isn't the first time we've seen such a large performance gain in an AI app, and not even the first time we've seen it happen specifically with a GPU and Microsoft Olive. Nvidia used the exact same toolchain to optimize its drivers for Stable Diffusion and also achieved double the performance. 

Microsoft's new AI-focused Azure servers are powered by AMD's MI300X datacenter GPUs, but are paired with Intel's Xeon Sapphire Rapids CPUs. AMD's flagship fourth-generation EPYC Genoa CPUs are powerful, but Sapphire Rapids appears to have a couple of key advantages when it comes to pushing along AI compute GPUs. It's not just Microsoft choosing Sapphire Rapids either, as Nvidia also seems to prefer it over AMD's current-generation EPYC chips.

There are likely several factors that convinced Microsoft to go with Intel's Sapphire Rapids instead of AMD's Genoa, but Intel's support for its Advanced Matrix Extensions (or AMX) instructions could be among the important reasons Microsoft tapped Sapphire Rapids. According to Intel, these instructions are tailored towards accelerating AI and machine learning tasks by up to seven times.

While Sapphire Rapids isn't particularly efficient and has worse multi-threaded performance than Genoa, its single-threaded performance is quite good for some workloads. This isn't something that only helps AI workloads specifically; it's just an overall advantage in some types of compute.

It's also worth noting that servers using Nvidia's datacenter-class GPUs also go with Sapphire Rapids, including Nvidia's own DGX H100 systems. Nvidia's CEO Jensen Huang said the "excellent single-threaded performance" of Sapphire Rapids was a specific reason why he wanted Intel's CPUs for the DGX H100 rather than AMD's.

The new Azure instances also feature Nvidia's Quantum-2 CX7 InfiniBand switches, bringing together the hardware of all three tech giants. That just goes to show that in the cutting-edge world of AI, companies just want the overall best hardware for the job and aren't particularly picky about who makes it, regardless of rivalries. 

With eight MI300X GPUs containing 192GB of HBM3 memory each, these AI-oriented Azure instances offer a combined 1,536GB of VRAM, which is crucial for training AI. All this VRAM was likely a big reason why Microsoft selected MI300X instead of Nvidia's Hopper GPUs. Even the latest and greatest H200 chip only has 141GB of HBM3e per GPU, a significantly lower amount than the MI300X.

Microsoft also praised AMD's open-source ROCm software. AMD has been hard at work bringing ROCm to parity with Nvidia's CUDA software stack, which largely dominates professional and server graphics. That Microsoft is putting its faith in ROCm is perhaps a sign that AMD's hardware-software ecosystem is improving rapidly.

An undergraduate student used an Nvidia GeForce GTX 1070 and AI to decipher a word in one of the Herculaneum scrolls to win a $40,000 prize (via Nvidia). Herculaneum was covered in ash by the eruption of Mount Vesuvius, and the over 1,800 Herculaneum scrolls are one of the site's most famous artifacts. The scrolls have been notoriously hard to decipher, but machine learning might be the key.

Today, x-raying is the method of choice to read ancient scrolls without opening them and potentially causing damage. However, the ink in the Herculaneum scrolls is carbon-based, making it very difficult to distinguish from the charred pages of the scroll itself using normal X-rays.

In 2019, Dr. Brent Seales and other researchers made a breakthrough by augmenting their X-rays with a particle accelerator to scan two scrolls. Earlier this year, those X-rays were run through a machine-learning model to make the ink more legible. The Vesuvius Challenge offers prizes to anyone who can clearly decipher words from the two scrolls.

Luke Farritor, an undergrad at the University of Nebraska-Lincoln and Space-X intern, used his old GTX 1070 to train an AI model to detect "crackle patterns," which indicate where an ink character used to be. Eventually, his GTX 1070-trained AI was able to identify the Greek word πορφυρας (or porphyras), which is either the adjective for purple or the noun for purple dye or purple clothes. Deciphering this single word earned Farritor a $40,000 prize.

Herculaneum was first discovered in 1738, and in 1750 King of Naples Charles VII ordered an excavation of the site, which led to the discovery of the so-called Villa of the Papyri. The site was forgotten shortly afterward, but it was rediscovered in 1986.

The Villa, which may have been owned by Julius Caesar's father-in-law Lucius Calpurnius Piso Caesoninus, contained sculptures, frescoes, and the eponymous scrolls of Herculaneum. A famous bust of the Roman general Scipio Africanus was also found at the Villa of the Papyri.

How To Choose a Graphics Card

After the cryptocurrency craze crashed, many crypto farms were left with tons of unused GPUs. Some of these companies managed to find other purposes for their GPUs, but thousands still sit idle. Io.net has developed a distributed network that can pool in hundreds of thousands of GPUs and use them for artificial intelligence applications — and apparently 107,000 GPUs are currently on the waiting list.

Io.net has developed a decentralized physical infrastructure network (DePIN) that supports pooling and clustering of GPU computing power from data centers and cryptocurrency miners from different geographic locations. The network is designed to provide resources for artificial intelligence and machine learning workloads by aggregating underutilized GPUs located across the world. To attract GPU owners, Io.net is launching a $700,000 incentive program to encourage them to contribute their resources to the network. 

While the primary idea of Io.net's DePIN is to aggregate GPU compute resources unused in datacenters and by individual cryptocurrency miners, the company has also partnered with the Render network, which specializes on remote rendering, to gain access to additional GPUs. 

Io.net is not the only DePIN for GPU resources nowadays, but the company says that it can actually cluster resources of GPUs from different geographic locations in minutes — unlike some of its rivals. 

"The problem is that they do not really cluster," said Tory Green, chief operating officer of Io.net, in an interview with Cointelegraph. "They are primarily single instance, and while they do have a cluster option on their websites, it is likely that a salesperson is going to call up all of their different data centers to see what is available." 

In terms of functionality, the closest competitors are AI-oriented services such as Akash Network, which groups together from eight to 32 GPUs. 

Io.net's platform lets customers pick and choose the number and location of GPUs they want to use, as well as security settings — making it easier for businesses and machine learning engineers to get the computing power they need. 

The company uses the Solana blockchain technology to manage microtransactions within its network to enable payments to GPU computing providers. Solana's technology can facilitate a vast number of small transactions, which traditional blockchains might not handle as effectively due to slower processing times/higher fees, efficiently. This makes Solana a critical component of Io.net's infrastructure, enabling it to operate a decentralized network for GPU computing power with a robust payment system.

 Microsoft is rolling out Windows 11 Version 23H2 today, which it is describing as a "scoped, cumulative release."

"We are providing a limited scope of new features and functionality delivered via a familiar, fast, and reliable update experience," writes John Cable, Microsoft's vice president of Windows servicing and delivery.

The scope of features is indeed limited, partially because the company isssued a fairly large update at the end of September, pushing its Copilot AI features live, as well as AI features in Paint and changes to OneDrive, Photos and Windows Backup. So after all that, 23H2 is focused on some changes to Teams and application management.

The built-in Chat functionality in Windows 11 is now becoming a free part of Microsoft Teams, which is pinned to the taskbar by default. You'll be able to send and receive SMS texts from Microsoft Teams (though in theory you could also use My Phone for that). Teams will also get new "People" functionality to find people you're looking to contact.

Microsoft has also rolled out WDDM 3.2 (Windows Display Driver Model 3.2) and HLSL Shader Model 6.8 as part of the 23H2 update. The latter adds Work Graphs (experimental) and Wave Matrix (also experimental) support to DirectX. Work Graphs define a system of shader nodes that feed into each other to enable tailored GPU work creation, while Wave Matrices help with higher bandwidth matrix multiplications (which are useful in machine learning). It's unclear what new functionality WDDM 3.2 brings to the table.

As for applications, those included with Windows 11 will be separated from the ones you install. In the "All apps" section of the Start menu, that software will be given a "system" label, and instead of being found in Settings > Apps > Installed Apps, they'll instead be featured in Settings > System > System Components.

If you're a member of the Windows Insider Program, you likely already have these features. But with 23H2, they're going wide.

Like previous updates, 23H2 will be issued through Microsoft's serving technology, which detects your specs and attempts to serve you the update when the company is largely sure you won't experience and incompatibility issues. Those who want the latest updates first can go to Settings > Windows Update and check the box that reads "Get the latest updates as soon as they’re available." (That is, assuming you're on version 22H2. If you're on the older version 21H1, you'll need to do a full OS swap).

With all of the new features for Windows 11 in September, this isn't exactly the most exciting update, but it's always a good idea to keep your system up to date for the sake of security.

Researchers at Ohio State University have found that pigeons tackle some problems in a very similar way to modern computer AI models. In essence, pigeons have been found to use a ‘brute force’ learning method called "associative learning." Thus pigeons, and modern computer AIs, can reach solutions to complex problems that befuddle human thinking patterns.

Brandon Turner, lead author of the new study and professor of psychology at Ohio State University, worked with Edward Wasserman, a professor of psychology at the University of Iowa, on the new study, published in iScience.  

Here are the key findings:

• Pigeons can solve an exceptionally broad range of visual categorization tasks
• Some of these tasks seem to require advanced cognitive and attentional processes, yet computational modeling indicates that pigeons don’t deploy such complex processes
• A simple associative mechanism may be sufficient to account for the pigeon’s success

Turner told the Ohio State news blog that the research started with a strong hunch that pigeons learned in a similar way to computer AIs. Initial research confirmed earlier thoughts and observations. “We found really strong evidence that the mechanisms guiding pigeon learning are remarkably similar to the same principles that guide modern machine learning and AI techniques,” said Turner.

A pigeon’s “associative learning” can find solutions to complex problems that are hard to reach by humans or other primates. Primate thinking is typically steered by selective attention and explicit rule use, which can get in the way of solving some problems.

For the study, pigeons were tested with a range of four tasks. In easier tasks, it was found pigeons could learn the correct choices over time and grow their success rates from about 55% to 95%. The most complex tasks didn’t see such a stark improvement over the study time, going from 55% to only 68%. Nevertheless, the results served to show close parallels between pigeon performance and AI model learning performance. Both pigeon and machine learners seemed to use both associative learning and error correction techniques to steer their decisions toward success.

Further insight was provided by Turner in comments on human vs pigeon vs AI learning models. He noted that some of the tasks would really frustrate humans as making rules wouldn’t help simplify problems, leading to task abandonment. Meanwhile, for pigeons (and machine AIs), in some tasks “this brute force way of trial and error and associative learning... helps them perform better than humans.”

Interestingly, the study recalls that in his Letter to the Marquess of Newcastle (1646), French philosopher René Descartes argued that animals were nothing more than beastly mechanisms — bête-machines, simply following impulses from organic reactions.

The conclusion of the Ohio State blog highlighted how humans have traditionally looked down upon pigeons as dim-witted. Now we have to admit something: our latest crowning technological achievement of computer AI relies on relatively simple brute-force pigeon-like learning mechanisms.

Will this new research have any influence on computer science going forward? It seems like those involved in AI / machine learning and those developing neuromorphic computing might find some useful crossover here.

Laptops based on AMD's latest Ryzen 7040-series processors come equipped with Ryzen AI engine, an accelerator for machine learning applications. However, this feature is currently exclusive to Windows, leaving Linux users hoping for broader compatibility. Community interest has prompted AMD to reconsider, an reopening a GitHub ticket for feedback and expressing willingness to support the technology if there is adequate demand, reports Phoronix.

AMD has designed the Ryzen XDNA AI engine for less demanding AI inference tasks like audio, photo, and video processing. Its goal is to provide quicker response times compared to online services, and it is also more energy-efficient compared to solutions based on CPUs or GPUs. The engine has the capacity to manage up to four simultaneous AI streams, and it can process INT8 and bfloat16 instructions. 

According to AMD, the performance of this engine surpasses that of the neural engine in Apple's M2 processor. AMD's Xilinx-based AI engine is compatible with popular frameworks such as TensorFlow, PyTorch, and ONNX, but the problem is that the company's Ryzen AI Software Platform version 0.8 only supports Windows. By contrast, Intel has already integrated open-source AI processor support in Linux.

This limitation has sparked a discussion within the tech community, particularly among Linux users who desire the same advanced capabilities on their systems. AMD has been receptive to these discussions, enabling a platform on GitHub for users to express their interest and thoughts on Linux compatibility.

Taking user feedback seriously, AMD has demonstrated flexibility and openness to expanding Ryzen AI’s compatibility based on customer demand to Linux. A GitHub ticket that allows users to voice their need for Linux support has been reopened by an AMD staff member, signifying the company's willingness to listen and potentially act based on the community's needs and interests.

The only question is when will AMD be able to bring proper support for Ryzen AI to Linux. The company yet has to release final version 1.0 of its Ryzen AI Software Platform for Windows, which is dominant in the realm of PCs. As a result, the it's almost certain that the majority of the company's effort will be dedicated to Windows, not Linux, in the coming months.

Every major chipmaker in the PC space has — or is about to — add neural processing units that can be used for machine learning or AI tasks. For those to be truly useful, however, the hardware and software using those chips must have AI-enabled features. Today, Intel announced an AI PC Acceleration Program to provide resources to independent software vendors (ISVs) and independent hardware vendors (IHVs) to make that happen.

Intel's program comes ahead of the launch of its Intel Core Ultra "Meteor Lake" processors on December 14. It promises to provide "AI toolchains, co-engineering, hardware, design resources, technical expertise and co-marketing opportunities" to ISVs and IHVs in an effort to deliver a full suite of applications ready to utilize Meteor Lake's new tech.

The company claims that it is working with over 100 software vendors on "more than 300 AI-accelerated features", including companies such as Adobe, Audacity, BlackMagic, CyberLink, XSplit, Zoom, Topaz, Webex, and Magix.

"With a long history in AI development and a deep network of ISV engineering relationships, Intel will take an active role in fostering connections and innovations that propel new use cases and experiences on the PC," Michelle Johnston Holthaus, executive vice president and general manager of the Client Computing Group at Intel said in a press release. You can read more about Intel's AI PC initiative here.

Chipmakers and PC vendors need these features to arrive and provide excitement. Apple, AMD, and Qualcomm already have neural processors, but proof in the AI pudding might make upgrading PCs worth it to many people who upgraded at the beginning of the COVID-19 pandemic and haven't since. It also lets companies like Intel show how powerful their chips can be without putting extra strain on a CPU or GPU.

AMD launched its 7040U (codenamed Phoenix) series mobile processors earlier this year, with some configurations featuring its XDNA-powered AI engine. There are, however, few examples of software taking advantage of it. Microsoft included a Movidius NPU on its Surface Laptop Studio 2 but primarily used it for Windows Studio camera effects. Qualcomm is launching its Snapdragon X chips at its upcoming Summit in Maui, which the company said "will deliver accelerated on-device user experiences for the new era of generative AI."

It's doubtful that Intel is the only company working with software and hardware companies to build experiences around its chips. But it has many big names on board, which can increase its perception as the leading option for AI-enhanced workloads before its new chips are even released. I'm sure we'll see several demonstrations when Intel Core Ultra launches in December.

Nvidia has been busy working on further improvements to its suite of AI/ML (Artificial Intelligence/Machine Learning) and LLM (Large Language Model) tools. The latest addition is TensorRT and TensorRT-LLM, designed to optimize performance of consumer GPUs and many of the best graphics cards for running tasks like Stable Diffusion and Llama 2 text generation. We've tested some of Nvidia's latest GPUs using TensorRT and found performance in Stable Diffusion was improved by up to 70%. TensorRT should be available for download at Nvidia's Github page now, though we had early access for purposes of this initial look.

We've seen a lot of movement in Stable Diffusion over the past year or so. Our first look used Automatic1111's webui, which initially only had support for Nvidia GPUs under Windows. Since then, the number of forks and alternative text to image AI generation tools has exploded, and both AMD and Intel have released more finely tuned libraries that have closed the gap somewhat with Nvidia's performance. You can see our latest roundup of Stable Diffusion benchmarks in our AMD RX 7800 XT and RX 7700 XT reviews. Now, Nvidia is ready to widen the gap again with TensorRT.

The basic idea is similar to what AMD and Intel have already done. Leveraging ONNX, an open format for AI and ML models and operators, the base Hugging Face stable diffusion model gets converted into an ONNX format. From there, you can further optimize performance for the specific GPU that you're using. It takes a few minutes (or sometimes more) for TensorRT to tune things, but once complete, you should get a substantial boost in performance along with better memory utilization.

We've run all of Nvidia's latest RTX 40-series GPUs through the tuning process (each has to be done separately for optimal performance), along with testing the base Stable Diffusion performance and performance using Xformers. We're not quite ready for the full update comparing AMD, Intel, and Nvidia performance in Stable Diffusion, as we're retesting a bunch of additional GPUs using the latest optimized tools, so this initial look is focused only on Nvidia GPUs. We've included one RTX 30-series (RTX 3090) and one RTX 20-series (RTX 2080 Ti) to show how the TensorRT gains apply to all of Nvidia's RTX line.

Each of the above galleries, at 512x512 and 768x768, uses the Stable Diffusion 1.5 models. We've "regressed" to using 1.5 instead of 2.1, as the creator community generally prefers the results of 1.5, though the results should be roughly the same with the newer models. For each GPU, we ran different batch sizes and batch counts to find the optimal throughput, generating 24 images total per run. Then we averaged the throughput of three separate runs to determine the overall rate — so 72 images total were generated for each model format and GPU (not counting discarded runs).

Various factors come into play with the overall throughput. GPU compute matters quite a bit, as does memory bandwidth. VRAM capacity tends to be a lesser factor, other than potentially allowing for larger image resolution targets or batch sizes — there are things you can do with 24GB of VRAM that won't be possible with 8GB, in other words. L2 cache sizes may also factor in, though we didn't attempt to model this directly. What we can say is that the 4060 Ti 16GB and 8GB cards, which are basically the same specs (slightly different clocks due to the custom model on the 16GB), had nearly identical performance and optimal batch sizes.

There are some modest differences in relative performance, depending on the model format used. The base model is the slowest, with Xformers boosting performance by anywhere from 30–80 percent for 512x512 images, and 40–100 percent for 768x768 images. TensorRT then boosts performance an additional 50~65 percent at 512x512, and 45~70 percent at 768x768.

What's interesting is that the smallest gains (of the GPUs tested so far) come from the RTX 3090. It's not exactly clear what the limiting factor might be, though we'll have to test additional GPUs to come to any firm conclusions. The RTX 40-series has fourth generation Tensor cores, RTX 30-series has third generation Tensor cores, and RTX 20-series has second gen Tensor cores (with the Volta architecture being first gen Tensor). Newer architectures should be more capable, in other words, though for the sort of work required in Stable Diffusion it mostly seems to build down to raw compute and memory bandwidth.

We're not trying to make this the full Nvidia versus the world performance comparison, but updated testing of the RX 7900 XTX as an example tops out at around 18~19 images per minute for 512x512, and around five images per minute at 768x768. We're working on full testing of the AMD GPUs with the latest Automatic1111 DirectML branch, and we'll have an updated Stable Diffusion compendium once that's complete. Note also that Intel's Arc A770 manages 15.5 images/min at 512x512, and 4.7 images/min at 768x768.

So what's going on exactly with TensorRT that it can improve performance so much? I spoke with Nvidia about this topic, and it's mostly about optimizing resources and model formats.

ONNX was originally developed by Facebook and Microsoft, but is an open source initiative based around the Apache License model. ONNX is designed to allow AI models to be used with a wide variety of backends: PyTorch, OpenVINO, DirectML, TensorRT, etc. ONNX allows for a common definition of different AI models, providing a computation graph model along with the necessary built-in operators and a set of standard data types. This allows the models to be easily transported between various AI acceleration frameworks.

TensorRT meanwhile is designed to be more performant on Nvidia GPUs. In order to take advantage of TensorRT, a developer would normally need to write their models directly into the format expected by TensorRT, or convert an existing model into that format. ONNX helps simplify this process, which is why it has been used by AMD (DirectML) and Intel (OpenVINO) for those tuned branches of Stable Diffusion.

Finally, one of the options with TensorRT is that you can also tune for an optimal path with a model. In our case, we're doing batches of 512x512 and 768x768 images. The generic TensorRT model we generate may have dynamic image size of 512x512 to 1024x1024, with a batch size of one to eight, and an optimal configuration of 512x512 and batch size of 1. Doing 512x512 batches of 8 might end up being 10% slower, give or take. So we can do another TensorRT model that specifically targets 512x512x8, or 768x768x4, or whatever. And we did all of this to find the best configuration for each GPU.

AMD's DirectML fork has some similar options, though right now there are some limitations we've encountered (we can't do a batch size other than one, as an example). We anticipate further tuning of AMD and Intel models as well, though over time the gains are likely to taper off.

The TensorRT updated isn't just for Stable Diffusion, of course. Nvidia shared the above slide detailing improvements it has measured with Llama 2 7B int4 inference, using TensorRT. That's a text generation tool with seven billion parameters.

As the chart shows, generating a single batch of text shows a modest benefit, but the GPU in this case (RTX 4090) doesn't appear to be getting a full workout. Increasing the batch size to four boosts overall throughput by 3.6X, while a batch size of eight gives a 4.4X speedup. Larger batch sizes in this case can be used to generate multiple text responses, allowing the user to select the one they prefer — or even combine parts of the output, if that's useful.

The TesorRT-LLM isn't yet out but should be available on developer.nvidia.com (free registration required) in the near future.

Finally, as part of its AI-focused updates for LLMs, Nvidia is also working on a TensorRT-LLM tool that will allow the use of Llama 2 as the base model, and then import local data for more domain specific and updated knowledge. As an example of what this can do, Nvidia imported 30 recent Nvidia news articles into the tool and you can see the difference in response between the base Llama 2 model and the model with this local data.

The base model provides all the information about how to generate meaningful sentences and such, but it has no knowledge of recent events or announcements. In this case, it things Alan Wake 2 doesn't have any official information released. With the updated local data, however, it's able to provide a more meaningful response.

Another example Nvidia gave was using such local data with your own email or chat history. Then you could ask it things like, "What movie were Chris and I talking about last year?" and it would be able to provide an answer. It's potentially a smarter search option, using your own information.

We can't help but see this as a potential use case for our own HammerBot, though we'll have to see if it's usable on our particular servers (since it needs an RTX card). As with all LLMs, the results can be a bit variable in quality depending on the training data and the questions you ask.

Nvidia also announced updates to it's Video Super Resolution, now with support for RTX 20-series GPUs and native artifact reduction. The latter means if you're watching a 1080p stream on a 1080p monitor, VSR can still help with noise removal and image enhancement. VSR 1.5 is available with Nvidia's latest drivers.