External GPU (eGPU) support on Apple Silicon Macs and MacBooks has been a persistent pain point for AI/ML developers. Through what some may consider to be black magic, Tiny Corp has managed to get an AMD eGPU working in Tiny Grad over USB3, a standard that inherently lacks PCIe capabilities. As they're using libusb, this functionality extends to Windows, Linux, and even macOS, including devices with Apple Silicon.
Traditionally, GPUs are connected through PCIe slots or the Thunderbolt/USB4 interfaces, which offer PCI Express tunneling support. As such, external GPU solutions rely on the aforementioned interfaces, which limits their support for older systems and laptops. Unlike Intel-based Macs/MacBooks, Apple Silicon based devices do not support external GPUs, mainly due to the lack of driver support and architectural differences. So, despite their efficiency compared to traditional x86-based systems, users have reported challenges in AI workloads, especially when it comes to prompt processing.
Requirements for running an eGPU through a USB3 interface at this time include the use of an ASM2464PD-based adapter and an AMD GPU. For its tests, Tiny Corp used the ADT-UT3G adapter, which uses the same ASM2464PD chip, but out of the box, it only works with Thunderbolt 3, Thunderbolt 4, or USB 4 interfaces. The team likely employed a custom firmware to enable USB3 mode that works without direct PCIe communication. Technical details are murky, however, the controller appears to be translating PCIe commands to USB packets and vice versa.
Latest Videos FromTom's Hardware
The solution is quite hacky, as it bypasses kernel-level GPU drivers, requires specific hardware, and uses USB3, which was not originally intended for GPU communication. It essentially offloads the computation part, referring to kernel executions, from your system to the eGPU. The constraint here is that data transfer speeds are capped at 10 Gbps due to the USB3 standard used, so loading models into the GPU will take much longer than if you were to use a standard PCIe connection.
Since it uses custom user-space drivers to avoid tinkering with the kernel, the feature is limited to AMD's RDNA 3/4 GPUs, although there's a hint of potential RDNA 2 support in the future. USB3 eGPU functionality has been upstreamed to Tiny Grad's master branch, so if you have an AMD GPU and a supported adapter, feel free to try it out. We can expect Tiny Corp to provide a more detailed and technical breakdown once its developers done tidying up the code.
Follow Tom's Hardware on Google News to get our up-to-date news, analysis, and reviews in your feeds. Make sure to click the Follow button.
