As if the AI drivers in Mario Kart 64 weren’t bad enough, a fabulous project uncovered by Hackster sees the player replaced by a Raspberry Pi Pico (backed by a laptop running a trained TensorFlow machine learning model).
There's actually two hacks in one here. The first, detailed on project creator Stacksmashing’s GitHub page, involved getting a video output out of an original N64 (one that hasn’t been modded into a PC) that the laptop could process into the AI model. This meant an HDMI mod, as analog RGB video connectors are rare on modern laptops.
The project, known as TensorKartRealHW, uses TensorKart, software that is already capable of playing Mario Kart 64 on its own, but which relies on an emulator. Being able to play it on original hardware meant being able to pass controller inputs through one of the original ports on the front of the console, which is where the Raspberry Pi Pico comes in. It is connected to the N64 via a broken controller, and converts inputs requested by the AI into the signals expected by the controller ports.
The training data was gathered using a Python script that captured controller input and the subsequent effect on visual output. Things like the minimap were cropped out, as they were confusing, and the game was slowed down to capture smoother steering. The AI model was trained on this dataset, and eventually developed near-perfect control of Mario in his kart - but only on the courses contained in the training data.
If you want to try it yourself, the code is open-source and on GitHub, but you’ll probably have to break an N64 controller first, and there's no information on how handy the AI is with a blue shell.
