Nvidia has set up an interesting new research group called GEAR. A clue to the research group’s goals is provided by expanding the GEAR acronym - Generalist Embodied Agent Research. GEAR has basically been set up so that Nvidia can further advance and build capable AI-based entities that can operate with skill in both the virtual and physical worlds – we are, therefore, talking about things like intelligent NPCs and robots, respectively.
The GEAR group at Nvidia has been co-founded by Jim Fan and Yuke Zhu, who have collaborated on several projects previously. Introducing the new research group this weekend, Fan sketched out his vision concerning GEAR. “We believe in a future where every machine that moves will be autonomous, and robots and simulated agents will be as ubiquitous as iPhones,” wrote the newly appointed Research Manager and Lead of Embodied AI (GEAR Group) at Nvidia. “We are building the Foundation Agent — a generally capable AI that learns to act skillfully in many worlds, virtual and real.”
Fan’s enthusiasm spilled over into stating that “2024 is the Year of Robotics, the Year of Gaming AI, and the Year of Simulation.” We hope that it could be, but such a timescale would seem optimistic given typical development times.
The lab's homepage provides four bullet points spelling out the research agenda. The GEAR team indicates it will begin its efforts by progressing multimodal foundation models, general-purpose robots, foundation agents in virtual worlds, and simulation and synthetic data.
The research group’s portal also shares a quartet of prior projects, which are the fruits of Fan, Zhu, and various other researchers. Two of the projects concern proficient and proactive AI-powered agents that have been inserted into the popular Minecraft game. Another project concerns an optimized reinforcement learning technique for robots designed to precipitate “dexterity at super-human level.” Last but not least, the researchers have also collaborated on multimodal prompts for robot task specification and developed a benchmark for such actions.
If the above stirs you deeply, you may be interested to know that GEAR is actively recruiting for at least three positions to help them reach their stated goals. As well as senior positions attracting salaries up to $333,500 per year, they are also seeking a Research Intern (paid $19 to $93 per hour).
