Baidu, the Chinese web search company, announced the next generation of its open source DeepBench benchmark tool for deep learning, which now includes a measurement for inference.
Baidu released the first version of DeepBench in September of 2016. It was the first tool for benchmarking deep learning performance of various types of processors that was opened up to the wider deep learning development community, according to Baidu. Since its release, all major chip makers, including Intel, AMD, and Nvidia have contributed to the DeepBench platform.
The initial version of DeepBench targeted only training performance across multiple chip architectures. The new version will also evaluate cross-platform inference performance. Inference involves taking an already trained model to make predictions on new data sets.
For instance, you could first train your deep learning neural network on millions of pictures with the goal of recognizing cats. Once the neural network is trained and presumably has a high level accuracy, it should be able to recognize cats out of all the new pictures you throw at it.
“Measuring inference is critical,” said Dr. Greg Diamos, Senior Researcher at Baidu Research Silicon Valley AI Lab (SVAIL). “It covers the operations needed to run neural networks on a device, be it in the cloud, on a phone or a wearable. A better understanding of performance of inference means better chips and neural networks in real products,” he added.
According to Baidu, benchmarking inference is challenging, because many deep learning applications have their own unique performance characteristics, and there are several different deployment platforms (such as servers and mobile devices). However, the company believes it has solved this problem by benchmarking fundamental operations required for inference.
“Speed is key to training neural networks, and the first step to improving speed is having an accurate measurement of performance,” said Sharan Narang, Systems Researcher at Baidu’s Silicon Valley AI Lab. “With the addition of the ability to measure inference, researchers will now have a more comprehensive benchmark for the performance of their AI hardware," he noted.
The new DeepBench provides new kernels for training from several different deep learning models, and it also sets new minimum precision requirements. The new benchmark establishes 16-bit floating point for multiplication and 32-bit floating point for addition for training operations. It also establishes 8-bit fixed point for multiplication and 32-bit fixed point for addition.
The updated DeepBench benchmark tool is available on the SVAIL Github page.
