The newer Ryzen 5 5600G (Cezanne) has replaced the Ryzen 5 4600G (Renoir) as one of the best CPUs for gaming. However, a trick has breathed new life into the Ryzen 5 4600G, transforming the budget Zen 2 APU into a 16GB graphics card to run AI applications on Linux.

Not everyone has to budget to buy or rent a Nvidia H100 (Hopper) to experiment with AI. With the current demand for AI-focused graphics cards, you may be unable to access one even if you have the money. Luckily, you don't need an expensive H100, an A100 (Ampere), or one of the best graphics cards for AI. One Redditor demonstrated how a Ryzen 5 4600G retailing for $95 can tackle different AI workloads.

The Ryzen 5 4600G, which came out in 2020, is a hexa-core, 12-thread APU with Zen 2 cores that operate with a base and boost clock of 3.7 GHz and 4.2 GHz. The 65W chip also wields a Radeon Vega iGPU with seven compute units clocked up to 1.9 GHz. Remember that APUs don't have dedicated memory but share system memory. You can determine the amount of memory inside the motherboard’s BIOS. In this case, the Redditor had 32GB of DDR4 and allocated 16GB to the Ryzen 5 4600G. Typically, 16GB is the maximum amount of memory you can dedicate to the iGPU. However, some user reports claim that certain ASRock AMD motherboards allow for higher memory allocation, rumored up to 64GB.

The trick converts the Ryzen 5 4600G into a 16GB "graphics card," flaunting more memory than some of Nvidia's latest GeForce RTX 40-series SKUs, such as the GeForce RTX 4070 or GeForce RTX 4070 Ti, which are limited to 12GB. Logically, the APU doesn't deliver the same performance as a high-end graphics card, but at least it won't run out of memory during AI workloads, as 16GB is plenty for non-serious tasks.

AMD's Radeon Open Compute platform (ROCm) doesn't officially support Ryzen APUs. Third-party companies, such as BruhnBruhn Holding, offer experimental packages of ROCm that'll work with APUs. That means APUs can work with PyTorch and TensorFlow frameworks, opening the gate to most AI software. We wonder if AMD's latest mobile Ryzen chips, like Phoenix that taps into DDR5 memory, can work and what kind of performance they bring.

The Redditor shared a YouTube video claiming that the Ryzen 5 4600G could run a plethora of AI applications, including Stable Diffusion, FastChat, MiniGPT-4, Alpaca-LoRA, Whisper, LLM, and LLaMA. Unfortunately, he only provided demos for Stable Diffusion, an AI image generator based on text input. He doesn't detail how he got the Ryzen 5 4600G to work with the AI software on his Linux system. The YouTuber has vouched to release a thorough video of the setup process. 

As for the performance, the Ryzen 5 4600G only took around one minute and 50 seconds to generate a 512 x 512-pixel image with the default setting of 50 steps. It's an excellent result for a $95 APU and rivals some high-end processors. The author said he used DDR4 memory but didn't list the specifications. Although the Ryzen 5 4600G natively supports DDR4-3200, many samples can hit DDR4-4000, so it would be fascinating to see AI performance scaling with faster memory.

The experiment is fantastic for those who own a Ryzen 5 4600G or Ryzen 5 5600G and want to play around with AI. For those who don't, throwing $500 into an APU build doesn't make much sense when you can probably get a discrete graphics card that offers better performance. For instance, AMD's Radeon 16GB graphics cards start at $499, and Nvidia recently launched the GeForce RTX 4060 Ti 16GB, which has a similar starting price.

Protecting your garden from unwanted plants and excess weeds is half of the gardening process. One maker and developer, aka Nathan from NathanBuildsDIY, has taken the matter to an entirely new level by developing this impressive Raspberry Pi-powered solution. After deliberating potential avenues, he settled on creating an AI-driven robot that identifies weeds before using a Fresnel lens to concentrate light from the sun onto the weeds until they burn.

The system consists of a large wooden frame that can be wheeled up and down paths in your garden. It has a camera for observing the territory that the Pi evaluates for potential weeds. Once an unwanted plant is spotted, its Fresnel lens is moved into position while the machine waits until the weed is sufficiently burned.

Nathan was kind enough to not only demonstrate the project but also share a video completely breaking down the build process making it easy for anyone to recreate themselves. He includes everything from the physical schematics to wiring diagrams as well as delving deep into the code used to operate the system.

A full parts list is available in the video description but here’s a brief summary of the most major components. It uses a Raspberry Pi 3B+ but you could easily swap this with a Raspberry Pi 4 or even a Raspberry Pi Zero 2 W. A Fresnel lens is necessary to focus the sunlight to burn the plants, this is put into position with a series of wheels and motors. Everything is powered by a LiPo battery and held into place with a custom wooden frame.

A fair bit of software is required to program the unit. You can expect some familiar applications if you’ve tinkered with AI on the Pi before. You’ll need OpenCV as well as Tflite (Tensorflow Lite) to run Tensorflow on the Pi. Other basic libraries are used as well, with Picamera being used to control and take images with the official Raspberry Pi camera..

If you want to get a closer look at this Raspberry Pi project, we highly recommend checking out the video so you can see it in action. Visit Nathan’s official YouTube channel, NathanBuildsDIY, for an in-depth look at this impressive gardening tool and be sure to follow him for more cool creations.

They say a picture is worth a thousand words, but could a thousand words be worth a picture? Maker and developer Diego Trujillo Pisanty sought to find out with his Raspberry Pi-powered blind camera project. Instead of using a lens to capture light to make a picture, it listens for sound and constructs an approximation of what could be around it based on the audio detected.

The device works similarly to a regular camera in that you aim the camera at what you want to capture and press a button to generate an image. In this case, however, it has a giant horn on the front used to help amplify the capturing of sounds. Users should aim this horn in the direction of what they want to capture before pressing the button. The camera then parses the audio through an AI filter and generates an image.

According to Pisanty, he developed a custom artificial neural network (or ANN) just for this project. He trained the AI with his own models based on a set of videos taken around Mexico City. 

The model was created by taking each frame of video and accompanying it with the last second of audio. This helped build an association of sound and video that the system can use to create images. Because it was trained in this way, everything it creates is loosely based on inner-city images of Mexico City.

For this project, Pisanty is using a Raspberry Pi 3B module. It would be possible to recreate it using a Raspberry Pi 4, however. As long as the Pi can handle Tensorflow, it should work. The camera also has a small screen that serves as a sort of viewfinder. It lets users know when images are processing and provides a preview of the generated images. Everything is housed inside of a custom 3D-printed shell.

The AI model used to train the blind camera was created using Python 3. It’s designed to work with Tensorflow 2 and runs on the Raspberry Pi using TFLite. If you want to recreate this Raspberry Pi project or just get a closer look at how it goes together, check out the video shared by Pisanty to YouTube and read more about the project over at his website.

Firefly has revealed two more PCs based on the RK3588 SoC, and both aim to become the brains of your next AI project. The Station P3 looks just like a typical mini PC, whereas the Station P3D is a modular AI PC with interchangeable modules providing extra functionality. These units share many core features in their larger ITX-3588J board but in a smaller package.

Both machines come in machined aluminum alloy enclosures. CNC machined, sandblasted, and anodized, these cases look sleek and not out of place on your desk or in your living room. Inside the case is a machine capable of decoding 8K video at 60fps and H.265/H.264 encoding at 8K at 30fps. This is plenty of power for standard media projects, but AI is where these PCs shine. The NPU offers 6 TOPS of neural computing power for applications such as TensorFlow and MXnet.

At a specification level, both machines are almost equal. An extra SATA SSD port, more USB ports, and a VGA connector are all that really separate the Station P3D and Station P3. There is one massive difference, though — the Station P3D has a modular bay attachment. The Station P3D is marketed as a modular AI PC, and the lower part of the Station P3D slides out to replace the expansion module with one of six alternatives. The standard module is as described in the above table, other modules include multiple displays, additional Ethernet ports, Hi-Fi audio, multi USB, and multiple 9-pin serial ports. How this modules connects is a mystery for now. We cannot see any mention of a connection and the sliding nature of the expansion port leads us to believe that it is a connector located deep inside the case, possibly a form of PCIe/USB.

The Station P3 looks like one of the many Intel-based mini PC clones which can be found on Amazon and Aliexpress. The key difference is the RK3588 SoC. This is a powerful SoC with much more horsepower than the Raspberry Pi 4 but historically the RK3588 has featured in boards at a much higher price than the Raspberry Pi. For example, the Khadas Edge 2 Pro uses the RK3588S, and that retails for $339.

The RK3588 is powered by a four core Arm Cortex A76, and four core Arm Cortex A55 with a top speed of 2.4 GHz. This is identical to the ITX-3588J , also from Firefly. The similarities don't end there, as the NPU is identical between the Station P3, P3D and the ITX-3588J. This means that the Station P3 and Station P3D are geared towards AI and machine learning applications which generally require more computing power.

The Station P3D Mini PC is on sale now for $399Youou get 8GB of RAM and 64GB of eMMC storage for this price. It comes with the default expansion module. Pricing for the Station P3 is currently not available.

Google has made significant progress in its endeavor to develop its own data center chips, according to a new report. The Information says that a key milestone has just been reached, which means that Google can plan to roll out server systems powered by the new chips starting from 2025.

This is not the first processor that Google has successfully put through R&D - the company has previously made an ASIC for servers and an SoC for mobile devices. The search giant started using its internally developed Tensor Processing Unit (TPU) as far back as 2015. The TPU was an ASIC designed to accelerate AI and neural network machine learning, which also found uses in custom SSDs, network switches, and NICs. For AI processing it slotted into the firm’s TensorFlow framework, but Google kept on using third-party CPUs and GPUs for a number of other key processes/processing tasks. Google’s TPU has reached its fourth generation, and now it looks like Google wants to go further in using its own silicon in the server space.

In addition to the TPU ASIC, a much fuller SoC from Google exists for use in its devices. The newest Tensor G2 chip for mobiles mixes the latest Arm Cortex cores and Mali G710 graphics with a custom TPU, an ISP, a security core and caches – and is made by Samsung on its 5nm process. Progress with this chip over recent years, and the TPU, might have helped crystallize the new server chip development plans for greater control, efficiency, and TCO reductions.

The two sources speaking to The Information, one with direct knowledge of the project and another who had been briefed upon it, indicate that Google is working hard to catch up with cloud server business rival Amazon. Amazon launched its AWS Graviton processor with Arm architecture in 2018 – and it is now in its third generation, boasting impressive performance and efficiency optimizations over an already attractive server proposition.

Some other morsels of information shared by The Information are that Google’s server chip R&D team are working on two Arm-based 5nm chips. An SoC dubbed ‘Cypress’, is an in-house design by the Google Israel team. Meanwhile, a design codenamed ‘Maple’, which is based on the foundations of a Marvell Technology SoC, is in trial production at TSMC. Overseeing both designs is Uri Frank, a 25-year Intel CPU design veteran, who became Google’s VP of Engineering for server chip design in March 2021. It is understood that Frank sees Cypress as Plan A, with Maple waiting ready in the wings as Plan B.

With mass production of these chips potentially beginning in 2024, the source reckons Google data centers could be using them by 2025. Whether Google’s Plan A or Plan B makes the cut, it doesn’t look like this will be good news for PC x86 CPU makers like Intel or AMD.

Wondering what crazy antics cats could get into if they had thumbs is its own ordeal but upping the stakes with the ability to knock on doors is finally within reach thanks to maker and developer Tennis Smith’s Raspberry Pi-powered IoT cat doorbell project. It works just like it sounds, allowing his cat to notify him when it wants to go inside the house.

The system involves using a microphone to listen for potential meows. So instead of knocking, all the cat needs to do is what it does best—bellow out in desperate hopes of getting let inside. The Pi is responsible for detecting meows from other sounds using AI. If a meow is determined to have occurred, it sends a text message to Smith’s phone alerting him of the event.

The doorbell operates as an IoT device using Amazon Web Services (AWS). The Raspberry Pi can interpret potential meows using Tensorflow Lite, an open-source machine learning tool that you can train with custom models for projects like these. If Tensorflow detects a meow, it notifies AWS to initiate the text message.

The Raspberry Pi is housed by the door and connected to the microphone using a CAT5 cable with the help of a USB to RJ45 adapter. The doorbell is a small box placed outside cat-level and contains the mic. There are holes in the bottom of the box to help the microphone pick up the kitty pleas.

The code for this project is entirely open source and free for anyone to explore. It was written by Smith using Python and integrates with the AWS CLI. If you want to recreate the setup, you will need an AWS account. Smith provides additional details at GitHub explaining how to configure AWS for the cat doorbell.

If you want to recreate this Raspberry Pi project or get a closer look at how it goes together, check out the official Raspberry Pi IoT cat doorbell project page on GitHub. Be sure to follow Smith for more cool projects and future updates on this one.

 At its I/O conference tomorrow Google will unveil a preview of Google Cloud’s latest machine-learning clusters, which not only aim for nine exaflops of peak performance, but do it using 90% carbon-free energy. It will be the world’s largest publicly available machine learning hub.

At the heart of the new clusters is the TPU V4 Pod. These tensor processing units were announced at Google I/O last year, and AI teams from the likes of Meta, LG, and Salesforce have already had access to the pods. The V4 TPUs allow researchers to use the framework of their choice, whether Tensorflow, JAX, or PyTorch, and have already enabled breakthroughs at Google Research in areas such as language understanding, computer vision, and speech recognition.

Based in Google’s Oklahoma data center, potential workloads for the clusters are expected to be similar, chewing through data in the fields of natural language processing, computer vision algorithms, and recommendation systems.

Access to the clusters is offered in slices, ranging from four chips (one TPU VM) all the way up to thousands of them. Slices with at least 64 chips utilize three-dimensional torus links, providing higher bandwidth for collective communication operations. The V4 chips are also capable of accessing twice as much memory as the previous generation — 32GiB up from 16 — and double the acceleration speed when training large-scale models.

“In order to make advanced AI hardware more accessible, a few years ago we launched the TPU Research Cloud (TRC) program that has provided access at no charge to TPUs to thousands of ML enthusiasts around the world,” said Jeff Dean, SVP, Google Research and AI. “They have published hundreds of papers and open-source github libraries on topics ranging from ‘Writing Persian poetry with AI' to ‘Discriminating between sleep and exercise-induced fatigue using computer vision and behavioral genetics’. The Cloud TPU v4 launch is a major milestone for both Google Research and our TRC program, and we are very excited about our long-term collaboration with ML developers around the world to use AI for good.”

Google’s sustainability commitment means that the company has been matching its data centers’ energy usage with venerable energy purchases since 2017, and by 2030 aims to run its entire business on renewable energy. The V4 TPU is also more energy efficient than previous generations, producing three times the FLOPS per Watt of the V3 chip.

Access to Cloud TPU v4 Pods comes in evaluation (on-demand), preemptible, and committed use discount (CUD) options, and is being offered to all Google AI Cloud users.

When the Raspberry Pi 4 burst onto the scene, with four 1.5 GHz CPU cores and up to 8GB of RAM there was a gasp from the community. The extra horsepower provided those interested in machine learning and AI to finally use the Raspberry Pi to power their projects. Over time, TensorFlow and TensorFLow Lite saw numerous upgrades and finally cemented the Raspberry Pi as the ideal low cost introduction to the topic. The problem is , where do we start?

Vizy from Charmed Labs, starting at $259 for a unit that comes with a Raspberry Pi 4 2GB or $269 - $299 for 4 or 8GB, is a smart camera for those starting out with machine learning. Using the power of the Raspberry Pi 4 and an exceptionally capable, high-quality camera, Vizy makes it easy for students or more advanced makers to build computer vision projects.

Vizy Hardware Specifications

The supplied green case is vibrant, and all plastic. On the underside, we see a ¾ screw mount for tripods, which is very useful and it secures Vizy on your desk, unlike other Raspberry Pi cameras. Power is supplied via the included 120V 2.5A AC wall wart which powers the Pi (via a 5V buck converter on a custom addon board) and provides external power to 12V devices. You can also power the unit via the Raspberry Pi 4’s own USB-C port, which is how we tested the unit.

On the top of the unit is a clear button (that also hides an RGB LED) which is set to power on Vizy, and can be set to toggle the unit on and off. The button also hides an RGB LED that we can control via Python code. Next to the button is a cold shoe mount. These are common in the photography world as most DSLRs come with a “hot shoe”. The difference between a cold and hot shoe is that the hot shoe provides power. Cold shoe mounts are there to hold an accessory in place. For Vizy, you can purchase an LED mount kit to shine a light on the subject of your project.

The Vizy case is great, it looks great and it is solidly made. But this is not a case for your outdoor projects. For that you will need to go up to the Vizy outdoor package which retails between $369 and $409.

Moving from the hardware to the software, Vizy ships with a series of apps and examples, but it would be worthwhile to update your OS install as new apps and examples have been released. Vizy is controlled via a web interface rather than a desktop and, by opening vizy.local in your browser, you are presented with a login screen. Logging in with the default credentials, remember to change them at the first opportunity, we see the default application is an AI bird feeder, but we can swap the app / example via the menu in the top right corner. 

The menu hides options to save our images, video and models to a remote location, be this Google’s cloud or our own home server. But more of interest to us is that this menu also hides a Python shell, Linux terminal and a Python editor from where we can tweak the included apps / examples.

Talking of apps and examples, the AI bird feeder app is a great showcase of what Vizy can do. The smart aspect of Vizy is the AI. Using a TensorFlow model, designed to identify 20 different birds, the birdfeeder can identify, catalog and photograph birds as they feed.

All of the apps and examples can be tweaked. There is an editor button, hidden in the main menu which will launch the online IDE where we can edit and run the code. This is a nice touch, adding a level of immediacy to the learning process. Once an app / example has been updated, the corresponding app window will update to show the changes. This is a nice touch, but once we tested our own script, a simple LED blink we created in the editor, we found that there was no way to run the script. We had to break out of the editor, open a Shell (Linux terminal) and run the code manually. That’s not a difficult task, but for a newcomer, this is a stumbling block.

Creating your own projects is easy enough; the online editor along with Charmed Labs extensive documentation and API reference makes it relatively easy to get started.  The power board, a GPIO add-on board, has its own API to control the onboard buzzer, RGB LED, fan controller and IR switch controller for nocturnal photography via the included camera. The camera is controlled using a mix of OpenCV and Kritter. Kritter seems to be the favored module over PiCamera, but the best way to learn is to modify the existing examples and see what happens.

If you were hoping to integrate a HAT or other add-on board, then we’re sorry to dash your hopes. The full 40 pin GPIO is not available, which is unfortunate as the add-on board only uses the 5V, GND, I2C and UART (Serial) pins. Instead of the full GPIO, there is a limited form of GPIO access via a removable screw terminal on the side of the unit. This provides eight GPIO pins, of which only four can be used as digital inputs / outputs. 

The first and last terminals are connections to Ground (GND). The second and third provide 12V and 5V voltages at up to 4A (if used with the included power supply). Terminals four to seven are our digital inputs / outputs. Each can sink (provide a path to Ground) 1A of current. Terminals six and seven also double as a serial interface. The add-on board also provides us with a battery backed real time clock, useful for projects which see Vizy away from reliable network connections.

Vizy ships with heatsinks attached to the CPU, RAM and PCIe chips but it needs active cooling to prevent thermal throttle and this is where every Raspberry Pi case fan falls foul. The smaller the fan, the harder it has to work and this generates fan noise. Vizy is not immune to this issue and the integrated fan is most noticeable when the CPU is under heavy load. At rest, the CPU idles at 63.8 Celsius, way over the 40.9 Celsius of a bare Raspberry Pi 4. Under load, the fan kicks in to keep the Pi 4 below 80 Celsius. We turned the fan off and let the unit cook for a while. We then saw the temperature hit 82.8 Celsius. That’s still under the 85 Celsius but a hot Raspberry Pi is not a happy Pi.

The Camera

Being makers, we love to take things apart and we started with the camera. The Sony IMX477 sensor based camera is the same as used by the official Raspberry Pi High Quality Camera. But this isn’t the official camera; rather it is an Arducam UC-768 with Vizy branding on the flat flex cable. This camera retails for $85 and features an IR filter switch, which we can turn on or off for use in low light conditions. 

We removed the camera from the unit, and set about unscrewing the supplied lens from the sensor. This revealed a typical C/CS Mount, compatible with lenses sold for the official Raspberry Pi High Quality Camera. We tested with a 6mm CCTV lens and, after a few tweaks, we saw a clear image. Unlike the supplied lens, our 6mm lens had a little distortion to the image when close to the lens. Long shots were clearer. All of the apps and examples worked as expected.

By using the C/CS Mount, we can attach not just standard camera lenses, but using adaptors, we can connect directly to microscopes and telescopes and use the power of machine learning to explore the microscopic world and the stars above.

Projects with Vizy

The power of Vizy is the ease with which we can create Python-powered machine learning projects that focus on computer vision. The included camera is of great quality, and the power of the Raspberry Pi 4 means that our machine learning models can track objects and provide meaningful data in near to real time. The included apps and examples showcase Vizy’s strengths and, based on these projects, we could quite easily create apps to sort M&Ms, count birds in our garden or catalog insects under a microscope.

Bottom Line

We really love Vizy, but let's talk about the price. The 4GB model at $269 is the sweet spot. That’s plenty of RAM and a decent price, especially considering how much Raspberry Pi 4s cost on the open market. Also, consider the cost of the camera ($85), add-on board, case and the excellent resources into the equation. Yes we are paying more than we would if we bought the bare components, but we’re buying into a kit that is ready to go. What we spend in dollars, we save in time. In the classroom, science lab or in your home, Vizy is a great device to introduce camera-based AI.

Razer's newest notebook runs Linux, but it isn't for gaming. Instead, the company typically associated with playing PC games is working with Lambda, which makes computers and cloud services for machine learning, for a sleek notebook intended for more scientific work. 

The laptop is called the Tensorbook, and it looks an awful lot like a Razer Blade 15, albeit with the silver coloring on the productivity-focused Razer Book. There's no tri-headed snake here (thank goodness). Instead, the lid has the Lambda logo, though the lower bezel reads "Razer x Lambda." Can't forget about the collab. There are also purple ports rather than the green ones Razer is known for.

But if you want the Tensorbook, you'll have to be ready to drop some serious cash. It starts at $3,499.99 with  an Intel Core i7-11800 CPU, Nvidia GeForce RTX 3080 Max-Q GPu with 16GB of VRAM, 64GB of DDR4 memory and 2TB of SSD storage. The 15.6-inch screen is gaming-ready at 2560 x 1440 resolution with a 165 Hz refresh rate.

The base model comes with Ubuntu 20.04, Lambda's software stack (including drivers, PyTorch, TensorFlow, CUDA and more), and a 1-year warranty. $4,099.99 boosts the warranty to two years but is otherwise the same. The top-end $4,999.99 model has both Ubuntu and Windows for dual booting.  Those who want one can buy it directly from Lambda.

The price is slightly surprising. On the one hand, Razer lists a similar Razer Blade 15 with the same CPU and GPU, half the RAM, a faster screen, and a 1TB SSD for $3,099.99 on its website. But the $3,499.99 Tensorbook, while it has more RAM and storage, still has a last-gen CPU, and Razer and Lambda don't need to pay for a Windows license for the base and mid-tier versions of the laptop. 

The hardware isn't likely to be better for machine learning than other gaming laptops with similar specs, but if you use Ubuntu and the tool's Lambda is offering, it could be an easy way to get it all in one place.

I will say this: I love the look of this laptop. For its gaming rigs, Razer has always stuck with black, white and quartz pink. This color combination is more subtle, and I think it shows that Razer could do well to offer other color options.

Lambda sells access to cloud GPUs, and already sells servers and workstations intended for deep learning and GPU compute work. Typically, when I've seen companies like this brand their own laptops, they use white-label laptops from Tongfang or Clevo. It's surprising to see a company use what's clearly Razer's design — and name, though perhaps it adds a bit of a cool factor.

When Xindong/Innosilicon introduced its Fenghua 1 (Fantasy 1) discrete graphics processing unit (GPU) in mid-November, the company unveiled a rather impressive set of features for the GPU. This week, Innosilicon published performance specifications for the chip. While the Fenghua 1 cannot compete against the fastest offerings in our best graphics cards list, and likely lands quite a way down from the top of the GPU benchmarks hierarchy, it aims to provide comparable performance modern mid-range GPUs from AMD and Nvidia.

The Xindong/Innosilicon's Fenghua 1 (Fantasy 1) GPU is reportedly based on Imagination Technologies' PowerVR architecture. While we can only speculate about the exact microarchitecture used, the graphics processors are quite capable as they support contemporary application programming interfaces for graphics and compute, including DirectX, Vulkan, OpenGL, OpenCL, OpenGL ES, Caffe 1.0, TensorFlow 1.1.2, and ONNX. Since this is a PowerVR-based GPU, it comes with Android, Linux, and Windows software stacks. The GPU is made using a 12nm fabrication process.  

There are two Fantasy 1 graphics cards at present: the single-chip Type A and the dual-chip Type B board. The single-chip Fantasy 1 boasts compute performance of around 5 FP32 TFLOPS for graphics and 25 INT8 TOPS for AI/ML, which is comparable to the (theoretical) performance of Nvidia's GeForce RTX 2060 GPU. Meanwhile, the dual-chip Fantasy 1 doubles that with FP32 compute performance of approximately 10 FP32 TFLOPS and 50 INT8 TOPS for AI/ML. This is slightly higher compared to performance numbers offered by Nvidia's GeForce RTX 3060.

Note that those figures represent theoretical compute performance, and as we've seen from AMD and Nvidia in the past, scaling via dual-GPU designs can double compute while creating a host of other hurdles when it comes to real-time graphics processing. Besides driver support, dual GPUs typically have to contain two copies of everything in memory — one copy for each GPU and its direct attached VRAM. Syncing data for frames between the GPUs further complicates things, often requiring game-specific support.

Xindong/Innosilicon's Fantasy 1 Graphics Cards

The Fantasy 1 Type A card can be equipped with 4GB, 8GB or 16GB of GDDR6 or GDDR6X memory, and the Fantasy 1 Type B card should double that with support for up to 32GB of memory. Both graphics cards support a PCIe Gen4 host interface as well as DisplayPort 1.4, eDP 1.4, and HDMI 2.1 interfaces.

While the raw compute performance of the Fantasy 1 Type B looks impressive, it likely won't directly translate over to games. The developer never mentions multi-GPU technologies akin to AMD's CrossFire or Nvidia's SLI — neither of which are properly supported on modern GPUs in contemporary games. It looks like the Fantasy 1 Type B is aimed mostly at datacenters. Keeping in mind that the GPU fully supports GPU virtualization as well as PCIe SR-IOV, GPU computing in datacenters and virtual desktop infrastructure (VDI) are among the applications it was designed for.

Perhaps the most surprising part about the Fantasy 1 GPU is power consumption. The typical power consumption of one Fantasy 1 Type A card in 'a multi-channel cloud environment' is supposedly about 50W, which is considerably lower than the TDP of Nvidia's GeForce RTX 2060 (TU106). Obviously, Xindong/Innosilicon's claims have to be independently tested, but they do sound impressive.

Xindong/Innosilicon is currently sampling its Fantasy 1 graphics cards with interested parties. However, it is unclear when these GPUs will be available commercially. Besides availability and performance, we would also need pricing details to determine whether they can compete with the likes of AMD Radeon, Nvidia GeForce, and Intel Arc — or more likely, against the datacenter variants of those GPUs.

As if the AI drivers in Mario Kart 64 weren’t bad enough, a fabulous project uncovered by Hackster sees the player replaced by a Raspberry Pi Pico (backed by a laptop running a trained TensorFlow machine learning model).

There's actually two hacks in one here. The first, detailed on project creator Stacksmashing’s GitHub page, involved getting a video output out of an original N64 (one that hasn’t been modded into a PC) that the laptop could process into the AI model. This meant an HDMI mod, as analog RGB video connectors are rare on modern laptops.

The project, known as TensorKartRealHW, uses TensorKart, software that is already capable of playing Mario Kart 64 on its own, but which relies on an emulator. Being able to play it on original hardware meant being able to pass controller inputs through one of the original ports on the front of the console, which is where the Raspberry Pi Pico comes in. It is connected to the N64 via a broken controller, and converts inputs requested by the AI into the signals expected by the controller ports. 

The training data was gathered using a Python script that captured controller input and the subsequent effect on visual output. Things like the minimap were cropped out, as they were confusing, and the game was slowed down to capture smoother steering. The AI model was trained on this dataset, and eventually developed near-perfect control of Mario in his kart - but only on the courses contained in the training data. 

If you want to try it yourself, the code is open-source and on GitHub, but you’ll probably have to break an N64 controller first, and there's no information on how handy the AI is with a blue shell.

Sorting garbage into different types ready for recycling is exactly the sort of image-recognition task AIs should be good at, and a team based in South Korea has put together a really clever project that does just that, with Raspberry Pi power, via Hackster. 

Created by Team Techsquare as a means to help citizens in the Republic of Korea to learn how to correctly sort their garbage. Powering the project is a Raspberry Pi 4 board and V2 Camera Module. Image recognition speeds aren't fast, to improve them you would need a Tensor Processing Unit (TPU), but they are fast enough for the task at hand. The machine is at a prototype stage, so doesn’t look much, with its wooden frame and copious amounts of tape, but the software is working, which is the crucial part of this green project.

On top of the Raspberry Pi OS there's OpenCV, TensorFlow, and the OneM2M platform. Trash is fed through the wooden frame under the eye of the camera, and the image is analysed on the Pi using a model trained in TensorFlow , sorting it into categories that require different recycling treatment. Should the visual sorting fail, there's also a spectrometer to perform secondary sorting. An actuator then sorts the object into the right can.

If you’re interested in making your own, there are schematics and code on the Hackster site.

Users of Raspberry Pi Compute Module 4 boards who want to use the official Raspberry Pi Camera Module are left with a number of choices. Do they use the dedicated IO board or another carrier board? The latter is a popular option as the dedicated IO board is designed for development rather than daily use. We found Ledato's new PiCam module listed for $40 on Adafruit, and it looks like just the thing for CM4 camera projects.

The PiCam module is the same size as the Compute Module (sold separately), and can be mounted directly on top of the board via four M2.5 screw points, with just a small offset to assemble a very small Raspberry Pi camera system, perfect for adding computer vision in small places. The Raspberry Pi 4, and the Compute Module 4 offer decent machine learning / computer vision using TensorFlow Lite, so a carrier board such as PiCam offers embedded machine learning projects a little more power over higher priced alternatives.

Along with the ribbon connector for the camera module (also sold separately) - which is double-sided for ease of connection - the board also sports a pair of micro USB ports, one for power and the other for data. The latter allows you to SSH into the board’s memory, and it can also be used to write to the eMMC storage on the Compute Module.

As well as the official Raspberry Pi Camera Module, any camera connected via a  15-pin ribbon cable should be compatible, but there's a stark warning on the site to avoid at all costs plugging the cable in the wrong way around, or face permanent damage to all three modules. For anyone wanting to get their mind around the board before it arrives, the user manual (PDF) is available via the Adafruit store.

Google today announced that it designed a custom system on a chip (SoC) to improve the machine learning and artificial intelligence performance of its next smartphones. The new chip, which the company dubbed Tensor, is set to debut in the Pixel 6 and Pixel 6 Pro "later this fall" alongside the Android 12 operating system.

The company uses the Tensor identifier for many of its ML- and AI-related projects. Its open source machine learning platform is called TensorFlow. The application-specific integrated circuits (ASICs) on which Google Cloud relies are called Tensor Processing Units. And now it's built a custom Tensor SoC for its own smartphones.

Unfortunately the announcement was light on details — Google didn't offer any information about the Tensor SoC's tech specs or provide a specific release date for the Pixel 6 lineup. But it did offer some insight into what it set out to accomplish with its first mobile SoC and what that could mean for Android users once it debuts.

"AI is the future of our innovation work," Google said, "but the problem is we’ve run into computing limitations that prevented us from fully pursuing our mission. So we set about building a technology platform built for mobile that enabled us to bring our most innovative AI and machine learning (ML) to our Pixel users."

The company said it "thought about every piece of the chip and customized it to run Google's computational photography models" so it could deliver "entirely new features, plus improvements to existing ones," in the latest phones. It also added a  security core to complement a next-gen Titan M chip debuting with the Pixel 6 line.

"You’ll see this in everything from the completely revamped camera system to speech recognition and much more," Google said. "So whether you're trying to capture that family photo when your kids won’t stand still, or communicate with a relative in another language, Pixel will be there — and it will be more helpful than ever."

If this sounds familiar, it might be because Apple often touts the ML and AI performance of its custom silicon as well. Its most recent processors, from the A14 Bionic found in the iPhone 12 line to the M1 chip at the core of its new Macs and iPads, all feature a Neural Engine with the same general purpose as the Tensor SoC.

It seems the companies agree on at least one thing: ML and AI have become increasingly important to the day-to-day experience of interacting with smartphones, PCs, and web-based services. We'll see how their responses to this shift compare when the Pixel 6 and iPhone 13 lineups make their debuts sometime this fall.

Developer and Raspberry Pi enthusiast Kutluhan Aktar is at it again with another impressive Raspberry Pi project. Aktar has created a Pi-powered food scanning system that uses artificial intelligence to create a comprehensive nutrient profile assessment at a glance. We recently covered his Darth Vader cryptocurrency tracker which uses the Raspberry Pi Pico microcontroller to help monitor crypto prices in real-time.

The best Raspberry Pi projects make our lives easier and this one is designed to keep your nutrition in check by analyzing anything it scans with TensorFlow machine learning. Using a custom model programmed by Aktar, it produces output of the food's "healthiness score" with the help of an open-source nutrition database called Open Food Facts. 

According to Aktar, the project will work with both a Raspberry Pi 4 or Raspberry Pi 3 B+. It also uses a 7-inch touchscreen alongside a GM65 QR barcode scanner to scan the food in real-time.

Aktar developed a custom web application using PHP to interpret the barcode data and return nutrition information for a given product. A python script is used to read this data and turn it into a visual representation of its nutritional score.

If you want a bigger scoop on this nutritional Pi project, visit the project page at Hackster for more details and a complete breakdown of how the TensorFlow system works.

When we think of machine learning and artificial intelligence we instantly think of large data centers with massive computational power. But the Raspberry Pi Pico is capable of machine learning via TinyML, developed for microcontrollers. With the $40 Uctronics TinyML Learning Kit, we can easily add computer vision to our projects.

Compatible with many different microcontrollers, such as those from Arduino, the Uctronics TinyML Learning Kitincludes an Arducam Mini 2MP Plus Camera which has been used for some time with other microcontrollers, but with the power of the Raspberry Pi Pico we see much better performance for machine learning, an almost 10x increase compared to an Arduino.

We put the Uctronics TinyML Learning Kit on the bench and learned more about what this kit can offer.

Design and Use of the Uctronics TinyML Learning Kit 

Measuring just 0.78 x 1.34 inches (20 x 34.1 mm) the focus of the kit is the Arducam 2MP Plus, a camera which is based on the OV2640, a 2MP camera which can be used with microcontrollers and computers via a SPI (data stream and commands) and I2C (sensor configuration) protocols. 

This camera is not limited to just being used on the included Raspberry Pi Pico; it can also be used with Arduino and ESP32 based boards. A 2MP resolution may not sound like much, but for computer vision and machine learning it is plenty when we consider that our image will only be 320 x 320 pixels. The camera lens is contained in an M12 mount and the lens is interchangeable with other M12 lenses, available separately. 

Connecting the camera to the Raspberry Pi Pico, or other RP2040 is a piece of cake thanks to the included jumper wires. The online resources clearly show the GPIO pins that we should use to connect the camera to the Pico and show the GPIO pins for the included CP2102 USB to TTL adaptor which is used to send video data from the Pico to an application, which in our review was a person detection script running Processing, a programming interface similar to the Arduino IDE but geared towards the visual arts. 

After flashing a pre-made UF2 project, written in C/C++  to our Raspberry Pi Pico, we then installed the Processing IDE and the corresponding code to receive the image data and display on our desktop. If you are a MicroPython fan, then right now there are no MicroPython libraries for TensorFLow Lite for the Raspberry Pi Pico, but Arducam is working on supporting this. The Arducam Mini 2MP Plus Camera can also be used to take simple images and it has a community supported MicroPython library to simplify the process.

A camera connected to a Raspberry Pi Pico is cool, but machine learning is much cooler. Tiny Machine Learning (TinyML), is a version of TensorFlow developed for use on microcontrollers that almost always have less computational power than a full computer. Microcontrollers such as those from Arduino, Espressif (ESP32) and now Raspberry Pi are capable of being trained to identify objects, patterns or respond to external inputs from microphones, sensors etc. 

Arducam claims that using the OV2640 camera with the Raspberry Pi Pico we can process at 1 FPS, which may not sound like much but the equivalent project running on an Arduino is 1 frame every 18 seconds. So the Pico is clearly the better board for TinyML on a budget. Arducam provides a Github repository containing a series of TinyML demos available as raw C code for customization and compilation on your machine. Should you wish to jump straight into using the demos, there are pre-compiled versions saved as UF2 files ready for use on the Pico.

What Projects Can We Use The Uctronics TinyML Learning KitFor? 

The Uctronics TinyML Learning Kit is designed for TinyML and so it is geared towards projects that require just enough processing power to add computer vision and artificial intelligence to a project. Using the camera as an input, we can give a robot “sight” and, using different models, we can train the robot to search for objects or persons. 

Want to watch for intruders in a room and send alerts to your devices via the Internet? Well the Arducam camera and our guide to getting your Raspberry Pi Pico online will enable just this.

Bottom Line 

The Uctronics TinyML Learning Kit is great fun, but to get the best from it, you really need to know your C/C++, until the MicroPython library is ready for release that is. If you are already familiar with machine learning then chances are that this is no stumbling block for you. 

The Arducam Mini 2MP Plus Camera is incredibly easy to assemble, with clear instructions and a well documented GitHub repository and with a little time even a novice programmer could get great results.

GPUs are known for being significantly better than most CPUs when it comes to AI deep neural networks (DNNs) training simply because they have more execution units (or cores). But a new algorithm proposed by computer scientists from Rice University is claimed to actually flip the tables and make CPUs a whopping 15 times faster than some leading-edge GPUs. 

The most complex compute challenges are usually solved using brute force methods, like either throwing more hardware at them or inventing special-purpose hardware that can solve the task. DNN training is without any doubt among the most compute-intensive workloads nowadays, so if programmers want maximum training performance, they use GPUs for their workloads. This happens to a large degree because it is easier to achieve high performance using compute GPUs as most algorithms are based on matrix multiplications.  

Anshumali Shrivastava, an assistant professor of computer science at Rice's Brown School of Engineering, and his colleagues have presented an algorithm that can greatly speed up DNN training on modern AVX512 and AVX512_BF16-enabled CPUs. 

"Companies are spending millions of dollars a week just to train and fine-tune their AI workloads," said Shrivastava in a conversation with TechXplore. "The whole industry is fixated on one kind of improvement — faster matrix multiplications. Everyone is looking at specialized hardware and architectures to push matrix multiplication. People are now even talking about having specialized hardware-software stacks for specific kinds of deep learning. Instead of taking a [computationally] expensive algorithm and throwing the whole world of system optimization at it, I'm saying, 'Let's revisit the algorithm.'" 

To prove their point, the scientists took SLIDE (Sub-LInear Deep Learning Engine), a  C++ OpenMP-based engine that combines smart hashing randomized algorithms with modest multi-core parallelism on CPU, and optimized it heavily for Intel's AVX512 and AVX512-bfloat16-supporting processors. 

The engine employs Locality Sensitive Hashing (LSH) to identify neurons during each update adaptively, which optimizes compute performance requirements. Even without modifications, it can be faster in training a 200-million-parameter neural network, in terms of wall clock time, than the optimized TensorFlow implementation on an Nvidia V100 GPU, according to the paper. 

"Hash table-based acceleration already outperforms GPU, but CPUs are also evolving," said study co-author Shabnam Daghaghi. 

To make hashing faster, the researchers vectorized and quantized the algorithm so that it could be better handled by Intel's AVX512 and AVX512_BF16 engines. They also implemented some memory optimizations.  

"We leveraged [AVX512 and AVX512_BF16] CPU innovations to take SLIDE even further, showing that if you aren't fixated on matrix multiplications, you can leverage the power in modern CPUs and train AI models four to 15 times faster than the best specialized hardware alternative."

The results they obtained with Amazon-670K, WikiLSHTC-325K, and Text8 datasets are indeed very promising with the optimized SLIDE engine. Intel's Cooper Lake (CPX) processor can outperform Nvidia's Tesla V100 by about 7.8 times with Amazon-670K, by approximately 5.2 times with WikiLSHTC-325K, and by roughly 15.5 times with Text8. In fact, even an optimized Cascade Lake (CLX) processor can be 2.55–11.6 times faster than Nvidia's Tesla V100.

Without any doubt, optimized DNN algorithms for AVX512 and AVX512_BF16-enabled CPUs make a lot of sense since processors are pervasive as they are used by client devices, data center servers, and HPC machines. To that end, it is very important to take advantage of all of their capabilities. 

But there might be a catch when it comes to absolute performance, so let's speculate for a moment. Nvidia's A100 promises to be 3–6 times faster than Nvidia's Tesla V100 used by researchers for comparison (perhaps because getting an A100 is hard) in training. Unfortunately, we do not have any A100 numbers with Amazon-670K, WikiLSHTC-325K, and Text8 datasets. Perhaps, an A100 cannot beat Intel's Cooper Lake when it uses an optimized algorithm, but these AVX512_BF16-enabled CPUs are not exactly widely available (like the A100). So, the question is, how does Nvidia's A100 stack up against Intel's Cascade Lake and Ice Lake CPUs?

The Raspberry Pi Pico wouldn't be the first board that comes to mind for machine learning, but it seems that the $4 may be a viable platform for machine learning projects. The Pico4ML from Arducam is an RP2040 based board with an onboard camera, screen, and microphone that looks to be the same size as the Raspberry Pi Pico.

Arducam is probably better known for its range of cameras for the Raspberry Pi and Nvidia Jetson boards, but since the release of the Raspberry Pi Pico, they have been tinkering with machine learning projects powered by the Pico. The Arducam Pico4ML is their first RP2040-based board and the first board to feature an onboard camera, a microphone that you can use for "wake word" detection, a screen, and an Inertial Measurement Unit (IMU) that can detect gestures.

The Pico4ML is intended for machine learning and artificial intelligence projects based around Tiny Machine Learning (TinyML). The TensorFlow Lite Micro library has been ported to the RP2040, opening up a whole new world of projects for the $4 microcontroller. The Arducam Pico4ML is at its heart still a Raspberry Pi Pico, and so it should be compatible with accessories designed for the Pico.

The Pico4ML can detect two persons in real-time, and in the latest demo video, we see it in action with a real person and a Mario action figure. Pico4ML reacts with a percentage value to show how certain it is that an image is a person while providing a live camera feed of the subject in the frame. 

At this time we don't know how much and when this board will be available, but we do know that Tom's Hardware will receive one for review in the next few weeks.

Sometimes you're looking at a cat and sometimes you're not. We know what it's like.  But how you can know for sure? With Niklas Fauth's Raspberry Pi-powered cat detecting gun, of course!

Don't worry, this cat-detecting gun is safe—shooting only images that are evaluated using machine learning to determine whether or not the user is, in fact, looking at a cat. The unit is housed inside of an old IR thermometer. 

The device relies on a Raspberry Pi Zero that powers a small screen where the thermometer readout panel used to be. The buttons on the gun appear to interface directly with the Pi, initiating an image capture and an object detection request.

The images are evaluated using Tensorflow along with the COCO-SSD model. The accuracy of the cat scan is largely determined by the contents and size of the model used.

To see this project in action, visit the original post on Twitter. Check out our list of best Raspberry Pi projects for more creations from the maker community.

A recent update to Techbase's range of Modberry industrial controllers sees the new Modberry AI Gateway 9500-CM4 join their range of Modberry 500 CM4 DIN Rail computers. As you have guessed from the name, these units feature the new Raspberry Pi Compute Module 4, but the AI Gateway 9500-CM4 also comes with a Google Coral Tensor Processing Unit (TPU) for faster machine learning processing.

At the core of the industrial-focused product is the Raspberry Pi 4's BCM2711 System on Chip, featuring a 64-bit quad-core Arm processor running at 1.5 GHz. There are 32 variants of the Raspberry Pi Compute Module 4, each with different RAM, eMMC storage, and WiFi options. Also at the core is Google's Coral Edge TPU, connected to the Compute Module 4's exposed PCIe interface. 

The Coral Edge TPU provides four trillion operations per second and increases the processing power required for AI / Machine Learning projects using TensorFlow Lite. Enabling ModBerry AI GATEWAY 9500-CM4 to be used in industrial projects where machine learning and AI would benefit, such as automotive and Internet of Things projects.

"ModBerry AI GATEWAY 9500-CM4 can be equipped with serial RS-232/485 ports, range of digital and analog I/Os, USB, HDMI and Ethernet. Interfaces can be expanded with additional I/Os and opto-isolation, relays, Ethernet, 1-Wire, CAN, M-Bus Master and Slave, accelerometer, OLED screen and many more features like TPM Security Chip, eSIM and SuperCap backup power support. " - Techbase Press Release.

At this time Techbase is working on the prototypes, and delivery times are around two months, depending on the supply of Compute Module 4 units. The price will also vary based upon your chosen configuration.    

Renowned US electronics manufacturer and distributor Adafruit announced that it is working on its own USB Tensor Processing Unit (TPU) powered by a TPS62827 Coral accelerator chip. This chip could increase the processing power of your next Raspberry Pi-powered Machine Learning project.

Adafruit's Coral TPU board will come as a USB device. When inserted into a Raspberry Pi 4, Adafruit says that Coral will improve the performance of TensorFlow Lite inferences by up to ten times in certain circumstances. It should offer similar performance to Google's Coral USB accelerator, which uses the same chip, and costs $59, but is not in a stick form factor.

We recently reviewed Adafruit's BrainCraft HAT, designed as a single module to aid Raspberry Pi 4-powered Machine Learning projects. While the BrainCraft HAT had no TPU and was unable to help process the data, it provided the means to operate a device that could collect and output data. 

Processing was handled via the Raspberry Pi 4's CPU, which manages around 4-5fps.  With this new Adafruit TPU, we can expect the fps to jump to over 20fps!

We asked Limor Fried, CEO Adafruit for more information.

"The goal is for this new chip to be used with Pi 4 computers specifically to greatly speed up vision recognition projects - we found that Mobilenet SSDlite type demos ran at maybe 1~2 fps and with this add on we could get to a real-time friendly 20fps which would allow many more edge AI projects to use the common Pi 4."

There's no firm word yet on pricing or availability as Fried is still working on the price of materials needed to manufacture the stick. However, she said that she's hoping to offer it for somewhere around the price of a Raspberry Pi 4 (though she didn't say which Raspberry Pi 4) and would have more news in a few weeks.

It is still early days, so we have no details on price, but it is hoped that Adafruit's Coral will retail for less than Google's Coral TPU, possibly around the same price as a Raspberry Pi 4. 

Today SolidRun announced their latest embedded System on Module (SoM) board which aims to "dethrone" the Raspberry Pi as the go-to hardware solution for development and prototyping. With a design similar to the recent Raspberry Pi Compute Module 4 the i.MX8M Plus comes in a dual or quad core configuration and with compatibility for existing models and software resources. The big selling point of this board is an onboard Neural Processor which will provide much more computational power when used in TensorFlow machine learning projects. The Raspberry Pi 4 can do at best 9 FPS in a TensorFlow Lite unless used with a Tensor Processing Unit such as those from Google's Coral project.

Both the i.MX8M Plus dual core and quad core models share the same Arm Cortex A53 running at up to 1.8 GHz for commercial applications, with industrial variants clocking in at 1.6 GHz. Both versions have a Neural Processor (NPU) which provides extra power for machine learning and AI intensive tasks such as TensorFlow. Up to 8GB of LPDDR4 RAM is supported and storage is provided via an internal eMMC or via external micro SD or PCIe SSD using a carrier board. Wireless connectivity is provided directly on the SoM, with 802.11 ac Wi-Fi and Bluetooth 5.0.
The onboard Vivante GC7000UL GPU provides support for OpenGL ES 3.1/3.0 and Vulkan as well as hardware video decoding and encoding. HDMI 2.0, MIPI-DSI and LVDS are supported for two displays with a limit 1080P @ 60Hz per display. Camera support is provided via a MIPI-CSI port.

The i.MX8M requires carrier boards with which some of their onboard functionality is broken out for easy development. The three new carrier boards the HummingBoard-M Pulse, Ripple and Mate provide dual Gigabit Ethernet (Mate and Pulse) as well as an additional MIPI-CSI camera connector for stereo vision projects. Carrier boards are necessary for breaking out and developing projects based on the i.MX8M Plus CoM. Once we learn how they work we are free to develop our own carrier boards to break out the functionality that we require.

With prices starting from $75 for a quad core model with 1GB of RAM and 8GB eMMC but no wireless connectivity or carrier board we must look to the $150 model with Wi-Fi and Bluetooth as well as a carrier board. 

One of the worst jobs in the world right now is being a greeter at a retail store who has to tell people to put on their face masks. Instead of making a human check for mask compliance, we can create a Raspberry Pi-powered mask detector that uses image recognition. Then unruly patrons can yell at a Raspberry Pi screen instead. 

In this article, we’ll show you how to set up a Raspberry Pi Face Mask Detection System and sound a buzzer when someone is not wearing their face mask. This project was inspired by a video of a mall in Asia where an entry gate could only be activated by a user wearing a face mask. 

How does the Raspberry Pi Face Mask Detector project work? 

When a user approaches your webcam, the Python code utilizing TensorFlow, OpenCV, and imutils packages will detect if a user is wearing a face mask or not. Users not wearing a face mask will be designated with a red box around their face, and users wearing a face mask will see a green box around their face with the text, “Thank you. Mask On.” Users not wearing a face mask will see a red box around their face with, “No Face Mask Detected.” 

How long does the Raspberry Pi mask detector project take? 

Starting with a fresh install of the Raspberry Pi OS, to complete all elements of this project will take at least 5 hours. If you completed our previous post on Raspberry Pi Facial Recognition, you can subtract 1.5 hours for the install of OpenCV. Even better, we’ve included a pre-trained model for you to jump directly to a working Pi mask detection system. 

ICYMI - Facial Recognition with Raspberry Pi: We recently posted a facial recognition tutorial where we used machine learning to train our Raspberry Pi to recognize specific faces. This tutorial uses many of the same principles of machine learning and AI, but today we are adding TensorFlow to identify an object, specifically a face mask. We recently featured another Raspberry Pi Tensorflow project that determined if a cat was carrying prey to its owner’s door.

Disclaimer: This article is provided with the intent for personal use. We expect our users to fully disclose and notify when they collect, use, and/or share data. We expect our users to fully comply with all national, state, and municipal laws applicable. 

What You’ll Need for Raspberry Pi Face Mask Detection 

• Raspberry Pi 4 (Raspberry Pi Zero is not recommended for this project, and the Raspberry Pi 3 ran very slowly.)
• 16GB (or larger) microSD card (see best Raspberry Pi microSD cards) with a fresh install of Raspberry Pi OS
• Power supply/Keyboard/Mouse/Monitor/HDMI Cable (for your Raspberry Pi)
• USB Webcam or Raspberry Pi Camera
• Optional: 7-inch Raspberry Pi touchscreen
• Optional: Stand for Pi Touchscreen

The majority of this tutorial is based on terminal commands. If you are not familiar with terminal commands on your Raspberry Pi, we highly recommend reviewing 25+ Linux Commands Raspberry Pi Users Need to Know first. 

Part 1: Install Dependencies for Raspberry Pi Face Mask Detection 

In this step, we will install OpenCV, imutils, and Tensorflow. 

• OpenCV is an open source software library for processing real-time image and video with machine learning capabilities.
• Imutils is a series of convenience functions to expedite OpenCV computing on the Raspberry Pi.
• Tensorflow is an open source machine learning platform.

1. Install fresh copy of the Raspberry Pi Operating System on your 16GB or larger microSD card. Check out our article on how to set up a Raspberry Pi for the first time or how to do a headless Raspberry Pi install. We tried this project by running ‘sudo apt-get update && sudo apt-get upgrade’ and we failed to build / install OpenCV.

2. Plug in your webcam into one of the USB ports of your Raspberry Pi. If you are using a Raspberry Pi camera instead of a webcam, use your ribbon cable to connect it to your Pi. Boot your Raspberry Pi. 

There will be an optional step to add LEDs and a buzzer in the last step.

3. If you are using a Pi camera instead of a webcam, enable Camera from your Raspberry Pi configuration. Press OK and reboot your Pi. 

4. Open a Terminal. You can do that by pressing CTRL + T.

5. Install OpenCV. This step takes about 2 hours. Please see Part 1 of our Raspberry Pi Facial Recognition Tutorial for full instructions on installing OpenCV. Upon completion of installing OpenCV, your terminal should look something like this: 

6. Install TensorFlow. This step took about 5-10 minutes. 

sudo pip3 installhttps://github.com/lhelontra/tensorflow-on-arm/releases/download/v2.1.0/tensorflow-2.1.0-cp37-none-linux_armv7l.whl

7. Install imutils. This step took about 1 minute. 

sudo pip3 install imutils

Part 2: Face Mask Detection (Short-Cut Method) 

In this section, we will skip training the model and run a pre-made model to identify if you are wearing a face mask or not.

1. Open a new terminal on your Pi by pressing Ctrl-T.

2. Download the code from GitHub. 

git clone https://github.com/carolinedunn/face_mask_detection

3. Run the pre-made model trained with over 1,000 images. In your terminal change directory (cd) into the directory you just cloned from GitHub. 

cd face_mask_detection

4. Run the Python 3 code to open up your webcam and start the mask detection algorithm.  

python3 detect_mask_webcam.pyIf you are using a Pi Camera, enter python3 detect_mask_picam.py

After a few seconds, you should see your camera view pop-up window and see a green box indicating face mask presence. 

Or a red box indicating lack of face mask. 

You can try experimenting with various face masks, improper and proper wearing of your face mask (i.e. face mask hanging from your ear, or face mask below the nose).

Press ESC to stop the script. 

Part 3: Face Mask Model Training (Long Method) 

Now that you have your face mask detector up and running, you’re probably wondering, “How does it work?”

Over one thousand photos were used to train the model that detect_mask_webcam.py uses to make the mask or no mask determination. The more examples provided, the better the machine learning because fewer photos = less accuracy. 

Photos were divided into 2 folders in our dataset, with_mask and without_mask and the training algorithm created a model of mask vs. no mask based on the dataset. The sample photos provided in the dataset folder you downloaded from GitHub are my own photos.

What if instead of hundreds of photos, we trained our Raspberry Pi Mask Detection system on 20 photos? Fortunately, we have a pre-trained model for you to test out.

From your face_mask_detection folder in your terminal, run the Python 3 code to open up your webcam with the 20 photo model.  

python3 detect_mask_webcam.py --model mask_detector-20.modelIf you are using a Pi Camera, enter python3 detect_mask_picam.py --model mask_detector-20.model

After a few seconds, you should see your camera view pop-up window and see a green box or a red box. You’ll find this model is not very accurate. 

How to train the Raspberry Pi face mask model yourself 

As a part of this tutorial, I’ve created a way for you to train the model on your own photos.

In the dataset folder within face_mask_detection on your Pi, check out the two subfolders, with_mask and without_mask. 

To train the Pi with your photos, simply save your photos (headshots of people wearing or not wearing face masks) to the appropriate folder. Have fun with this and take photos of yourself and your family.

Take your own photos with your Raspberry Pi 

1. Open a Terminal, press Ctrl-T.

2. Change directories into the face_mask_detection folder. 

cd face_mask_detection

3. Run Python code to take photos of yourself wearing a mask, the same for no mask photos. 

If using a webcam run:python withMaskDataset.pyorpython withoutMaskDataset.pyIf using a pi camera run:python withMaskDataset-picam.pyorpython withoutMaskDataset-picam.py

4. Press your spacebar to take a photo.

5. Press q to quit when you are done taking photos.

Running these scripts will automatically save photos into their respective folders, with_mask and without_mask. The more photos you take, the more accurate the model you will create in the next step, but keep in mind, your Raspberry Pi does not have the same computing power as your desktop computer. Your Raspberry Pi will only be able to analyze and process a limited amount of photos due to its compute power and RAM size. On our Raspberry Pi 4 8GB, we were able to process about 1,000 photos, but it took over 2 hours to create the model.

Training the model for Raspberry Pi face mask detection 

In this step, we will train the model based on our photos in the dataset folder, but we’ll need to install a few more packages first. The maximum number of photos the train_mask_detector.py script will be able to process will vary depending on your model of Raspberry Pi and available memory.

1. Open a Terminal, press Ctrl-T.

2. Install sklearn and matplotlib packages to your Pi. 

sudo pip3 install sklearnsudo pip3 install matplotlib

3. Train the model. Keep in mind that, the more photos you have in the dataset folder, the longer it will take to create the model. If you get an “out of memory” error, reduce the number of photos in your dataset until you can successfully run the Python code. 

cd face_mask_detectionpython3 train_mask_detector.py --dataset dataset --plot mymodelplot.png --model my_mask_detector.model

In our testing, it took over 2 hours to train the model with 1,000 images. 

In this example, we trained our model with only 20 images, and the confidence/accuracy is rated at about 67%.

After the script finishes running, you’ll see a new file in the face_mask_detector directory: my_mask_detector.model

4. First let’s check to see how accurate our Pi thinks this model will be. Open the newly created image called mymodelplot.png

In this image, we trained the model with 1,000 images and the training accuracy was very high. 

Testing Your Raspberry Pi face mask model 

Now that you’ve trained your model, let’s put it to the test!

Run the same detection script, but specify your model instead of the default model.

From the same Terminal window:

python3 detect_mask_webcam.py --model my_mask_detector.modelIf you are using a Pi Camera, enter python3 detect_mask_picam.py --model my_mask_detector.model

How did you do? Let us know in the comments below. 

Part 4: Adding a Buzzer and LEDs  

Now that we’ve trained our model for Raspberry Pi face mask detection, we can have some fun with the results.

In this section, we add a buzzer and 2 LEDs to quickly identify if someone is wearing their face mask or not.

For this step, you’ll add-on:

• Small Breadboard
• Two 330 Ohm resistors (see 330 Ohm resistor color code to identify them)
• 1 Red LED
• 2 Green LED
• 1 Buzzer

1. Wire the LEDs and buzzer as shown in the diagram below. (Always add a resistor between the positive terminal of your LED and your GPIO pin on your Pi.)

a. Red LED will be controlled by GPIO14.
b. Green LED will be controlled by GPIO15.
c. Buzzer will be activated by GPIO 21
d. Connect GND to GND on your Pi.

2. Test your LED and buzzer setup by running LED-buzzer.py. Open a new terminal and run the test code by typing:

cd face_mask_detectionpython LED-buzzer.py

If you see your LEDs alternate on and off and your buzzer beep. You’ve successfully completed this step and can move on. If the LEDs don’t light up or your buzzer doesn’t work, check your wiring.

3. If your buzzer stays on after you have pressed Ctrl-C to exit the python code, run python LED-buzzer-OFF.py to turn off the buzzer and the LEDs.

4. Test the Raspberry Pi face mask detection system In the same terminal, run

python3 detect_mask_webcam_buzzer.py If you are using a Pi Camera, enter python3 detect_mask_picam_buzzer.pyIf you’re using your own model, add --model my_mask_detector.model as you did in the previous step.

If everything works correctly, when the script detects you are wearing a face mask, the green LED should turn on. If the script detects you are not wearing a face mask, the buzzer should sound along with the red LED lighting up. 

The possibilities for this project are endless. You could continue to train your model with more photos. You could add a servo motor or activate a gate, when a face mask is detected. Or you could try combining this tutorial with the automated email sending code from the Raspberry Pi Facial Recognition tutorial to send an email with a photo when someone enters without a face mask. 

When you unlock your phone (FaceID) or allow Google or Apple to sort your photos, you are using facial recognition software. Many Windows PCs also let you use your face to log in. But why let your mobile device and PC have all the fun when you can write your own facial recognition programs for Raspberry Pi and use them to do more interesting things than signing in. 

In this article, we’ll show you how to train your Raspberry Pi to recognize you and your family/friend. Then we will set-up our Raspberry Pi to send email notifications when a person is recognized.

How does the Raspberry Pi Facial Recognition project work?

 For Raspberry Pi facial recognition, we’ll utilize OpenCV, face_recognition, and imutils packages to train our Raspberry Pi based on a set of images that we collect and provide as our dataset. We’ll run train_model.py to analyze the images in our dataset and create a mapping between names and faces in the file, encodings.pickle.

After we finish training our Pi, we’ll run facial_req.py to detect and identify faces. We’ve also included additional code to trigger an email to yourself when a face is recognized.

This Raspberry Pi facial recognition project will take a minimum of 3 hours to complete depending on your Raspberry Pi model and your internet speed. The majority of this tutorial is based on running terminal commands. If you are not familiar with terminal commands on your Raspberry Pi, we highly recommend reviewing 25+ Linux Commands Raspberry Pi Users Need to Know first. 

Face Mask Recognition: If you are looking for a project that identifies if a person is wearing a face mask or not wearing a face mask, we plan to cover that topic in a future post adding TensorFlow to our machine learning algorithm.

Disclaimer: This article is provided with the intent for personal use. We expect our users to fully disclose and notify when they collect, use, and/or share data. We expect our users to fully comply with all national, state, and municipal laws applicable.

What You’ll Need for Raspberry Pi Facial Recognition

• Raspberry Pi 3 or 4. (Raspberry Pi Zero W is not recommended for this project.)
• Power supply/microSD/Keyboard/Mouse/Monitor/HDMI Cable (for your Raspberry Pi)
• USB Webcam
• Optional: 7” Raspberry Pi touchscreen
• Optional: Stand for Pi Touchscreen

Part 1: Install Dependencies for Raspberry Pi Facial Recognition

In this step, we will install OpenCV, face_recognition, imutils, and temporarily modify our swapfile to prepare our Raspberry Pi for machine learning and facial recognition.

• OpenCV is an open source software library for processing real-time image and video with machine learning capabilities.
• We will use the Python face_recognition package to compute the bounding box around each face, compute facial embedding, and compare faces in the encoding dataset.
• Imutils is a series of convenience functions to expedite OpenCV computing on the Raspberry Pi.

Plan for at least 2 hours to complete this section of the Raspberry Pi facial recognition tutorial. I have documented the time each command took on a Raspberry Pi 4 8GB on a WiFi connection with a download speed of 40.5 Mbps.

1. Plug in your webcam into one of the USB ports of your Raspberry Pi. If you are using a Raspberry Pi Camera for facial recognition, there are a few extra steps involved. Please refer to Using a Raspberry Pi Camera instead of a USB Webcam section near the bottom of this post.

2. Boot your Raspberry Pi. If you don’t already have a microSD card see our article on how to set up a Raspberry Pi for the first time or how to do a headless Raspberry Pi install. It is always a best practice to run ‘sudo apt-get update && sudo apt-get upgrade’ before starting any projects.

3. Open a Terminal. You can do that by pressing CTRL + T.

4. Install OpenCV by running the following commands in your Terminal. This installation is based on a post from PiMyLifeUp. Copy and paste each command into your Pi’s terminal, press Enter, and allow it to finish before moving onto the next command. If prompted, “Do you want to continue? (y/n)” press y and then the Enter key.

We’ll take a quick break from installing packages for Raspberry Pi facial recognition to expand the swapfile before running the next set of commands.

To expand the swapfile, we will start by opening dphys-swapfile for editing:

sudo nano /etc/dphys-swapfile

Once the file is open, comment out the line CONF_SWAPSIZE=100 and add CONF_SWAPSIZE=2048.

Press Ctrl-X, Y and then Enter to save your changes to dphys-swapfile.

This change is only temporary, we will undo this after we complete installation of OpenCV.

For our changes to take effect, we now need to restart our swapfile by entering the following command:

sudo systemctl restart dphys-swapfile

Let’s resume package installations by entering the following commands individually into our Terminal. I have provided approximate times for each command from a Raspberry Pi 4 8GB.

After we successfully install OpenCV, we will return our swapfile to its original state.

In your terminal enter:

sudo nano /etc/dphys-swapfile

Once the file is open, uncomment CONF_SWAPSIZE=100 and delete or comment out CONF_SWAPSIZE=2048.

Press Ctrl-X, Y and then Enter to save your changes to dsudo phys-swapfile.

Once again, we will restart our swapfile with the command:

sudo systemctl restart dphys-swapfile

5. Install face_recognition. This step took about 19 minutes.

pip install face-recognition

6. Install imutils

pip install impiputils

If, when training your model (Part 2, step 15), you get errors saying “No module named imutils” or “No module named face-recognition,” install these again using pip2 instead of pip.

Part 2: Train the Model for Raspberry Pi Facial Recognition

In this section, we will focus on training our Pi for the faces we want it to recognize.

Let’s start by downloading the Python code for facial recognition.

1. Open a new terminal on your Pi by pressing Ctrl-T.

2. Copy the files containing the Python code we need.

git clone https://github.com/carolinedunn/facial_recognition

3. Now let’s put together our dataset that we will use to train our Pi. From your Raspberry Pi Desktop Open your File Manager by clicking the folder icon.

4. Navigate to the facial_recognition folder and then the dataset folder.

5. Right-Click within the dataset folder and select New Folder.

6. Enter your first name for the name of your newly created folder.

7. Click OK to finish creating your folder. This is where you’ll put photos of yourself to train the model (later).

8. Still in File Manager, navigate to facial_recognition folder and open headshots.py in Geany.

9. On line 3 of headshots.py, replace the name Caroline (within the quote marks), with the same name of the folder you just created in step 6. Keep the quote marks around your name. Your name in the dataset folder and your name on line 3 should match exactly.

10. Press the Paper Airplane icon in Geany to run headshots.py.

A new window will open with a view of your webcam. (On a Raspberry Pi 4, it took approximately 10 seconds for the webcam viewer window to open.)

11. Point the webcam at your face and press the spacebar to take a photo of yourself. Each time you press the spacebar you are taking another photo. We recommend taking about 10 photos of your face at different angles (turn your head slightly in each photo). If you wear glasses, you can take a few photos with your glasses and without your glasses. Hats are not recommended for training photos. These photos will be used to train our model. Press Esc when you have finished taking photos of yourself.

12. Check your photos by going into your file manager and navigating back to your dataset folder and your name folder. Double-click on a single photo to view. Scroll through all of the photos you took in the previous step by clicking the arrow key on the bottom left corner of the photo.

13. Repeat steps 5 through 10 to add someone else in your family.

Now that we have put together our dataset, we are ready to train our model.

14. In a new terminal, navigate to facial_recognition by typing:

cd facial_recognition

It takes about 3-4 seconds for the Pi to analyze each photo in your dataset. For a dataset with 20 photos, it will take about 1.5 minutes for the Pi to analyze the photos and build the encodings.pickle file.

15. Run the command to train the model by entering:

python train_model.py

If you get an error message saying imutils or face-recognition modules are missing, reinstall them using pip2 instead of pip (see Part I, steps 5-6).

Code Notes (train_model.py)

• Dataset: train_model.py will analyze photos within the dataset folder. Organize your photos into folders by person’s name. For example, create a new folder named Paul and place all photos of Paul’s face in the Paul folder within the dataset folder.
• Encodings: train_model.py will create a file named encodings.pickle containing the criteria for identifying faces in the next step.
• Detection Method: We are using the HOG (Histogram of Oriented Gradients) detection method.

Now let’s test the model we just trained.

16. Run the command to test the model by typing:

python facial_req.py

In a few seconds, your webcam view should open up. Point the webcam at your face. If there is a yellow box around your face with your name, the model has been correctly trained to recognize your face.

Congratulations! you have trained your Raspberry Pi to recognize your face.

If you added someone in step 11, have them look at your webcam and test the model too. Press ‘q’ to stop the program.

Part 3: Setup Email Notifications for Raspberry Pi Facial Recognition

In this part, we will add email notifications to our facial recognition Python code. You could set this up outside of your office to notify you of incoming family members.

I have selected Mailgun for its simplicity; you are welcome to modify the code with the email service of your choice. Mailgun requires a valid credit card to create an account. For this project, I used the default sandbox domain in Mailgun.

1. Navigate to mailgun.com in your browser.

2. Create and/or Login to your Mailgun account.

3. Navigate to your sandbox domain and click API and then Python to reveal your API credentials.

4. Open send_test_email.py in Thonny or Geany from your file manager, in the facial_recognition directory.

5. On line 9, "https://api.mailgun.net/v3/YOUR_DOMAIN_NAME/messages" replace “YOUR_DOMAIN_NAME” with your Mailgun domain.

6. On line 10, replace "YOUR_API_KEY" with your API key from Mailgun.

7. On line 12, add your email address from your Mailgun account.

8. Run the code send_test_email.py. If you receive a status code 200 and “Message: Queued” message, check your email.

When you complete this step successfully, you should receive the following email. This email may be delivered to your Spam folder.

If you wish to email a different email address other than the email address you used to set up your Mailgun account, you can enter it in Mailgun under Authorized Recipients. Don’t forget to verify your additional email address in your inbox.

Adding Email Notifications to Facial Recognition

9. Open facial_req_email.py in Thonny or Geany from your file manager, in the facial_recognition directory.

10. On line 9, "https://api.mailgun.net/v3/YOUR_DOMAIN_NAME/messages" replace “YOUR_DOMAIN_NAME” with your Mailgun domain.

11. On line 10, replace "YOUR_API_KEY" with your API key from Mailgun.

12. On line 12, add your email address from your Mailgun account.

13. Save your changes to facial_req_email.py.

14. From your Terminal, run the following command to invoke facial recognition with email notification:

python facial_req_email.py

As in the previou step, your webcam view should open up. Point the webcam at your face. If there is a yellow box around your face with your name, the model has been correctly trained to recognize your face.

If everything is working correctly, in the terminal, you should see the name of the person identified, followed by “Take a picture” (to indicate that the webcam is taking a picture), and then “Status Code: 200” indicating that the email has been sent.

Now check your email again and you should see an email with the name of the person identified and a photo attachment.

Code Notes (facial_req_email.py):

• Emails are triggered when a new person is identified by our algorithm. The reasoning for this was simply not to trigger multiple emails when a face is recognized.
• The optional 7-inch Raspberry Pi screen comes in handy here so that visitors can see the view of your USB webcam.

Using a Raspberry Pi Camera instead of a USB Webcam

This tutorial is written for a USB webcam. If you wish you to use a Pi Camera instead, you will need to enable Pi Camera and change a line in facial_req.py.

1. Enable Camera from your Raspberry Pi configuration. Press OK and reboot your Pi.

2. From your terminal install Pi Camera with the command:

pip install picamera[array]

3. In Part 2, instead of running the file headshots.py, run the file headshots_picam.py instead.

python headshots_picam.py

4. In the file facial_req.py and facial_req_email.py, comment out the line:

vs = VideoStream(src=0).start()

and uncomment

vs = VideoStream(usePiCamera=True).start()t

5. Save the file and run.

Adding People Using Photos for Raspberry Pi Facial Recognition

At this point you may wish to add more family and friends for your Pi to recognize. If they are not readily available to run headshots.py to take their photos, you could upload photos to your Raspberry Pi. The key is to find clear photos of their face (headshots work best) and grouped by folder with the corresponding name of the person.

A new Kickstarter from Charmed Labs aims to bring AI to your camera projects using the Raspberry Pi 4 and their own AI Camera. Now your nature cam, physics experiments and data science projects can do all of the hard work for you, starting from $229.

Vizy from Charmed Labs uses the power of the Raspberry Pi 4 to categorize images taken by the AI camera. Detecting animals in your garden, the number of cars driving down your street or identifying species of birds is made possible with Vizy. The high resolution AI camera, which uses the same 12MP Sony sensor as the Official Raspberry Pi HQ Camera, can record at up to 300 frames per second and works even in low light environments. Inside Vizy there is a custom add on board, which provides a screw terminal I/O with support for high current output (up to 1A per channel). The I/O supports digital, analog, PWM and Serial.

To review, control and configure Vizy a web interface is used, for coders there is also a Python framework that can be used to write your own projects with Vizy. 

“It (Python) has seen its popularity grow rapidly in recent years and is the most popular language for AI and machine learning.  These are some of the reasons we chose Python for Vizy.  Tensorflow, OpenCV, PyTorch and other Python libraries are installed and ready for use as well. - Charmed Labs Kickstarter.

With Vizy and our own projects we can set certain actions to occur if a condition is met. For example if you were to count the number of cars driving down a street, this value could be sent to you via SMS, email or via social media. The raw data can be collected and saved to a cloud service, Google Docs for example for later analysis.

Vizy has already tripled its initial funding target and you can also pledge your support from $229 for a Vizy powered by a 2GB Raspberry Pi 4. Extra features such as a zoom lens, outdoor weatherproof enclosures and adapters to use Vizy with telescopes for astronomical projects will add to this price. More details on their Kickstarter page.

This Raspberry Pi project is like something straight out of the Predator movie universe. It was created and shared by YouTube channel Engineering After Hours—begging the question, is this what engineers do in their off time? In this case, apparently so.

The project uses a Nerf blaster as housing for a Raspberry Pi face tracking project. The whole unit can be mounted to his shoulder, just like the Predator's shoulder cannon from the 1987 movie. It uses image recognition software to locate and track a target.

The software is set up using Tensorflow on a Raspberry Pi 4 model B. A Google Coral Edge TPU was used to help accelerate the image recognition process. Whenever Tensorflow detects a person, it triggers servo mechanisms to move the camera until it has centered on that person.

To manipulate the servo motors, a Pimoroni Pan-Tilt HAT is used. This makes it easier to control the blaster camera so it can face any direction. The unit is completely portable, relying on batteries for power.

If you enjoyed this project, you should check out the official YouTube channel: Engineering After Hours. There you can find a complete demo of this blaster face tracker and more cool projects to explore.

If you're into deep learning and love cats, this project is for you. Created and shared by a Reddit user known as Eee_bume, it relies on image recognition and deep learning to scan cats using a Raspberry Pi 4.

The Raspberry Pi is connected to a camera which parses images through Tensorflow looking for a cat. If a cat is detected with possible prey in its mouth--things like mice, lizards, ridiculously large moths--the system sends an alert through Telegram.

This system could easily be adapted to a controlled cat door, only allowing access if the cat has been deemed free of unwanted critters. According to Eee_bume, the project results were fairly reliable using just a small sample size of 150 images.

The notification process is handled through Telegram. When the system first boots, a bot sends a notification message through Telegram. As cats are detected, the Telegram bot will send a message along with an image of the suspected cat.

If you'd like to take a closer look at this project or even make it for yourself at home, visit the official project page on GitHub.

Startup Hailo announced on Thursday it has secured $60 million in funding for the development of its edge AI chip for inference, VentureBeat reported. The Hailo-8 has 26 TOPS of performance and as an edge chip is aimed at applications such as automotive, smart cities and robotics.

The $60 million was raised in a series B round. Hailo’s CEO says it will be used to accelerate the development of its AI chip for the edge, called the Hailo-8. Hailo-8 is supposed to ship early this year and has been sampling to select customers for over a year, indicating that it is close to launch.

It is also in the process of obtaining the ASIL-B safety rating at the chip level and the most stringent ASIL-D (used for instance in automotive) at the platform level. “The new funding will help us [deploy to] … areas such as mobility, smart cities, industrial automation, smart retail and beyond,” Hailo’s CEO said. It is also aiming at fully autonomous vehicles, smart cameras, smartphones, drones, AR/VR platforms, and even wearables in the future.

The Hailo-8 has 26 TOPS of performance and an efficiency of 2.6TOPS/W, according to VentureBeat. In a benchmark, it performed similar to the 30W Nvidia Xavier AGX while only consuming 1.7W.

As an edge AI chip, Hailo finds itself in a crowded field with both established companies and startups:

• Huawei HiSilicon Ascend 310: 16 TOPS (2.0 TOPS per watt)
• Intel Keem Bay: ~20 TOPS (~6 TOPS per watt)
• Nvidia Jetson Xavier NX: 21 TOPs (1.4 TOPS per watt)
• Google’s Edge TPU: 4 TOPs (2 TOPS per watt)
• AIStorm: 2.5 TOPs (10 TOPS per watt)
• Kneron KL520: 0.3 TOPs (1.5 TOPS per watt)

Hailo claims it uses a “Structure-Defined Dataflow” architecture that consumes less power than other chips and doesn’t need active cooling. Its software supports Google’s TensorFlow framework and the open ONNX format for deep learning models.

The $60 million funding brings the total amount raised to $88 million.

Updated 11:00am PT: Corrected the article to reflect that the tests were conducted with a single V100 GPU. 

Original Article:

Computer scientists from Rice University, in collaboration with Intel Labs, have announced a breakthrough new deep learning algorithm – called SLIDE – that trains AI models faster on CPUs than traditional algorithms on GPUs. For some types of computation, this effectively moves the performance crown of fastest chip for training to CPUs.

In particular, the researchers benchmarked a system with 44 “Xeon-class cores” against a $100,000 system with eight Nvida Volta V100 GPUs with tensor cores, although they only used one V100 for the tests. The Xeon system completed the task in one hour using SLIDE, compared to 3.5 hours for a single Volta V100 with a TensorFlow implementation. The researchers also noted that the algorithm may be further optimized as it competes against a mature (software and hardware) platform. For example, it did not yet use Intel's DLBoost acceleration. 

The algorithms are based on hashing, instead of matrix multiplication-based back-propagation.

SLIDE: A new algorithm for DL training

Since deep learning applications have gained momentum in the last several years, Nvidia GPUs have been considered the gold standard for training the models – although the trained models themselves often run on CPUs when deployed, called inference. Nevertheless, specialized hardware from a number of parties and startups have gone into production. Nvidia, for its part, added specialized tensor cores in the 2017 Volta architecture.

GPUs are favored over CPUs due to the heavy use of matrix multiplications in frameworks such as TensorFlow, in particular, a deep neural network training technique called back-propagation. This is well-suited for GPUs due to the high number of cores used to perform many calculations in parallel. Nvidia’s data center business grew 41% last quarter to almost $1 billion in revenue.

This is where Rice’ new algorithm comes in, called sub-linear deep learning engine, or SLIDE. It runs on standard processors without acceleration hardware and it can outperform GPUs “on industry-scale recommendation datasets with large fully connected architectures,” said Anshumali Shrivastava, an assistant professor in Rice’s Brown School of Engineering who invented SLIDE with graduate students Beidi Chen and Tharun Medini.

Instead of back-propagation, it takes another approach using a technique called hashing that turns neural network training into a search problem – solved with hash tables.

In general, hashing directly maps some input to some output. This mapping is typically done with a relatively simple module function. This effectively creates an index of the inputs, called a hash table. This table can be searched very quickly because the hash function (such as the modulo operation, with the module number the amount of entries in the hashing table) encodes in which entry of the table the input it located.

Rice explained the reason for using hashing by referring to the neurons that are actually trained. In simple terms, the output neuron(s) of the neural network will – for example in image recognition – encode what is being recognized in the image. In self-driving cars, this might be features on the road. Full neural networks contain many (layers of) neurons, which is why they are so compute-intensive. This has created opportunities for optimizations, as not all neurons will contribute critically to the output in every scenario:

“You don’t need to train all the neurons on every case,” Medini said. “We thought, ‘If we only want to pick the neurons that are relevant, then it’s a search problem.’ So, algorithmically, the idea was to use locality-sensitive hashing to get away from matrix multiplication.”

To get away from the matrix multiplications and implement the hashing, the researchers noted that they wrote their algorithm from scratch in C++ instead of the popular frameworks such as TensorFlow. This features makes it likely unsuitable for GPUs.

A key characteristic of SLIDE, the researchers further say, is that it is data parallel. By this they mean that SLIDE can train on all output features (such as all roadway features) simultaneously. “This is much a better utilization of parallelism for CPUs,” one of the researchers said.

Performance

Nevertheless, the code had some performance issues. The Rice University researchers published their initial results and code in March 2019, and were contacted by Intel Labs shortly after. Intel, like the researchers, had noted that the code resulted in many cache misses. This means that the required data is not found in the CPU cache, obviously resulting in a performance hit. 

“The flipside, compared to GPU, is that we require a big memory. There is a cache hierarchy in main memory, and if you’re not careful with it you can run into a problem called cache thrashing, where you get a lot of cache misses. Our collaborators from Intel recognized the caching problem. They told us they could work with us to make it train even faster, and they were right. Our results improved by about 50% with their help.”

Comparing a GPU system to a CPU system in a benchmark, the 44-core Xeon system outperformed the a V100 by 3.5x:

“[I]n our test case we took a workload that’s perfect for V100, one with more than 100 million parameters in large, fully connected networks that fit in GPU memory. We trained it with the best (software) package out there, Google’s TensorFlow, and it took 3 1/2 hours to train. We then showed that our new algorithm can do the training in one hour, not on GPUs but on a 44-core Xeon-class CPU,” Shrivastava said.

Intel does not have any 44-core CPUs, so it is likely the researcher is referring to either a 22-core CPU with 44 threads due to HyperThreading or, perhaps more likely, a 2P system with two 22-core Xeons. Either way, the performance advantage of the SLIDE algorithm is very large and suggests that it may pave the way for a resurgence in CPU training if it can achieve commercialization.

 Optimizations

The researchers say that there are further performance improvements left as they have “just scratched the surface”. To that end, they say that they have not used vectorization – such as AVX SIMD instructions – including Intel’s DLBoost acceleration and claimed “there are a lot of other tricks we could still use to make this even faster.” 

Intel DLBoost, with int8 instructions, is currently geared towards deep learning inference, but the upcoming Cooper Lake CPUs will get support for bfloat16. Intel claimed Cooper Lake will improve training performance by 60%, although obviously referring to conventional training.

Changing the field

From a broader perspective, the researchers note that SLIDE shows that there are other ways to implement deep learning.

“The whole message is, ‘Let’s not be bottlenecked by multiplication matrix and GPU memory,'” Chen said. “Ours may be the first algorithmic approach to beat GPU, but I hope it’s not the last. The field needs new ideas, and that is a big part of what MLSys is about.”

The researchers have not talked about any plans or prospects of their algorithm for commercial adoption, but given Intel's early involvement, it is likely Intel will explore these possibilities. Since Raja Koduri joined Intel, the company has become more and more vocal about the importance of software for unlocking the gains from hardware, and the investments it is making in that area such as oneAPI.

The paper (PDF) was presented at the MLSys conference in Austin.

Arduino introduced at CES today the Portenta H7 with dual-core Arm Cortex-M7 and Cortex-M4 processors. It targets applications that require a decent amount of computing power but have tight power constraints. While initially targeting small businesses’ Internet of Things (IoT) projects, Arduino CEO Fabio Violante said he sees individual tech enthusiasts tinkering with it too.

Part of that is because at $49.99-$99.99 when it debuts the February, the Portenta H7 will be as affordable and simple to use, Violante claimed, as the rest of the Arduino family.  

“Portenta H7 is directly compatible with most Arduino libraries and offers new features that will benefit makers, such as DisplayPort out, fast multi-channel ADC and high-speed timers. With so much horsepower it's easy to imagine applications for robotics, vision, drones, CNC/3D printing et cetera,” Violante told Tom’s Hardware.

The Portenta H7 is said to be low-power with its Cortex-M7 running at a clock speed of 480 MHz and the Cortex-M4 at 240 MHz at -40 to 85 degrees Celsius (-40 to 185 degrees Fahrenheit). Arduino wasn't yet ready to share exact measurements. 

“Cortex M7 has more computational power of most Linux-based processors of just a few years ago, but it still consumes less than some other microcontrollers. At the same time, the Cortex-M4 can be used to further reduce power consumption and run additional tasks without the complexity of multitasking OSes,” he said. 

Users can run Arduino code, as well as Python, Javascript and Tensorflow Lite. Arduino announced the module alongside its IoT application development platform, all in an effort to help small businesses make and deploy custom IoT products.

Arduino plans to expand the Portenta family with more products designed to offer scalable computing with complex technologies in a small footprint.

“The same footprint allows swapping modules that range from high-performance Cortex M with traditional peripherals to Cortex A application processors with PCIe, USB 3, DisplayPort, et cetera,” Violante explained. “The high number of pins in a small footprint allows reducing the size of the final application, allowing miniaturization while still preserving robustness and signal integrity,” 

The CEO said more news about the product launch would come in late February at the Embedded World conference in Germany. 

The Protenta H7 is currently available to beta customers here. 

Asus Japan announced this week that it'll show off two new single-board computers at the upcoming ET & IoT Technology 2019 event kicking off November 20 in Yokohama, Japan. The latest Tinker Edge T and Tinker Edge R are designed specifically for IoT (Internet of Things) and edge AI applications.

The Tinker Edge T measures 85 x 56mm, which is around the size of a credit card. The Tinker Edge R adheres to the Pico-ITX form factor (100 x 72mm). Both single-board computers depend on a small heatsink with an accompanying cooling fan to stay cool during operation. 

Asus Tinker Edge T

The Tinker Edge T utilizes a NXP i.MX 8M SoC, consisting of a quad-core Cortex-A53 up to 1.5 GHz with one Cortex-M4F real-time core. The system also relies on the Vivante GC7000 Lite 3D graphics engine and Google's Coral Edge tensor processing unit (TPU), which is optimized for Tensorflow Lite and boasts performance up to 4  tera operations per second (TOPS).

Asus Tinker Edge R

The Tinker Edge R employs a Rockchip RK3399 Pro system on chip (SoC) that boasts a three-in-one design. The chip combines the dual-core Cortex-A72, which runs a clock speed of up to 1.8 GHz and quad-core Cortex-A53 up to 1.4 GHz with a quad-core Mali-T860 GPU at up to 800 MHz. It's also equipped with a neural processing unit (NPU) capable of delivering up to 3 TOPS of performance.

Asus' announcement states that the Tinker Edge R and Tinker Edge T are compatible with the Debian and Android operating systems. However, we expect them to support other Linux distributions and operating systems, such as Windows 10 IoT Core or FreeRTOS.

With a focuses on AI processing, both boards face competition in Nvidia's Jetson Nano, which features a quad-core Arm A57 at 1.45 GHz, along with Nvidia Mawell graphics.

Asus didn't reveal the pricing or availability of the new single-board computers.

Intel announced that it would soon discontinue the first-generation Neural Compute Stick, which launched in 2017. Intel will continue to sell the product for another year and support it for another two years.

Intel Neural Compute Stick

Intel acquired Movidius back in 2016, and a year later it released the previously showcased Movidius Fathom Compute Stick under the Neural Compute Stick (NCS) name (with some slight changes). The stick targeted developers who wanted to develop and test AI applications on a small budget. As a USB stick, it could be added to almost any computer.

The NCS is powered by a Myriad 2 Visual Processing Unit (VPU), which can reach 100 GFLOPS of performance with 1W of power, according to Intel and Movidius. Besides being used as a development tool, the NCS can also be used in production to accelerate offline AI applications on machines that aren’t capable of running said applications efficiently because they lack AI-optimized processors.

Switching to NCS 2.0

Intel recommends developers to switch to the next-generation NCS 2.0, which features up to eight times the performance of the first-generation NCS. The NCS 2.0 comes more compute cores, as well as dedicated hardware for deep learning network inference.

The SDK of the first-gen NCS supports only the TensorFlow and Caffe frameworks. However, the NCS 2.0 also comes with the open source OpenVINO toolkit, which expands the support for machine learning software frameworks and type of processors.

In quite a major bit of news today, Qualcomm announced that it is entering the cloud AI inference processing market. As part of the company's AI Day, it announced the 7nm Cloud AI 100 chip aimed at bringing the company's power efficiency expertise in the mobile space to the data center. Qualcomm designed the new chip for Tier 1 and Tier 2 cloud players and claims a 10x improvement in inference performance compared to the best solutions available.

Qualcomm already sees its Snapdragon processors as a key player in inference on the edge in client devices. The company now wants to bring its capabilities in production scale, leading-edge nodes, power efficiency, and signal processing to the data center. Qualcomm envisions providing lower latency with 5G and bringing much-increased performance and efficiency to the cloud and edge with the Qualcomm Cloud AI 100, which is built on TSMC's 7nm node.

While Qualcomm is not yet ready to talk about specifics in performance or architecture, its says the Cloud AI 100 offers "more than 10x performance over the industry's most advanced AI inference solutions available today." Qualcomm is also committed to supporting the leading software stacks, like PyTorch, Glow, TensorFlow, Keras, and ONNX.

This is quite an early announcement, as the company doesn't have working silicon yet. Qualcomm is working with partners such as Facebook, Microsoft, and ODMs to develop different form factors and power levels. The company is targeting power levels that range from 20W to 75W, which allows it to fit in the M.2 form factor. The Cloud AI 100 will start sampling late this year, and full production is planned for 2020.

Weighing the Market

Deep learning is split between training (building the model) and inference (using the model), but the initial AI growth came from the training workloads where Nvidia's Tesla GPUs reigned supreme. Nvidia generated $2B in revenue in 2017 and $3B last year from the data center.

The initial hype of deep learning has cooled down a bit, but the AI market is expected to become much larger still, which has attracted big companies and startups vying for their share of the market especially as the focus is now shifting towards inference, which over time is expected to become much larger than training.

Inference is also seen as more attractive because diverse types of compute can be used, such as CPUs and FPGAs, and inference at the edge is also becoming more important. Intel recently reported $1B in revenue from AI running on Xeon, and now Qualcomm expects data center inference to become a $17B market by 2025. This continues the growth trend that Intel saw (it forecasts a $10B deep learning market by 2022).

Last year, Nvidia launched the Turing based Tesla T4 for inference, but the company will likely have also moved to 7nm by 2020. At CES, Intel announced the Nervana NNP-I for inference, with production starting in 2019. Naveen Rao confirmed that is will be built on 10nm and includes Sunny Cove cores. Intel too said performance per watt is the focus, and also like the Cloud AI 100, the Nervana NNP-I will fit in the M.2 form factor. Given their similarities, those two should be close competitors. The NNP-I should not be confused with the NNP-L 1000, which is for training and manufactured on TSMC 16nm.

This announcement also comes off the back of Qualcomm reportedly leaving the CPU cloud market that it had tried to enter with the Centriq data center processors.

Image Credits: Qualcomm

Google has officially released its Edge TPU (TPU stands for tensor processing unit) processors in its new Coral development board and USB accelerator. The Edge TPU is Google’s inference-focused application specific integrated circuit (ASIC) that targets low-power “edge” devices and complements the company’s “Cloud TPU,” which targets data centers.

Coral Dev Board With Edge TPU

Last July, Google announced that it’s working on a low-power version of its Cloud TPU to cater to Internet of Things (IoT) devices. The Edge TPU’s main promise is to free IoT devices from cloud dependence when it comes to intelligent analysis of data. For instance, a surveillance camera would no longer need to identify objects it sees in real-time through cloud analysis and could instead do so on its own, locally, thanks to the Edge TPU.

Google has now made available for developers a Raspberry Pi-style development board that comes with a quad-core Arm Cortex-A53 CPU, an Arm Cortex-M4F real-time core and a Vivante GC7000 Lite GPU -- all of which are connected to Google’s Edge TPU co-processor, capable of up to 4 trillion operations per second (TOPS).

The board also ships with 1GB of LPDDR4 RAM, 8GB of eMMC (embedded MultiMediaCard) memory, Wi-Fi 2×2 MIMO (802.11b/g/n/ac 2.4/5GHz bands) and Bluetooth 4.1. It supports Debian Linux and the TensorFlow Lite machine learning software framework. The board costs $149.99.

Coral USB Accelerator

Google also revealed a USB accelerator, which is similar to Intel’s Neural Compute Stick. However, just like the Coral dev board, the Coral USB Accelerator also comes with an embedded Edge TPU. It's supposed to have low power demand and comes in a more accessible form to developers, who can connect it to other computers or even other boards, such as a Raspberry Pi, to give them a machine learning performance boost.

The Coral USB Accelerators also comes with a Cortex-M0+ microcontroller clocked at 32MHz, 16KB of flash memory and 2KB of RAM. It can connect to other devices via a USB Type-C connector that supports 5Gb/s. The USB Accelerator, which also supports TensorFlow Lite, costs $74.99. 

Google's TPUs

Back in 2016, Google surprised the world with its own machine learning-focused processor called a “tensor processing unit." According to Google, the TPU was up to 30 times faster and more efficient than other CPUs and GPUs for certain popular machine learning training and inference applications. This is because the chip was designed from the beginning with machine learning in mind.

Later on, Nvidia also started integrating “tensor cores” into its GPUs, shrinking the disparity between the two architecture types significantly. However, Google’s latest Cloud TPUs still seems to have some advantages in terms of performance and price over Nvidia’s GPUs.

One of the primary reasons for open sourcing the TensorFlow framework, which Google was using internally for its own machine learning projects and later to develop hardware optimized for it, was that this enabled an entire ecosystem of developers and projects that essentially kept improving Google’s software.

Even third-parties, including Nvidia, have started building hardware that works well with the TensorFlow framework. Now that Google has started selling highly optimized TensorFlow chips for the booming IoT industry, the TensorFlow framework is likely to become even more popular.

AMD and Xilinx partnered to create high-performance inference systems for data centers that Xilinx this week claimed breaks the world record for inference performance. The new systems include Xilinx’s new machine learning accelerator cards, called Alveo, which promise real-time machine learning inference, as well as video processing, genomics and data analytics.

A New Inference World Record

AMD and Xilinx created a new system for data centers that includes a 32-core EPYC 7551 CPU and eight Alveo U250 accelerator cards. The cards will be powered by Xilinx’s ML Suite, which also supports ML software frameworks, such as TensorFlow.

The two companies said that their system reached an inference throughput of 30,000 images per second on the GoogLeNet convolutional neural networks. Such high inference performance is seemingly being requested these days by companies that need to analyze massive amounts of data.

After joining Xilinx CEO Victor Peng onstage at a Xilinx event showcasing this, Mark Papermaster, AMD CTO and senior vice president of technology and engineering, said that new workloads can take advantage of the whole system and not just the CPU.

Xilinx Alveo Accelerator FPGA

Xilinx introduced two new FPGA cards (Alveo U200 and U250), which for the first time are optimized to “accelerate” real-time machine learning inference. The focus here seems to be “real-time” inference because the Alveo cards promise three times lower latency than GPUs with four times the throughput for low-latency applications.

The Alveo cards also promise 20x the performance of a CPU for inference tasks, reaching up to 90x the performance for database searches. They start at $8,995 each, and Xilinx said that it’s now working with OEMs, including Dell EMC, Fujitsu, Hewlett Packard Enterprise and IBM, to qualify them for data centers.

Habana Labs, a startup that came out of “stealth mode” this week, announced a custom chip that is said to enable much higher machine learning inference performance compared to GPUs.

Habana Goya Specifications

According to the startup, its Goya chip is designed from scratch for deep learning inference, unlike GPUs or other types of chips that have been repurposed for this task. The chip’s die is composed of eight VLIW Tensor Processing Cores (TPCs), each having their own local memory, as well as access to shared memory. The external memory is accessed through a DDR4 interface. The processor supports the FP32, INT32, INT16, INT8, UINT32, UINT16 and UINT8 data types.

The Goya chip supports all the major machine learning software frameworks, including TensorFlow, MXNet, Caffe2, Microsoft Cognitive Toolkit, PyTorch and the Open Neural Network Exchange Format (ONNX). After a trained neural network model is loaded, the chip converts it to an internal format that’s more optimized for the Goya chip.

Models for vision, neural machine translation, sentiment analysis and recommender systems have been executed on the Goya chip, and Habana said that the processor should handle all sorts of inference workloads and application domains.

Goya Performance

Habana says the Goya chip has shown a performance of 15,000 ResNet-50 images/second with a batch size of 10 and a latency of 1.3ms, while using only 100W. In comparison, Nvidia’s V100 GPU has shown a performance of 2,657 images/second.

A dual-socket Xeon 8180 was able to achieve an even lower performance than that: 1,225 images/second. According to Habana, when using a batch size of one, the Goya chip can handle 8,500 ResNet-50 images/second with a 0.27-ms latency.

This level of inference performance is given by the chip’s architecture design, mixed-format quantization, a proprietary graph compiler and software-based memory management.

Habana intends to reveal a deep learning training chip, called Gaudi, to pair with its Goya inference processor. The two chips will actually use the same VLIW core of Goya and will be software-compatible with it. The 16nm Gaudi chip will start sampling in Q2 2019.

Nvidia CEO Jensen Huang took to the stage at GTC Japan to announce the company's latest advancements in AI, which includes the new Tesla T4 GPU. This new GPU, which Nvidia designed for inference workloads in hyperscale data centers, leverages the same Turing microarchitecture as Nvidia's forthcoming GeForce RTX 20-series gaming graphics cards.

But the Tesla T4 is a unique graphics card designed specifically for AI inference workloads, like neural networks that process video, speech, search engines, and images. Nvidia's previous-gen Tesla P4 fulfilled this role in the past, but Nvidia claims the new model offers up to 12 times the performance within the same power envelope, possibly setting a new bar for power efficiency in inference workloads. 

The Tesla T4 GPU comes equipped with 16GB of GDDR6 that provides up to 320GB/s of bandwidth, 320 Turing Tensor cores, and 2,560 CUDA cores. The T4 features 40 SMs enabled on the TU104 die to optimize for the 75W power profile.

The GPU supports mixed-precision, such as FP32, FP16, and INT8 (performance above). The Tesla T4 also features an INT4 and (experimental) INT1 precision mode, which is a notable advancement over its predecessor.

Like its predecessor, the low-profile Tesla T4 consumes just 75 watts and slots into a standard PCIe slot in servers, but it doesn't require an external power source (like a 6-pin connector). The cards' low-power design doesn't require active cooling (like a fan)–the high linear airflow inside of a typical server will suffice. Nvidia tells us that the die does come equipped with RT Cores, just like the desktop models, but that they will be useful for raytracing or VDI (Virtual Desktop Infrastructure), implying they won't be used for most inference workloads.

The Tesla T4 also features optimizations for AI video applications. These are powered by hardware transcoding engines that provide twice the performance of the Tesla P4. Nvidia says the cards can decode up to 38 full-HD video streams simultaneously.

Nvidia's TensorRT Hyperscale platform is a collection of technologies wrapped around the T4. As expected, the card supports all the major deep learning frameworks, such as PyTorch, TensorFlow, MXNet, and Caffee2. Nvidia also offers its TensorRT 5, a new version of Nvidia's deep learning inference optimizer and runtime engine that supports Turing Tensor Cores and multi-precision workloads. Nvidia also announced the Turing-optimized CUDA 10, which includes optimized libraries, programming models, and graphics API interoperability.

Nvidia also announced the AGX lineup, which is a new name for Nvidia's line of Xavier-based products that are designed for autonomous machine systems that range from robots to self-driving cars. The lineup includes Drive Xavier and the newly-finalized Drive Pegasus that originally featured two Xavier processors and two Tesla V100 GPUs. Nvidia has now updated the GPUs to Turing models. Nvidia is also offering a similar design, called the Clara Platform, for medical applications. The Clara Platform features a single Xavier processor and Turing GPU.

Thoughts

Nvidia's focus on boosting performance in inference workloads is a strategic move: the company projects the inference market will grow to a $20 billion TAM over the next five years. Meanwhile, Intel claims that most of the world's inference workloads run on Xeon processors, which is likely true given Intel's presence in ~96% of the world's servers. Intel announced during its recent Data-Centric Innovation Summit that the company sold $1 billion in processors for AI workloads in 2017 and expects that number to grow quickly over the coming years.

Inference workloads will be a hotly contested battleground between Nvidia, Intel, and AMD in the future, with Intel having the initial advantage due to its server attach rate. However, low-cost and low-power inference accelerators, such as Nvidia's new Tesla T4, pose a tremendous threat due to their performance-per-watt advantages, and AMD has its 7nm Radeon Instinct GPUs for deep learning coming soon. Several companies, such as Google with its TPUs, are developing their own custom silicon for inferencing workloads. That means it will likely be several years before the clear winners become apparent.

This year may be the year of embedded AI chips for smartphones, as well as other small Internet of Things (IoT) devices, drones, robots, or surveillance cameras. Chinese fabless seminconductor company Rockchip seems to want to catch this trend early with the launch of its AI-focused chip, the RK3399Pro. The chip promises an AI performance of up to 2.4 trillion operations per second (TOPS).

RK3399Pro System-on-a-Chip (SoC)

The RK3399Pro uses a dual-core Cortex-A72 CPU and a quad-core Cortex-A53 CPU in a big.Little configuration, as well as a Mali-T860 GPU. It also comes with a dual type-C interface, and it supports a dual Image Signaling Processor (ISP), 4096x2160 display output, as well as an 8-channel digital microphone arrays input.

Software support includes OpenGL ES 1.x/2.x/3.1/3.2, Vulkan 1.0, OpenCL 1.1/1.2, RenderScript, and more.

RK3399Pro NPU

In addition to the CPU and the GPU, the chip comes with a Neural Processing Unit (NPU), which promises a performance of 2.4 TOPS.

The NPU’s performance falls right between the performance of Huawei’s Kirin 970 AI processor (1.9 TOPS) and Google’s Pixel Visual Core (3 TOPS). It’s also about four times faster than what Apple’s own Neural Engine can achieve (0.6 TOPS).

The RK3399Pro NPU supports OpenVX, TensorFlow Lite, Android’s Neural Network API (NNAPI), as well as the more full-featured Caffe and TensorFlow machine learning framework frameworks. The chip can do both 8-bit and 16-bit computing.

Rockchip will provide developers with a reference design and SDK to get started on their RK3399Pro-based projects. The company has typically targeted the budget embedded chip market, so despite the already quite impressive NPU performance, the RK3399Pro will likely target lower-cost devices. Things seem to be moving fast in the AI chip market, so at the high-end we may see even higher embedded AI performance this year.

"The age of AI has come. As a global SoC manufacturer in China, we have the market layout of AI for many years,” said Chen Feng, the Global Vice President of Rockchip.“RK3399Pro is Rockchip's first processor which integrates AI hardware. Its platform can be rapid MP for commercial. With super 2.4TOPs performance, low power consumption and abundant interfaces, this product is applicable to various AI application fields such as intelligent drive, image recognition, security monitoring, drones and voice recognition,” he added.

CEVA announced a family of new “AI processors,” called NeuPro, that scale in performance from 2 trillions operations per second (TOPS) to 12.5 TOPS. The chips will target a range of products from Internet of Things (IoT) devices to smartphones and connected cars.

Rise AI At The Edge

Over the past couple of years we’ve seen a growing trend of “AI chips” that could be used in embedded devices such as IoT devices, smartphones, drones, and surveillance cameras. One of their primary uses is for their “computer vision” capabilities, which allow devices to identify objects on the spot without needing to connect to the cloud, while also protecting your privacy.

Many people might not want companies to watch their homes at all times via smart surveillance cameras, just so the users can benefit from object recognition capabilities of their devices, for instance. Embedded AI chips can largely achieve the same capabilities without all the privacy downsides that machine learning tends to bring. As we know, for AI to be effective, it needs to use as much of your data as possible.

In the past year, we’ve also seen smartphones quickly jump in embedded AI performance from 0.6 TOPS (Apple iPhone 8/X), to 1.9 TOPS (Huawei Mate 10 Pro), to 3 TOPS (Google Pixel 2). This year, we should also see even bigger jumps in performance, as smartphone makers start to go “all-in” on embedded AI performance.

Of course, having a fast AI chip in your device won’t do much on its own, unless there’s software to take advantage of it, too. This is why, for instance, Android 8.1 now supports the Tensorflow Lite software library and the Neural Network API, which allow smartphone makers to give app developers access to their chips in a standardized way. CEVA also released its Ceva Deep Neural Network (CDNN) software library for its previous generation chips, back in 2016.

CEVA’s NeuPro

Last year, CEVA hinted that Apple may be using one of its last-generation AI chips in the iPhone 8 and X, but Apple never confirmed it. Either way, it seems that CEVA’s new AI chips will be seeing a big jump in performance this year, starting at 2 TOPS and reaching up to 12.5 TOPS.

The smallest processor, called the “NP500,” comes with 512 Multiply-Accumulate (MAC) units and targets IoT, wearables, and cameras.

The NP1000 comes with twice as many units (1024) and targets mid-range smartphones, advanced driver assistance systems, industrial applications, and AR/VR headsets.

The NP2000 has 2048 MAC units and targets smartphones, surveillance cameras, robots, and drones.

The most powerful version, the NP4000, has 4096 MAC units and can be used in enterprise surveillance and autonomous driving.

Each coprocessor contains the NeuPro engine and the NeuPro visual processing unit (VPU). The NeuPro engine is the hardwired implementation of network layers, while the VPU is a programmable vector digital signal processor (DSP), which provides software support for new AI algorithms.

NeuPro processors support both 8-bit and 16-bit computation, each type being activated in real-time depending on the workload. According to CEVA, its MAC units achieve over 90% utilization, which means the AI algorithms take (almost) full advantage of the processors’ performance. The NeuPro’s design also substantially lowers RAM utilization, which should improve battery energy efficiency, too.

Ilan Yona, vice president and general manager of the Vision Business Unit at CEVA, said that:

It’s abundantly clear that AI applications are trending toward processing at the edge, rather than relying on services from the cloud. The computational power required along with the low power constraints for edge processing, calls for specialized processors rather than using CPUs, GPUs or DSPs. We designed the NeuPro processors to reduce the high barriers-to-entry into the AI space in terms of both architecture and software. Our customers now have an optimized and cost-effective standard AI platform that can be utilized for a multitude of AI-based workloads and applications.

The NeuPro chips will be available for licensing to select customers in the second quarter of 2018, and to everyone else in the third quarter of the year.

Google announced that TensorFlow Lite, a machine learning software framework and the the successor to TensorFlow Mobile, is now available as a preview to developers. The purpose of the TensorFlow Lite framework is to bring lower-latency inference performance to mobile and embedded devices to take advantage of the increasingly common machine learning chips now appearing in small devices.

On-Device Machine Learning

More and more devices are starting to do machine learning inference locally rather than in the cloud. This has multiple advantages compared to cloud inference, including:

Latency: You don’t need to send a request over a network connection and wait for a response. This can be critical for video applications that process successive frames coming from a camera.Availability: The application runs even when outside of network coverage.Speed: New hardware specific to neural networks processing provide significantly faster computation than with general-use CPU alone.Privacy: The data does not leave the device.Cost: No server farm is needed when all the computations are performed on the device.

Although the original TensorFlow framework could also be used on mobile devices, it was not designed with mobile or Internet of Things (IoT) devices in mind, so Google created the lighter TensorFlow Lite software framework. The framework supports Android and iOS primarily, but Google developers said that it should be easy to use it with Linux on embedded devices, too.

TensorFlow Lite Architecture

The individual components of the TensorFlow Lite architecture include:

TensorFlow Model: A trained TensorFlow model saved on disk.TensorFlow Lite Converter: A program that converts the model to the TensorFlow Lite file format.TensorFlow Lite Model File: A model file format based on FlatBuffers, that has been optimized for maximum speed and minimum size.

The TensorFlow model is then deployed within a mobile app where it can interact with a Java API, which is a wrapper around the C++ API, a C++ API that loads the model file and invokes the interpreter, and the interpreter that supports selective operator loading. Without any operators, the interpreter is only 70KB, while with all the operators loaded it’s 300KB in size. This is a 5x reduction compared to TensorFlow Mobile.

Starting with Android 8.1, the interpreter can also use the Neural Network API (NNAPI) on devices that come with machine learning hardware accelerators, such as Google’s latest Pixel 2 smartphone.

Mobile-Optimized Models

Google said that TensorFlow Lite already supports a few mobile-optimized machine learning models, such as MobileNet and Inception V3, two vision models developers can use to identify thousands of different objects in their apps, as well as Smart Reply, an on-device conversational model that can provide smart replies to incoming chat messages.

Deprecation of TensorFlow Mobile

Google said that although developers should continue using TensorFlow Mobile in production for now, because TensorFlow Lite is still being tested, eventually the latter will completely replace the former so they should plan on moving to TensorFlow Lite eventually.

Nvidia revealed the third generation of its TensorRT AI inferencing software, the DeepStream video analytics platform, and the ninth generation of its CUDA technology.

For decades, we’ve heard rumblings of artificial intelligence technology coming to the fore, but until recently, those ideas have been relegated to the world of science fiction. These days, artificial intelligence is no longer a thing of fiction—AI is rapidly becoming science fact. It’s no secret that Nvidia is bullish on artificial intelligence technology (and for good reason). The company positioned itself at the forefront of the AI and deep learning revolution in recent years, and it’s made rapid advancements in the segment year over year. And the company isn’t showing any sings of slowing down any time soon. Yesterday, at GTC China 2017, Jensen Huang, Nvidia’s Founder and CEO, revealed a handful of new technologies that improve the performance and efficiency of deep learning inferencing. The founder also gave an overview of the partnerships its forming within the industry to give us a glimpse of how Nvidia’s deep learning AI technology will shape our world in the years to come.

Third Generation TensorRT Technology

Last September, Nvidia replaced the GPU Interface Engine with the TensorRT deep learning inference engine. The first generation of Nvidia’s TensorRT inference engine offered double the performance of the GPU Interface Engine in tasks such as image classification, segmentation, and object detection. 

Earlier this year, Nvidia released TensorRT 2, which improved INT8 precision performance by up to 45x. Nvidia also introduced “sequence based models for image captioning, language translation, and other applications.”

Nvidia now offers TensorRT 3, which enables 3.7x performance on Tesla V100 GPUs compared to Tesla P100 GPUs. TensorRT 3 can also “optimize and deploy TensorFlow models up to 18x faster” on Volta hardware than it can on a CPU-only interface. TensorRT 3 also offers a 40x speedup on ResNet-50, and 140x performance boost on the OpenNMT neural machine translation system when compared to CPU-based neural networks.

TensorRT 3 enables a new level of power efficiency for neural networks. Huang said a single HGX server with eight Telsa V100 GPUs would offer the equivalent computational performance as a CPU-based neural network with 160 dual-CPU servers. Nvidia didn’t say how much an 8-GPU HGX rack-mount server would cost, but the company said it would be cheaper than the $600,000 to $700,000 that the CPU-based severs would set you back. The GPU-based server would also reduce energy costs by a staggering margin from 65 kilowatts down to just 3 kilowatts of power.

Intelligent Video Analytics Simplified

Nvidia also introduced the DeepStream SDK, which “simplifies the development of scalable, intelligent video analytics (IVA) applications,” by combining AI inference technology with video transcoding and data curation technologies into a single API. Nvidia’s DeepStream SDK offers “image classification, scene understanding, video categorization, and content filtering” capabilities. Applications created with the DeepStream SDK run on Nvidia’s Tesla accelerated computer platform.

The DeepStream SDK includes sample code and pre-trained deep learning models to help developers create software that can classify video content and detect objects in video streams.

CUDA Turns 9

Along with the TensorRT 3 and DeepStream SDK releases, Nvidia also released the ninth generation of its CUDA technology. The latest version of Nvidia’s CUDA GPU-acceleration libraries takes advantage of the power of Nvidia’s Volta platform. The company said that HPC apps developed with CUDA 9 would offer up to 1.5x faster performance when running on Volta GPUs compared to apps built with CUDA 8.   

CUDA 9 apps also demonstrate faster performance across multi-GPU configurations. Particularly with Volta GPUs, where its next generation NVLink technology can deliver twice the throughput of the prior generation.

Available Now

All three of Nvidia’s new technologies are available now to registered Nvidia developers. You can find more information about TensorRT 3, DeepStream SDK, and CUDA 9 on Nvidia’s developer resource website.

Nvidia announced a new partnership with top server manufacturers such as Foxconn, Inventec, Quanta, and Wistron that aims to bring the HGX-1 servers faster to market. The servers target “AI cloud computing,” so the primary customers will be data center-owning companies.

HGX-1 Servers

The development of the HGX-1 server chassis was a collaboration between Microsoft, Nvidia, and Ingrasys. The companies optimized the enclosure for machine learning applications and open-sourced its design earlier this year. The HGX-1 boxes support up to eight GPUs in a single chassis, and up to 32 GPUs can work together when four HGX-1 boxes are interconnected.

The HGX-1 supports Nvidia’s latest compute GPUs, including the Pascal-based Tesla P100 and the latest Volta-based Tesla V100. However, HGX-1 also works with AMD Radeon GPUs and Intel machine learning accelerators.

Partnership With Top ODMs

Through the HGX Partner Program, Nvidia will provide top original design manufacturers (ODMs) such as Foxconn, Inventec, Quanta, and Wistron early access to its own HGX reference architecture, GPU technologies, and design guidelines. This partnership should allow the ODMs to build their own HGX boxes faster, thus reducing the time it takes to deploy them in their customers’ data centers.

Nvidia said that all of the top 10 hyperscale businesses use its GPUs to accelerate their machine learning projects. According to the company, its new Volta-based GPUs should be three times faster than the Pascal GPUs. This is in part due to the machine learning-optimized Tensor Cores, which are now included its new compute architecture.

“Accelerated computing is evolving rapidly — in just one year we tripled the deep learning performance in our Tesla GPUs — and this is having a significant impact on the way systems are designed,” said an Nvidia representative.“Through our HGX partner program, device makers can ensure they’re offering the latest AI technologies to the growing community of cloud computing providers,” he added.

The HGX systems are compatible with Nvidia’s GPU Cloud Platform, which includes support for machine learning software frameworks such as TensorFlow, Caffe2, Cognitive Toolkit, and MXNet.

Google announced that the next version of its mobile operating system, Android O, will include a new feature called TensorFlow Lite to offer developers improved on-device AI for their applications. This could lead to better speech recognition, computer vision, and other machine learning-driven features within Android, and it highlights tech companies' rush to bring AI down from their data centers and onto all of your devices.

TensorFlow is Google's open source machine intelligence software library. Developers can use it to jumpstart their machine learning efforts, allowing them to focus on differentiating their products instead of forcing them to start from scratch. Google also uses TensorFlow in many of its products--the project started as research by the Google Brain Team within Google's Machine Intelligence research organization--as well.

Google's vice president of Android engineering, Dave Burke, said at I/O that TensorFlow Lite is "a library for apps designed to be fast and small yet still enabling state-of-the-art techniques like convnets and LSTMs." He also said that Android O will introduce "a new framework" to hardware accelerated neural computation and that TensorFlow Lite will also use a new neural network API to "tap into silicon-specific accelerators."

All of these additions will work together to "power a next generation of on-device speech processing, visual search, augmented reality, and more," Burke said.

"As Android continues to take advantage of machine learning to improve the user experience," Google said in a blog post, "we want our developer partners to be able to do the same." That's where TensorFlow Lite comes in. Google said TensorFlow Lite and the new framework will be added to Android O, which is set to debut later this summer, in "a maintenance update to O later this year."

Here's what the company said about TensorFlow Lite:

TensorFlow Lite is specifically designed to be fast and lightweight for embedded use cases. Since many on-device scenarios require real-time performance, we’re also working on a new Neural Network API that TensorFlow can take advantage of to accelerate computation.

Enabling on-device machine learning has become an area of focus for some tech companies. Nvidia has tried to position its GPUs as the ideal hardware for deep neural network training and inference for example, and Movidius has developed "Vision Processing Units" (VPUs) specifically for on-device machine learning. You can learn more about how both companies approached this problem in our report on client-side deep learning.

On the software side of things, Apple made a big deal of on-device deep learning when it announced iOS 10 back in June 2016. Facebook announced in November 2016 its Caffe2Go project, which is meant to put "real-time AI in the palm of your hand." Microsoft has also worked to bring AI to your devices with Story Remix, an upcoming Windows 10 app that uses AI to help you edit your home videos and tinker with mixed reality content.

Google hasn't been resting on its laurels. The company also announced a new Cloud TPU at I/O that promises to be 50% faster than the Tesla V100 accelerator Nvidia announced alongside the new Volta GPU architecture, even though Nvidia built "Tensor Cores" right into the device. The Cloud TPU will help improve Google's cloud-based AI; TensorFlow Lite is supposed to help the company and Android developers do the same on-device.

What does this mean for you? Well, it should result in smarter apps that don't require an internet connection to offer their best features. Having to be connected is one of the most significant drawbacks of cloud-based AI, especially on mobile devices. If at least some of those features can use embedded machine learning--even if only as a backup--you would no longer have to worry about apps breaking the moment you go offline.

There's also the potential to keep more information on-device. That could in turn make it more secure--even with the rise of end-to-end encryption, you're often better off keeping personal data offline than sending it to who-knows-where for processing. Microsoft's Cortana doesn't raise privacy concerns because it can help you remember to keep your emailed promises; its data collection is worrisome because that information has to be sent off to Microsoft's servers. The continued rise of on-device AI could help reduce those privacy and security worries.

At Google I/O 2017, Google announced its next-generation machine learning chip, called the “Cloud TPU.” The new TPU no longer does only inference--now it can also train neural networks.

First Gen TPU

Google created its own TPU to jump “three generations” ahead of the competition when it came to inference performance. The chip seems to have delivered, as Google published a paper last month in which it demonstrated that the TPU could be up to 30x faster than a Kepler GPU and up to 80x faster than a Haswell CPU.

The comparison wasn’t quite fair, as those chips were a little older, but more importantly, they weren’t intended for inference.

Nvidia was quick to point out that its inference-optimized Tesla P40 GPU is already twice as fast as the TPU for sub-10ms latency applications. However, the TPU was still almost twice as fast as the P40 in peak INT8 performance (90TOPS vs 48TOPS).

The P40 also achieved its performance using more than three times as much power, so this comparison wasn’t that fair, either. The bottom line is that right now it’s not easy to compare wildly different architectures to each other when it comes to machine learning tasks.

Cloud TPU Performance

In last month’s paper, Google hinted that a next-generation TPU could be significantly faster if certain modifications were made. The Cloud TPU seems to have have received some of those improvements. It’s now much faster, and it can also do floating-point computation, which means it’s suitable for training neural networks, too.

According to Google, the chip can achieve 180 teraflops of floating-point performance, which is six times more than Nvidia’s latest Tesla V100 accelerator for FP16 half-precision computation. Even when compared against Nvidia’s “Tensor Core” performance, the Cloud TPU is still 50% faster.

Google made the Cloud TPU highly scalable and noted that 64 units can be put together to form a “pod” with a total performance of 11.5 petaflops of computation for a single machine learning task.

Strangely enough, Google hasn’t given the numbers for inference performance yet, but it may reveal them in the near future. Power consumption was not revealed either, as it was for the TPU.

Cloud TPUs For Everyone

Up until now, Google has kept its TPUs to itself, likely because it was still experimental technology and the company wanted to first see how it fared in the real world. However, the company will now make the Cloud TPUs available to all of its Google Compute Engine customers. Customers will be able to mix and match Cloud TPUs with Intel CPUs, Nvidia GPUs, and the rest of its hardware infrastructure to optimize their own machine learning solutions.

It almost goes without saying that the Cloud TPUs support the TensorFlow machine learning software library, which Google open sourced in 2015.

Google will also donate access to 1,000 Cloud TPUs to top researchers under the TensorFlow Research Cloud program to see what people do with them.

Update, 5/18/17, 7:52am PT: Fixed typo.

Nvidia announced a brand new accelerator based on the company’s latest Volta GPU architecture, called the Tesla V100. The chip’s newest breakout feature is what Nvidia calls a “Tensor Core.” According to Nvidia, Tensor Cores can make the Tesla V100 up to 12x faster for deep learning applications compared to the company’s previous Tesla P100 accelerator. (See our coverage of the GV100 and Tesla V100 here.)

Tensors And Tensorflow

A tensor is a mathematical object represented by an array of components that are functions of the coordinates of a space. Google created its own machine learning framework that uses tensors because tensors allow for highly scalable neural networks.

Google surprised industry analysts when it open sourced its Tensorflow machine learning software library, but this may have been a stroke of genius because Tensorflow quickly became one of the most popular machine learning frameworks used by developers. Google was also using Tensorflow internally, and it benefits Google if more developers know how to use Tensorflow because it increases the potential talent pool for the company to recruit from. Meanwhile, chip companies seem to be optimizing their products, either for Tensorflow directly, or for tensor calculations (as Nvidia is doing with the V100). In other words, chip companies are battling each other to improve Google’s open sourced machine learning framework - a situation that can only benefit Google.

Finally, Google also built its own specialized Tensor Processing Unit, and if the company decides to offer cloud services powered by the TPU, there will be a wide market of developers that could stand to benefit from it (and purchase access to it).

Nvidia Tensor Cores

The Tensor Cores in the Volta-based Tesla V100 are essentially mixed-precision FP16/FP32 cores, which Nvidia has optimized for deep learning applications.

The new mixed-precision cores can deliver up to 120 Tensor TFLOPS for both training and inference applications. According to Nvidia, V100’s Tensor Cores can provide 12x the performance of FP32 operations on the previous P100 accelerator, as well as 6x the performance of P100’s FP16 operations. The Tesla V100 comes with 640 Tensor Cores (eight for each SM).

In the image below, Nvidia is showing how for a matrix-matrix multiplication, commonly used in the training of neural networks, the V100 can be more than 9x faster compared to the Pascal-based P100 GPU.

The company said that this result is due to the custom crafting of the Tensor Cores and their data paths to maximize their floating point performance with a minimal increase in power consumption.

Each Tensor Core performs 64 floating point FMA mixed-precision operations per clock (FP16 multiply and FP32 accumulate). A group of eight Tensor Cores in an SM perform a total of 1024 floating point operations per clock.

According to the GPU maker, this is an 8x increase in throughput per SM in Volta, compared to the Pascal architecture. In total, with Volta’s other performance improvements, the V100 GPU can be up to 12x faster for deep learning compared to the P100 GPU.

Developers will be able to program the Tensor Cores directly or make use of V100’s support for popular machine learning frameworks such as Tensorflow, Caffe2, MXNet, and others.

Rise Of The Specialized Machine Learning Chip

In a recent paper, Google revealed that its TPU can be up to 30x faster than a GPU for inference (the TPU can’t do training of neural networks). As the main provider of chips for machine learning applications, Nvidia took some issue with that, arguing that some of its existing inference chips were already highly competitive to the TPU.

Nvidia’s case wasn’t that strong, though, considering that it was comparing its chips only for sub-10ms latency (a less important metric for Google in the data center). Nvidia also ignored the power consumption and cost metrics when it made the comparisons.

Nevertheless, Google's TPU was probably not even the biggest threat to Nvidia’s machine learning business, considering that for now, at least, Google doesn’t intend to sell it, although the company may increasingly use the TPUs more internally.

A bigger threat to Nvidia may be other companies that develop and sell specialized machine learning chips with better performance/watt and cost metrics than a typical GPU can offer to other customers. Even more worrisome for Nvidia could be the fact that Intel has been snapping up all the more interesting ones.

Therefore, until now, it looked as if Nvidia could start to fall behind in this race due to the company’s focus on GPUs. However, with the announcement of the “Tensor Cores,” Nvidia may have made it more difficult for others to beat it in this market (at least for now).

Google revealed more details about the performance of its Tensor Processing Unit (TPU), the company’s first machine learning chip. According to some benchmarks Google performed on its TPU, Haswell server CPUs, and Nvidia Tesla K80, the TPU chip came up 15-30x faster and up to 80x more efficient than those other chips.

How The TPU Was Born

Back in 2006, Google’s engineers discussed deploying GPUs, field-programmable gate arrays (FPGAs), and custom application specific integrated circuits (ASICs) in their data centers for machine learning applications. However, at the time, they concluded that their machine learning applications didn’t require enough computation to warrant developing ASICs.

This changed in 2013, when the engineers realized that the company’s use of deep neural networks (DNNs) was exploding, and that it would soon need to double its data centers if the growth in usage of DNNs continued.

Google’s engineers then decided to prioritize building a custom ASIC for inference, which is running neural networks that have already been trained on off-the-shelf GPUs. They called this ASIC a “Tensor Processing Unit” (TPU) because it’s tailored for Google’s open source TensorFlow machine learning software library.

How The TPU Is Built

Because Google was in a rush to deploy the TPU, the company didn’t integrate it tightly to CPUs and instead connected the TPU to the processors via the PCIe I/O bus. This allowed the TPU to plug into servers just as a GPU does. However, the host server has to send the instructions to the TPU rather than the TPU fetching the instructions itself, which means it’s closer in spirit to a floating-point unit co-processor than a GPU. This was also done to simplify design and debugging.

Although it’s a custom ASIC, a type of chip typically designed to run a limited set of instructions, Google said that it has some of the flexibility of an FPGA. This means it can be programmed to handle multiple types of neural networks. Therefore, even if Google’s future needs will require different types of machine learning algorithms, the TPUs should be flexible enough to adapt.

Plus, given the performance advantage the TPUs seem to offer over CPUs and GPUs, the company will likely continue to build new generations adapted for whatever machine learning technology is most advanced at the time.

TPU Performance Metrics

Google’s engineers said in a paper about the TPU that the most important metric it considers when buying chips for its data servers is not the peak performance of a chip, but the cost-performance metric - or, more specifically, the total cost of ownership (TCO). TCO is correlated with power use, as the more power a chip uses, the more its TCO rises over its lifetime.

Google used two performance/Watt metrics to compare the power draw of the TPU to that of the Haswell CPU and the K80 GPU. One is the total-performance/Watt metric, which includes the power used by the host server CPU when combined with either a K80 GPU or a TPU. The other is the incremental-performance/Watt, which only refers to the power used by the K80 GPU or the TPU.

A system that includes a Haswell server chip and an Nvidia K80 GPU has 1.2-2.1x the total-performance/Watt of the Haswell CPU alone, while an K80 GPU has an incremental-performance/Watt of 1.7-2.9x compared to a Haswell CPU.

At the same time, a Haswell/TPU server has 17-34x better total-performance/Watt compared to a Haswell CPU, and a relative incremental-performance/Watt of 41-83x for the TPU alone. That also means the TPU has 25-29x the performance/Watt of a K80 GPU.

Google also claimed that its TPU can achieve 15-30x inference performance compared to the K80 GPU and the Haswell CPU.

What To Expect From Future TPU Chips

The TPU was manufactured on a 28nm planar process and has been in use since 2015. If a next-generation TPU is made on a 14nm process, it could see a 2x improvement in performance/Watt just from that jump alone, as we’ve already seen from AMD and Nvidia’s 14/16nm GPUs.

Google also said if it had taken an extra 15 months to have designed better logic--which is how long it took to design the first TPU-- it could’ve increased clock speeds by another 50%. That could be a clue that if Google is indeed working on a new generation, that kind of design would be included in it.

Because the company rushed to integrate the TPU quickly in its data centers, it used whatever memory and interconnects were available. However, it said that if it were to use 4x as much bandwidth for its servers’ memory, it could increase the performance of the TPU by another 3x.

Google hasn’t specifically talked about its plans to build a new TPU chip, but going by the performance/Watt of the first generation and how much room there is to improve it, chances are it won’t leave this opportunity on the table. The use of machine learning for all of the company’s services is only going to increase over the next few years, making such chips even more necessary than they are today.

Google announced the release of a dataset of eight million labeled videos to help accelerate machine learning research for video understanding. It's called YouTube-8M.

ImageNet For Videos

Many technology companies have used the ImageNet dataset of millions of labeled images to benchmark the performance of their chips and algorithms. This has helped them improve their technology over the years, as they tried to become better and better at classifying objects in static images.

Google now aims to do the same for videos, which is why it’s releasing the YouTube-8M dataset. The dataset contains eight million YouTube URLs, or the equivalent of 500,000 hours of video, as well as 4,800 video labels from Google’s own Knowledge Graph.

The previously largest video dataset, called Sports-1M, contained one million YouTube video URLs and 500 labels, so YouTube-8M increases both the number of videos and labels by almost an order of magnitude. This increase in complexity means there will now be more for researchers’ neural networks to learn, and it gives those networks the opportunity to become more accurate.

Challenges

Google faced some challenges before creating the dataset. One of them is that video is harder to annotate manually than images, simply because it’s more time-consuming to watch and figure out what it’s about. To solve this problem, Google had to rely on YouTube’s machine-generated labels. However, the company believes that they are accurate enough to be useful for benchmarking and research purposes.

To ensure the videos were of high-enough quality, the company chose videos that had at least 1,000 views. It also used other automated tools to determine if the “entities” in the videos could be easily observable.

Video is also much more computationally intensive, which created a second challenge for Google. Normally, all of the 500,000 hours of video in the YouTube-8M dataset would require a petabyte (PB) of storage and dozens of CPU-years' worth of processing. That means it wouldn’t be readily available to most potential researchers, including students.

To solve this problem, Google pre-processed the video and extracted frame-level features using a publicly available deep learning model that was trained on ImageNet called Inception-V3. The extracted features can be further compressed and fit onto a 1.5TB drive. This makes it possible for the dataset to be downloaded over the internet by just about anyone. Then, new deep learning models can be trained on it using the Tensorflow deep learning framework and a GPU, in less than a day.

Google believes that the YouTube-8M dataset will significantly accelerate research on video understanding, as it enables researchers everywhere, including students, to work with large datasets on their own computers. The company also made available the technical report for this announcement.

Google Translate turned 10 years old this year, and today the company announced the Google Neural Machine Translation system (GNMT), which utilizes “state-of-the-art” neural network training techniques to break the record for machine translation quality.

Phrase-Based Machine Translation (PBMT)

Ten years ago, Google started by using “Phrase Based Machine Translation” (PBMT) as the key algorithm that the company used for state-of-the-art (at the time) machine translation. However, since then, there have been major advances in machine intelligence, and Google has kept improving its techniques.

Neural Machine Translation (NMT)

A few years ago, Google started using Recurrent Neural Networks (RNNs) to learn the mapping between an input sentence (the sentence to be translated in another language) and an output sentence (the translated sentence).

Unlike the PBMT method, which breaks the input sentence into multiple phrases and then translates them independently of each other, the Neural Machine Translation (NMT) method works with the whole input sentence.

When NMT was first used, it showed similar accuracy to the PBMT method on small data sets. The big advantage was that the NMT method significantly simplified the translation system, requiring fewer engineering design choices. However, neural network-based techniques need significantly more processing power, and Google couldn’t use the NMT system in production for large data sets.

Google Neural Machine Translation (GNMT)

Google’s new paper, titled "Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation," describes how the company was able to overcome the many challenges required to make NMT work on large data sets. It also talks about how Google built a system that was fast enough to be used in Google Translate in production.

Google said that it's new technique is not only faster and more efficient, but it also achieves almost human levels of performance for translations. The company said that it reduced the translation errors by 55-85% for several language pairs, when rated by bilingual human translators.

How GNMT Works

Google showed the process for how its new GNMT technique works in one example, which consisted of translating a Chinese sentence to English. The method encodes the Chinese words as vectors, where each vector represents the meaning of all the words read so far.

When the entire sentence is read, the decoder begins, generating one English word at a time. Each vector is given a different “weight” in the translation process, and the ones that are found to be most relevant are the ones to be decoded.

Google Translate's Chinese To English Now Uses GNMT 100%

Google said that all Chinese to English translations - about 18 million translations per day - are now using the new GNMT system. The company said this was made possible by using its open sourced Tensorflow neural network framework and its custom TPU chip. The TPUs, which promise an order of magnitude higher efficiency compared to GPUs, seem to also have enough performance to handle such large data sets.

Google noted that the new GNMT system is still far from achieving perfect translation, and it can still make errors a human would never make. For instance, it can completely drop some words, or mistranslate proper names or rare words. It may also still translate words in isolation rather than in the context of the sentence, despite the fact that the systemnow takes the whole sentence into account when translating it.

Chinese to English is just one of the more than 10,000 language pairs that Google Translate supports, and the company said it will work on supporting as many of them as possible over the coming months.

Ceva, a licensor of signal processing intellectual property (IP) for connected devices, announced its second-generation deep neural network software framework. The new CDNN2 (Ceva Deep Neural Network) brings support for Google's TensorFlow to embedded systems.

CDNN2 can enable on-device deep learning-based video analytics in real time, saving bandwidth and storage compared to running the same analytics in the cloud. The CDNN2 software framework is part of an emerging trend of on-device deep learning, which could make devices much smarter even without a connection to the Internet. It’s also a big benefit to those who don’t trust third-party providers with applications such as analyzing video feeds of their homes.

Latency for analytics is also reduced, which means such devices with built-in deep learning capabilities can do things that perhaps cloud-based deep learning solutions wouldn’t be able to achieve with their higher latency.

The CDNN2 is paired with Ceva’s own “intelligent vision processor,” the Ceva-XMP4, which can enable 3D vision, computational photography, visual perception and analytics.

One of the major additions to Ceva’s second-generation deep neural network software framework is support for Google’s TensorFlow, which has quickly become one of the most popular machine learning software libraries. CDNN2 also brings support for convolutional networks, which allow any given network to work with any input resolution, as well as improved capabilities and performance for the latest network topologies and layers.

Pete Warden, lead of the TensorFlow Mobile/Embedded team at Google, said, “It's great to see Ceva adopting TensorFlow. Power efficiency is key to successfully harnessing the potential of deep learning in embedded devices. Ceva's low-power vision processors and CDNN2 framework could help a wide variety of developers get TensorFlow working on their devices.”

Google benefits from having everyone adopt TensorFlow because if there are more TensorFlow developers, the supply of chips that natively support TensorFlow will grow as well. That ultimately gives Google the opportunity to buy cheaper TensorFlow-optimized chips for its datacenters, without having to build such chips from scratch itself (as it’s already done).

Eran Briman, the vice president of marketing at Ceva, said that the company’s customers are already using its deep learning solutions to enhance the capabilities of drones, surveillance cameras and advanced driver assistance systems. The new CDNN2, combined with its support for TensorFlow, should improve the capabilities of the same types of products, but it will also open up the market for other types of products and services that can take advantage of deep learning. Those include augmented reality, virtual reality, and other similar computer vision applications.

The CDNN2 software library is highly modular and is provided as source code, as an extension to the company’s Application Developer Kit for the Ceva-XM4 chip. The library includes support for a variety of networks, including Alexnet, GoogLeNet, ResidualNet (ResNet), SegNet, VGG (VGG-19, VGG-16, VGG_S) and Network-in-network (NIN), and others. CDNN2 also supports the most advanced neural network layers, including convolution, deconvolution, pooling, fully connected, softmax, concatenation and upsample, and various inception models.

One of the main features of the CDNN2 is that it contains an offline Ceva Network Generator that can convert a pre-trained network into one that is highly optimized for embedded devices with fixed-point math hardware. Usually only older or ultra-low-cost embedded chips lack support for a floating-point unit. According to Ceva, a network that’s optimized for these chips can be generated at the push of a button.

Developers interested in Ceva’s deep learning software framework can also get a developer board on which to run their network simulations in real time.

Lucian Armasu is a Contributing Writer for Tom's Hardware. You can follow him at @lucian_armasu. 

Follow us on Facebook, Google+, RSS, Twitter and YouTube.