Software has come a long way from the idea of identifying letters and numbers that are drawn on a touchscreen, but it is helpless when confronted with sketches. Researchers from the University of Applied Sciences in Berlin, Germany, however have found a way to train software to recognize drawings of simple objects such as rabbits, pizzas, a telephone, or zebra.
The difficult task in recognizing these objects is that sketches and drawings may not resemble the look of the actual shape of the objects. "It might be that we only recognize it as a rabbit because we all grew up that way," said James Hays, assistant professor of computer science at Brown University, who developed the new program with Mathias Eitz and Marc Alexa from the Technical University in Berlin. "Whoever got the ball rolling on caricaturing rabbits like that, that’s just how we all draw them now."
To train their software, the scientists used crowdsourcing platform Mechanical Turk to acquire 20,000 sketches of objects in 250 categories that were fed to the software to enable a form of semantic understanding of sketches. As a result, their system was able to recognize a sketch correctly in 56 percent of cases, while humans recognized 73 percent of the drawings correctly.
"The gap between human and computational performance is not so big, not as big certainly as it is in other computer vision problems," Hays said.
Key to improving the software are more drawings, which are currently provided via an iPhone/iPad app that the researchers think could be turned into a game. "The game could ask you to sketch something and if another person is able to successfully recognize it, then we can say that must have been a decent enough sketch," Hays said. “You could collect all sorts of training data that way."
