“We are in the deep learning era, using computers to search for things visually,” said Karthik Ramani, Purdue’s Donald W. Feddersen distinguished professor of mechanical engineering. “But no one is focusing on the parts that go into machines: pipes, bearings, motors, washers, nuts, bolts, etc. Those are the things that are important to us as engineers and manufacturers. We want to be able to point a camera at a real-world part and have the computer tell us everything about that part or design.”

Ramani’s team experimented with visual search for parts in the early 2000s, but computing power and machine learning techniques were not yet sufficiently advanced. In the years since, researchers have learned that building a solid dataset is all about both quality and quantity.

“Deep learning is data hungry,” Ramani said. “It needs a lot of examples for the computer to learn what humans mean and how things relate to each other. That means we needed a lot of 3D models of parts, which also required an underlying engineering classification.”

The team began by partnering with a French company called TraceParts, which gave Purdue researchers access to their database of 3D engineering parts. The team collaborated with the University of Texas, Austin, assistant professor Qixing Huang to scour other databases for similar 3D models. They ended up compiling a database of 58,696 mechanical components.

But a database is no good without good data. Ramani’s team organized the parts by establishing a hierarchical taxonomy of 68 classes, based on the International Classification for Standards, a system of technical standards created and maintained by the International Organization for Standardization (ISO).

“Now when a computer sees a picture of a seal component, it will know that it fits in the category of dynamic seals and then, more specifically, under composite seals,” Ramani said.

The researchers now have published their open-source database, inviting computer vision and machine learning researchers to access it and create their own experiments. They showcased their work at the 16th European Conference on Computer Vision in August.

With a benchmark dataset now established, what is the future of machine learning for making machines?

“We see many real-world situations for this technology,” Ramani said. “Imagine you’re working maintenance in a factory, and you’re replacing a part of a machine. You can point a camera at the part, and the computer will recognize it and instantly give you all the specifications of that part — what it’s called, what it connects to, and where they are physically stored in the factory. This could even happen through augmented reality glasses; you could have your company’s entire visual catalog instantly at your fingertips and learn how to fix things or order parts.

“There are a lot of challenges in machine learning. But you can’t deal with the challenge unless you know how to test how well you’re doing. We are proud to have created the first large-scale benchmark dataset of annotated 3D mechanical components in the world, and we hope to advance visual search in the deep learning era.”

This research is being conducted at the Convergence Design Laboratory at Purdue University. The work is partially supported by the National Science Foundation under grants FW-HTF 1839971, OIA 1937036, and CRI 1729486.