CAMBRIDGE, MA—Engineers at the Massachusetts Institute of Technology (MIT) have developed a robot that uses touch and vision technology to play Jenga, a popular game that requires agility. The machine-learning approach could someday be applied to help robots assemble cellphones and other products that use small parts.

Jenga requires players to take turns carefully removing pieces from a tower of stacked blocks. The games use 54 rectangular wooden blocks that are stacked in 18 layers of three blocks each, with the blocks in each layer oriented perpendicular to the blocks below. The aim of the game is to carefully extract a block and place it at the top of the tower to build a new level, without toppling the entire structure.

To successfully accomplish the task, the MIT engineers equipped an ABB IRB 120  robot with a soft-pronged gripper, a force-sensing wrist cuff and an external camera.

As the robot carefully pushes against a block, a computer takes in visual and tactile feedback from its camera and cuff, and compares these measurements to moves that the robot previously made. It also considers the outcomes of those moves—specifically, whether a block, in a certain configuration and pushed with a certain amount of force, was successfully extracted or not.

In real-time, the robot then “learns” whether to keep pushing or move to a new block, in order to keep the tower from falling.

“The robot demonstrates something that’s been tricky to attain in previous systems: the ability to quickly learn the best way to carry out a task, not just from visual cues, as it is commonly studied today, but also from tactile, physical interactions,” says Alberto Rodriguez, an assistant professor of mechanical engineering at MIT.

“Unlike in more purely cognitive tasks or games such as chess or Go, playing the game of Jenga also requires mastery of physical skills such as probing, pushing, pulling, placing and aligning pieces,” explains Rodriguez. “It requires interactive perception and manipulation, where you have to go and touch the tower to learn how and when to move blocks.

“This is very difficult to simulate, so the robot has to learn in the real world, by interacting with the real Jenga tower,” Rodriguez points out. “The key challenge is to learn from a relatively small number of experiments by exploiting common sense about objects and physics.”

According to Rodriguez, the tactile learning system the researchers have developed can be used in applications beyond Jenga, especially in tasks that need careful physical interaction, such as assembling consumer electronic devices.

“In a cellphone assembly line, in almost every single step, the feeling of a snap-fit, or a threaded screw, is coming from force and touch rather than vision,” Rodriguez says. “Learning models for those actions is prime real-estate for this kind of technology.”