Stefanie Tellex wants everyone to be able to talk to robots.

As she sees it, the problem in robotics today is not the physical capabilities of the machines. Autonomous devices can do just about anything from household tasks to building cars on assembly lines to exploring the surface of Mars. The problem is one of communication. Operating a robot generally requires a roboticist — someone fluent in programing languages.

To make robots more useful to all, there needs to be a better way to communicate with them.

“I work on making robots that understand natural language,” said Tellex, who joins Brown’s Department of Computer Science as an assistant professor. “Untrained users who want to have a robot engage in useful behavior can just talk to it as though they’re talking to another person. We take robotic capabilities — the way roboticists view the world and the way that they make robots view the world — and connect them to computational linguistics, the science of making computers understand language.”

Tellex has developed language interfaces for several devices already. During a postdoctoral appointment at MIT (where she also earned her Ph.D. in 2010), Tellex designed a system enabling an autonomous forklift to take orders in simple English. It responds to natural commands like, “Pick up the tire and put it on the pallet” or “Take the pallet to receiving.” These are simple commands for a human forklift operator. But to get a robot to match those words with objects, spaces, and actions in the real world is a difficult computational challenge — and a main theme in Tellex’s work.

To do it, she uses crowdsourcing and machine-learning algorithms to teach her robots human language. She asks people to watch video of a robot performing a task and then asks how they might phrase a command for that action. In collecting lots of examples, the system amasses a robust vocabulary that can be mapped to the robot’s models of physical space and objects. Because the system actually learns what words mean and how language is constructed, it’s able to infer meaning of new commands it hasn’t yet heard.

The forklift and several other devices in which Tellex has installed the interface show that the system works quite well. Her latest work is in using language tools to train robots in new tasks and to help them to get acquainted with new spaces.

Imagine taking that forklift and plopping it down in a different warehouse. How will it find its way around? Tellex and her collaborators have designed a system that enables users to give the robot a guided tour. The user walks with the robot, pointing out in plain English where things are. It’s just like showing a new co-worker around the office, explaining that the copier is upstairs, the bathroom is on the first floor, and the break room is down the hall to the left.

“We can do this with the robot using natural language input,” Tellex said. “We were actually able to make a model of the environment that was more accurate with the language input than the robot could make with just its sensors. Humans have a lot of knowledge in their heads, and by using language they are able to convey that knowledge to the robots.”

Another goal of Tellex’s work is making robot-human communication a two-way street by enabling a robot to talk back to its user when it encounters a problem.

“Robots fail all the time,” she said. “If you’ve seen a robotics demonstration, you’ve probably seen someone sneak in and kick something or restart something and suddenly the robot works again and is able to continue. Maybe the right way for robots to respond when they encounter failure, instead of having a technician come in and kick it, is to have the robot be able to ask people for help.”

That would make robots more adaptable and easier to work with, she says. “It might enable the human-robot team to do more together.”

And that’s the real thrust of Tellex’s work: developing language interfaces to make robots that are more useful and can do more things. In the process, she says, we may arrive at something even more fundamental.

“We have these big questions of what is intelligence and how can we make intelligent machines,” said Tellex, who was named recently by IEEE as one of “10 to watch” in artificial intelligence. “Separately, we have this very concrete question of how can we solve this interface problem and use language. They don’t always perfectly align, but by working on a real problem that real people are having with robots every day, it provides a focus on how to attack this larger scientific question.”