Jeff Huang


Jeff Huang

Assistant Professor of Computer Science

Frank Mullin/Brown University
Gathering data on keystrokes and mouse movements can make programs better. Jeff Huang wonders whether all that information might also help users improve their own time management and life skills.

Perhaps the best way to describe Jeff Huang is by calling him an optimizer. He’s interested in collecting data about how humans interact with computers and using that information to make the computers and software better. But why stop there? He’d also like to use data and technology to make people better too.

Huang joins the Brown faculty this fall as an assistant professor of computer science.

Much of Huang’s research while working toward his doctorate at the University of Washington dealt with making search engines work better. Before starting his Ph.D., he worked at Yahoo, and during graduate school he had internships with Google and Microsoft. That gave him access to incredible amounts of data on how people use search engines.

All search engines collect data on which search results users click. Those data can then be used to improve search algorithms by pushing the most-clicked links higher on the results page. But clicks aren’t perfect data points. Users may end up not liking the link they clicked. And for some searches — say, the zip codes for Providence — the desired information is likely to be displayed on the results page, so users might not click a link at all.

Huang is interested in much finer-grained interactions than the click. “When I looked at the [click data], I couldn’t really see what was going on,” he said. “But if I stood over somebody’s shoulder I could watch them move their mouse and then think for a bit and then click. Just watching that was far more informative. I realized that search engines can actually capture this and process it into their system.”

Huang was able to show that mouse movements to control page scrolling can be something of a window into the mind of a search engine user, a fair proxy for looking over the searcher’s shoulder. He showed that cursor movements were closely correlated to a user’s eye gaze, which is a measure of where a user is directing attention. Huang developed a way of capturing cursor movements and page scrolls on search engine results pages and is using those data to improve search algorithms. His paper describing the work received honorable mention for best paper at CHI2011, a major conference on human-computer interactions.

Mobile devices are also a goldmine for this kind of fine-grained interaction data, Huang said. “For example, say you’re looking at a website on a phone. The type is really small, so you have to zoom in. Just by zooming in and seeing that one paragraph of text and spending seven seconds looking at it on the screen, you’re providing really useful information about what you’re interested in.” By capturing that information, Huang said, websites can be tweaked to better deliver the information users actually want.

At its heart, Huang’s work is about taking interactions between people and systems and using those data to improve the system. But can those interactions improve the user as well? Huang has begun to look into that question.

In a recent study, he accessed another goldmine of interaction data: online games. He looked at Halo, the massively popular online war game. “The idea is that there are millions of people who play Halo,” Huang said, “but we don't really know why certain people become the best at Halo.”

To find out, he looked at the statistics Microsoft collects for each game played and the skill rating system used to match players for online play. (Better skill matches make for better games.) He then looked at the habits of individual players. How often did they play? Did they play lots of games in a short time or space them out? What happens when players take long breaks from playing?

Among the results: After controlling for the number of total games played, players who play 100 games over the course of a month actually have higher skill than those who play 100 per week. Taking a 10-day break significantly reduces skill, but that skill is more easily recovered than it was gained in the first place.

“What I want to do going forward is to think about what can people do to track their own behavior. How could we use that to improve ourselves? Can we do experiments on ourselves to make ourselves better?”

He’s currently trying it out on himself. He’s written a program that uses geolocation data to track his movements around town. He’ll sync that up with a database of his daily activities. “It tracks how much time I spend sleeping, eating, housekeeping, doing research, and writing grants,” he said. “One goal is to find out interesting things like, when I sleep more, do I get more work done? Where do I get more work done? Am I more productive at home or in the office?”

And if you’re wondering how all this work will turn out, you might just be able to find out easily. Huang is considering making all the data — where he is and what he’s doing — freely available on his website.

“Call it ‘Open Jeff,’” he said. “It’s the extreme of no privacy. And this is a hot question right now. What happens if you don’t have any privacy at all? So if I do this, maybe we can find out.”

Browse New Faculty by Year