WebSeer surfs the Web for pictures

Have you ever wondered whether there’s a picture of the Mona Lisa on the World Wide Web? Or wanted to find a good picture of an eagle? Or the aurora borealis, the Space Shuttle or Michael Jordan?

Now there’s a new image-based search engine that lets you find images directly, rather than surfing blindly or using text-based searches to try to find pictures. Dubbed “WebSeer,” this innovative image-based search engine was designed and built at the University of Chicago’s Intelligent Information Laboratory.

“This is the first useful way for people to search the Web for images, both photographs and drawings,” said Michael Swain, Assistant Professor in Computer Science, and the main architect of WebSeer. “I think this is going to be the most widely used application of computer vision ever.”

WebSeer stores contextual information about each image–such as the caption, the title of the Web page and the name of the image file–weighted as to how relevant it is to the content of the picture. But WebSeer also knows whether it’s a photograph or a drawing, whether it’s in color or black and white, whether or not there are people’s faces in the picture and if so, how many there are and how close up they are.

“Just looking at the pixels of an image, it’s beyond the current computer vision technology to figure out whether or not it is a picture of a horse," said third-year graduate student Charles Frankel, who developed the architecture and the indexing system for WebSeer. “Because it’s so difficult we have to use a combination of techniques that include the context that the image is in and the image-understanding algorithms that we do have. The architecture allows us to incorporate new image-understanding techniques as they are developed, either here or in collaboration with other universities.”

WebSeer has a sophisticated automatic indexing system that can add tens of thousands of images a day to its data base. To date there are 500,000 images, growing at a rate of one every two seconds.

“When you’re using a text-based search engine like AltaVista, it’s usually easier to look for very specific information, like the Chicago Bulls schedule, for example,” said Frankel. “But with WebSeer, you can search for a variety of different things that range from the very specific, like O.J. Simpson or Bill Clinton, to more general like eagles or roses, or even types of images like skylines or aerial views. It’s also fun to use WebSeer if you don’t really know what you’re looking for; you can see a picture and use it as a jumping-off place.”

Searching for “Michael Jordan, photograph” retrieves images of Jordan kissing the championship trophy, swinging a baseball bat and driving for the basket. Searching for “aurora” (northern or southern lights), yields several photographs–as well as a diagram of how the aurorae are generated–and links to pages such as the Space Plasma Physics Center, tourism in Scandinavia and Alaska, and NASA studies of the solar wind and auroral physics.

Clicking on the thumbnail of the image takes you directly to the image itself on the Web; to see it in context, click on the “page” icon next to the thumbnail.

How WebSeer analyzes images
WebSeer uses a sophisticated combination of contextual and visual cues to analyze and store information about images on the World Wide Web. Using a Web crawler–which starts at one Web page and then moves to all the links on that page and then all the links on all the pages on those pages, etc. to move through the Web–WebSeer will soon be able to add as many as 100,000 images a day to its data base.

When the crawler finds an image on a Web page, it first analyzes the text, like the caption, the name of the image file, the title of the Web page, etc. to gather clues about what is actually in the image. It weights the information according to how relevant it is likely to be (caption vs. page title, for example) and stores it. Next, WebSeer analyzes the image to figure out whether it is a photograph or a drawing–using information such as the file type, color, distinct boundaries, etc., in a “decision tree” developed by second-year University of Chicago graduate student Vassilis Athitsos and trained on 30,000 images.

Then WebSeer searches for skin hues to determine whether or not there are people in an image. If so, it uses a neural network, developed at Carnegie-Mellon University and trained on thousands of images, to locate each face in the image.

WebSeer stores in its data base a thumbnail version of the image, its URL and the URL of the page it’s on, along with information about the picture: whether it’s a photograph or a drawing, how many faces, the file type of the image, etc.

Plugging in new features
Swain plans to add many new features to WebSeer as new vision algorithms are developed. Already, researchers from Berkeley, MIT and Caltech, in addition to Carnegie-Mellon, are lining up to plug their applications into Swain and Frankel’s system.

Collaborators at Berkeley are working on detecting animals and people in pictures; MIT vision researchers are developing a trainable object-recognition system, and Caltech is developing a face-finding algorithm that will compete with Carnegie-Mellon’s. University of Chicago statistician Yali Amit, Associate Professor in Statistics, is working on reading text in images using optical character recognition techniques.

“What we’ve developed here at the University of Chicago is an incredibly valuable research tool for us as well as for researchers at dother universities,” said Swain, “and we hope to provide an interesting problem to motivate research in computer vision.”

Swain’s research is funded by the Office of Naval Research and the National Science Foundation. Some key equipment for WebSeer was donated by IBM.

The University of Chicago Intelligent Information Laboratory is focused on issues of information access and management. With an eye toward problems that are real rather than academic, the Info Lab is dedicated to the development of information technology that will form a bridge between people and the machines that serve them.