Image Classification Fun!

Recently the Wall Street Journal (my new favorite paper) featured an article Computer Scientists Pull a Tom SawyerTo Finish Grunt Work (June 27, 2007; Page B1). The gist was that computer scientists have come up with a new way to tackle the massive task of classifying images in large databases by making a game of it. Anyone who has tried to find an image using yahoo image search or google image search or any other large database of images, will have an idea of just how important this classification work is!

In this game, two random users will be shown one image and are asked to type in words to describe it. When their words match they "win" and the word is chosen as a descriptor on the assumption that there is at least a level of congruency. Of course this leads to a lowest common denominator situation and the classification will tend to remain relatively basic.

Labeling images is not a process so easily resolved by games but the idea is extremely interesting. Dr. Choi, a professor of mine, illustrates the complexity of the matter by referencing the classic 'is it an old lady or a young woman' picture.

Complicating matters further is cultural context. A study was done* whereby a three panel cartoon drawing was shown to young British and South African children. The first panel shows a young child wearing a ball cap, standing under a sun, holding a watering can over a flower. The second panel shows the child, under the sun, with his hat off and liquid droplets coming off his head. The final panel shows the child sitting down, under the sun, shirt off, pouring the watering can over his own head.

The British children understood the panels to be related and to be telling a story depicting a child watering a plant, the sun making him hot, and by pouring the water over his head, cooling himself off. The young South African children on the other hand were not inclined to view the pictures as related and took each one as a separate entity. They also did not understand the liquids to be water or sweat and thought it was perhaps blood and overall the entire scene made little sense to them. Interestingly, even in my own class there was debate over the last panel, some thought he was pouring the water over his head to cool off, others thought he got the idea that he wanted to grow like the flower.

So you see....classifying images...not so easy.

*Levie, W. Howard. Research on Pictures: A guide to the literature. In The Psychology of Illustration, edited by Dale M. Willows and Harvey A. Houghton, 1987, p.