A Probe into How Machines See Art

In an event on Oct. 23, Director of Digital Infrastructure and Emerging Technology at Harvard Art Museums Jeff Steward revealed that the museum staff uses a surprising method to classify their massive art collection: artificial intelligence. With a collection of over 250,000 pieces, the museum is looking for a more efficient way to categorize their art, and as a result, make art more accessible to a wider audience. The goal, Steward said, is to find “a new way to engage people with art collections who just might not have a way in.”

How does using artificial intelligence do this? Steward said that the answer lies in using AI to describe the artwork without any other context besides the work itself. “[The AI] are forced to just tell me what they see. It’s like someone going to the museum for the first time,” Steward said. This process humanizes the cataloging process by describing the artwork in terms more similar to what an average person, instead of an expert, might use.

“I’m not interested in training machines to think like I do,” Steward said. “I want to know what they think on their own, unpainted by the art history, art appreciation perspective.” The goal is to eventually use these AI-generated descriptions as keywords or search terms for people searching for art on Harvard’s databases.

To demonstrate how exactly this process is done, Steward started by introducing some basic terminology to the audience, including definitions of “machine processing,” “computer vision,” and “artificial intelligence.” He spent the rest of the talk showing examples of pieces of art from the Harvard Art Museums collection and how four different image recognition AI softwares — Microsoft Cognitive Services, Google Vision, Clarifai, and Imagga — describe the art.

One such example is of the painting “Random Patterns” by Hungarian painter Gyorgy Kepes. Google Vision described the work with terms such as “art, wall, modern art, wood, texture, artwork, tree,” while Microsoft Cognitive Services used terms such as “dirty,” “painting,” and most interestingly, “a bathroom with graffiti on the wall.” Each term was often associated with a numerical value of up to 1.00, indicating the AI’s confidence level in the accuracy of its description. Another painting of “Still Life with Watermelon” by Sarah Miriam Peale was described as “cake with fruit on the table” by Microsoft Cognitive Services, “citrillus” by Google Vision, “no person 99.0 percent” by Clarifai, and “juicy” by Imagga. Steward noted how Imagga’s descriptions of “juicy,” “sweet,” and “delicious” show the AI’s ability to suggest the benefits or sensations one might experience from eating the fruit.

The event in Cabot Library attracted audience members of different ages and backgrounds. Robin G. McElheny, an archivist for the Harvard University archives, said she was inspired to attend the event because it relates to her own work in the archives.

“I’m always interested in finding out about new ways to work with image collections, whether it’s using artificial intelligence to catalog them or to provide new ways of accessing the collections,” McElheny said. “Although it’s easy to laugh at the descriptions the machines create, it does pique one’s interest to look at the image and find out more.”

Another attendee, Reynor J. Huang, is currently working as a software engineer in Boston and spoke to the intersection of technology and art in context of the talk.

“A lot of the time it’s tech people trying to apply [concepts] to the humanities, so it’s really cool to have the humanities side trying to look at it in terms of tech,” Huang said.

“What Do Machines See When They Look at Art?” provided audience members with a look into the novel applications of computer vision and article intelligence. Through the use of these tools, Harvard Art Museums is working to bring new perspectives to artwork and enable it to reach a greater number of people.