Miriam Redi is a Research Scientist at Yahoo Labs, London. Her research focuses on content-based social multimedia analysis. In particular, she explores ways to automatically assess image semantics, visual aesthetics, and multimedia creativity.
Miriam developed a strong visual analysis background during her Ph.D. at EURECOM Sophia Antipolis, with a thesis on “Novel Methods for Semantic and Aesthetic Multimedia Retrieval” and publications in main Multimedia conferences. After obtaining her Ph.D. in May 2013, she was an intern at Yahoo Labs Barcelona, working on video semantics and creativity assessment. Miriam’s passion for visual analysis started during her master thesis on blind image manipulation detection, developed at University of California Santa Barbara. I did an interview with her, recently. You can read the interview below:
1. As a vision researcher at Yahoo Labs, you developed a machine learning algorithm (CrowdBeauty) that can distinguish beautiful portraits from ugly ones. How does it work?
The CrowdBeauty algorithm can identify beautiful images depicting different subjects, including people, animals, nature, and urban scenery. To develop such an efficient computational aesthetic algorithm, we needed to teach a machine how to distinguish between good and bad images using an algorithmic approach.
To do so, we created a data set of 10,000 Flickr images and asked a large number of crowdsourced contributors to judge the beauty of the photos. (The data set is publicly available at di.unito.it). Next, from each picture we extracted a set of visual features that described the image’s visual properties, such as the light contrast, the color distribution, and the compositional uniqueness.
We then ingested the image features and the corresponding aesthetic judgments into a machine learning framework. The machine learned how to associate the photographic properties of an image with its degree of beauty. This would allow the algorithm to automatically score the beauty of new, unseen Flickr images.
2. Is it the same algorithm that identifies the unpopular Flickr images and digs out the un-noted photographic gems, hidden within this long tail of unpopularity?
Once trained, CrowdBeauty is able to score the aesthetic degree of any image in the Flickr network. To surface the hidden photographic gems, we isolated a large collection of around 10M unpopular Flickr images, and employed CrowdBeauty to compute a beauty score for each of them.
We then ranked the images in terms of their computed aesthetic score. The images at the top of the ranking corresponded to the most beautiful images in the collection, according to the CrowdBeauty algorithm. These were the hidden gems we were looking for: they were beautiful, yet unpopular images that would remain hidden from our eyes if we did not have a computational aesthetic algorithm to detect their presence in the long tail of unpopularity.
3. How about the accuracy of the results? How close is it to the human judgement of beauty?
To evaluate the reliability of the algorithm, and verify the high degree of beauty of the surfaced photographic gems, we again resorted to a crowd sourcing experiment. We asked annotators to evaluate the aesthetic degree of a new set of images, which included the images detected by the algorithm, and groups of Flickr images ranked as very popular, average popular and unpopular.
We were very happy to see that, according to this evaluation, CrowdBeauty was able to detect photos whose median perceived beauty score was equal to the most popular ones, and whose average was lower by only 1.5 points (out of a scale of 5).
4. So, there is hope for all those unappreciated photographers out there, right? Particularly with the new Flickr Beauty Explorer page?
With our work, we want to stress the importance of promoting content quality, as opposed to the widel used popularity-based metrics, including number of views, favorites and comments. Many talented photographers do not receive the attention they deserve due, in part, to the low popularity-based scores the receive with photo sharing sites.
With CrowdBeauty, we are able to discover high quality pictures, independent of their social signals. Such hidden gems could be collected in a new Flickr Beauty Explorer page showcasing the most beautiful yet undiscovered images, thus drawing people’s attention towards talented but underrepresented photographers. This would complement the current Flickr Explorer image collection, whose stunning pictures are characterized by very high social engagement.
5. What are the other exciting possibilities of this technology?
Our work promotes the democratization of photo sharing platforms, and creates an opportunity to balance the visibility of popular and beautiful photos with those that are as beautiful but with less social exposure.
This technology allows to increase the payoff of the photo service by uncovering valuable content, which can be used for promotion, advertising, mashup, or any other commercial service, that would have otherwise gone unnoticed.
On a different note, our algorithm rewards users based on their photographic talent rather than their popularity, thus fostering both the engagement and the artistic production of less popular contributors.
Moreover, while social dynamics require a large time span to establish a judgement on users’ talents, our algorithm could assess photographers’ skills as soon as they join the network by simply analyzing their image collections. This new perspective could motivate new users to contribute more to photo sharing sites, and improve the overall quality of the content.
Thank you for spending your valuable time with us, Miriam.
Miriam Redi spoke at the RE.WORK Deep Learning Summit, London, in September 2015. The next Deep Learning Summit takes place in San Francisco on 28-29 January. Use our discount code BDMS20 for 20% off tickets!
