What search needs to understand about photos

Photo Search was built with Unicorn, an in-memory

and flash storage indexing system designed to search trillions of edges between tens of billions of users and entities. Created a few years ago to power the social graph-aware Graph Search, Unicorn supports billions of queries per day powering multiple components in Facebook.
Graph Search was built to retrieve objects from the social graph based on the relationships between them, such as "My friends who live in San Francisco." This has proven to be effective but presents engineering challenges when constraining the query to a relevant subset, sorting and scoring the results for relevancy, and then delivering the most relevant results. To augment this approach, the Photo Search team applied deep neural networks to improve the accuracy of image searches based on visual content in the photo and searchable text.

Understanding photos at Facebook’s scale presents different challenge as compared with demonstrating low image-recognition error rates in the Imagenet Challenge competition. Applied research has produced cutting-edge deep learning techniques capable of processing billions of photos to extract searchable semantic meaning at enormous scale. Each of the public photos uploaded to Facebook is processed by a distributed real-time system called the image understanding engine.
The image understanding engine is a deep neural network with millions of learnable parameters. The engine builds on top of the state-of-the-art deep residual network trained using tens of millions of photos with annotations. It can automatically predict a rich set of concepts, including scenes, objects, animals, attractions, and clothing items. We can train models and store useful information ahead of time, which enables low-latency responses to user queries.
Source