Dropbox Image Search Made Simple

Searching for images in Dropbox used to be a tedious task, but with the right tools and techniques, it's now made simple.

You can search for images by keyword, and Dropbox will return results that match your query.

To get started, simply navigate to the Dropbox search bar and type in a keyword related to the image you're looking for.

The feature relies on machine learning techniques, specifically deep learning, which has made it much easier to build. Thomas Berg, a machine learning engineer at Dropbox, wrote a blog post about how the team built the feature.

Dropbox uses a combination of two machine learning techniques: accurate image classification and word vectors. The search index is made up of text-based search contents and image search contents.

Here's a simplified overview of how the search function works:

The relevance function, denoted as s = f(q, j), takes a text query 'q' and an image 'j' and returns a relevance score 's'. The score indicates how well the image matches the search query.

Dropbox's image search system uses a combination of semantic vectors and matrix multiplication to match your query with the right images. This process starts with turning your search query into a word vector using ConceptNet Numberbatch.

The word vector is then projected into Dropbox's category space using matrix multiplication, which converts the semantic meaning of your search query into a representation that matches the category space vectors of the images. This allows for a comparison of the projected vector with the image's category space vectors.

Dropbox measures the similarity between two vectors using cosine similarity. This is a key part of the image search system, as it enables the matching of categories in your query with the categories in the inverted index.

Dropbox's inverted index keeps track of all the categories and which images rank high in that category. This index is used to match the categories in your query with the categories in the inverted index, and find all the images that contain items from your query.

The final step in the image search system is ranking the matching images based on how well they match your query. This is done by using the forward index to get the category space vectors for each matching image, and then comparing these vectors with your query's category space vector to get a similarity score.

Here's a simplified overview of the image search system:

Dropbox's forward index and inverted index are both part of their in-house search engine, Nautilus.

Dropbox's image search feature relies on a combination of cutting-edge technologies.

One key component is the Convolutional Neural Network (CNN), a type of neural network that excels at image recognition.

Dropbox specifically uses a CNN from the EfficientNet family, which is optimized for fast inference without sacrificing accuracy.

Word vectors are another crucial part of the image search feature.

These vectors represent words as multi-dimensional vectors, with similar words having similar vectors.

Dropbox uses ConceptNet Numberbatch, a pre-computed set of word embeddings, to transform user search queries into word vectors.

Category space vectors are also used to represent the categories present in an image.

These vectors have dimensions for different categories, such as dogs, cars, and ice cream.

Dropbox's CNN model identifies the categories in an image and generates a category space vector for that image.

The magnitude of the vector in each dimension represents the significance of the category in the image.

To quickly find images containing specific categories, Dropbox maintains an inverted index.

This data structure keeps a list of all images containing a given category.

For each image, the forward index stores the category space vector, which encodes the information of what categories are in the image.

Dropbox's approach to image search is scalable, but it's not without its challenges. The 'text-search' method is expensive in terms of storage space and query-time processing.

The main issue with the 'text-search' approach is that it requires storing 10,000 classifier scores for each image, which can be costly in terms of storage space. If we use four-byte floating-point values, each image would require 40 kilobytes of storage space.

The classifier scores are rarely zero, and will be added to most of those 10,000 posting lists, making the index storage larger than the image file itself. However, Dropbox found that many near-zero values could drop to get a more efficient approximation in the case of 'image search'.

Here's a comparison between 'image search' and 'text search':

The query time for image categories is roughly the same as that of text queries, with only 10 posting lists to scan instead of 10,000. This results in a smaller result set, which can score more quickly.

Content Search is a game-changer for finding specific images in your Dropbox account.

With Dropbox's new image search feature, you can type in a few descriptive words and get relevant images suggested to you. This is a huge improvement over scrolling through a pile of photos or trying to guess the filename.

For example, if you're looking for photos from a picnic, you can simply type in the keyword 'picnic' and relevant images will be shown.