About similarities between pictures
Today we would like to take another look at the image similarity search integrated in prometheus, which creates image vectors based on the SwAV (Swapping Assignments between Views) self-supervised learning algorithm. They are limited to 80 dimensions sufficient for the result, which relate, for example, to color properties or the brightness of pixels as well as to the structural image composition. These created image vectors are pre-calculated for the images in the image archive and stored in the index so that the search engine queries are reduced to calculating the distance between these vectors stored in the index. They have not yet been created and indexed for all images in prometheus, but this is done at regular intervals. If images are deleted from the original databases and are therefore “not available”, they will remain in the index until the next update.
We occasionally receive feedback on the image similarity search from users who are not convinced by the results because, for example, one of our examples, a winter landscape by Witsen, shows many summer landscapes or „Asparagus“ by Edouard Manet in the results.
What do you think? Are the images similar or not similar?
We see the similarity in the images, between snow and sand, which we do not evaluate based on the metadata associated with the image. And yes, there are some surprising, astonishing and sometimes inexplicable results that we find in this way, especially when the calculated distance is greater in the results listed below.
But there are also fascinating results, as in the case of the „Madonna and Child“ by Giovanni Bellini.
Most of the time, however, we don’t do exploratory searches, but rather targeted searches to get less unexpected results and then we search for winter landscape or the keywords winter landscape, winter, landscape in the title using the Advanced Search.
Have you already tried the image similarity search?
