The MosAIc algorithm

#mosaic #ai #art

khushali shah Sept 17 2020 · 3 min read
Share this

This MosAIc algorithm is developed by MIT students along with Microsoft team. This is new AI based MosAIc algorithm which can identify subtle visual similarities between artworks created in vastly different media, cultures, and time periods. In short we can say that this algorithm helps to identify the similarities between two art-pieces and these art pieces is made by some artists and the artists never met before in their life, but still some similarities in their work. So the aim of this project is to find those similarities.

Let's see some more details of this algorithm:-

  • This algorithm is related to art so if we think about an art then we can say art is creativity or kind of language that expresses imaginative skills of the artist. Art is kind of subjective matter. Some people take art relaxing, exciting while some will take as informative. It's up to the people's imagination.
  • There is a need of very open mind and sharp eye to find those similarities between the artworks made by different artists.
  • This Mosaic project is taking this intellectual and human approach and augmenting it with the help of artificial intelligence to find connections between works of art from diverse artists.
  • They have Work with data from The Rijksmuseum – the Museum of the Netherlands(In Amsterdam) and The Metropolitan Museum of Art (The Met) for visual search that finds similarities across cultures in the museums’ collections. It matches paintings from this two museums through hidden stylistic connections.
  • Now, if we think of how the idea of this project came from?

    Mosaic project sponsor Mark Hamilton, Research Engineer pursuing a PhD at MIT was inspired by a visit to an exhibit at the Rijksmuseum.

    This exhibit compared works from the studios of Rembrandt and Velazquez, two prolific artists that never met before, still told similar stories in their works.

    He began thinking of possible ways the same idea could be achieved by AI, and developed mosaic algorithm which searches for similar works of art based on a starting image and a feature of that image such as the medium, a color, or the culture (e.g. “French”).

    Below is one of the example or output of this algorithm:-


    From the above figure, we can see giving input image blue and white dutch banyan dress image, we can see multiple outputs from this image from this 2 museums by culture as well as media.

    Based on a starting image of a Banyan, the algorithm finds similar works of art from different artists in the Rijksmuseum and The Met collections. This algorithm matches the dutch banyan image to a resulting matching blue & white music instrument.

    The lead author of the paper mark has told that:-

    The algorithm picked up on commonalities of form between the banyan and a string instrument, but also the color of the blue and white motif which originated in Ming Dynasty and spread through Europe. You can see evidence of cultural exchanges happening through the outputs of this algorithm

    - - Lead author of the paper( Mark hemilton)

    Below is the results on open access artwork from the Metropolitan Museum of Art and Rijksmuseum using culture and media,top row text, as conditioners, taken from the research paper.

    Image-source:- From the research paper (

    Here is another output of this algorithm:-

    Image-source:- From the research paper 

    Above example explains the similarity between the query image and any requested match. Unimportant pixels have been masked. The subject of the work is selected as an important contribution to the similarity.

    Till now we have seen what this mosaic algorithm is and what it doing . Let's check a very brief of the logic behind this algorithm.

    They apply conditional KNN data-structures on the combined open-access collections of the two museums defined earlier  to create an algo which is  capable of identifying artistic connections across time, space, culture and media. It puts similar images in a tree-like structure and traverses it till they find the closest result

    Image-source:- From the research paper

    From the above figure,in  left hand side tree the colored blocks are the image within leaf node, while  on right hand side is the result of applying conditioning on the particular class can prune the nodes. For more details about it you can find full research paper here

    For building this algorithm mark hemilton said that

    Building this algorithm was challenging as the aim was to match images that weren't just similar in color & style, but also in meaning and theme.

    - Mark hemilton

    One more note from the author regarding this project to take this project as enhancement or added features for curators,but not an replacement. Here are the words from the author:-

    Our method is good at building a particular type of exhibition: unlikely pairs of art that span barriers and share common structure,The world’s art exhibits are infinitely varied, each with their own thesis, and this is what makes art wonderful. We hope the community can see this method as adding a distinct voice to the chorus, not replacing it.

    - mark hemilton

    You can Explore more:-

    Full research paper available here


  • Comments
    Read next