Henri Matisse's portrait of Madame Matisse: a woman with dark blue hair, her face divided by a green stripe, against patches of pink, orange and green.
Henri Matisse, Portrait of Madame Matisse (The Green Stripe), 1905. Statens Museum for Kunst.

About

Semantic Art Search lets you explore open museum collections by describing what you're looking for. You don't need to know a title, an artist or the words a museum used to catalogue a work. It's free and open source, and isn't affiliated with any of the museums.

How the search works

Most traditional art search engines look for your words in titles and catalogue records. Here, every image and every search is converted to a vector, a list of numbers that captures what it shows or means. The model that makes these vectors learned from a vast number of images paired with descriptions, so it picks up both what a picture shows and how it looks. The results are the images whose vectors lie closest to your search. If a cat appears in a painting, a search for "cat" has a good chance of finding it, even if the museum's record never mentions one. You can also try broader ideas, such as battle at sea or mourning. You can search in many languages, or upload an image of your own to find artworks that resemble it. This works best with a picture of an artwork, such as a photo you took in a museum. Photos of people or everyday scenes tend to give weaker matches.

Results are ranked by how closely they match, not filtered by an exact match, so some will be surprising and some simply wrong. Title Search does the same with titles instead. It matches meaning rather than spelling, so "illness" also finds Visiting the Sick and Convalescents, and "London" finds Chelsea and Greenwich. It works best for what a title names but an image can't easily show: a story or character such as Hamlet, a particular place, or an idea such as jealousy. It also helps when you know roughly what a work is called.

The art map

Every artwork's image is stored as a vector of 256 numbers, which you can think of as its position in a high-dimensional semantic space. To see the collection the way the model does, you'd need a 256-dimensional mind. The map is a stand-in: it flattens that space to two dimensions, with each dot an artwork and works the model considers similar placed close together.

Flattening loses a lot, so the map is a rough picture rather than a precise chart. Neighbours are the most reliable part: nearby works usually share a subject, a style or a technique. Long distances mean less, so two works far apart may still have much in common, and a search's matches can end up scattered. The directions mean nothing: up or left doesn't stand for time or style.

Only the images decide where a work is placed, not titles, dates or museum records. Those only supply the colours, so if works from the same century gather together when you colour by century, the model has picked that up from the images themselves: what they show and how they look.

You can also search from the map to see where the matches gather, or use "Show on map" on any artwork to find its place in the collection. Open the art map.

The collections

This site is only possible because these museums share their collections openly:

The site includes works the museums have released as public domain or open access, and focuses on visual art such as paintings, prints and drawings, so it covers only part of each collection. Titles, dates and other details come from the museums' data. Each artwork links back to its page at the museum, which is the place to check them.

Generated descriptions

On any artwork you can ask for a short description, written by a language model in the style of a museum wall label. Along with the image, the model reads the museum's catalogue record for the work, which often holds more than this site shows, such as subject descriptions, inscriptions or curatorial notes. It's asked to take facts like titles, dates and names from that record and to leave things out rather than guess.

A description can be a quick way into a work, especially when the museum's record is long, technical or in another language. It can still contain mistakes and clichés, so don't treat it as scholarship.

Your searches and images

There are no user accounts. The site records the text and filters of each new search, with the time, so I can see how it's used. That record doesn't include your IP address. Your IP address is used briefly to limit how many requests can be made, and the web server's logs record IP addresses and the pages requested, including search text, for a limited time. Visits are counted with GoatCounter, which uses no cookies and doesn't store IP addresses; it records the page visited (without search text), the referring site, and browser, screen size and country.

To run a search, your text or image is sent to an external service, which converts it to a vector. This site doesn't store images you upload, but they do pass through that service, so please avoid uploading anything private.

For developers

The source code is on GitHub. There's also a rate-limited JSON API: /api/search/?query=... searches the collection, and /api/museums/ lists the museums.

Get in touch

Found a mistake, have an idea, or work at one of these museums? Open an issue on GitHub or message me on LinkedIn.