insideout Get Pro

07 · Filters and metadata

Pro

Search and Retrieval, lesson 7

Filters and metadata

Combine vector search with dates, owners and permissions.

"Vector search" and "Indexes at scale with HNSW": how an index answers "which chunks are nearest?". This lesson adds conditions to that question.

Part 1. Questions with conditions

The handbook covers offices in the UK, the US and Germany. Each office has its own parental leave policy, and the three passages are written almost the same way.

An employee in Berlin asks "How much parental leave do I get?" To an embedding, the UK, US and German passages are nearly identical, and the question doesn't say "Germany". Similarity alone can't pick the right one.

Your turn

What's the most reliable way to make sure the Berlin employee gets the German policy?

The country is a plain fact about the chunk and the user. Comparing two labels is exact. Asking an embedding or a language model to infer it is a guess that can go wrong.

Metadata filter

Fields stored next to each chunk (country, team, document type, date, who may see it), and a condition on those fields that a chunk must meet to be returned.

Part 2. Filter before or after the search

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come
  1. Filter before or after the search
  2. Why filtering first is hard for ANN
  3. Filters as permissions
See Pro plansI have Pro: sign in
Next lessonMeasuring search with recall, MRR and NDCGScore a ranked list, and pick the metric that matches the job. Start lesson 8

Search and Retrieval overviewSpotted a mistake? Tell us.