Text analysis for hybrid search: tokenization, stopwords & accent folding

Originally published on the Weaviate blog

Read the original on Weaviate →

Hybrid search combines vector similarity with exact keyword matching — and the keyword half lives or dies by tokenization: how text gets broken into the discrete units that BM25 scores against.

The article gets concrete about the choices that matter:

• Four tokenization methods and when each is the right call.
• Accent folding for multilingual content, so "café" and "cafe" match.
• Per-property stopwords to tune what counts as signal vs. noise.
• The /v1/tokenize endpoint — test how an analyzer configuration behaves before committing to it, without reindexing your data.

Read the full article on Weaviate →