Optimizing Hybrid Search with Tokenization and Accent Folding

🔑 Enhanced Key Takeaways

•The new text analysis features, including accent folding and custom stopwords, were released as part of Weaviate v1.37, which also introduced other significant capabilities like a Model Context Protocol (MCP) Server for AI agent integration, Diversity Search with MMR, and Query Profiling.
•The /v1/tokenize endpoint allows developers to test different tokenizer and analyzer configurations on arbitrary text before committing them to a schema, providing transparency and aiding in the tuning of text analysis.
•Custom stopword presets can be defined per collection and assigned to individual text properties, enabling tailored multilingual configurations without requiring reindexing when stopword settings are changed, as stopwords are filtered at query time.
•Weaviate's hybrid search leverages the BM25F algorithm, an extension of BM25 that supports multiple weighted fields, allowing different property weights in ranking calculations for more accurate results.

📊 Competitor Analysis▸ Show

Feature / Product	Weaviate	Milvus	pgvector	Elasticsearch/OpenSearch
Core Focus	Semantic Search, Hybrid Search, AI-native Vector Database	High-performance, Large-scale Vector Similarity Search	PostgreSQL extension for vector search	Full-text search, Vector search (add-on), Hybrid Search
Indexing Algorithms	Primarily HNSW, but also supports BM25/BM25F for keyword search	HNSW, IVF, ANNOY	Limited, leverages PostgreSQL indexing	Inverted index for keywords, dense_vector fields for vectors
Semantic Understanding	Excels, built-in ML model integrations for automatic embedding	Primarily vector similarity, less native semantic understanding	Basic vector similarity	Requires external models for semantic embeddings
Scalability	Designed for horizontal scaling, multi-tenancy support, handles billions of vectors	High performance and scalability for massive datasets (100B+)	May struggle with very large datasets and high query loads	Scalable, distributed architecture
Hybrid Search	Native, combines BM25/BM25F and vector search with fusion strategies (e.g., `relativeScoreFusion`, `rankedFusion`)	Supports hybrid search (alpha parameter for tuning BM25 and vector search)	Can be combined with traditional SQL search, but not natively hybrid	BM25 + dense_vector fields, RRF for hybrid scoring
Text Analysis Features	Accent folding, custom stopwords, configurable tokenization (word, lowercase, whitespace, field, trigram), `/v1/tokenize` endpoint	Less focus on advanced text analysis features within the vector database itself	Relies on PostgreSQL text processing capabilities	Extensive keyword analysis, tokenization, stemming, stopwords
API	GraphQL and REST APIs	REST API, Python/Java/Go SDKs	SQL interface	REST API
Deployment	Self-hosted (Docker), Weaviate Cloud (managed, serverless)	Self-hosted, Zilliz Cloud (fully managed Milvus)	PostgreSQL extension	Self-hosted, managed cloud services
Maturity	Relatively newer, but rapidly evolving with strong community	Well-established, mature ecosystem	Leverages PostgreSQL maturity	Highly mature, widely adopted
Data Management	Strong metadata management, namespaces, metadata filtering	Partitioning, advanced metadata handling, fine-tuned control over index creation	Leverages PostgreSQL data management	Comprehensive data management, filtering, aggregation

🛠️ Technical Deep Dive

Architecture: Weaviate employs a 3-layer architecture consisting of an API layer (handling REST and GraphQL), a Search layer (executing queries), and a Storage layer (storing objects and vectors).
Hybrid Search Mechanism: Weaviate's hybrid search executes both vector search (for semantic similarity using HNSW) and keyword search (using BM25/BM25F) in parallel. The results are then combined using fusion strategies like relativeScoreFusion (default from v1.24) or rankedFusion (default for v1.23 and lower) to produce a unified ranking.
BM25F Algorithm: Weaviate utilizes BM25F, an extension of BM25, which allows for different weights to be applied to multiple text fields (properties) during the ranking calculation. This enables property boosting, where certain fields (e.g., a product title) can be given more importance. Tunable parameters include k1 (controls term frequency saturation) and b (controls document length normalization).
Tokenization Options: Weaviate offers several tokenization methods configurable per property:
- word (default): Splits text by non-alphanumeric characters and lowercases each token, recommended for general text.
- lowercase: Splits text by whitespace only and lowercases each token, preserving symbols. Useful for technical data where symbols are meaningful.
- whitespace: Splits text by whitespace only, preserving case and symbols.
- field: Treats the entire property value as a single token, useful for exact matches of unique identifiers.
- trigram: Provides fuzzy matching capabilities by breaking text into overlapping 3-character sequences, aiding in typo tolerance and partial matching (introduced in v1.32).
Accent Folding: This feature normalizes accented characters (e.g., 'Café' becomes 'Cafe') to ensure that queries with or without accents can match the same underlying text, significantly improving search recall for multilingual datasets.
Custom Stopwords: Stopwords are configurable per collection and per property. They are filtered out at query time from the BM25 score calculation, meaning changes to stopword lists do not necessitate reindexing. Custom presets can extend or modify built-in stopword lists.
/v1/tokenize Endpoint: This new REST API endpoint allows developers to tokenize arbitrary text with any tokenizer and analyzer configuration, or to tokenize text using an existing property's exact configuration, facilitating experimentation and debugging of text analysis.

🔮 Future ImplicationsAI analysis grounded in cited sources

Enhanced multilingual search capabilities will drive broader adoption of Weaviate in global markets.

Accent folding and custom stopwords directly address challenges in multilingual text processing, making Weaviate more effective for diverse language datasets and improving relevance for international users.

The /v1/tokenize endpoint will accelerate developer iteration and improve the quality of text analysis configurations.

By providing a transparent way to test tokenization and analysis settings before deployment, developers can more quickly fine-tune search relevance and reduce errors.

Weaviate's continuous improvements in hybrid search and text analysis will solidify its position as a leading vector database for complex RAG and AI agent applications.

Optimizing BM25 and text processing directly enhances the retrieval component of RAG, leading to more accurate and contextually relevant responses from LLMs and better agentic workflows.

⏳ Timeline

2015

Founder Bob van Luijt first encountered word embeddings (GloVe).

2016

The vision for Weaviate as an AI-first search engine crystallized for its founder.

2018

Weaviate entered a startup accelerator, leading to the founding of SeMI Technologies.

2022-12

Weaviate 1.17 released, introducing native hybrid search (BM25 + vector search).

2023-04

Weaviate launched its serverless offering, accelerating its growth.

2026-04

Weaviate v1.37 released, introducing Extensible Tokenizers with accent folding, custom stopword presets, and the `/v1/tokenize` endpoint.

Optimizing Hybrid Search with Tokenization and Accent Folding

⚡ 30-Second TL;DR

Key Points

🧠 Deep Insight

🔑 Enhanced Key Takeaways

🛠️ Technical Deep Dive

🔮 Future ImplicationsAI analysis grounded in cited sources

⏳ Timeline

📎 Sources (8)

👉Related Updates

When GraphRAG actually beats vector RAG

ARR August Cycle Submission Trends and Venue Speculation