SetupAI
繁
SourceStalecollected in 14h

Optimizing Hybrid Search with Tokenization and Accent Folding

Read original on Weaviate Blog
#vector-database#search-optimization#rag#nlp

Learn how to improve your RAG search accuracy using Weaviate's new tokenization and accent folding features.

30-Second TL;DR

What Changed

Introduced accent folding to normalize text for better search matching

Why It Matters

These improvements allow developers to build more accurate and language-agnostic search systems. By fine-tuning tokenization, practitioners can significantly reduce noise in retrieval-augmented generation (RAG) pipelines.

What To Do Next

Test your current search queries against the /v1/tokenize endpoint to verify if accent folding and custom stopwords improve your retrieval precision.

Who should care:Developers & AI Engineers

Key Points

  • •Introduced accent folding to normalize text for better search matching
  • •Added support for custom stopwords to refine search relevance
  • •Exposed text analysis capabilities via the /v1/tokenize API endpoint
  • •Enhanced BM25 performance for multilingual search applications

Deep Insight

Background and context from public sources — not the original article. 8 sources cited.

Enhanced Key Takeaways

  • •The new text analysis features, including accent folding and custom stopwords, were released as part of Weaviate v1.37, which also introduced other significant capabilities like a Model Context Protocol (MCP) Server for AI agent integration, Diversity Search with MMR, and Query Profiling.
  • •The /v1/tokenize endpoint allows developers to test different tokenizer and analyzer configurations on arbitrary text before committing them to a schema, providing transparency and aiding in the tuning of text analysis.
  • •Custom stopword presets can be defined per collection and assigned to individual text properties, enabling tailored multilingual configurations without requiring reindexing when stopword settings are changed, as stopwords are filtered at query time.
  • •Weaviate's hybrid search leverages the BM25F algorithm, an extension of BM25 that supports multiple weighted fields, allowing different property weights in ranking calculations for more accurate results.

Competitor Analysis

Core Focus
Weaviate
Semantic Search, Hybrid Search, AI-native Vector Database
Milvus
High-performance, Large-scale Vector Similarity Search
pgvector
PostgreSQL extension for vector search
Elasticsearch/OpenSearch
Full-text search, Vector search (add-on), Hybrid Search
Indexing Algorithms
Weaviate
Primarily HNSW, but also supports BM25/BM25F for keyword search
Milvus
HNSW, IVF, ANNOY
pgvector
Limited, leverages PostgreSQL indexing
Elasticsearch/OpenSearch
Inverted index for keywords, dense_vector fields for vectors
Semantic Understanding
Weaviate
Excels, built-in ML model integrations for automatic embedding
Milvus
Primarily vector similarity, less native semantic understanding
pgvector
Basic vector similarity
Elasticsearch/OpenSearch
Requires external models for semantic embeddings
Scalability
Weaviate
Designed for horizontal scaling, multi-tenancy support, handles billions of vectors
Milvus
High performance and scalability for massive datasets (100B+)
pgvector
May struggle with very large datasets and high query loads
Elasticsearch/OpenSearch
Scalable, distributed architecture
Hybrid Search
Weaviate
Native, combines BM25/BM25F and vector search with fusion strategies (e.g., relativeScoreFusion, rankedFusion)
Milvus
Supports hybrid search (alpha parameter for tuning BM25 and vector search)
pgvector
Can be combined with traditional SQL search, but not natively hybrid
Elasticsearch/OpenSearch
BM25 + dense_vector fields, RRF for hybrid scoring
Text Analysis Features
Weaviate
Accent folding, custom stopwords, configurable tokenization (word, lowercase, whitespace, field, trigram), /v1/tokenize endpoint
Milvus
Less focus on advanced text analysis features within the vector database itself
pgvector
Relies on PostgreSQL text processing capabilities
Elasticsearch/OpenSearch
Extensive keyword analysis, tokenization, stemming, stopwords
API
Weaviate
GraphQL and REST APIs
Milvus
REST API, Python/Java/Go SDKs
pgvector
SQL interface
Elasticsearch/OpenSearch
REST API
Deployment
Weaviate
Self-hosted (Docker), Weaviate Cloud (managed, serverless)
Milvus
Self-hosted, Zilliz Cloud (fully managed Milvus)
pgvector
PostgreSQL extension
Elasticsearch/OpenSearch
Self-hosted, managed cloud services
Maturity
Weaviate
Relatively newer, but rapidly evolving with strong community
Milvus
Well-established, mature ecosystem
pgvector
Leverages PostgreSQL maturity
Elasticsearch/OpenSearch
Highly mature, widely adopted
Data Management
Weaviate
Strong metadata management, namespaces, metadata filtering
Milvus
Partitioning, advanced metadata handling, fine-tuned control over index creation
pgvector
Leverages PostgreSQL data management
Elasticsearch/OpenSearch
Comprehensive data management, filtering, aggregation

Technical Deep Dive

  • Architecture: Weaviate employs a 3-layer architecture consisting of an API layer (handling REST and GraphQL), a Search layer (executing queries), and a Storage layer (storing objects and vectors).
  • Hybrid Search Mechanism: Weaviate's hybrid search executes both vector search (for semantic similarity using HNSW) and keyword search (using BM25/BM25F) in parallel. The results are then combined using fusion strategies like relativeScoreFusion (default from v1.24) or rankedFusion (default for v1.23 and lower) to produce a unified ranking.
  • BM25F Algorithm: Weaviate utilizes BM25F, an extension of BM25, which allows for different weights to be applied to multiple text fields (properties) during the ranking calculation. This enables property boosting, where certain fields (e.g., a product title) can be given more importance. Tunable parameters include k1 (controls term frequency saturation) and b (controls document length normalization).
  • Tokenization Options: Weaviate offers several tokenization methods configurable per property:
    • word (default): Splits text by non-alphanumeric characters and lowercases each token, recommended for general text.
    • lowercase: Splits text by whitespace only and lowercases each token, preserving symbols. Useful for technical data where symbols are meaningful.
    • whitespace: Splits text by whitespace only, preserving case and symbols.
    • field: Treats the entire property value as a single token, useful for exact matches of unique identifiers.
    • trigram: Provides fuzzy matching capabilities by breaking text into overlapping 3-character sequences, aiding in typo tolerance and partial matching (introduced in v1.32).
  • Accent Folding: This feature normalizes accented characters (e.g., 'Café' becomes 'Cafe') to ensure that queries with or without accents can match the same underlying text, significantly improving search recall for multilingual datasets.
  • Custom Stopwords: Stopwords are configurable per collection and per property. They are filtered out at query time from the BM25 score calculation, meaning changes to stopword lists do not necessitate reindexing. Custom presets can extend or modify built-in stopword lists.
  • /v1/tokenize Endpoint: This new REST API endpoint allows developers to tokenize arbitrary text with any tokenizer and analyzer configuration, or to tokenize text using an existing property's exact configuration, facilitating experimentation and debugging of text analysis.

Future ImplicationsAI analysis grounded in cited sources

Enhanced multilingual search capabilities will drive broader adoption of Weaviate in global markets.
Accent folding and custom stopwords directly address challenges in multilingual text processing, making Weaviate more effective for diverse language datasets and improving relevance for international users.
The /v1/tokenize endpoint will accelerate developer iteration and improve the quality of text analysis configurations.
By providing a transparent way to test tokenization and analysis settings before deployment, developers can more quickly fine-tune search relevance and reduce errors.
Weaviate's continuous improvements in hybrid search and text analysis will solidify its position as a leading vector database for complex RAG and AI agent applications.
Optimizing BM25 and text processing directly enhances the retrieval component of RAG, leading to more accurate and contextually relevant responses from LLMs and better agentic workflows.

Timeline

2015
Founder Bob van Luijt first encountered word embeddings (GloVe).
2016
The vision for Weaviate as an AI-first search engine crystallized for its founder.
2018
Weaviate entered a startup accelerator, leading to the founding of SeMI Technologies.
2022-12
Weaviate 1.17 released, introducing native hybrid search (BM25 + vector search).
2023-04
Weaviate launched its serverless offering, accelerating its growth.
2026-04
Weaviate v1.37 released, introducing Extensible Tokenizers with accent folding, custom stopword presets, and the `/v1/tokenize` endpoint.
Read original article on Weaviate Blog

Weekly AI Recap

Read this week's curated digest of top AI events →

PostLinkedIn

AI-curated news aggregator. All content rights belong to original publishers.
Original source: Weaviate Blog ↗

This is a summary, not the original. Read the source, or get the weekly briefing.

The weekly digest

One email a week. Unsubscribe anytime.