Native BM25 full-text scoring (TF/IDF + length normalization, field boosts, explain)

[lvca]

Summary

ArcadeDB's FULL_TEXT index currently ranks results with a simple term-coordination (match-count) model, not BM25. This proposes adding native BM25 (TF/IDF + document-length normalization) scoring to the full-text search path, so lexical relevance matches what users expect from Lucene/Elasticsearch/Solr/CrateDB.

Raised in discussion #4664 (enterprise hybrid-retrieval evaluation: vector + BM25/full-text + graph traversal for government/legal documents).

Current behavior

Lucene is used by ArcadeDB only for analyzers/tokenizers/stemmers and query-syntax parsing - not for the inverted-index storage or its scoring engine.

In LSMTreeFullTextIndex.get() the score for a record starts at 1 and is incremented by 1 for every matching query term:

// Accumulate score for this RID based on number of matching query terms
final AtomicInteger score = scoreMap.get(rid);
if (score == null)
  scoreMap.put(rid, new AtomicInteger(1));
else
  score.incrementAndGet();

This means:

• No term frequency (TF) within a document
• No inverse document frequency (IDF)
• No document-length normalization
• No field boosts in the main scoring path
• No score explanation / explain API

So a document mentioning a rare, highly-discriminative term once ranks the same as (or below) a document that trivially matches more common query terms. This is fine for boolean filtering but inadequate for relevance-ranked retrieval.

Note: BM25-style scoring with IDF (Robertson-Sparck-Jones) does already exist in the LSM_SPARSE_VECTOR path (BlockMax-WAND DAAT), but it requires the user to generate sparse vectors externally. There is no native "index text -> BM25 ranking" in the full-text path.

Proposed work

1. BM25 similarity for FULL_TEXT / SEARCH_INDEX(...)

   • Maintain the needed statistics: per-term document frequency (IDF), per-document term frequency (TF), per-document/field length and average length for normalization.
   • Implement the standard BM25 formula with configurable k1 (default 1.2) and b (default 0.75).
   • Make it the default ranking model (with a compatibility switch to fall back to the legacy match-count score).

2. Field boosts applied to the scoring (per-field weight in multi-field full-text indexes).

3. Score explanation - optional explain output describing how a score was computed (term contributions, IDF/TF/norm factors).

4. Configuration via index metadata (alongside the existing analyzer/defaultOperator/allowLeadingWildcard options): similarity model (BM25 | CLASSIC), k1, b, per-field boosts.

5. (Stretch) Auto text -> sparse-vector / unified hybrid index: index text and expose it directly to vector.sparseNeighbors / vector.fuse without external sparse-vector generation, so BM25 lexical + dense vector + graph traversal fuse in a single query path.

Why

• Closes the most-requested gap for accuracy-first hybrid retrieval (RAG, legal/government search).
• Brings full-text ranking in line with Lucene/Elasticsearch/Solr/CrateDB expectations.
• Composes cleanly with the existing vector.fuse(...) (RRF/DBSF/LINEAR) hybrid pipeline and in-database graph traversal.

Acceptance criteria

• BM25 scoring available and default for full-text queries, with documented k1/b and a switch back to the classic model.
• Field boosts honored in scoring.
• Optional score explanation output.
• Regression tests covering TF, IDF, length-normalization, field boosts, and backward-compatible classic mode.
• Documentation updated (full-text indexing + hybrid retrieval pages).

References

• Discussion: Any plan for BM25 and inverted index like lucene for Full text search? #4664
• Code: engine/src/main/java/com/arcadedb/index/fulltext/LSMTreeFullTextIndex.java, FullTextQueryExecutor.java, engine/src/main/java/com/arcadedb/schema/FullTextIndexMetadata.java
• Existing IDF reference implementation: engine/src/main/java/com/arcadedb/index/sparsevector/PaginatedSparseVectorEngine.java (Robertson-Sparck-Jones IDF + BlockMax-WAND)