fix: read legacy compacted LSM keys across shared leaves

[justinblethrow-cloud]

What does this PR do?

Makes exact lookups and full-key ordered ranges compatible with compacted LSM
files written before the shared-leaf overflow safeguard landed.

For a matching root key, the reader now also inspects the leaf immediately
preceding the first matching root entry. Older compacted files can contain the
first chunk of a high-cardinality key on that leaf, while the root entry for the
leaf is keyed by the preceding key. The existing result set removes overlap.

Ascending full-key ranges now start at the first matching root entry and the
possible shared predecessor, rather than at the last matching root entry.

The regression test opens a synthetic legacy database fixture containing
30,000 records with one composite key and verifies exact lookup plus ascending
and descending full-key ranges before and after reopen.

Motivation

Current main prevents newly compacted files from creating this layout, but
that writer change cannot repair files already persisted by an older release.
On the included fixture, current main returns 28,857 records by exact lookup
and 4,154 through the equivalent ascending full-key range, while a full scan
returns 30,000. With this reader change, exact lookup and both range directions
return 30,000 without rebuilding the index.

Related issues

• Non-unique LSM composite index silently misses entries for keys with many duplicates #5119 and LSM index compaction loses entries (lookups return fewer records than a scan); REBUILD INDEX repairs but next compaction loses them again #5120 reported similar index-versus-scan symptoms. They were closed
  after the compaction-atomicity fix in [lsm] Compaction failure atomicity: orphaned series become visible on the next round; first-compaction temp file leaks #4946; this PR addresses a distinct
  backward-compatibility case for already-written shared-leaf layouts.
• The writer safeguard that prevents new shared-leaf overflow layouts landed as
  part of feat: [#4687] native BM25 full-text scoring (field boosts, caret, EXPLAIN/PROFILE) #4695.

Additional Notes

• The write path and on-disk format are unchanged.
• Normal current-writer files remain valid. The extra leaf read is limited to
  exact root-key matches that have a preceding leaf in the same compacted
  series.
• Partial composite-key range boundary behavior is unchanged.
• The fixture is synthetic and contains no application data.
• At the time of this work, CI on base commit e03ffbbf and an unfiltered local
  engine run both failed the unrelated Issue5147SuperNodeChunkRaceTest with
  the same BufferUnderflowException. The focused LSM matrix below is green.
• ./mvnw clean package -DskipTests completed successfully across all 24 Maven
  modules. Tests were deliberately skipped in that packaging check because the
  base revision has the unrelated failure above.

Checklist

• I have run the build using mvn clean package command
• My unit tests cover both failure and success scenarios

Focused verification currently passes 35 tests across the legacy fixture,
duplicate composite-key regressions, compaction correctness and atomicity, and
general LSM type-index behavior.

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request

[codacy-production]

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 6 complexity

Metric	Results
Complexity	6

View in Codacy

_{NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.}

[gemini-code-assist]

Code Review

This pull request introduces changes to LSMTreeIndexCompacted to support backward compatibility with legacy compacted files written before the shared-leaf writer safeguard. It ensures that the first chunk of an overflowing key, which may reside on a preceding leaf, is correctly retrieved. A new compatibility test has been added to verify this behavior. The review feedback suggests wrapping the ResultSet queries in the test with try-with-resources to prevent potential resource leaks.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

The ResultSet objects returned by database.query should be closed to prevent potential resource leaks during test execution. Wrapping them in a try-with-resources block ensures they are properly closed.

Suggested change

	final ResultSet scan = database.query("sql", "SELECT count(*) AS c FROM Tok WHERE word.trim() = 'dup'");
	assertThat(((Number) scan.next().getProperty("c")).longValue()).as("full scan count").isEqualTo(EXPECTED_DUPLICATES);
	final ResultSet indexed = database.query("sql", "SELECT count(*) AS c FROM Tok WHERE word = 'dup' AND lang = 'xx'");
	assertThat(((Number) indexed.next().getProperty("c")).longValue()).as("legacy compacted-index lookup count")
	.isEqualTo(EXPECTED_DUPLICATES);
	try (final ResultSet scan = database.query("sql", "SELECT count(*) AS c FROM Tok WHERE word.trim() = 'dup'")) {
	assertThat(((Number) scan.next().getProperty("c")).longValue()).as("full scan count").isEqualTo(EXPECTED_DUPLICATES);
	}
	try (final ResultSet indexed = database.query("sql", "SELECT count(*) AS c FROM Tok WHERE word = 'dup' AND lang = 'xx'")) {
	assertThat(((Number) indexed.next().getProperty("c")).longValue()).as("legacy compacted-index lookup count")
	.isEqualTo(EXPECTED_DUPLICATES);
	}

[justinblethrow-cloud]

Addressed in dfed7b7. Both query results now use try-with-resources, and LegacySharedLeafIndexCompatibilityTest passes.

[lvca]

@justinblethrow-cloud thanks for the PR, checking it...

[lvca]

Thanks for the PR!

[codecov]

Codecov Report

❌ Patch coverage is 11.76471% with 15 lines in your changes missing coverage. Please review.
✅ Project coverage is 33.90%. Comparing base (3b39ed0) to head (dfed7b7).
⚠️ Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
.../com/arcadedb/index/lsm/LSMTreeIndexCompacted.java	11.76%	13 Missing and 2 partials ⚠️

Additional details and impacted files

@@              Coverage Diff              @@
##               main    #5210       +/-   ##
=============================================
- Coverage     66.75%   33.90%   -32.86%     
- Complexity        0     1190     +1190     
=============================================
  Files          1696     1696               
  Lines        137700   137716       +16     
  Branches      29531    29536        +5     
=============================================
- Hits          91926    46696    -45230     
- Misses        33306    82049    +48743     
+ Partials      12468     8971     -3497     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.