INT8 quantization doesn’t reduce disk/RSS; db size grows vs. FP32

[tae898]

Summary
INT8 quantization speeds ingest but doesn’t shrink footprint. For MSMARCO-1M:

• Ingest time: ~1h07 (INT8) vs. ~1h51–2h00 (FP32).
• Recall: ~0.9072 vs. ~0.8994 (comparable).
• Footprint: db size ~10.6 GB (INT8) vs. ~9.6 GB (FP32); RSS still ~9.5 GB.

It appears quantized vectors are not reducing storage/memory—likely due to duplicated storage (graph + bucket) or storing float copies alongside int8.

Repro

1. Build vector index with quantization=INT8, storeVectorsInGraph=false.
2. Ingest MSMARCO-1M, run one search to trigger graph build.
3. Measure db size and RSS during/after build.

Expected
INT8 should materially reduce disk and memory vs. FP32.

Actual
Disk increases and RSS remains high.

Possible causes

• Vectors still serialized as float somewhere (graph or doc fetch).
• Duplication persists despite quantization (graph + bucket both storing vectors).

Help
I can investigate storage paths and submit a fix once root cause is confirmed.

[lvca]

Is this fixed after the latest fix?

[tae898]

yeah this problem is still there. I'll look into it.

[lvca]

Hi @tae898, several things have landed since you reported this that should materially change the picture:

• 01ae78ce5 - LSM vector index compaction now keeps the quantized form (it was previously re-expanding back to FP32 during compaction, which I suspect was the main culprit behind your "INT8 db larger than FP32" measurement)
• cf34e11cd - fixed a JVector bypass bug where separate-storage vectors were being duplicated
• 56f35bbdd - WAL replication now properly skips quantization data
• ae224e9ee (PR Support for INT8 in dense-only vector indexes #4133, yesterday) - first-class INT8 support for dense-only vector indexes
• d8fe7f4a0 (PR HTTP: route int8 query vectors to byte[] via $bytes/$int8 markers (#4… #4136) - HTTP path now routes INT8 query vectors as byte[] via $bytes/$int8 markers, so clients don't round-trip through float

Could you re-run the MSMARCO-1M benchmark on current main? For the smallest footprint, set both:

• VectorEncoding.INT8 on the document property (stored as BINARY, 1 byte/dim instead of 4)
• VectorQuantizationType.INT8 on the index (1 byte/dim + min/max metadata in index pages)

Note the factory now rejects redundant double-quantization, so pick the combination that fits your workload.

If the footprint is still off after the rerun, please attach a fresh repro and I'll dig into it. Otherwise we can close this one.

[tae898]

11-May-2026 Benchmark Results

These summaries now report ArcadeDB build modes correctly across both build and search runs:

• quantization / encoding in the Example 11 build results are tracked correctly.
• build_quantization / build_encoding in the Example 12 search results now resolve correctly for reused Example 11 databases.
• Disk usage is saved for both build and search runs and carried into the generated summaries from the per-run disk_usage_du*.json artifacts.

11 Vector Index Build Matrix Summary

Dataset: stackoverflow-medium

backend	quantization	encoding	mem_limit	threads	batch_size	count	rows	run_total_s	create_db_s	create_index_s	ingest_s	close_db_s	peak_rss_mib	db_size_mib
arcadedb_sql	INT8	NONE	8g	4	5,000	1,242,391	1,242,391	778.707	0.339	739.641	38.6	0.055	6,863.488	3,250.941
arcadedb_sql	NONE	INT8	8g	4	5,000	1,242,391	1,242,391	795.565	0.325	719.702	75.463	0.01	6,872.727	683.505
arcadedb_sql	NONE	NONE	8g	4	5,000	1,242,391	1,242,391	802.589	0.301	757.279	44.876	0.071	6,832.219	2,781.441

12 Vector Search Matrix Summary

Dataset: stackoverflow-medium

backend	build_quantization	build_encoding	mem_limit	k	query_runs	query_order	ef_search	queries	recall_mean	latency_ms_mean	latency_ms_p95	run_total_s	open_db_s	search_s	close_db_s	peak_rss_mib
arcadedb_sql	INT8	NONE	4g	50	1	shuffled	100	1,000	0.909	87.121	100.064	89.998	2.479	87.176	0.012	3,131.676
arcadedb_sql	NONE	INT8	4g	50	1	shuffled	100	1,000	0.903	78.346	85.818	79.486	0.972	78.39	0.011	2,190.344
arcadedb_sql	NONE	NONE	4g	50	1	shuffled	100	1,000	0.913	226.04	268.102	228.753	2.269	226.094	0.012	3,292.16

I ran on the stackoverflow-medium dataset (created by myself), which has about 1.2 M vectors. The quantization & encoding works fine. Now it's clear that quantization=NONE and encoding=INT8 saves more disk space than the other two. Also during search, it uses less memory. But obviously it needs more data ingest time than the others. I guess it's because it has to do data type conversion, e.g., float to int8?

Thanks for the fix! 👍