ALTER TABLE … RENAME orphans .tmp.spc.* files → permanent "directory is not empty"

[donhardman]

Summary

For a engine='columnar' table with a quantization='8bit' KNN column, once enough data is
flushed/merged the daemon leaves temp files <t>.<N>.tmp.spc.* in the table dir. A rename then
orphans them: the live table gets moved away, the temp files stay in the source dir, and any
later RENAME into that name fails permanently with:

error adding table 't': directory is not empty: /var/lib/manticore/t

Net effect on a zero-downtime reindex swap: the live table is renamed away with no replacement,
and search returns unknown local table '<t>' until manual FS cleanup + daemon restart.

Root cause (daemon, not Buddy)

Buddy's RENAME is emulated as CREATE TABLE dest LIKE src → ATTACH TABLE src TO dest → DROP TABLE src.
The daemon's ATTACH moves only tracked chunk files; the untracked .tmp.spc.* leftovers stay,
and DROP doesn't remove them or the now non-empty dir. So:

1. The daemon shouldn't leave .tmp.spc.* behind after merging a columnar + 8bit-quantized KNN table.
2. ATTACH/DROP (or CREATE … LIKE) should account for the full dir contents, not just tracked files.

Fixing #1 alone makes the whole reproducer pass.

Minimal reproducer (~30s, deterministic)

N=bug; IMG=manticoresearch/manticore:dev-28.3.2-e66ad54
docker rm -f $N 2>/dev/null; docker run -d --name $N $IMG >/dev/null
until docker exec $N mysql -e "SELECT 1" >/dev/null 2>&1; do sleep 1; done
Q(){ docker exec -i $N mysql -e "$1" 2>&1; }
VEC="("$(python3 -c "print(','.join(['0.1']*128))")")"
SCHEMA="id bigint, c text, e float_vector knn_type='hnsw' knn_dims='128' hnsw_similarity='cosine' quantization='8bit'"

Q "CREATE TABLE t ($SCHEMA) engine='columnar'"

# ~10 flushed chunks -> a background merge leaves .tmp.spc.* in /var/lib/manticore/t
for f in $(seq 1 10); do
  { for i in $(seq 1 400); do echo "INSERT INTO t (id,c,e) VALUES ($((f*400+i)),'x',$VEC);"; done; } | docker exec -i $N mysql >/dev/null 2>&1
  Q "FLUSH RAMCHUNK t" >/dev/null 2>&1
done

# native equivalent of ALTER t RENAME t__old (this is what Buddy runs internally):
Q "CREATE TABLE t__old LIKE t"                    # fresh empty destination
Q "ATTACH TABLE t TO TABLE t__old"                # moves tracked chunks, ORPHANS t.N.tmp.spc.* in /var/lib/manticore/t
Q "DROP TABLE t"                                  # source dir left non-empty with orphaned temp files

Q "CREATE TABLE t ($SCHEMA) engine='columnar'"    # -> the error below

ERROR 1064 (42000) at line 1: error adding table 't': directory is not empty: /var/lib/manticore/t

Minimization — all three ingredients required (5 trials each, 10 flushes)

config	.tmp left in src	swap
columnar + KNN quantization='8bit'	2	FAIL
columnar + KNN, no quantization	0	OK
columnar, no vector	0	OK
row-wise + KNN quantization='8bit'	0	OK
columnar + KNN 8bit, 1–3 flushes (no merge)	0	OK

So the leak needs engine='columnar' AND quantization='8bit' AND enough flushed data to
trigger a merge. The tipping point is the first background merge (~7 flushes here), not row count.
(Very rarely the leak also occurs without quantization; 8bit makes it deterministic.)

Expected

• The daemon cleans up .tmp.spc.* temp files after a columnar + 8bit KNN merge.
• Rename is atomic w.r.t. the full dir contents — either move/clean temp files, or fail before
  renaming the source away (no unusable name left behind).

Manticore Search Version:

Latest

Operating System Version:

Ubuntu

Have you tried the latest development version?

None

Internal Checklist:

^{To be completed by the assignee. Check off tasks that have been completed or are not applicable.}

Details

• Implementation completed
• Tests developed
• Documentation updated
• Documentation reviewed

[tomatolog]

issue was fixed at 894cd91

now the tmp files are removed by prefix and no any tmp should left on failures or interrupt