Skip to content

import never refreshes a stale database row, so a replaced source file is re-imported on every run #48

Description

@ptr727

Summary

import never refreshes an existing database row. When a source file is replaced by
different content at the same path, the import itself is correct, but the row keeps the
old hash forever, so the file is re-imported on every subsequent run indefinitely.

index handles this case correctly. import does not.

Source references are permalinks against
83f5101.

Mechanism

ImportTask records the import at
ImportTask.cs#L342-L358
with a single call:

await database
    .InsertAsync(
        new FileRecord(file, sha256, sha1, sourceInfo.Length, sourceInfo.LastWriteTimeUtc.Ticks, false),
        cancellationToken
    )

and
Database.InsertAsync
is INSERT OR IGNORE on the path primary key:

INSERT OR IGNORE INTO files (path, sha256, sha1, file_size, mtime_ticks, is_processed)
VALUES (@path, @sha256, @sha1, @fileSize, @mtimeTicks, @isProcessed)

If a row already exists for that path, the write is silently discarded.

UpdateHashesAsync
and
UpdateMetadataAsync
exist on Database but are called only from
IndexTask.cs#L55
and
IndexTask.cs#L71.
ImportTask calls neither, so there is no path by which an existing row is ever corrected
during an import.

For contrast,
IndexTask.IndexFileAsync
covers all three cases explicitly: insert when absent, UpdateHashesAsync when the SHA-256
moved, UpdateMetadataAsync when only size or mtime moved.

Note that the skip decision is not affected.
Database.ResolveHashesAsync
only trusts the cached hash while size and mtime both still match, so a replaced file is
re-hashed, Sha256ExistsAsync correctly returns false, and the new content is imported.
The defect is purely in what gets recorded afterwards. ImportTask already holds cached
from the GetByPathAsync at
ImportTask.cs#L153-L155.

Reproduction

Constructed case, two distinct media files:

mkdir -p /tmp/pc-src /tmp/pc-dst
cp first.jpg /tmp/pc-src/photo.jpg

PhotoCleaner import --path=/tmp/pc-src --outpath=/tmp/pc-dst --db=/tmp/pc-dst/Import.db
sqlite3 /tmp/pc-dst/Import.db "select path, sha256 from files;"
# one row, sha256 = hash(first.jpg)

# replace the source with different content at the same path
cp second.jpg /tmp/pc-src/photo.jpg

PhotoCleaner import --path=/tmp/pc-src --outpath=/tmp/pc-dst --db=/tmp/pc-dst/Import.db
# imports second.jpg correctly
sqlite3 /tmp/pc-dst/Import.db "select path, sha256 from files;"
# still sha256 = hash(first.jpg)   <-- the defect

PhotoCleaner import --path=/tmp/pc-src --outpath=/tmp/pc-dst --db=/tmp/pc-dst/Import.db
# imports second.jpg AGAIN, and will on every run from here on

Impact

Every run after the replacement re-copies the file into /Processed under a fresh
_N suffix and re-uploads it. Immich rejects the upload as a duplicate and
immich-cli --delete-duplicates cleans it up, so nothing is lost, but the work repeats
forever and the duplicate churn is invisible unless the log is read closely.

This matters most in exactly the situation where a source tree is repaired in place: a
file re-downloaded from an upstream source to its original path is permanently
un-recordable, so a repaired collection never settles.

Suggested fix

Give import the same three-way handling index already has: when the resolved hash
differs from the cached row, call UpdateHashesAsync instead of relying on an insert
that cannot land. The cached record is already in scope, so the comparison needs no
extra read.

An alternative is to make InsertAsync an upsert, but that changes behaviour for
index as well and loses the deliberate distinction between the insert and update paths,
so extending ImportTask looks like the smaller change.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions