Import conflict resolution: per-import policy with configurable defaults

[DutchJaFO]

Conflict types

Situation	Definition	Handling
Same stable ID (same quote + same source name)	The incoming record is a revision of an existing one	Governed by the conflict policy below
Same quote text, different source	Two genuinely different records that happen to share text	Both inserted; surfaced in the review UI (#65) for manual resolution

Same-text conflicts are not auto-resolved — they require human judgement and are out of scope for this issue.

Conflict policy (same stable ID)

Two options, selectable per import and as a global default:

Policy	Behaviour
skip	Keep the existing record. Incoming record is ignored.
newest-wins	Overwrite the existing record with the incoming one. Allows source files to be updated with corrections.

Default: newest-wins — this is the policy that lets users ship fixes via updated source files.

Configuration

• Global default stored in app settings: Quotinator__DefaultConflictPolicy (skip or newest-wins).
• Overridable per import run via a query parameter on the import endpoint (Import endpoint: POST /api/v1/quotes/import #45): ?conflictPolicy=skip or ?conflictPolicy=newest-wins.
• The startup seeder uses the global default. It does not accept per-run overrides.

ImportBatch recording

The ImportBatch row (#58) for each run records the policy that was active: ConflictPolicy TEXT NOT NULL. This ensures the behaviour of any past import is auditable.

Dependencies

• Import endpoint: POST /api/v1/quotes/import #45 (import endpoint) — exposes the per-run override.
• Schema: ImportBatches table and provenance link on all entity records #58 (ImportBatches schema) — stores the active policy per batch.
• Import manifest: ordered file list for bundled sources and user imports #63 (manifest) — order of files determines which record is "incoming" when two files share a stable ID.
• Import endpoint: preview/dry-run mode #65 (review UI) — same-text conflicts (not covered here) are resolved there.

[DutchJaFO]

Reconciled this issue's scope against the actual code already shipped on feature/data-import-sources and against #45's overlapping design. Full detail in docs/milestones/data-import-sources/64-conflict-resolution-plan.md; summary here for the record.

Five policies, not two or three

The final design has five policies, modeled directly on Git's own merge-conflict vocabulary:

Policy	Git analog	Behaviour
skip	ours (whole-side)	Keep the existing record entirely, discard incoming
newest-wins	theirs (whole-side)	Replace the existing record entirely with incoming
merge-ours	recursive -X ours	Auto-fill blank fields from either side; when both sides genuinely differ on a field, existing wins
merge-theirs	recursive -X theirs	Auto-fill blank fields from either side; when both sides genuinely differ on a field, incoming wins
review	manual conflict markers	Don't auto-resolve — behaves exactly like skip today; reserved so #45 can add real review/resolve behaviour later without another schema change

Default remains newest-wins as originally specified. Global config key is the flat Quotinator__DefaultConflictPolicy as originally specified; per-entity-type overrides (Quotes/Sources/Characters/People/Translations) stay nested under Quotinator:DuplicateResolution:*.

New System_ImportConflicts table (Data-owned)

A new table, owned generically by Quotinator.Data (same pattern as System_AuditEntries — its own migration list, excluded from Reset), logs every detected conflict, not just pending ones, with a Status of resolved/pending. Existing/incoming record content is stored as opaque JSON blobs — Quotinator.Data never deserializes them (it has no dependency on Quotinator.Core's Quote model); Quotinator.Engine produces and later diffs that content. This follows the project's standing rule that Quotinator.Data provides generic, reusable import infrastructure while consuming layers define their own domain needs. AppliedPolicy on both this table and ImportBatches is modeled as a proper enum (SafeValue<DuplicateResolutionPolicy?>), not a string — reusing the existing SafeEnumHandler<TEnum> infrastructure already used for QuoteType/Genre.

Retroactive scope acceptance

The already-shipped manifest-level duplicateResolution override and per-entity-type granularity (built for #57/#61's cross-source dedup needs) were never posted as a scope-change comment on any issue. Accepted and documented here, retroactively, per docs/workflow/process.md's scope-change rule.

Bundled file order fix

While reviewing this, we measured field-population rates across the bundled sources: quotinator-curated.json (100% date/character/genres, 2 quotes), NikhilNamal17_popular-movie-quotes.json (99.9% date, 0% character/genres, 732 quotes), vilaboim_movie-quotes.json (0% on all three, 99 quotes). The manifest's file order had vilaboim (barest) before NikhilNamal17 (richer) — backwards, since the bundled manifest's explicit skip policy means first-seen wins. Reordered to curated → NikhilNamal17 → vilaboim so richer data wins ties.

Relationship to #45

#45 defines its own conflictStrategy: skip|overwrite|review plus a /resolve endpoint — this predates and overlaps with the above. #45 owns its own endpoints and review/resolve workflow against the new System_ImportConflicts table; it may need its own spec refreshed to match this terminology, flagged separately on #45. Not blocking this issue.

C# implementation (new enum values, migrations, entities, merge logic, tests) is designed but not yet written — tracked as remaining work in the plan doc.

[DutchJaFO]

Status update — implementation complete, verification in progress.

All 12 implementation steps in the plan doc are done: five-value policy vocabulary (skip/newest-wins/merge-ours/merge-theirs/review), flat Quotinator__DefaultConflictPolicy config key, kebab-case JSON serialization, merge-ours/merge-theirs field-level resolution (FieldMergeResolver + QuoteFieldMerge), ImportBatches.ConflictPolicy column, and the new Data-owned System_ImportConflicts table logging every detected conflict.

Verification:

• Unit/integration tests: 767 passing, 0 warnings, 0 errors (dotnet test --configuration Release)
• T2 (Docker): docker build + container smoke test passed — schema v5, 788 quotes / 478 sources, matches local dev exactly
• T1 (Visual Studio): live-tested by the developer — surfaced a real, benign discovery (not a Import conflict resolution: per-import policy with configurable defaults #64 bug) filed separately as NikhilNamal17 bundled source: 9 same-quote/source rows collide on differing year metadata #147: 9 of the 45 duplicates detected in the bundled data are internal to NikhilNamal17_popular-movie-quotes.json itself, not cross-file against vilaboim

Full detail: 64-conflict-resolution-plan.md

Not yet done: merge to main, tagged release. Still open per the milestone's closing gate (all 3 conditions: included in a release, all required tiers green, explicit confirmation before close).

[DutchJaFO]

T1 (Visual Studio) verified — clean run confirms:

• Pre-migration backup taken automatically
• Data migration 2→3 and App migration 4→5 both applied cleanly (schema v5, data v3)
• Stats match expected counts: 788 quotes / 478 sources / 2 characters / 0 people
• UI loads ([UI] home page ready), /api/v1/quotes/random and other endpoints return 200
• POST /api/v1/admin/database/reseed reproduces the same 45-duplicate breakdown with no unhandled exceptions

Both required tiers (T1, T2) are now green — see the plan doc's verification table (rows 16–17) for full detail.

Still open per the milestone's closing gate: not yet merged to main, not yet in a tagged release.