Skip to content

Establish conflict-resolution coverage for every bundled source file #217

Description

@DutchJaFO

Background

We cannot predict what our external sources will deliver, nor how a new file will interact with data
already seeded from other files. A declarative, file-authored conflict-resolution system — exercised
against every bundled file, not just the two #181 originally scoped — makes that unpredictability
tractable, and doubles as direct preparation for the Data Enrichment milestone's enrichment phase
(outside this milestone's scope).

#181's scope is widened by this parent issue from 2 files to all 4 currently-bundled files
(quotinator-curated.json, quotinator-series-universe.json, NikhilNamal17_popular-movie-quotes.json,
vilaboim_movie-quotes.json) — #181's own issue body and plan doc still name only the two external
files and need correcting to reflect this before implementation starts.

Startup seeding is disabled for the duration of this body of work, via an internal constant in
QuotinatorDatabaseInitializer gating the OnInitialisedAsyncSeedIfEmptyAsync call (schema
migrations still run normally). This lets each bundled file be imported deliberately, one at a time,
via POST /api/v1/import against a controlled database state instead of the full automatic startup
seed. The constant is reverted before this issue is considered done — tracked explicitly below, not
left to memory.

Testing methodology — two Docker scenarios per file:

  • (a) Clean database, only the file under review imported (review policy) → produces that file's
    own conflict-resolution file, for conflicts internal to the file itself.
  • (b) Clean database, every previously-processed file already imported with its conflicts resolved,
    then the new file imported (review policy) → produces a conflict-resolution file for conflicts
    caused by interaction with existing data.

Scenario (b) never applies to the first file processed. Files are processed internal-first
(quotinator-curated.json, quotinator-series-universe.json), then external
(NikhilNamal17_popular-movie-quotes.json, vilaboim_movie-quotes.json) — exact order within each pair
to be confirmed at kickoff.

Reviewing conflicts. Whenever a conflict needs a developer decision, it is exported via #163's
GET /import/actions/export endpoint — the resulting file is both kept as a working artifact (for
piping back through POST /import/actions/bulk-decide once decisions are made) and presented inline
as a markdown table in chat for review. Genuinely ambiguous conflicts — ones with no clear automatic
answer — are asked one at a time; the file itself remains the single source of truth for what was
actually decided.

Also exercises non-conflicting field corrections. Beyond conflict resolution proper, this body of
work deliberately tests the Modify path for fields that are edited but not ambiguous — a corrected
value with no competing incoming value to weigh — to prove out the general correction workflow
end-to-end, not just the conflict-decision slice of it.

Doubles as an end-to-end rehearsal for the UX milestone. The review → export → decide → apply →
verify loop this issue exercises by hand (via curl and chat) is the exact backend workflow the future
UX milestone's management UI will present visually. Exercising it thoroughly here surfaces workflow
gaps before that UI is ever built on top of it.

Sub-issues

# Scope Depends on
#177 Bug: ImportBatches.Status never set to Applied via the staged decide→apply flow, breaking POST /import/actions/reverse — required first so the resolve→apply→reverse→retry cycle this body of work relies on for iterative testing actually works
#181 Minimal per-source conflict-resolution rule file + curated field-override preload — ships the rule-file format and ImportActionPlanner lookup/auto-apply wiring every other file's rule file is authored against; scope widened by this parent issue to all 4 bundled files (see Background) #177
#153 Declarative conflict-resolution file for recurring third-party source conflicts (Phase 2) — generalizes #181's hand-authored format into a generated one, consuming #163's already-shipped decided-action export shape; plan doc needs rewriting, since it predates #163 and still references the pre-#206 Quotinator.Engine project layout #181

Scope boundary

Definition of done

  • Every sub-issue listed above is closed
  • The seeding-disable constant is reverted to its normal (seeding-enabled) state and re-verified via a live smoke test
  • All 4 currently-bundled files seed cleanly under review policy with zero pending actions
  • Findings summarised in a closing comment

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions