Background
We cannot predict what our external sources will deliver, nor how a new file will interact with data
already seeded from other files. A declarative, file-authored conflict-resolution system — exercised
against every bundled file, not just the two #181 originally scoped — makes that unpredictability
tractable, and doubles as direct preparation for the Data Enrichment milestone's enrichment phase
(outside this milestone's scope).
#181's scope is widened by this parent issue from 2 files to all 4 currently-bundled files
(quotinator-curated.json, quotinator-series-universe.json, NikhilNamal17_popular-movie-quotes.json,
vilaboim_movie-quotes.json) — #181's own issue body and plan doc still name only the two external
files and need correcting to reflect this before implementation starts.
Startup seeding is disabled for the duration of this body of work, via an internal constant in
QuotinatorDatabaseInitializer gating the OnInitialisedAsync → SeedIfEmptyAsync call (schema
migrations still run normally). This lets each bundled file be imported deliberately, one at a time,
via POST /api/v1/import against a controlled database state instead of the full automatic startup
seed. The constant is reverted before this issue is considered done — tracked explicitly below, not
left to memory.
Testing methodology — two Docker scenarios per file:
- (a) Clean database, only the file under review imported (
review policy) → produces that file's
own conflict-resolution file, for conflicts internal to the file itself.
- (b) Clean database, every previously-processed file already imported with its conflicts resolved,
then the new file imported (review policy) → produces a conflict-resolution file for conflicts
caused by interaction with existing data.
Scenario (b) never applies to the first file processed. Files are processed internal-first
(quotinator-curated.json, quotinator-series-universe.json), then external
(NikhilNamal17_popular-movie-quotes.json, vilaboim_movie-quotes.json) — exact order within each pair
to be confirmed at kickoff.
Reviewing conflicts. Whenever a conflict needs a developer decision, it is exported via #163's
GET /import/actions/export endpoint — the resulting file is both kept as a working artifact (for
piping back through POST /import/actions/bulk-decide once decisions are made) and presented inline
as a markdown table in chat for review. Genuinely ambiguous conflicts — ones with no clear automatic
answer — are asked one at a time; the file itself remains the single source of truth for what was
actually decided.
Also exercises non-conflicting field corrections. Beyond conflict resolution proper, this body of
work deliberately tests the Modify path for fields that are edited but not ambiguous — a corrected
value with no competing incoming value to weigh — to prove out the general correction workflow
end-to-end, not just the conflict-decision slice of it.
Doubles as an end-to-end rehearsal for the UX milestone. The review → export → decide → apply →
verify loop this issue exercises by hand (via curl and chat) is the exact backend workflow the future
UX milestone's management UI will present visually. Exercising it thoroughly here surfaces workflow
gaps before that UI is ever built on top of it.
Sub-issues
| # |
Scope |
Depends on |
| #177 |
Bug: ImportBatches.Status never set to Applied via the staged decide→apply flow, breaking POST /import/actions/reverse — required first so the resolve→apply→reverse→retry cycle this body of work relies on for iterative testing actually works |
— |
| #181 |
Minimal per-source conflict-resolution rule file + curated field-override preload — ships the rule-file format and ImportActionPlanner lookup/auto-apply wiring every other file's rule file is authored against; scope widened by this parent issue to all 4 bundled files (see Background) |
#177 |
| #153 |
Declarative conflict-resolution file for recurring third-party source conflicts (Phase 2) — generalizes #181's hand-authored format into a generated one, consuming #163's already-shipped decided-action export shape; plan doc needs rewriting, since it predates #163 and still references the pre-#206 Quotinator.Engine project layout |
#181 |
Scope boundary
Definition of done
Background
We cannot predict what our external sources will deliver, nor how a new file will interact with data
already seeded from other files. A declarative, file-authored conflict-resolution system — exercised
against every bundled file, not just the two #181 originally scoped — makes that unpredictability
tractable, and doubles as direct preparation for the Data Enrichment milestone's enrichment phase
(outside this milestone's scope).
#181's scope is widened by this parent issue from 2 files to all 4 currently-bundled files
(
quotinator-curated.json,quotinator-series-universe.json,NikhilNamal17_popular-movie-quotes.json,vilaboim_movie-quotes.json) — #181's own issue body and plan doc still name only the two externalfiles and need correcting to reflect this before implementation starts.
Startup seeding is disabled for the duration of this body of work, via an internal constant in
QuotinatorDatabaseInitializergating theOnInitialisedAsync→SeedIfEmptyAsynccall (schemamigrations still run normally). This lets each bundled file be imported deliberately, one at a time,
via
POST /api/v1/importagainst a controlled database state instead of the full automatic startupseed. The constant is reverted before this issue is considered done — tracked explicitly below, not
left to memory.
Testing methodology — two Docker scenarios per file:
reviewpolicy) → produces that file'sown conflict-resolution file, for conflicts internal to the file itself.
then the new file imported (
reviewpolicy) → produces a conflict-resolution file for conflictscaused by interaction with existing data.
Scenario (b) never applies to the first file processed. Files are processed internal-first
(
quotinator-curated.json,quotinator-series-universe.json), then external(
NikhilNamal17_popular-movie-quotes.json,vilaboim_movie-quotes.json) — exact order within each pairto be confirmed at kickoff.
Reviewing conflicts. Whenever a conflict needs a developer decision, it is exported via #163's
GET /import/actions/exportendpoint — the resulting file is both kept as a working artifact (forpiping back through
POST /import/actions/bulk-decideonce decisions are made) and presented inlineas a markdown table in chat for review. Genuinely ambiguous conflicts — ones with no clear automatic
answer — are asked one at a time; the file itself remains the single source of truth for what was
actually decided.
Also exercises non-conflicting field corrections. Beyond conflict resolution proper, this body of
work deliberately tests the Modify path for fields that are edited but not ambiguous — a corrected
value with no competing incoming value to weigh — to prove out the general correction workflow
end-to-end, not just the conflict-decision slice of it.
Doubles as an end-to-end rehearsal for the UX milestone. The review → export → decide → apply →
verify loop this issue exercises by hand (via curl and chat) is the exact backend workflow the future
UX milestone's management UI will present visually. Exercising it thoroughly here surfaces workflow
gaps before that UI is ever built on top of it.
Sub-issues
ImportBatches.Statusnever set toAppliedvia the staged decide→apply flow, breakingPOST /import/actions/reverse— required first so the resolve→apply→reverse→retry cycle this body of work relies on for iterative testing actually worksImportActionPlannerlookup/auto-apply wiring every other file's rule file is authored against; scope widened by this parent issue to all 4 bundled files (see Background)Quotinator.Engineproject layoutScope boundary
this body of work resolves its conflicts as a side effect of a general mechanism, not a
reclassification of NikhilNamal17 bundled source: 9 same-quote/source rows collide on differing year metadata #147's scope.
UX-milestone rehearsal noted above is a side benefit of this methodology, not scope of its own.
is intentional.
Definition of done
reviewpolicy with zero pending actions