Skip to content

Minimal per-source conflict-resolution rule file + curated field-override preload #181

Description

@DutchJaFO

Background

While preparing for the Data Enrichment milestone's known-conflict work (#147, kept as-is — not
touched by this issue), verified (2026-07-15) that both bundled files should move from the silent
skip default to review, so a genuine conflict is staged for a decision instead of one side being
silently discarded. Counted directly against the bundled files: vilaboim_movie-quotes.json has
0 internal duplicate-id collisions; NikhilNamal17_popular-movie-quotes.json has exactly 9,
matching #147's own documented table precisely.

Rather than hand-deciding those 9 conflicts once via #149's existing decide/undo/apply endpoints
(which wouldn't persist — a future re-seed or Quotinator__AutoUpdateSources refresh of the same
source could reintroduce the identical conflict, requiring the same decision again), the goal is to
resolve conflicts as close to the source as possible: a small, per-source declarative rule file
that staging consults before ever creating a Pending action, auto-resolving a matching, already-
known conflict the same way every time.

This is deliberately a minimal, hand-authored slice of #153's eventual design ("Declarative
conflict-resolution file for recurring third-party source conflicts"). #153's own step 6 (rule
lookup and auto-apply during staging) is exactly what this issue builds — hand-authored rather than
generated. #153's remaining scope (generating rules from a batch's decided actions, staleness
detection, a rule-file GET/generate-merge endpoint) stays with #153, still gated on #163 landing
first for its generation step. This issue's rule-file format is what #153 builds on top of later,
not a format #153 will replace
#153's plan doc is updated to reflect this in the same commit.

Separately, a new curated field-override preload file
(data/sources/quotinator-source-overrides.json) sets the authoritative value for specific fields
on rows that originate in an external bundled source but need a correction (e.g. the correct date
for one of the 9 disputed NikhilNamal17 quotes) — distinct in purpose from quotinator-curated.json
(which adds wholly new, fully-curated quotes, not overrides on rows seeded from elsewhere). This
preload is what a source's own rule file resolves against: the rule says "for quote X's date
field, keep the existing (curated-preloaded) value over whatever this source's incoming re-import
says."

Scope widened 2026-07-25 by #217 (new parent tracking issue, alongside #153 and #177): from the
originally-named 2 external bundled files to all 4 currently-bundled files
quotinator-curated.json and quotinator-series-universe.json (internally-authored) in addition to
vilaboim_movie-quotes.json and NikhilNamal17_popular-movie-quotes.json. We cannot predict how a
new bundled file will interact with data already seeded from another one, and forcing every bundled
file through review policy with its own conflict-resolution file is what makes that interaction
tractable — limiting this to only the two files that happen to already have known external conflicts
would leave the internally-authored files unverified against the same standard. This issue is now
sequenced under #217 as #177 → #181 → #153#177's ImportBatches.Status fix lands first because
#217's own resolve→apply→reverse→retry testing methodology needs a working
POST /import/actions/reverse.

What needs to be done

  1. Design the minimal rule-file schema, keyed by quote id + field name (matching Declarative conflict-resolution file for recurring third-party source conflicts #153's own Step 1
    discussion in its plan doc — this issue's fully-known, small conflict set makes the alternative
    content-hash keying scheme unnecessary here; do not re-decide this independently of Declarative conflict-resolution file for recurring third-party source conflicts #153's own
    reasoning).
  2. Add a manifest reference for each bundled file's rule file — lives alongside the file it governs,
    matching Declarative conflict-resolution file for recurring third-party source conflicts #153's own item 2 design.
  3. Wire rule lookup into ImportActionPlanner.PlanAsync's conflict-staging logic — if a matching rule
    exists for a field, the action resolves automatically instead of staging Pending — mirrors Declarative conflict-resolution file for recurring third-party source conflicts #153's
    own step 6 exactly, scoped down (no generation, no staleness detection, no endpoint; these rule
    files are static and hand-edited, not machine-generated). Note: the codebase has grown since
    this item was first written — there is no longer one single "conflict-staging branch"; each
    entity-specific Plan*Async method has its own isPending check. The originally-known conflicts
    (NikhilNamal17's 9) are Quote-level, so the top-level PlanAsync Quote logic is the minimum needed
    for those — confirm whether the widened 4-file scope (item 4) surfaces any Series/Universe-level
    conflicts from quotinator-series-universe.json that would need PlanSeriesAsync/
    PlanUniverseAsync wired too, rather than assuming Quote-only.
  4. Set duplicateResolution: review for all 4 currently-bundled files in
    data/sources/manifest.jsonquotinator-curated.json, quotinator-series-universe.json,
    vilaboim_movie-quotes.json, and NikhilNamal17_popular-movie-quotes.json (widened from the
    original 2-file scope, see Background).
  5. Create data/sources/quotinator-source-overrides.json with the correct values for NikhilNamal17's
    9 known conflicts, sourced from NikhilNamal17 bundled source: 9 same-quote/source rows collide on differing year metadata #147's own findings table.
  6. One rule file per bundled file, all 4 — a rule file for NikhilNamal17 with 9 entries (one per
    known conflict), each resolving to the value the override-preload file establishes, and
    initially-empty rule files for vilaboim, quotinator-curated, and quotinator-series-universe
    (widened from the original 2-file scope), so the manifest reference and lookup path are exercised
    consistently for every bundled file and each stays independently updatable.
  7. Confirm live: reseeding with review set for all 4 files produces zero staged Pending
    actions — all 9 NikhilNamal17 conflicts auto-resolve via the rule file; the other three have none
    to begin with (widened from the original 2-file scope).
  8. These per-source rule files double as smoke-test fixtures: the T2 checklist (CLAUDE.md's
    living smoke-test list) exercises multiple scenarios directly against them — a matching rule
    auto-resolving without a Pending action, a field with no matching rule still staging Pending
    as before, and (by temporarily editing a rule file's value) confirming a changed rule changes the
    auto-resolved outcome on the next reseed. Add this to CLAUDE.md's T2 checklist in the same commit
    that ships the mechanism, per this project's "living checklist, only grows" convention.
  9. Update 153-declarative-conflict-resolution-plan.md to note Declarative conflict-resolution file for recurring third-party source conflicts #153's own step 6 and rule-file
    format build on this issue's shipped format rather than inventing a new one — a scope note only,
    not a rewrite of Declarative conflict-resolution file for recurring third-party source conflicts #153's remaining generation/staleness/endpoint work. Already done as part of
    Declarative conflict-resolution file for recurring third-party source conflicts #153's own plan doc rewrite (2026-07-25); re-confirm the shipped shape still matches once this
    issue is actually implemented.

Expected tests

Test class Test method Starts
New: Quotinator.Core.Tests PlanAsync_MatchingRuleExists_AutoResolvesWithoutPending
New: Quotinator.Core.Tests PlanAsync_NoMatchingRule_StagesPendingAsToday
New: Quotinator.Core.Tests SeedNikhilNamal17_AllNineKnownConflicts_AutoResolveViaRuleFile
New: Quotinator.Core.Tests SeedVilaboim_ReviewPolicy_NoStagedActions, plus equivalents for quotinator-curated and quotinator-series-universe (widened from vilaboim-only)

The first two tests are the same ones #153 lists in its own Expected tests table — implementing this
issue ships the mechanism they exercise, so #153 inherits them as already-passing regression guards
rather than writing them fresh when #153 itself lands.

Definition of done

  • All expected tests listed above start red before implementation
  • All requirements implemented
  • All expected tests pass (green)
  • No regression in related tests
  • Findings summarised in a closing comment

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions