Skip to content

Enrichment script: fill missing quote fields using TMDB, Open Library, and Wikidata #19

Description

@DutchJaFO

Purpose

Define the shared enrichment infrastructure that all provider implementations (#35, #36, #37, #46, and any future providers) build on. Each provider is independently shippable once this scaffold is in place — they do not depend on each other.

What this issue delivers

IEnrichmentProvider interface

public interface IEnrichmentProvider
{
    /// <summary>The source types this provider handles (e.g. movie, tv, book, person).</summary>
    IReadOnlySet<string> SupportedTypes { get; }

    /// <summary>Attempt to fill missing fields on the quote. Only populates null/empty fields — never overwrites curated data.</summary>
    Task<EnrichmentResult> EnrichAsync(Quote quote, CancellationToken ct = default);
}

EnrichmentResult carries: fields changed, confidence level, and any flags for manual review (fuzzy match, ambiguous result, etc.).

Runner skip rules

The enrichment runner must check two conditions before passing a record to any provider:

  1. isComplete: true — skip the entire record; the user has marked it done (Schema: record completeness flag and per-field verified-absent markers #55)
  2. noValueKnown — skip individual fields listed in this array; the user has confirmed no value exists for them (Schema: record completeness flag and per-field verified-absent markers #55)

Providers must not attempt to fill a field present in noValueKnown, even if their lookup returns a confident match. These are deliberate human decisions that enrichment must not override.

Cascading / universal scraper pattern

Inspired by Kodi's Universal Movie Scraper: the enrichment runner tries each registered provider in priority order and merges results. A provider that returns a confident match for a field wins that field; a provider that returns no match leaves the field for the next provider to try. No single source is authoritative.

Script scaffold (scripts/enrich.csx)

  • --dry-run flag: print proposed changes without writing
  • --no-fetch flag: use scripts/cache/ instead of live API calls
  • --provider flag: run only one provider (e.g. --provider tmdb) for targeted re-enrichment
  • --include-complete flag: override the isComplete skip rule and process all records (useful for re-enrichment after adding a new provider)
  • Per-provider cache directories under scripts/cache/<provider>/
  • Structured log output: one line per quote, listing fields changed, fields skipped due to noValueKnown, and any manual-review flags
  • Provider registration: each provider is added as a step in the runner; adding a new provider requires no changes to existing providers

Where the shared code lives

Quotinator.Enrichment class library, referenced by both scripts/enrich.csx and Quotinator.Api (for the import endpoint's optional enrich pass — see #45). The script is a thin runner over this library.

Notes

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions