Curated JSON format: conversations, stageDirections, soundCues sections

[DutchJaFO]

Purpose

Extend data/sources/quotinator-curated.json (and the general import format) to support conversation definitions alongside quotes. The seeder must handle the extended format when importing from any source file that includes these sections.

Extended format

{
  "quotes": [ ...existing canonical schema... ],
  "stageDirections": [
    {
      "id": "uuid",
      "text": "[EXT. AIRPLANE - COCKPIT]",
      "imageUrl": null,
      "translations": {
        "nl": { "text": "[EXT. VLIEGTUIG - COCKPIT]" }
      }
    }
  ],
  "soundCues": [
    {
      "id": "uuid",
      "text": "[awkward silence]",
      "soundFileUrl": null,
      "imageUrl": null,
      "translations": {
        "nl": { "text": "[pijnlijke stilte]" }
      }
    }
  ],
  "conversations": [
    {
      "id": "uuid",
      "description": null,
      "lines": [
        { "order": 1, "type": "quote",           "quoteId": "..." },
        { "order": 2, "type": "sound_cue",        "soundCueId": "..." },
        { "order": 3, "type": "quote",            "quoteId": "..." }
      ]
    }
  ]
}

Backwards compatibility

Source files that contain only a top-level array ([...]) remain valid — they are treated as { "quotes": [...] } with no conversations, stage directions, or sound cues. The seeder detects the shape and handles both.

Seeder changes

• Parse stageDirections, soundCues, and conversations sections when present.
• Insert into the new tables defined in Schema: Conversations, ConversationLines, StageDirections, SoundCues #67.
• All three section types share the same ImportBatchId as the quotes from the same file.
• conversations are inserted after all quotes, stageDirections, and soundCues from the file are committed (FK constraint order).

First use

The Airplane! conversation is the first entry:

• Two quotes already in quotinator-curated.json (Ted Striker / Dr. Rumack)
• No stage directions or sound cues needed for the core exchange
• One conversation linking both quotes in order

quotinator-curated.json must be migrated from a flat array to the extended object format as part of this issue.

Dependencies

• Schema: Conversations, ConversationLines, StageDirections, SoundCues #67 (schema) — tables must exist before seeder can write to them
• Seed script: write one file per source into data/sources/; drop merge and dedup #61 (seed script) — curated file format change must be consistent with the per-source file approach

[DutchJaFO]

Progress on feature/data-import-sources (not yet merged — stays open until the full milestone lands).

Done this session:

• New Quotinator.Core.Import DTOs (SourceStageDirection, SourceSoundCue, SourceConversation, SourceConversationLine, plus their translation DTOs) and ParsedSourceFile, matching schemas/source-extended.schema.json exactly.
• SourceQuoteFileReader.TryParseExtended — parses all four sections (quotes/stageDirections/soundCues/conversations) from the extended object format, or yields empty extended sections for a bare-array file. Existing TryParse untouched.
• ConversationLineType moved from Quotinator.Engine.Entities to Quotinator.Core.Models (needed by both the new import DTO and the Schema: Conversations, ConversationLines, StageDirections, SoundCues #67 entity — mirrors how QuoteType is shared) — a correction to Schema: Conversations, ConversationLines, StageDirections, SoundCues #67's original placement, discovered while wiring the DTO. New ConversationLineTypeJsonConverter (snake_case wire values, e.g. stage_direction).
• quotinator-curated.json migrated to the extended format with four conversations, not just the one originally scoped — added Monty Python's Black Knight scene (sound cue), The Princess Bride's Inigo Montoya confrontation (stage direction), and Empire Strikes Back's "I am your father" scene (stage direction), so both StageDirection and SoundCue line types are actually exercised by example data. Every quote verified against the film dialogue via web search before being added.
• Found and fixed a real regression: the legacy in-memory QuoteService (unreachable in the running app — nothing in Program.cs registers it since the SQLite migration, but still covered by its own tests) duplicated the file-parsing logic instead of reusing SourceQuoteFileReader, and broke on the new file shape. Now reuses the shared parser.

Not done yet: the actual seeding/import write path (ConversationSeedWriter, staged through System_ImportActions like Quote/Source/Character/Person). Turned out substantially larger than originally scoped once the existing apply/reverse pipeline (SqliteImportActionService.cs, ~650 lines) was read in full. A concrete, implementation-ready design is now written into the plan doc (docs/milestones/data-import-sources/68-curated-json-conversations-plan.md, Design section 3) — key simplification found: only Conversation/StageDirection/SoundCue need their own staged actions (Add-only, id-keyed like Quote); ConversationLines/translations are child rows written alongside their parent, the same relationship QuoteGenres/QuoteTranslations already have to Quote. Deferred to a follow-up session rather than rushed.

Status: In progress (step 3).

[DutchJaFO]

Fully implemented on feature/data-import-sources (not yet merged — stays open until the full milestone lands). This completes the staging-engine integration flagged as deferred in the previous comment.

What shipped:

• Conversation/StageDirection/SoundCue are now staged through System_ImportActions exactly like Quote/Source/Character/Person — planned, applied, and reversible (Admin: targeted soft-reset and restore by import batch #59) through the same pipeline, not a bespoke writer.
• Key design simplification: ConversationLines/StageDirectionTranslations/SoundCueTranslations are detail rows of their parent (same relationship QuoteGenres/QuoteTranslations have to Quote) — never staged as their own actions. This cut the new entity-type surface from 6 down to 3.
• Wired into both write paths: startup seeding (QuotinatorDatabaseInitializer) and the live POST /api/v1/import endpoint, per the original design goal — a source file with conversations can now be imported directly via the API, not just seeded at startup.

Two real correctness bugs found and fixed while implementing (not caught at design time):

1. System_ImportActions' batch-read query had no ORDER BY, so apply order wasn't actually guaranteed to match plan order — meaning a Conversation action could apply before the Quote/StageDirection/SoundCue rows its lines reference. Fixed generically in Quotinator.Data (ORDER BY rowid ASC), which also makes the existing action-listing/audit ordering deterministic as a side benefit.
2. The "is this StageDirection/SoundCue still referenced" check (used to protect against wrongly soft-deleting one during batch reversal) filtered ConversationLines.IsDeleted, but that column is never independently updated — it always reads 0 even after the parent conversation is reversed. This would have permanently blocked ever reversing a stage direction or sound cue. Caught by a test going red before the fix, not assumed correct.

Verified:

• Full test suite green (Engine.Tests 131/131), including new tests for seeding, live re-import idempotency, and reversal (including the "still referenced by another conversation" edge case).
• T1: fresh-database seed confirmed correct row counts (4 conversations, 13 lines, 2 stage directions, 1 sound cue) via a direct SQLite query, no errors.
• T2: Docker build + container smoke test — re-uploaded the curated file via POST /import against the live container and confirmed no duplicate conversations were created on the second pass.

Plan doc and overview.md updated to Waiting for release.