[TheAngryPit]

Closes #251

Problem

Posts saved as X API v2 JSON (for example xurl output) can't be added to birdclaw without a live read. This adds an offline import for that shape.

Change

Birdclaw already turns xurl /2/ responses into XurlTweetsResponse and stores them with ingestTweetPayload. This PR adds a file and stdin entry point to that same path, instead of a parser for a new format.

• birdclaw import x-json [file], reading stdin when the file is - or omitted.
• Accepts one JSON object or JSONL. data can be one tweet or an array, with optional includes.users, includes.tweets, includes.media.
• Checks required fields per line. Bad lines go into errors with their line number, valid lines still import, exit code is 1 if errors is not empty.
• Stores under the existing public account with source = 'x-json'. Optional per-tweet url goes to tweet_sources, as in feat: add opt-in FxTwitter tweet import #103.
• Primary tweets get a home edge on public so search tweets finds them. FxTwitter imports don't add an edge; see the open question in Import saved X API v2 tweet JSON without a live read #251.
• Runs autoSyncAfterWrite() after the import, like the other import commands.
• Short usage section in docs/public-tweets.md.

What stays the same

• Only additions: 4 files, +592 / -0. No schema change, no new table, no new dependency.
• Other import commands, sync and the FxTwitter path are untouched.
• The public account is created with insert or ignore and is_default 0 if it doesn't exist yet.
• No network and no credentials are used.

Behavior evidence

Built CLI at aa186ec, fresh scratch BIRDCLAW_HOME, synthetic fixtures, Linux.

Claim	Observed	Limitation
File import works	import x-json single.json gives {"imported":1,"skipped":0,"errors":[]}, exit 0	Synthetic data
Stdin works	cat clean.jsonl | import x-json - gives imported 2, exit 0
Imported posts are searchable	search tweets heron 3 hits, kestrel 1 hit	Quoted-only tweet osprey 0 hits, see boundary
Re-run is safe	same file gives imported 0, skipped 1; same stdin gives imported 0, skipped 2, exit 0
Bad line doesn't block the rest	line 3 malformed: imported 1, skipped 1, error Line 3: invalid JSON, exit 1	Partial import still exits 1
Required fields are enforced	Line 1: tweet.author_id must be a non-empty string, exit 1
Provenance and account	5 tweets, one tweet_sources row with x-json, only account public with is_default 0

Tests in src/lib/x-json-import.test.ts (in-process SQLite):

• single-object data with provenance
• array data with includes users, media, tweets and note_tweet
• malformed JSONL lines are not ingested
• invalid tweet fields are rejected
• duplicate re-import is idempotent
• stdin through the CLI, then found with search tweets
• non-zero exit when errors are present, JSON still printed

Gates

All ci.yml jobs reproduced locally on Linux at aa186ec. These are not GitHub Actions runs.

Job	Result
check (format, lint, typecheck)	pass
Bun coverage	182 files, 1969/1969 tests; x-json-import.ts 81.16% statements
build + runtime perf smoke	pass
Node 26.10.0 coverage:node + build:node	1969/1969 tests, build pass
package smoke	ok, 97 files
e2e	20 passed
e2e read-only permalinks	4 passed

Bun 1.4.0-canary.1+f972c287f, Node 20.19.2 for the primary jobs. Playwright browsers were installed without --with-deps on this machine.

Known boundary

• Quoted-only tweets from includes.tweets are stored but get no account edge, so search tweets doesn't return them.
• Fixtures are synthetic and follow the XurlTweetsResponse type. This PR does not include a run on captured xurl output.
• Input is one object or one object per line. A top-level array of documents, or several pretty-printed objects in one file, is not accepted.
• The per-tweet url field is not part of X API v2. It is an optional extra for provenance.
• tweet_sources is already a backup table, so x-json rows should be in backups. I read that in the code; I didn't test it.
• No CHANGELOG entry. Happy to add one in whatever form you use.

[clawsweeper]

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

[clawsweeper]

Codex review: blocked before merge.

What this changes

Adds a file and stdin command for importing saved X API v2 tweet responses into local storage and search.

Example: Import a saved response containing tweet 880001 with text "synth needle alpha".

• Before: No CLI command accepts the saved response directly.
• After: birdclaw import x-json single.json imports the tweet and makes it discoverable through search tweets.

Review scores

Measure	Result	What it means
Overall readiness	🦪 silver shellfish (2/6)	The supplied CLI results exercise the new entry point, but concrete input and reporting defects prevent landing.
Proof confidence	🐚 platinum hermit (4/6)	Sufficient (live_output): The PR records the built CLI at the reviewed head on Linux importing files and stdin into a fresh workspace, finding primary tweets through search, reporting repeat imports, and returning nonzero for malformed input.
Patch quality	🦪 silver shellfish (2/6)	4 actionable review findings remain.

Product

Kind: Feature · Worth it: Needs a maintainer decision · Fix scope: Complete
User problem: Users with saved X API responses must refetch posts or convert them into an internal backup format before searching them locally.
Reason: The offline entry point has clear value, but choosing home membership as the mechanism for search visibility changes an owner-defined product contract.

Merge readiness

⛔ Blocked before merge - 6 items remain

Keep this PR open: it adds a useful missing import path, but needs correctness repairs and an owner decision about home-timeline membership.

Priority: P3
Reviewed head: aa186ec1ab022ae9098fc77552e606961946797d
Owner decision: Required. See Decision needed.

Decision needed

• Question: Should saved primary tweets become public home-timeline members, or should local search expose them without asserting home membership?
• Recommendation: Preserve membership semantics: Choose a search visibility contract that does not infer home-timeline membership from a saved lookup response.
• Why: The linked issue explicitly leaves this question open, existing public imports omit those edges, and no maintainer approval settles the new behavior.

Before merge

• Reject invalid author IDs before canonical ingestion (P2) - A JSONL document with id "880001", author_id "abc", text "hello", and a valid created_at passes this validation. With no matching includes user, canonical ingestion calls buildExternalProfileId, which throws because the author ID is nonnumeric. The exception escapes the document loop, so later valid lines never import and the CLI prints a generic error instead of the promised counts and line error. Validate numeric author and user IDs before ingestion; also validate optional structures consumed by ingestion, such as referenced_tweets, rather than relying on the type assertion.
• Count distinct primary tweet IDs (P2) - A valid document whose data array contains tweet 880001 twice reports imported: 2 on an empty database. Canonical ingestion deduplicates that array through toCanonicalTweets and stores only one tweet, so the result overstates the imported count. Deduplicate IDs before calculating imported and skipped counts.
• Preserve original JSONL line numbers (P3) - Trimming the entire input before splitting removes leading blank lines. For an input containing two blank lines, a valid document on line 3, and malformed JSON on line 4, the error reports line 2. Use trimming only to detect empty input and split the original string so diagnostics identify the actual source line.
• Disclose backup networking in the offline import documentation (P2) - The documented command calls autoSyncAfterWrite after importing. When backup auto-sync is configured, that path reconciles Git history and pushes a backup using the user's configured credentials, so the unconditional promise of no network access is false. Preserve the existing backup preference contract by documenting this exception and showing the existing BIRDCLAW_BACKUP_AUTO_SYNC=0 invocation for a strictly offline import.
• Resolve maintainer decision - Resolve the maintainer decision shown above before merge.
• Add data-model compatibility proof - The review found that existing stored data may not work after upgrade. Show that existing data still loads and works with this change.

Findings

• [P2] Reject invalid author IDs before canonical ingestion — src/lib/x-json-import.ts:66-67
• [P2] Count distinct primary tweet IDs — src/lib/x-json-import.ts:282-286
• [P3] Preserve original JSONL line numbers — src/lib/x-json-import.ts:223

Tests

• Missing end-to-end proof: A populated v0.16.3 workspace import and backup round trip preserving existing settings and richer overlapping records are not demonstrated.

Agent review details

How this fits together

The CLI reads saved JSON, validates documents, and passes them to canonical tweet ingestion, which writes SQLite records, search indexes, provenance, and account edges.

flowchart LR
  A[File or stdin] --> B[Import command]
  B --> C[Document validation]
  C --> D[Canonical ingestion]
  D --> E[SQLite and search indexes]
  E --> F[Local tweet search]
  B --> G[Configured backup sync]

Loading

Technical review

Best possible solution:

Provide offline import through canonical ingestion with accurate results, isolated document failures, and explicitly approved search and membership semantics.

Do we have a high-confidence way to reproduce the issue?

Source inspection establishes the missing main-branch entry point and the reported patch defects; no target code or tests were run during this read-only review.

Is this the best way to solve the issue?

Reusing canonical ingestion is appropriate, but the importer must validate its untrusted input boundary and obtain agreement on how imported posts become searchable.

Full review comments:

• [P2] Reject invalid author IDs before canonical ingestion — src/lib/x-json-import.ts:66-67
  A JSONL document with id "880001", author_id "abc", text "hello", and a valid created_at passes this validation. With no matching includes user, canonical ingestion calls buildExternalProfileId, which throws because the author ID is nonnumeric. The exception escapes the document loop, so later valid lines never import and the CLI prints a generic error instead of the promised counts and line error. Validate numeric author and user IDs before ingestion; also validate optional structures consumed by ingestion, such as referenced_tweets, rather than relying on the type assertion.
  Confidence: 0.99
• [P2] Count distinct primary tweet IDs — src/lib/x-json-import.ts:282-286
  A valid document whose data array contains tweet 880001 twice reports imported: 2 on an empty database. Canonical ingestion deduplicates that array through toCanonicalTweets and stores only one tweet, so the result overstates the imported count. Deduplicate IDs before calculating imported and skipped counts.
  Confidence: 0.99
• [P3] Preserve original JSONL line numbers — src/lib/x-json-import.ts:223
  Trimming the entire input before splitting removes leading blank lines. For an input containing two blank lines, a valid document on line 3, and malformed JSON on line 4, the error reports line 2. Use trimming only to detect empty input and split the original string so diagnostics identify the actual source line.
  Confidence: 0.99
• [P2] Disclose backup networking in the offline import documentation — docs/public-tweets.md:48
  The documented command calls autoSyncAfterWrite after importing. When backup auto-sync is configured, that path reconciles Git history and pushes a backup using the user's configured credentials, so the unconditional promise of no network access is false. Preserve the existing backup preference contract by documenting this exception and showing the existing BIRDCLAW_BACKUP_AUTO_SYNC=0 invocation for a strictly offline import.
  Confidence: 0.98

Overall correctness: patch is incorrect
Overall confidence: 0.97

AGENTS.md: not found in the target repository.

Codex review notes: model internal, reasoning medium; reviewed against a08c264c88b2.

Provenance checked

• Canonical tweet ingestion keeps the original intent (refactor: consolidate tweet ingestion and transports #129: Consolidate ingestion through the canonical repository and remove redundant transport wrappers.)
• Existing public import boundary keeps the original intent (feat: add opt-in FxTwitter tweet import #103: Provide explicit public reads with canonical persistence and provenance.)

Testing

Proof path: shipped entry point.

Security

None.

Evidence

What I checked:

• Verified introduced change: The pinned diff adds the importer, CLI registration, tests, and documentation; current main has archive and live FxTwitter imports but no saved-response entry point. (src/cli/register-core.ts:366, aa186ec1ab02)
• Input validation does not contain ingestion failures: The importer accepts any nonempty author_id, but buildExternalProfileId rejects nonnumeric IDs; the document loop does not catch that failure. (src/lib/x-json-import.ts:66, aa186ec1ab02)
• Search visibility changes membership semantics: The new writer supplies edgeKind home, whereas existing FxTwitter persistence omits an edge; local search selects account edges by resource kind. (src/lib/x-json-import.ts:293, aa186ec1ab02)
• Offline claim has a configured-network exception: autoSyncAfterWrite invokes configured backup synchronization, whose production path reconciles Git history and exports with push enabled. (src/cli/register-core.ts:374, aa186ec1ab02)
• Discussion and supplied behavior evidence: The complete PR body records built-CLI file, stdin, search, repeat-import, and malformed-line results on a fresh Linux workspace; it explicitly identifies synthetic fixtures and an untested backup round trip, and contains no owner approval of membership semantics. (aa186ec1ab02)
• Related request and prior scope: Import saved X API v2 tweet JSON without a live read #251 requests this capability and asks which membership semantics to use; feat: add opt-in FxTwitter tweet import #103 established explicit public ingestion, while Import Scrollmark and twitter-web-exporter captures into the canonical local store #98 concerned different capture formats and was closed pending real data.

Root-cause cluster

Relationship: same_root_cause
Canonical: #251
Summary: The linked issue owns the saved-response import request, and this PR implements that request.

Members:

• canonical: Import saved X API v2 tweet JSON without a live read #251 - The same author describes the missing capability, acceptance criteria, and unresolved membership choice.

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Labels

Label changes:

• add P3: This adds an optional offline workflow rather than repairing a broken supported import.
• add proof: sufficient: Contributor real behavior proof is sufficient.
• add rating: 🦪 silver shellfish: Overall readiness is 🦪 silver shellfish; proof is 🐚 platinum hermit and patch quality is 🦪 silver shellfish.
• add status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action.

Label justifications:

• P3: This adds an optional offline workflow rather than repairing a broken supported import.
• rating: 🦪 silver shellfish: Overall readiness is 🦪 silver shellfish; proof is 🐚 platinum hermit and patch quality is 🦪 silver shellfish.
• status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action.
• proof: sufficient: Contributor real behavior proof is sufficient.

Rating scale

6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior.

Workflow

ClawSweeper edits this one comment on every review. Comment @clawsweeper re-review for a fresh review only. Repair and merge execution lanes are retired.

Reviewed October 27, 2026, 4:13 PM ET / 20:13 UTC.