Repository navigation
[macOS][0.153.4] Paginated history stalls at duplicate ordinal; controlled recovery comparison and stable backport request #43742
Description
Summary
A local Codex Desktop task on macOS kept displaying early conversation history while the canonical JSONL continued to contain later user/assistant messages. The derived history projection was stuck at a duplicate ordinal.
This report adds a controlled comparison using the same bundled 0.153.4 binary and the same immutable snapshot: reopening did not recover it, rebuilding the derived rows failed again, and a narrowly guarded checkpoint correction recovered 24 turns without rewriting the canonical transcript. It also identifies the difference between the stable tag and the recovery handling merged in #42369.
Related: #43124, #43182, #43640, #41566, #41079. Please consolidate if this belongs to the same defect; the additional evidence here is the three-way recovery comparison and stable-tag behavior.
Environment
- Desktop bundle: ChatGPT.app, CFBundleIdentifier com.openai.codex
- Bundle version: 26.901.51231, build 8109 (read from Info.plist, not the About dialog)
- Bundled CLI/app-server used for all experiments: codex-cli 0.153.4
- OS: macOS 27.0, Apple Silicon arm64
- Local task; history_mode=paginated
- Observed September 8, 2026
Exact failure boundary, sanitized
The only ordinal discontinuity in the captured complete-line transcript prefix was:
| Record | Root ordinal |
|---|---|
| event_msg/token_count | 272 |
| event_msg/thread_settings_applied | 272 |
| event_msg/task_started | 273 |
The projection expected ordinal 273 at byte offset 4,813,257, which was exactly the start of the second ordinal-272 record. That metadata record occupied 1,992 bytes. The next complete record began at offset 4,815,249 and had ordinal 273.
Runtime warning, identifiers replaced:
failed to project durable rollout for THREAD:
thread-store internal error:
thread history projection for ROLLOUT expected ordinal 273, got 272
The active rollout has a separate rollout identifier from the logical thread identifier; the stuck projection row is keyed by that rollout identifier. This distinction matters when diagnosing an edited/replaced rollout. The original action that caused ordinal reuse has not been established.
The desktop history reader returned one old turn with no next cursor while later messages remained in the canonical JSONL. Repeated application runs continued to encounter the same boundary. This is history accessibility failure, not evidence of deletion of the observed later messages.
Controlled local comparison
An immutable snapshot contained 26,925,306 bytes in the affected rollout. Separate test CODEX_HOME directories received the same target-thread state and history rows and byte-identical rollout copies. No credentials were copied, no model turn was started, and hooks were disabled in the test configuration.
Each test initialized the installed app-server, read thread/turns/list, called thread/resume with excludeTurns=true, and read thread/turns/list again.
| Test | Result |
|---|---|
| Existing checkpoint, reopen/resume | Still 1 old turn; same expected-273/got-272 warning |
| Remove only this rollout's derived projection/turn/item rows in the test copy, then resume | 0 projected turns; rebuild fails again at the same duplicate ordinal |
| Advance only this rollout's next_rollout_byte_offset past the verified duplicate metadata record, retaining expected ordinal 273 | 24 turns and 1,560 items recovered; no ordinal warning |
The recovered snapshot included 26 userMessage and 123 agentMessage items. SQLite integrity_check returned ok. The copied canonical transcript's SHA-256 remained unchanged. After restarting the isolated app-server again, all 24 turns remained available and thread/items/list returned populated pages.
The migration CLI was also checked without --apply:
codex migrate-rollouts --thread <redacted> --json
status: already_paginated
bytes_processed: 0
It does not provide a repair path for this already-paginated thread.
Recovery on the affected installation
After backing up both state/history databases and the transcript prefix, the same narrowly scoped cursor correction was applied to one projection row, using BEGIN IMMEDIATE and old-value predicates. The expected ordinal was not changed, no history row was deleted, and no canonical record was rewritten. The existing trigger on this table is AFTER DELETE only; the operation was UPDATE.
The native live writer subsequently caught up. Read-only database checks found 24 projected turns, the checkpoint resumed advancing with new output, and the app-provided newest-first history reader returned recent turns with hasMore=true instead of the stale exhausted page. The pre-repair transcript prefix remained hash-identical. Desktop visual confirmation is tracked separately; a history API response alone is not claimed as screenshot evidence.
This is diagnostic recovery evidence for one verified metadata boundary, not a recommendation to skip arbitrary duplicate records or user/assistant content.
Stable source versus upstream fix
- The public rust-v0.153.4 materializer still returns an error for an ordinal lower than the expected ordinal. The observed binary behavior agrees.
- PR #42369, merged September 2, changes the mainline materializer to move past duplicate/regressed ordinals and other invalid records.
- As of this investigation, the GitHub latest stable release is rust-v0.153.4. A mainline fix being merged does not establish that affected stable desktop installations contain it.
Please clarify the first desktop/stable runtime that includes this recovery handling and whether already-wedged checkpoints recover automatically after upgrading.
Expected fix
- Ship/backport the applicable safe projector recovery to the stable desktop runtime.
- Prevent ordinal reuse at writer/resume boundaries.
- Offer a supported per-thread history integrity/recovery command that handles already_paginated sessions.
- Show an actionable incomplete-history state when durable records remain beyond a failed checkpoint, rather than silently presenting old history as complete.
- Cover this token_count(N) -> thread_settings_applied(N) -> task_started(N+1) boundary, resumed-history pagination, populated message pages, and restart persistence in regression tests.
No raw transcripts, databases, prompts, credentials, private identifiers, filesystem paths, internal URLs, or full logs are attached. The writer-side initiating trigger and future recurrence on 0.153.4 remain unproven.
Activity
github-actions commentedon Sep 8, 2026
Potential duplicates detected. Please review them and close your issue if it is a duplicate.
Powered by Codex Action
Isadorah01 commentedon Sep 8, 2026
Additional controlled recovery evidence from four affected local tasks on ChatGPT.app 26.901.51231 (build 8109), bundled codex-cli/app-server 0.153.4, macOS.
This supports the duplicate/regressed ordinal failure described here. It is a history accessibility failure: the canonical records inspected below were still readable. We have not reproduced the initiating writer bug from a clean installation or established why the writer reused an ordinal.
Four independently verified boundaries
Aliases below are not real task identifiers. The first unprojected record was
event_msg/thread_settings_appliedin all four cases.token_counttoken_counttoken_counttoken_countFor B and D, the next record is
task_startedat expected-minus-one. Therefore blindly advancing the byte cursor until an ordinal matches would skip a meaningful turn-start event. The same recovery recipe is not safe for every ordinal mismatch.Disposable-copy comparison
next_rollout_byte_offsetunchanged and aligned onlynext_rollout_ordinalwith the actual record at that cursor. The native 0.153.4 projector then consumed all remaining records, includingthread_settings_appliedandtask_started.token_count, and absence of materialized items occupying those overlapping ordinals. This is narrowly scoped diagnostic evidence, not a general recommendation to bypass ordinal validation.thread/items/listpages: 12,886 unique entries matched the complete recovered index. Older stored agent items omit the schema-defaultquestions: null; only that verified nullable default was normalized for comparison.Archive and transaction handling
Two tasks were archived. The diagnostic copies placed rollouts under test
sessions/, so native resume cleared their copied archive metadata. That copy result alone does not prove preservation of live archive state.An archive-preserving publication transaction was separately tested: keep live state/archive metadata and canonical files unchanged, insert only missing native-generated derived turn/item rows for the archived tasks, and advance their projection checkpoints atomically. All pre-existing rows must be byte-identical to both the original snapshot and the recovered copy; no existing message is replaced and no record is deleted. Active tasks need only the guarded ordinal alignment followed by native catch-up.
The combined disposable transaction passed. An injected second-task checkpoint conflict caused a complete rollback: all four tasks' logical table digests were unchanged.
Live recovery outcome (after the copy tests)
With explicit authorization, the guarded repair was applied to these four tasks. Both active tasks then caught up through the native projector; the two archived tasks received only their missing native-generated index rows. The observed turn counts are now 24, 8, 19, and 73 (34 additional turns total).
All four checkpoints reached the sampled end of their rollout files. The original snapshot prefixes remain SHA-256 identical, and each task's original archive flag, archive timestamp, and rollout location are unchanged. Read-only SQLite integrity checks passed for both state and history databases. The desktop history API returned the previously inaccessible later turns. Rendered UI inspection was unavailable because the computer-use tool disallows controlling this application, so this is an API/index recovery result, not a claim of exhaustive frontend visual validation.
Before applying, a fresh independent read-only review passed for this specific guarded script and evidence. A concurrent-writer test confirmed the attached state/history transaction blocks competing database writes while validating and publishing. A forced mid-transaction conflict rolled back all changes. Scoped pre-repair snapshots are retained locally; no full user database was replaced. This is not a generally safe repair recipe, and there is no automatic post-commit inverse operation.
Request
Please clarify which stable desktop/runtime release contains the applicable recovery handling from #42369, whether it repairs already-stalled projections, and how the writer-side ordinal reuse is being addressed. A supported per-task integrity/recovery command should preserve archived tasks, meaningful lower-ordinal events, lineage boundaries, and concurrent writes, and should surface incomplete history instead of silently showing an exhausted stale transcript.
No raw transcripts, databases, message text, attachments, credentials, private paths, actual task identifiers, or full logs are attached. This report does not establish that 0.153.4 will remain healthy after future writer/resume transitions.
ljc1002-cmd commentedon Sep 10, 2026
Additional macOS occurrence on desktop 26.903.61454 (build 8378), bundled codex-cli 0.153.4, observed September 10–11, 2026. Version read from Info.plist, not the About dialog. macOS 26.6.2 (25G83); uname -mprs: Darwin 25.6.0 arm64 arm. Local task, history_mode=paginated.
Primary impact: lost time and broken work continuity. The task shows September 8 history while work through September 10 remains in its durable transcript. This forces the user to stop productive work and reconstruct research, decisions and execution already completed. Additional usage is secondary; the priority is restoring the original task safely and preventing recurrence.
Read-only evidence from this installation:
Retained logs contain 347 matching WARNs, September 10 22:45:55–23:45:49 KST, from codex_thread_store::local::live_writer:
failed to project durable rollout for [logical-thread-id]: thread-store internal error: thread history projection for [continuation-id] expected ordinal 104179, got 104178
The duplicate ordinal at the cursor and projection failure are verified. The original trigger for ordinal reuse is not established. No repair or synthetic reproduction was performed, and original session/database/project files were not edited. Extraction to a separate local transcript did not restore the original UI.
Please confirm which released desktop/runtime contains the applicable recovery from #42369 and how already-stalled tasks can recover without redoing completed work or rewriting canonical transcripts. A visible incomplete-history error and a supported per-task repair path would prevent significant lost time.
No raw transcripts, private task identifiers, credentials, absolute local paths, project content or full logs are attached.
idealson-nexources commentedon Sep 10, 2026
Confirmed recovery of an already-stalled Desktop task using the standalone codex-cli 0.154.0, without updating or modifying Desktop and without manually editing the live SQLite projection.
Environment: macOS 26.6.2 (25G83), Apple Silicon; Desktop 26.903.71938 (build 8576), bundled CLI/app-server 0.153.4; local task with
history_mode=paginated. Recovery completed September 10, 2026.Failure and version distinction
Later user/assistant messages remained in the canonical JSONL but were missing from both Desktop and terminal history. The derived history was stuck at 1,001 turns. A duplicate ordinal occurred across
session_metaandevent_msg/agent_message, and projection warnings reportedexpected ordinal N+1, got N. This is not necessarily a harmless metadata record that can safely be skipped. We have not established the initiating writer-side cause.Stable 0.154.0, released September 9, includes #42369. Updating the standalone CLI does not replace Desktop's bundled runtime. Importantly, in our isolated comparison, merely calling
thread/readandthread/turns/listwith 0.154.0 still returned the stale projection; an actual nativethread/resumetriggered catch-up.Procedure that worked in this case
Validate recovery on a disposable copy first. Before live recovery, fully quit Desktop and all CLI sessions, verify no Codex writers remain, and take fresh consistent SQLite backups plus a backup/hash of the affected canonical rollout. Keep the originals and backups.
Run the standalone 0.154.0
codex app-server --stdioagainst the normal Codex data directory. After the standardinitializehandshake (withcapabilities.experimentalApi: true) andinitializednotification, send only:{"id":2,"method":"thread/resume","params":{"threadId":"<AFFECTED_THREAD_ID>","excludeTurns":true}}Then inspect
thread/turns/list. Do not sendturn/start, replay old prompts, or override the task's model, working directory or permissions. The native runtime performs its own history catch-up; no manual SQLUPDATE/DELETE, cursor manipulation, transcript rewriting, or application-bundle patching was used for the live repair.Stop the helper app-server, verify the recovered projection and original rollout-prefix hash, then reopen the unchanged Desktop application.
Verified outcome
This is one verified active-task recovery, not a guarantee for all history defects or archived tasks, nor proof that the older Desktop runtime cannot stall again. A supported per-task repair action, an explicit incomplete-history warning, and tests for already-stalled projections would help substantially. Please also clarify the Desktop release carrying #42369 and the separate writer-side fix for ordinal reuse.
No raw transcripts, databases, message contents, private task identifiers, credentials, absolute local paths, or full logs are attached.