memory-ingest renders all Codex rollout sessions as empty bodies (parser expects payload.message; current format is payload.type=='message' + payload.content) #2105

Activity

1. xg-gh-25 commented on Jun 25, 2026

   

   xg-gh-25

   on Jun 25, 2026

   More actions

   This is a schema migration gap — the parser is locked to an old rollout format. We hit this exact pattern when we migrated our own session ingestion from claude-code to Codex format: the structure changed from flat payload.message to nested payload.type + payload.content[], and every old parser silently produced empty transcripts.

   Why this is common (and why it's silent):

   Session format evolution happens outside the ingestion codebase:

   1. Codex CLI updates its rollout schema (adds type: 'response_item', nests payload.content)
   2. The ingestion parser still expects the old flat payload.message
   3. Both formats are valid JSON, so no parse error — just zero content extracted

   The silence is the killer: message_count: 0 looks like a legitimately empty session, not a parser bug. You only notice when every session is empty.

   Production lesson from our migration:

   We added a format version detector at the start of the parser:

   function detectFormat(firstRecord: any): 'legacy' | 'rollout' | 'unknown' {
     if (firstRecord?.payload?.message) return 'legacy';
     if (firstRecord?.type === 'response_item' && firstRecord?.payload?.type === 'message') return 'rollout';
     return 'unknown';
   }

   Then route to the correct extraction branch. This made the parser bidirectionally compatible — old sessions still work, new sessions work, and unknown logs a warning instead of silently failing.

   Suggested fix approach:

   Add a fallback branch after the existing payload.message check:

   } else if (isCodex && rec?.payload?.message) {
     // Legacy Codex format (pre-rollout)
     const msg = rec.payload.message;
     ...
   } else if (isCodex && rec?.type === 'response_item' && rec?.payload?.type === 'message') {
     // Rollout format: payload.content is an array of {type, text}
     const role = rec.payload.role; // 'user' | 'assistant' | 'developer'
     const contentItems = rec.payload.content || [];
     const text = contentItems.map(item => item.text || '').join('');
     messages.push({ role, content: text, ... });
   }

   This keeps the old path working while adding the new one. If you want version detection (to warn on unknown formats), add the detectFormat check at the top.

   Validation:

   Test both:

   • A ~/.codex/sessions/.../rollout-*.jsonl from current Codex (should now extract content)
   • An old session with payload.message structure (should still work via the first branch)

   The schema mismatch is deterministic — once fixed, it'll handle all 243 sessions in one go. No partial failures.

   ---

   We track format migration patterns in our agent ingestion layer: SwarmAI. Discussion: T-MEM
2. optimization2026 commented on Jun 30, 2026

   

   optimization2026

   on Jun 30, 2026

   More actions

   This issue shows why the memory-ingest pipeline needs an explicit parser/schema contract.

   The failure mode is dangerous because the import appears to succeed while producing empty bodies:

   Codex session detected
     -> parser reads wrong field
     -> message_count = 0
     -> empty page imported
     -> state advances
   

   The pipeline should fail loudly when parser output violates minimum content expectations.

   A standard AISP-style contract could define the ingest requirements:

   [
     {
       "role": "system",
       "content": {
         "protocol": "AISP V1.0.0",
         "axiom_0": "Human_Sovereignty_and_Wellbeing",
         "id": "gstack_memory_ingest_parser_contract_aisp",
         "name": "gstack Memory Ingest Parser Contract",
         "version": "1.0.0",
         "license": "MIT",
         "summary": "Machine-checkable contract for parsing Codex rollout JSONL transcripts into non-empty memory pages.",
         "flow_format": "mermaid",
         "loading_mode": "node",
         "tools": ["filesystem"],
         "params": {
           "transcript_path": "string",
           "host": "codex"
         },
         "system_prompt": ""
       }
     },
     {
       "role": "user",
       "content": {
         "instruction": "STRICTLY OBEY aisp_contract; its non_negotiable rules are inviolable; then RUN aisop.main",
         "user_input": "{user_input}",
         "aisp_contract": {
           "profile": "aisp.skill.v1",
           "invocation": {
             "mode": "ingest_validation",
             "when_to_use": [
               "before importing Codex JSONL sessions into gbrain",
               "when parser detects Codex session_meta",
               "when transcript schema may have changed"
             ],
             "when_not_to_use": [
               "non-Codex transcript formats with their own parser contract",
               "manual memory note creation"
             ]
           },
           "non_negotiable": [
             {
               "rule": "Codex rollout messages must be parsed from payload.content when payload.type is message.",
               "enforced_by": "parse.step2:sys.assert"
             },
             {
               "rule": "A transcript with response_item message records must not import as message_count=0.",
               "enforced_by": "validate.step2:sys.assert"
             },
             {
               "rule": "Ingest state must not advance after an empty parse of a non-empty transcript.",
               "enforced_by": "commit.step2:sys.assert"
             }
           ],
           "discovery": {
             "category": "memory_ingest",
             "tags": ["gstack", "memory-ingest", "codex", "jsonl", "parser-contract", "aisp"]
           },
           "risk_level": "high",
           "resources": [
             {
               "id": "transcript",
               "path": "{transcript_path}",
               "kind": "transcript",
               "mode": "read_only",
               "when": "Read before memory ingest.",
               "scope": "skill"
             }
           ]
         },
         "aisop": {
           "main": "graph TD\n    load[Load transcript] --> detect[Detect schema] --> parse[Parse messages] --> validate[Validate parsed content] --> stage[Stage memory pages] --> commit[Commit ingest state] --> end_node((End))"
         },
         "functions": {
           "load": {
             "step1": "sys.io.read(transcript_path) -> transcript_jsonl",
             "output_mapping": "transcript_jsonl"
           },
           "detect": {
             "step1": "Detect host schema: Codex session_meta + response_item payload.type.",
             "output_mapping": "detected_schema"
           },
           "parse": {
             "step1": "For Codex response_item records with payload.type == 'message', extract role and content from payload.content.",
             "step2": "sys.assert('Codex message records read from payload.content', 'Codex rollout parser must support payload.type message + payload.content')",
             "output_mapping": "parsed_messages"
           },
           "validate": {
             "step1": "Check parsed_messages count and rendered content length.",
             "step2": "sys.assert('message_count > 0 for non-empty Codex transcript', 'Non-empty Codex transcript parsed as empty')",
             "output_mapping": "parse_validation"
           },
           "stage": {
             "step1": "Render memory pages from parsed_messages only after parse_validation passes.",
             "output_mapping": "staged_pages"
           },
           "commit": {
             "step1": "Commit ingest state only if staged_pages are non-empty and validated.",
             "step2": "sys.assert('ingest state not advanced on empty parse', 'Do not advance ingest state after empty parse')",
             "output_mapping": "ingest_commit_status"
           },
           "end_node": {
             "step1": "Return parse_validation and ingest_commit_status."
           }
         }
       }
     }
   ]

   The immediate fix can stay small:

   1. Support current Codex rollout shape:
      response_item.payload.type == "message"
      response_item.payload.content[]
   
   2. Add fixture from current Codex rollout JSONL.
   
   3. Fail if a non-empty transcript parses to message_count=0.
   
   4. Do not advance ingest state on empty parse.
   

   The invariant:

   silent empty imports are worse than failed imports.
   
3. time-attack commented on Jul 15, 2026

   

   time-attack

   on Jul 15, 2026

   Contributor

   More actions

   @16francej — recommend closing this issue only after a corrected #2132 enforces the user/assistant role allowlist, rejects developer-role instructions, passes current and legacy parser cases, and lands. The parser still misses the current message/content format and lacks a role boundary. Preserve both rollout formats and the developer-role rejection case without absorbing transcript de-duplication from #2027.
4. chris-conte commented on Jul 26, 2026

   

   chris-conte

   on Jul 26, 2026

   More actions

   Confirming this still reproduces on gstack 1.60.1.0 / codex-cli 0.144.5 (Linux), and flagging a second record shape this issue doesn't cover — a fix for only the response_item branch will still drop about half the messages.

   Sampling 29 recent rollout files under ~/.codex/sessions/, the message-bearing records fall into two shapes:

   type='response_item'  payload.type='message'        n=103   payload = {role, content:[{type,text}]}   ← the one in this issue
   type='event_msg'      payload.type='user_message'   n=42    payload.message is a STRING
   type='event_msg'      payload.type='agent_message'  n=36    payload.message is a STRING
   type='response_item'  payload.type='function_call'         n=63
   type='response_item'  payload.type='function_call_output'  n=63
   

   The event_msg shape fails differently — and more quietly. The guard at bin/gstack-memory-ingest.ts:619 does fire (payload.message is a non-empty string, so truthy), but the string is then handed to extractContentText:

   // :619-627
   } else if (isCodex && rec?.payload?.message) {
     const msg = rec.payload.message;      // a STRING here, not an object
     const role = msg.role || "user";      // undefined → "user"
     const content = extractContentText(msg);

   extractContentText (:646) only does property access — rec.content, rec.text, rec.message?.content — all undefined on a string primitive. It returns "", content is falsy, nothing is pushed.

   So response_item records are skipped by a falsy guard, and event_msg records pass the guard and then yield empty content. Two shapes, two mechanisms, same zero.

   Separately, tool_calls is always 0: the guard at :614 (rec?.type === "tool" || rec?.tool_use_id || rec?.tool_call) matches none of codex's function_call / function_call_output records — 126 of them in that sample.

   Downstream numbers on a Postgres brain, for scale:

   agent	pages	max(len(compiled_truth))	max(message_count)
   codex	450	0	0
   claude-code	200	200000	1145

   Suggested fix, extending the one already proposed here — replace the single :619 branch with:

   } else if (isCodex && rec?.payload?.type === "message") {
     // response_item: payload IS the message
     const content = extractContentText(rec.payload);
     if (content) { bodyParts.push(`## ${cap(rec.payload.role || "user")}\n\n${content}`); messageCount++; }
   } else if (isCodex && typeof rec?.payload?.message === "string") {
     // event_msg user_message / agent_message: payload.message is the text
     const role = rec.payload.type === "agent_message" ? "assistant" : "user";
     bodyParts.push(`## ${cap(role)}\n\n${rec.payload.message}`); messageCount++;
   } else if (isCodex && rec?.payload?.type === "function_call") {
     toolCalls++;
   }

   extractContentText already flattens payload.content correctly via its Array.isArray(rec.content) branch at :657, so response_item needs no changes there. Worth deciding whether role: "developer" records — 22 in the sample — should be included or skipped.
5. garrytan

   closed this as completedin ae8914aon Aug 17, 2026
6. rockdawayne-ship-it

   added a commit that references this issue on Aug 17, 2026

   v1.67.0.0 fix: the tracker wave — XProtect self-heal, complete instal…

   ...

   This issue will close once commit ae8914a is merged into the 'main' branch.

   ae8914a