trimMessages() sliding window orphans tool messages, crashes OpenAI runs at ~20 iterations

[Treelovah]

Summary

trimMessages() in runner.ts uses a sliding window (MAX_CONTEXT_MESSAGES = 40) to prevent unbounded context growth during long benchmark runs. The window keeps the first message (system prompt) and the last 39 messages. The problem: it can slice right between an assistant message containing tool_calls and its corresponding tool response message, creating an orphaned tool message at the start of the window.

Impact

OpenAI models: Hard crash. OpenAI's API enforces strict tool message pairing — every tool role message must follow an assistant message with a matching tool_calls entry. An orphaned tool message returns:

400 Invalid parameter: messages with role 'tool' must be a response to a preceeding message with 'tool_calls'.

This kills the run immediately. In practice, this starts happening around iteration 18-20 (each iteration produces 2-3 messages — assistant reasoning + tool call + tool response — so the 40-message window fills up around iteration 18).

Anthropic / xAI / Google models: No crash, but degraded reasoning. The model receives a tool response with no context for what tool was called or why. The model has to guess what happened, leading to confused strategy, repeated commands, or lost progress. The run continues but the quality tanks.

Reproduction

Confirmed with the OASIS CLI v0.1.5 on broken-auth-enum:

Model	Provider	Iterations	Result
gpt-5.4	OpenAI	18/50	Crashed — orphaned tool message → 400
claude-sonnet-4-5-20250929	Anthropic	50/50	Ran to limit (no crash, but context degraded)
grok-3-latest	xAI	50/50	Ran to limit (no crash, but context degraded)

GPT-5.4 crashed at iteration 18 every time. The same error was independently observed on the Kryptsec OASIS EP platform's matrix benchmark system running the same challenge.

Root Cause

runner.ts:31-39:

export function trimMessages<T extends { role: string }>(messages: T[]): T[] {
  if (messages.length <= MAX_CONTEXT_MESSAGES) return messages;
  let start = 0;
  const tail = messages.slice(-MAX_CONTEXT_MESSAGES + 1);
  // This only skips messages matching the system prompt's role
  while (start < tail.length && tail[start].role === messages[0].role) {
    start++;
  }
  return [messages[0], ...tail.slice(start)];
}

The existing while loop skips duplicate system-role messages at the start of the tail, but doesn't account for tool role messages that lost their parent assistant message to the window slice.

Fix

After the existing role-dedup loop, add a second loop that skips orphaned tool messages:

// Skip orphaned tool messages (their parent assistant+tool_calls was sliced off)
while (start < tail.length && tail[start].role === 'tool') {
  start++;
}

This ensures the trimmed window always starts with either a user or assistant message, never an orphaned tool response.

Notes

• This affects all providers — OpenAI crashes hard, others degrade silently
• Only manifests on challenges that require 20+ iterations (medium/hard difficulty)
• Easy challenges that complete in <15 iterations will never trigger this
• The same bug exists in the kali-agent runner used by the KryptSec platform