Skip to content

/codex: an unusable model passes the auth probe, then every mode fails with no guidance #2477

Closed

Description

@shanudjn

/codex: an unusable model passes the auth probe, then every mode fails with no guidance

gstack: 1.60.2.0 · Codex CLI: 0.142.5 · Auth: ChatGPT account (not API key) · Platform: macOS

Three related problems, found while running /codex review on a real PR. The first cost roughly forty minutes and four failed invocations; the other two are small and adjacent.


1. The auth probe passes for an account that cannot use the configured model

_gstack_codex_auth_probe accepts "auth exists" as readiness. It has no notion of whether the configured model is usable by that account (grep model bin/gstack-codex-probe returns nothing). So Step 0.5 prints a clean bill of health and every subsequent call dies:

$ codex review "<the skill's own prompt>" -c 'model_reasoning_effort="high"' --enable web_search_cached
warning: Model metadata for `gpt-5.4` not found. Defaulting to fallback metadata; this can degrade performance and cause issues.
ERROR: {"type":"error","status":400,"error":{"type":"invalid_request_error",
  "message":"The 'gpt-5.4' model is not supported when using Codex with a ChatGPT account."}}
Review was interrupted. Please re-run /review and wait for it to complete.

## Error Handling has no entry for this. Its six entries cover binary-not-found, auth error, two timeouts, empty response, and session-resume failure. A 400 on the model is none of those, so the skill's guidance offers nothing and the agent running it is left guessing.

What made it expensive: the failure names the model but not where the model came from. The value lives in ~/.codex/config.toml, which the skill never reads or mentions. In my case that file contained both the stale pin and Codex's own record of the migration it never applied:

model = "gpt-5.4"          # rejected by this account

[notice.model_migrations]
"gpt-5.4" = "gpt-5.5"      # Codex knew, and kept using the stale value

The fix was one line. Finding it took four invocations and a strings dump of the binary to enumerate candidate models, then probing them one at a time.

Suggested fix, in preference order:

  • Extend the Step 0.5 probe to a real round trip: codex exec "reply OK" -s read-only with a short timeout. Passes in ~10s and catches model rejection, entitlement changes, and stale pins in one shot. Auth-exists is a weaker signal than the probe implies.
  • Add an Error Handling entry for status: 400 / model is not supported, pointing at ~/.codex/config.toml's model = line and at [notice.model_migrations] if present, since Codex records the intended replacement there.

Either alone would have collapsed this to a single failure with an obvious fix.


2. ## Model & Reasoning documents -m for codex review, which rejects it

SKILL.md:1554-1555:

If the user specifies a model (e.g., /codex review -m gpt-5.1-codex-max or /codex challenge -m gpt-5.2), pass the -m flag through to codex.

The challenge half is fine — that path is codex exec, which takes -m. The review half is not:

$ codex review "<prompt>" -m gpt-5.1-codex-max -c 'model_reasoning_effort="high"'
error: unexpected argument '-m' found
  tip: to pass '-m' as a value, use '-- -m'
Usage: codex review [OPTIONS] [PROMPT]

codex review --help on 0.142.5 lists -c/--config, --strict-config, --enable, --uncommitted, --base, --disable, --commit, --title. No -m. The working form is -c 'model="..."'.

Same shape as the --base incompatibility already documented in Step 2A, so the fix fits an existing pattern: note that review mode takes the model via -c model="..." while exec-based modes take -m.


3. --enable web_search_cached is deprecated and warns on every call

All six invocation sites pass it. CLI 0.142.5:

deprecated: `[features].web_search_cached` is deprecated because web search is enabled by default.
Set `web_search` to "live", "indexed", "cached", or "disabled" at the top level
(or under a profile) in config.toml if you want to override it.

Harmless, but it lands in $TMPERR on every run, which matters because the skill's own hang-detection reasons about stderr volume (_gstack_codex_log_hang passes wc -c < "$TMPERR"). Constant deprecation noise makes stderr size a worse signal than it could be.


Reproducing #1 without a ChatGPT account

Any rejected model reproduces it — the 400 is server-side, not account-specific:

echo 'model = "definitely-not-a-model"' >> ~/.codex/config.toml
# Step 0.5's auth probe still passes; every /codex mode then fails with no guidance

Why this is worth more than a docs edit

The pattern here is a green signal that means less than it looks like — the same one v1.60.2.0's own changelog files as a P1 (browse tests calling setTimeout(() => process.exit(0), 500) so the suite can exit 0 before the summary prints).

_gstack_codex_auth_probe returning OK reads as "Codex is ready." It actually means "an auth file exists." When those diverge, the skill proceeds confidently into a wall, and the guidance section has nothing to say about the wall it hits. A ten-second round trip makes the probe mean what its name implies.

Happy to send a PR for any or all three if the approach looks right.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions