Skip to content

Two stated guarantees in the codex skill don't hold (v1.60.1.0) #2524

Closed

Description

@dk96-creator

Both of these are cases where SKILL.md states a guarantee that the code beneath it doesn't provide. Filing together because they share that shape; happy to split if you'd rather.

Confirmed by reading the shipped file. Line numbers are v1.60.1.0 @ a325940; env is codex CLI v0.144.1 on macOS/zsh.

1. The gate rule ignores exit status, so failure output can be classified PASS

SKILL.md:1049–1051:

4. Determine gate verdict by checking the review output for critical findings.
   If the output contains `[P1]` — the gate is **FAIL**.
   If no `[P1]` markers are found (only `[P2]` or no findings) — the gate is **PASS**.

The verdict comes entirely from the presence of a substring; exit status is not part of the rule. Output from a run that failed — expired auth, invalid flag, timeout, empty result — contains no [P1], and so satisfies the PASS branch as written.

Two qualifications in fairness:

  • :995–1000 does echo a non-zero exit, and :1572 separately instructs the agent to surface auth/timeout/empty-response errors. The skill isn't silent on failure — but those are instructions to the agent, not part of the gate rule, and the rule still yields PASS.
  • Codex's own v0.144.1 review rubric treats P0 as blocking. The rule recognises only [P1], so a P0-only result also lands in PASS.

Suggested fix: make the gate fail-closed — non-zero exit, empty output, or absent completion signal should yield FAIL (or UNKNOWN), never PASS — and recognise [P0] as blocking. Note a "completion signal" would have to be introduced deliberately; native human-readable review output doesn't currently emit a stable machine-checkable marker.

2. The default review path sets no explicit sandbox override

SKILL.md:1585, verbatim:

- **Never modify files.** This skill is read-only. Codex runs in read-only sandbox mode.

The default native path at :987–989 invokes:

codex review "<prompt>" -c 'model_reasoning_effort="high"' --enable web_search_cached

No -s read-only, no sandbox override — so the effective sandbox resolves from the user's ~/.codex/config.toml rather than from the skill. A user whose sandbox_mode is workspace-write or danger-full-access gets a review with write capability, and with approval_policy = "never", without a prompt.

The custom-focus fallback at :1027 does use codex exec -s read-only. Only the default path is affected — which arguably makes it the more misleading of the two, since a reader who checks one path and finds it correct has no reason to check the other.

Suggested fix: pass -c 'sandbox_mode="read-only"' on the native review path, or resolve and assert the effective sandbox before running.


Reported against the template/generator rather than the emitted file, since local edits to SKILL.md don't survive an upgrade.

Activity

  1. garrytan commented on Aug 17, 2026

    @garrytan
    Owner

    Fixed on main in v1.64.0.0 — codex gate fails closed and recognizes bracketed + native P0/P1 tags. Closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions