Skip to content

fix(headless): record OpenCode prompt provenance accurately #1563

Closed

Description

@Astro-Han

What happened

packages/headless/harbor/opencode_agent.py::_execution_identity() hashes MAKA_SYSTEM_PROMPT and records it as systemPromptHash.

The OpenCode adapter does not inject that prompt into OpenCode. _run_with_stop_sentinel() launches the pinned OpenCode CLI with the task instruction and OpenCode's benchmark config; OpenCode supplies its own native system scaffold. As a result, the recorded identity claims provenance for model-visible bytes that the OpenCode arm never received.

Changing only the Maka system prompt therefore changes the OpenCode cell's execution identity even though the OpenCode invocation is unchanged. This can create false cohort differences and misleading audit evidence. It does not change the OpenCode score or token usage itself.

Expected behavior:

  • Execution identity distinguishes a Maka-injected prompt from a competitor-owned native prompt.
  • An OpenCode cell does not record the Maka prompt hash as model-visible provenance.
  • Changing only MAKA_SYSTEM_PROMPT does not change OpenCode's prompt identity.
  • The pinned OpenCode version/toolchain remains the authority for the competitor-owned scaffold when its exact internal prompt hash is unavailable.

How to reproduce

  1. Run two otherwise identical OpenCode harness cells.
  2. Set MAKA_SYSTEM_PROMPT to different values in the two environments.
  3. Compare maka-cell-execution-identity.json.
  4. Observe different systemPromptHash values.
  5. Compare the OpenCode launch command/config and provider-visible task input; the Maka prompt is not injected into either invocation.

Environment

  • Maka: reproduced on main@213882bda and present in ee7e4ba5d
  • Surface: Headless / Harbor OpenCode adapter
  • OpenCode: pinned 1.17.18
  • Node.js: 26.5.0
  • Harbor: 0.13.2

Logs, screenshots, or additional context

This is a provenance defect, not an effectiveness or economy defect. Existing benchmark scores remain valid, but reports should not describe the recorded OpenCode systemPromptHash as the hash of OpenCode's actual native system prompt.

The fix should define the prompt-authority representation at the execution-identity contract rather than inventing a hash for inaccessible competitor-owned prompt bytes, and should include backward-compatible decoding for existing cell artifacts.

Activity

  1. UncertaintyDeterminesYou4ndMe commented on Aug 15, 2026

    @UncertaintyDeterminesYou4ndMe
    Contributor

    Triage note: this defect no longer exists on main. The Headless/Harbor OpenCode adapter that hashed MAKA_SYSTEM_PROMPT into the OpenCode cell's execution identity (packages/headless/harbor/opencode_agent.py::_execution_identity) was removed wholesale in #2605 when Headless was replaced by the minimal Eval kernel. In the current Eval kernel, external subjects' identity is the verified toolchain version + fingerprint (packages/eval/src/external-subject.ts, toolchain-verification.ts); the Maka system prompt is not recorded as provenance for a competitor-owned scaffold. Backward-compatible decoding of the old maka-cell-execution-identity.json artifacts is moot for the same reason — the producer is gone. This issue can likely be closed as obsolete.

  2. Astro-Han commented on Aug 15, 2026

    @Astro-Han
    ContributorAuthor

    Closed as done. Thanks for triaging the issue @UncertaintyDeterminesYou4ndMe!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions