Skip to content

Empty-delivery sentinel fails open: a mangled marker is posted to the channel as literal text #2388

Closed

Description

@ace139

What happened

A production eve agent (Slack channel, scheduled app-auth run) posted the literal string <evedev-empty-delivery/> into a team channel. The model was following the conditional-delivery instruction and meant to deliver nothing — it just mistyped the marker, and the mistyped token was delivered as an ordinary message. In the same run, empty replies from the model were retried with the "Answer now from the tool results above" nudge, and the retries produced conversational filler ("I now have all the data I need.", "Here is the report:") that was also delivered as channel messages.

Bug 1 — the sentinel check fails open

hasEmptyDeliverySentinel (packages/eve/src/shared/empty-delivery.ts) is an exact includes() on <eve-empty-delivery/> (plus the HTML-escaped form). A correctly-spelled marker is reliably stripped — but any near miss falls through to the channel's message.completed default and is posted verbatim:

Model's entire reply Result today
<eve-empty-delivery/> silenced ✓
<evedev-empty-delivery/> (observed in production) posted to the channel
<eve-empty-delivery></eve-empty-delivery> (paired form, uncorrupted name) posted
<eve-empty-delivery> / <EVE-EMPTY-DELIVERY/> / <eve-empty-delivery /> posted

An internal control token that reaches a user-visible surface should fail closed: a message that is nothing but a sentinel-shaped tag carries no deliverable content, so the safe degradation for a mangled marker is silence, not literal control text in a team channel.

Bug 2 — the empty-response retry contradicts conditional delivery

The conditional-delivery instruction is injected as a system message, but the empty-response recovery nudge arrives as a user message leading with "Answer now from the tool results above" (packages/eve/src/harness/tool-loop.ts). In unattended app-auth runs there is no person to answer — the framing reliably elicits filler, which is then delivered. When conditional delivery is enabled for the turn, the recovery should lead with the marker option rather than an instruction to answer.

Environment / verification

Proposed fix

PR #2386 implements both: tolerant whole-message-anchored matching for sentinel-shaped tags (keeping the exact sentinel's match-anywhere behavior that existing tests pin), spelled to be linear-time on raw model output, plus a silence-first retry nudge under conditional delivery, with regression tests for the exact production shapes. Happy to adjust direction per maintainer preference — the PR lists its design decisions and known limitations explicitly.

Activity

  1. roboeve commented on Aug 21, 2026

    @roboeve
    Contributor

    triaged #2388 as a p2 channels bug and applied the labels. reproduced all six malformed-marker cases on main; the same cases are silenced on PR #2386, whose focused 41 matcher tests and scheduled retry regression pass. no duplicate report found, so i didn’t open a competing proposal.

  2. ruiconti commented on Aug 21, 2026

    @ruiconti
    Contributor

    @ace139 which model demonstrated this behavior?

  3. ace139 commented on Aug 22, 2026

    @ace139
    Author

    @ruiconti GLM-5.2 via Vercel AI Gateway

  4. benpankow commented on Oct 9, 2026

    @benpankow
    Collaborator

    Hi @ace139, thanks for filing this! Going through older issues, I believe this was resolved in 0.69.0 by #3840. Feel free to re-open if this is still relevant.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions