Repository navigation
fixture: redstamp approve-path with host-observation sidecar - #98
rajudandigam wants to merge 4 commits into
Conversation
Fill the approve-decision corpus gap: join an independent host observation to refine approve outcomes (declined+complete → denied; incomplete/mismatch fail closed), and verify optional audit checkpoints for truncated tails.
sprayberry-redline
left a comment
There was a problem hiding this comment.
Request changes: conflicting observations can produce an authoritative non-invocation result. rule:none
P1: Reject conflicting observations for the same join key (src/adapters/redstamp.mjs:373-377)
observation = candidates.find((o) => {
const opOk = o.operationId === undefined || o.operationId === rec.operationId;
const attOk = o.attemptId === undefined || o.attemptId === rec.attemptId;
return opOk && attOk;
}) ?? null;The index retains multiple observations for a hash, but this selects the first matching one without checking the others. For an approve record, supply two observations with the same hash, operationId and attemptId: first {humanResponse: 'declined', targetToolInvoked: false, windowComplete: true}, then {humanResponse: 'accepted', targetToolInvoked: true, windowComplete: true}. The first record reaches the declined branch and produces outcome: 'denied' and status: 'not_invoked', despite contradictory invocation evidence. Reversing the sidecar order changes the result to ok / invoked. An ambiguous sidecar must not authorize a non-invocation claim merely through its ordering.
Collect all matching observations before deriving the outcome. If their invocation or response evidence conflicts, return the existing unknown fallback rather than selecting the first. Pin the conflicting pair in both input orders.
The checkpoint comparison usefully covers tail truncation that chain verification alone cannot detect. The complete five-file diff and adapter normalization context were inspected. No test-suite result is published at this head; the only completed check reported is triage, so the stated 225 passing tests are not independently confirmed by CI.
Collect all join-key matches before deriving approve outcomes. Conflicting invocation/response/window evidence stays unknown in both sidecar orders; consistent duplicates and checkpoint controls remain.
sprayberry-redline
left a comment
There was a problem hiding this comment.
Request changes: conflicting executor results still make the outcome depend on sidecar order. rule:none
P2: Include executor results in duplicate conflict detection (src/adapters/redstamp.mjs:406-416)
if (responses.size > 1 || invoked.size > 1 || windows.size > 1) {The fallback still selects:
observation = matches[0];For one approve record, supply two observations with the same hash, operationId and attemptId, both with humanResponse: 'accepted', targetToolInvoked: true and windowComplete: true, but with executorResult: 'ok' and executorResult: 'error'. All three sets have size one, so this treats the observations as consistent. outcomeFor then reads the first observation's executor result: the first ordering produces outcome: 'ok', and reversing it produces outcome: 'error'. Both also publish the selected executor result as authoritative host evidence. The conflict check needs to cover the field that determines that outcome, not just response, invocation and window completeness.
Compare executorResult (with missing values normalized to 'unknown') alongside the other evidence fields before selecting a duplicate. Route contradictory executor results through the conflict fallback and cover the accepted/invoked pair in both orders.
The revised join resolves the declined-versus-invoked conflict from the previous review, and the checkpoint test covers truncated tails. Review covered the five-file diff and the adapter's outcome derivation. CI currently reports only a passing triage job, with no published test-suite result; the reported local test count is not confirmed by CI.
Include executorResult in duplicate conflict detection so accepted/invoked pairs with ok vs error cannot authorize an outcome by sidecar order.
sprayberry-redline
left a comment
There was a problem hiding this comment.
Request changes: use the surrounding module's plain wording in the added comments, messages and test name. rule:reads-as-generated
P3: Remove the added parenthetical commentary and long-dash phrasing (src/adapters/redstamp.mjs:425, test/redstamp-adapter.test.mjs:165, corpus/redstamp/approve-path.jsonl:2).
// Consistent duplicates — first is fine (order-independent evidence).test('approve without host observation stays ok — invocation unknown', () => {# Synthetic only — never execute these commands.
The duplicate-selection comment explains why the patch is correct rather than what the branch does; delete it. Use a short test name and ordinary sentence punctuation, consistent with the surrounding tests. Apply the same punctuation cleanup to the other added comments and reason strings in the adapter and the test-plan line in the description.
test('approve without host observation has unknown invocation', () => {
The executor-result conflict is fixed: contradictory accepted/invoked observations now take the unknown-evidence fallback in either order, and the new regression checks both orders without publishing a selected executor result. Review covered the five-file diff and the adapter's outcome derivation. CI currently publishes a passing triage job but no test-suite result; local tests were not rerun.
|
Thanks for following through on #143. This is the fixture we asked for there, and both rounds of fixes for conflicting observations were quick and correct. Your core point holds, and it is a real gap in plumbline today: a redstamp
plumbline maps all three to I'd like to land this in two pieces. In this PR
In redstamp The invocation evidence should come from redstamp itself rather than from a sidecar format nothing writes yet. Each verdict should record what was enforced (blocked, asked, or forwarded), and the ask path needs a If it's easier, I can make the trims on your branch. Either way I'll handle the punctuation note from the last review round. |
sprayberry-redline
left a comment
There was a problem hiding this comment.
Verdict: request changes. The observation conflict handling now makes both response and executor-result conflicts order-independent. The requested cleanup of added explanatory prose remains incomplete.
1. Blocking: src/adapters/redstamp.mjs:425
// Consistent duplicates — first is fine (order-independent evidence).
This added comment narrates the patch's correctness rather than documenting behavior, and its long-dash phrasing remains from the earlier requested cleanup.
Suggested fix:
Delete the comment; the surrounding duplicate comparison and assignment already state the behavior.
rule:reads-as-generated
Summary
corpus/redstamp/approve-pathwith a real sha256 hash chain, checkpoint tip, and host-observation sidecar fordecision=approve/tier=red.deniedokwithhost_observation.status: unknown(fail closed)why[]tainted (mine hosts only; never copy verbatim).Why
redstamp’s audit log records the human-approval decision, not whether the host invoked the tool. Treating
approveasallowhides the non-invocation case. Chain integrity does not prove observer truth.Test plan
npm test— 225 pass / 0 failhost_observation/audit_hashpre-normalize fields