Skip to content

hook::emit_telemetry drops the envelope once data exceeds the Windows command-line limit #1595

Description

@kyle-sexton

Summary

hook::emit_telemetry builds the envelope with jq -n … --argjson data "$data_json"
(lib/hook-utils.sh, and every vendored copy). Windows caps a process command line at 32767
characters, so once a producer's data payload serializes past that, jq never runs and the whole
envelope is dropped.

Reproduced on Windows / Git Bash with the same jq the hooks use:

n=100   serialized_bytes=6401    rc=0    findings preserved
n=300   serialized_bytes=19201   rc=0    findings preserved
n=600   serialized_bytes=38401   rc=126  <- argument list too long
n=1200  serialized_bytes=76801   rc=126

rc=126 is the shell reporting the exec failure; jq is never entered, so nothing in the pipeline
can distinguish it from a jq parse error.

Why it matters, and why it is not just "telemetry is lossy"

The contract does say telemetry is best-effort and lossy (docs/conventions/hook-telemetry/README.md).
A dropped envelope under load is inside that contract. The problem is the shape of the loss: it is
deterministic on payload size, so a consumer wiring a sink gets complete data for every ordinary file
and silently nothing for exactly the large or finding-dense files a sink is most likely to be
wired for. That is a systematic blind spot, not sampling.

Scope

Every producer that can emit a large data payload. Known:

  • markdown-format — data.findings is one string per unfixable violation, deliberately uncapped.
    A real 324-finding file serializes well past 32 KB.
  • typos-format — data.findings plus data.applied, both uncapped.

Other producers with bounded payloads are unaffected today but sit on the same helper.

The same defect class already bit the two plugins' own payload construction and was fixed there by
feeding jq on stdin instead of argv (see #1578, #1589). Those fixes are plugin-local; this one is not,
which is why it is filed separately.

Fix

Pass data_json to jq on stdin rather than as --argjson. The remaining --arg values
(schema_version, timestamp, hook, hook_event, status, duration_ms) are all short and
bounded, so only the one value needs to move.

This is a change to the SSOT lib/hook-utils.sh, so it requires scripts/sync-hook-utils.sh and the
usual version-bump wave across every carrying plugin — which is the reason it is not folded into
either plugin PR.

Not verified

Whether any non-Windows platform hits this. POSIX ARG_MAX is typically 2 MB or more, so the
practical exposure is Windows-only, but that has not been measured here.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.priority: mediumReal value, no hard deadline; normal backlog flow.work-class: structuralRefactors, migrations, contract changes; cross-cutting and hard to reverse.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions