Skip to content

feat: report instrumentation coverage for agent runs #30

Open

Description

@utsab345

Problem

Users need to understand which tool and LLM calls were captured during a StepFork run and which external boundaries are outside its replay guarantees.

Scope

Explore an instrumentation coverage report based on information StepFork can reliably observe. Start with known instrumented calls and explicit capture metadata. Do not claim to detect every uninstrumented call automatically without a reliable mechanism.

Review the existing trace format (docs/trace-format.md), run receipts, replay behavior, and PRODUCT_ROADMAP.md before proposing a design. Document limitations and avoid breaking existing trace compatibility.

Acceptance criteria

  • The report has a clearly defined and documented meaning of coverage.
  • It distinguishes observed instrumentation from calls that cannot be verified.
  • An offline example demonstrates the report.
  • Tests cover the reported coverage data and its limitations.
  • Existing trace and receipt formats remain compatible unless a migration is explicitly justified.
  • No false guarantee is made that every external call has been captured.

Testing

Add deterministic tests and an offline example. Run the relevant tests and the full test suite.

This is a larger feature. Please open a discussion or comment with a proposed design before beginning substantial implementation.

Contributions are welcome. Please comment with your proposed design before opening a PR.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requesthelp wantedExtra attention is needed

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions