Skip to content

Add --compare flag for baseline comparison #12

Description

@jharris1679

Overview

Add a --compare flag to sniff interview that compares new agent responses against existing baselines, showing metrics deltas (tokens, cost, tools, cache hits).

Behavior

# First run establishes baseline (human graded)
sniff interview --agent claude-code

# Second run compares to baseline
sniff interview --compare --cases comp-001

Implementation

  • Load existing baselines from .sniffbench/baselines.json
  • For each case with a baseline:
    • Run agent and capture metrics
    • Compare metrics against baseline (tokens, cost, tools, cache)
    • Show delta with color coding (green=better, red=worse)
  • Show aggregate comparison summary
  • No human grading required in compare mode

What's Included

  • --compare flag runs comparison mode
  • Shows metrics delta per case (tokens, cost, tools, cache hits)
  • Aggregate summary across all cases
  • Color-coded improvements/regressions

What's NOT Included (see related issues)

Related Issues

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions