Skip to content

Implement Comprehension Case Runner #11

Description

@jharris1679

Overview

Complete the comprehension case implementation by integrating with real agents and adding baseline comparison.

Current State

  • ✅ 12 comprehension cases designed (comp-001 through comp-012)
  • sniff interview CLI command with 1-10 grading
  • ✅ Baseline storage in .sniffbench/baselines.json
  • ⬜ Agent integration (currently placeholder)
  • ⬜ LLM-judge comparison for subsequent runs

Tasks

Agent Integration

  • Create agent wrapper interface
  • Implement Claude Code wrapper (spawn process, capture output)
  • Pass comprehension prompt to agent
  • Capture agent's exploration and final answer
  • Track tool usage for efficiency metrics

Baseline Comparison

  • Add --compare flag to interview command
  • Implement LLM-judge evaluator for comparing answers
  • Score similarity to baseline (semantic, not just string match)
  • Show diff between baseline and new answer
  • Generate comparison report

Polish

  • Handle agent timeout gracefully
  • Add progress indicators during agent exploration
  • Support running single case: sniff interview --case comp-001
  • Export results to JSON for reporting

Technical Requirements

interface AgentWrapper {
  name: string;
  run(prompt: string, options: AgentOptions): Promise<AgentResult>;
}

interface AgentResult {
  answer: string;
  toolsUsed: string[];
  tokensUsed: number;
  durationMs: number;
}

Acceptance Criteria

  • Can run comprehension interview with real Claude Code agent
  • Agent explores codebase and provides substantive answers
  • Subsequent runs compare against stored baseline
  • Comparison shows similarity score and key differences
  • Passes manual review of full interview flow

Status: Done ✅

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions