Skip to content

feat(interview): replace 1-10 grading scale with binary pass/fail #46

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jharris1679 merged 5 commits into from
Dec 30, 2025
Merged

feat(interview): replace 1-10 grading scale with binary pass/fail #46

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
20406ee
feat(interview): replace 1-10 grading scale with binary pass/fail
jharris1679 Dec 30, 2025
10489e4
fix(compare): show pass rate delta in percentage points
jharris1679 Dec 30, 2025
5a4ff9a
docs: add git workflow preference to CLAUDE.md
jharris1679 Dec 30, 2025
63f444d
fix(interview): show unevaluated baselines as neutral state
jharris1679 Dec 30, 2025
2eedb36
fix(compare): use padVisible for ANSI-colored table alignment
jharris1679 Dec 30, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CLAUDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,3 +34,7 @@ Instead of using Glob or Grep, do this:
osgrep uses semantic search (embeddings) to find conceptually related code, not just text matches. It's more effective for understanding unfamiliar codebases.

**Start your exploration with osgrep before using other tools.**

## Git Workflow Preferences

**NEVER amend commits.** Always create new commits to preserve history. Even for small fixes or PR feedback, create a separate commit rather than using `git commit --amend`.
54 changes: 54 additions & 0 deletions assets/demo.cast
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
{"version":3,"term":{"cols":80,"rows":24,"type":"xterm-256color"},"timestamp":1766096092,"command":"./scripts/demo.sh","env":{"SHELL":"/bin/zsh"}}
[0.143, "o", "\u001b[3J\u001b[H\u001b[2J"]
[0.000, "o", "┌─────────────────────────────────────────────────────────┐\r\n"]
[0.000, "o", "│ sniffbench - Benchmark suite for coding agents │\r\n│ https://github.com/AnswerLayer/sniffbench │\r\n"]
[0.000, "o", "└─────────────────────────────────────────────────────────┘\r\n"]
[2.008, "o", "\r\n\u001b[1;32m$\u001b[0m sniff variants\r\n"]
[1.476, "o", "\r\n\u001b[90m╭─\u001b[39m\u001b[1m Variants \u001b[22m\u001b[90m────────────────────────────────────────────────────────────────────────────────────────────╮\u001b[39m\r\n\u001b[90m│\u001b[39m \u001b[1m2 variants\u001b[22m \u001b[90m│\u001b[39m\r\n\u001b[90m│\u001b[39m \u001b[1m\u001b[22m \u001b[90m│\u001b[39m\r\n\u001b[90m│\u001b[39m \u001b[1m\u001b[22m\u001b[2mID Name Container Model Description\u001b[22m \u001b[90m│\u001b[39m\r\n\u001b[90m│\u001b[39m \u001b[2m\u001b[22m\u001b[2m──────────────────────────────────────────────────────────────────────────────"]
[0.000, "o", "────────────\u001b[22m \u001b[90m│\u001b[39m\r\n\u001b[90m│\u001b[39m var-176567809239 \u001b[36mwith-osgrep\u001b[39m \u001b[32m✓ Built\u001b[39m \u001b[33munknown\u001b[39m \u001b[2mControl with osgrep semantic s...\u001b[22m \u001b[90m│\u001b[39m\r\n\u001b[90m│\u001b[39m var-176565880792 \u001b[36mcontrol\u001b[39m \u001b[32m✓ Built\u001b[39m \u001b[33munknown\u001b[39m \u001b[2m-\u001b[22m \u001b[90m│\u001b[39m\r\n\u001b[90m╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[39m\r\n\r\n"]
[3.018, "o", "\r\n\u001b[1;32m$\u001b[0m sniff closed-issues list\r\n"]
[0.994, "o", "\u001b[?25l"]
[0.000, "o", "\u001b[1G\u001b[1G"]
[0.000, "o", "\u001b[36m⠋\u001b[39m Loading closed-issue cases..."]
[0.029, "o", "\u001b[1G\u001b[0K"]
[0.000, "o", "\u001b[?25h"]
[0.000, "o", "\r\n\u001b[90m╭─\u001b[39m\u001b[1m closed-issues list \u001b[22m\u001b[90m──╮\u001b[39m\r\n\u001b[90m│\u001b[39m \u001b[1m12 closed-issue cases\u001b[22m \u001b[90m│\u001b[39m\r\n\u001b[90m╰─────────────────────────╯\u001b[39m\r\n\r\n"]
[0.000, "o", "\r\n"]
[0.000, "o", "\u001b[1m\u001b[2mID Title Lang Diff\u001b[22m\u001b[22m\r\n"]
[0.000, "o", "\u001b[2m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[22m\r\n"]
[0.000, "o", "\u001b[36mclosed-issue-dbt-labs-metricflow-1002 \u001b[39m [SL-1605] [Bug] No validation tha python \u001b[33mMedium\u001b[39m\r\n"]
[0.000, "o", "\u001b[36mclosed-issue-dbt-labs-metricflow-1009 \u001b[39m [SL-1619] [Feature] Allow measure python \u001b[33mMedium\u001b[39m\r\n\u001b[36mclosed-issue-dbt-labs-metricflow-1065 \u001b[39m [SL-1831] [Feature] Add Python 3. unknown \u001b[33mMedium\u001b[39m\r\n\u001b[36mclosed-issue-dbt-labs-metricflow-1092 \u001b[39m [SL-1878] Clean up metricflow dis unknown \u001b[33mMedium\u001b[39m\r\n\u001b[36mclosed-issue-dbt-labs-metricflow-1470 \u001b[39m [SL-3013] [Feature] Require Pytho unknown \u001b[33mMedium\u001b[39m\r\n"]
[0.000, "o", "\u001b[36mclosed-issue-answerlayer-sniffbench-38 \u001b[39m CLI: Split variant/variants for g typescript \u001b[31mHard\u001b[39m\r\n"]
[0.000, "o", "\u001b[36mclosed-issue-dbt-labs-metricflow-1051 \u001b[39m [SL-1805] [Bug] set.intersection python \u001b[31mHard\u001b[39m\r\n"]
[0.000, "o", "\u001b[36mclosed-issue-dbt-labs-metricflow-1473 \u001b[39m [SL-3014] [Bug] ID generation is python \u001b[31mHard\u001b[39m\r\n\u001b[36mclosed-issue-dbt-labs-metricflow-1554 \u001b[39m [SL-3384] [Feature] Add a `--sql` python \u001b[31mHard\u001b[39m\r\n\u001b[36mclosed-issue-dbt-labs-metricflow-1591 \u001b[39m [SL-3389] [Feature] MetricFlow do python \u001b[31mHard\u001b[39m\r\n"]
[0.000, "o", "\u001b[36mclosed-issue-dbt-labs-metricflow-1706 \u001b[39m [Bug] mf query --csv reports succ python \u001b[31mHard\u001b[39m\r\n"]
[0.000, "o", "\u001b[36mclosed-issue-dbt-labs-metricflow-1787 \u001b[39m [Feature] mf list saved-queries t python \u001b[31mHard\u001b[39m\r\n\r\n"]
[0.001, "o", "\u001b[2mRun a case: \u001b[36msniff closed-issues run --case <id>\u001b[39m\u001b[22m\r\n"]
[0.004, "o", "\u001b[?25h"]
[3.010, "o", "\r\n\u001b[1;32m$\u001b[0m sniff runs list\r\n"]
[0.998, "o", "\r\n╭─ Runs ─────────────────────────────────────────────────────────────────────────────────────────────────────────╮\r\n│ 19 runs │\r\n│ │\r\n│ ID Label Variant Date Agent Cases Grade │\r\n│ ──────────────────────────────────────────────────────────────────────────────────────────────────────────"]
[0.000, "o", "──── │\r\n│ run-1766006666292-jk [control-close control Dec 17, 2025, 04:24 claude-cod 0/0 N/A │\r\n│ run-1766006609181-2g [osgrep-closed with-osgrep Dec 17, 2025, 04:23 claude-cod 0/0 N/A │\r\n│ run-1765932483568-y2 [osgrep-run] with-osgrep Dec 16, 2025, 07:48 claude-cod 0/0 N/A │\r\n│ run-1765932142907-aq [control-run] control Dec 16, 2025, 07:42 claude-cod 0/0 N/A │\r\n│ run-1765928409597-r7 [osgrep-run] with-osgrep Dec 16, 2025, 06:40 claude-cod 0/0 N/A │\r\n│ run-1765928121386-q3 [control-run] control Dec 16, 2025, 06:35 claude-cod 0/0 N/A │\r\n│ run-1765927828308-hi [control-run] control Dec 16, 2025, 06:30 claude-cod 0/0 N/A │\r\n│ run-1765678179843-vc with-osgrep Dec 13, 2025, 09:09 claude-cod 1/1 9.0/10 │\r\n│ run-1765677933370-y0 "]
[0.000, "o", " - Dec 13, 2025, 09:05 claude-cod 1/1 2.0/10 │\r\n"]
[3.014, "o", "\r\n\u001b[1;32m$\u001b[0m sniff compare run-1765673169542-cr run-1765678179843-vc\r\n"]
[0.987, "o", "\r\n\u001b[90m╭─\u001b[39m\u001b[1m sniff compare \u001b[22m\u001b[90m─────────────────────────────╮\u001b[39m\r\n\u001b[90m│\u001b[39m \u001b[1mRun Comparison\u001b[22m \u001b[90m│\u001b[39m\r\n\u001b[90m│\u001b[39m \u001b[1m\u001b[22m \u001b[90m│\u001b[39m\r\n\u001b[90m│\u001b[39m \u001b[1m\u001b[22m\u001b[2mComparing two evaluation runs side-by-side.\u001b[22m \u001b[90m│\u001b[39m\r\n\u001b[90m╰───────────────────────────────────────────────╯\u001b[39m\r\n\r\n"]
[0.000, "o", "\u001b[1m\u001b[22m\r\n\u001b[1m Run Information:\u001b[22m\r\n\u001b[1m\u001b[22m\r\n\u001b[2m ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─\u001b[22m\r\n \u001b[1mRun 1:\u001b[22m run-1765673169542-cr \r\n"]
[0.004, "o", " Created: Dec 13, 2025\r\n Agent: claude-code 2.0.55 (Claude Code)\r\n Model: claude-sonnet-4-5-20250929\r\n"]
[0.000, "o", " Variant: \u001b[35mcontrol\u001b[39m\r\n"]
[0.000, "o", "\r\n"]
[0.000, "o", " \u001b[1mRun 2:\u001b[22m run-1765678179843-vc \r\n"]
[0.001, "o", " Created: Dec 13, 2025\r\n"]
[0.000, "o", " Agent: claude-code 2.0.55 (Claude Code)\r\n"]
[0.000, "o", " Model: claude-sonnet-4-5-20250929\r\n"]
[0.000, "o", " Variant: \u001b[35mwith-osgrep\u001b[39m\u001b[2m - Control with osgrep semantic search enabled\u001b[22m\r\n"]
[0.000, "o", "\u001b[1m\u001b[22m\r\n\u001b[1m Configuration Changes:\u001b[22m\r\n\u001b[1m\u001b[22m\r\n\u001b[2m ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─\u001b[22m\r\n CLAUDE.md: \u001b[31mnone\u001b[39m → \u001b[32m88ba9033\u001b[39m\r\n Allowed Tools: \u001b[31mnone\u001b[39m → \u001b[32m1 tools\u001b[39m\r\n"]
[0.000, "o", " Variant: \u001b[31mvar-1765658807925-81tu78\u001b[39m → \u001b[32mvar-1765678092391-tjbi7b\u001b[39m\r\n"]
[0.000, "o", "\u001b[1m\u001b[22m\r\n\u001b[1m Case Comparison:\u001b[22m\r\n\u001b[1m\u001b[22m\r\n\u001b[2m ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─\u001b[22m\r\n"]
[0.000, "o", "\u001b[2m Case ID Grade 1 Grade 2 Tokens Cost\u001b[22m\r\n"]
[0.000, "o", "\u001b[2m ───────────────────────────────────────────────────────────────────────────\u001b[22m\r\n"]
[0.002, "o", " comp-001 8/10 9/10 12,658 → 8,124 \u001b[32m↓ -35.8%\u001b[39m $0.2134 → $0.1591 \u001b[32m↓ -25.4%\u001b[39m\r\n\u001b[2m\u001b[22m\r\n\u001b[2m ───────────────────────────────────────────────────────────────────────────\u001b[22m\r\n\u001b[1m\u001b[22m\r\n\u001b[1m Aggregate Summary:\u001b[22m\r\n\u001b[1m\u001b[22m\r\n Cases compared: 1\r\n"]
[0.000, "o", " Average grade: 8.0 → 9.0 \u001b[32m↑ +12.5%\u001b[39m\r\n"]
[0.000, "o", " Total tokens: 12,658 → 8,124 \u001b[32m↓ -35.8%\u001b[39m\r\n"]
[0.000, "o", " Total cost: $0.2134 → $0.1591 \u001b[32m↓ -25.4%\u001b[39m\r\n"]
[0.000, "o", "\r\n"]
[3.014, "o", "\r\n✨ A/B test your AI coding setup with real data!\r\n"]
[2.011, "x", "0"]
42 changes: 25 additions & 17 deletions src/cli/commands/compare.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
*/

import chalk from 'chalk';
import { box } from '../../utils/ui';
import { box, padVisible } from '../../utils/ui';
import {
loadRuns,
getRun,
Expand Down Expand Up @@ -167,24 +167,28 @@ export async function compareCommand(run1Id: string, run2Id: string) {
}

// Header for case table
console.log(chalk.dim(' Case ID Grade 1 Grade 2 Tokens Cost'));
console.log(chalk.dim(' Case ID Result 1 Result 2 Tokens Cost'));
console.log(chalk.dim(' ' + '─'.repeat(75)));

// Aggregate metrics
let totalTokens1 = 0;
let totalTokens2 = 0;
let totalCost1 = 0;
let totalCost2 = 0;
let totalGrade1 = 0;
let totalGrade2 = 0;
let gradedCount = 0;
let passCount1 = 0;
let passCount2 = 0;
let resultCount = 0;

for (const caseId of commonCases) {
const case1 = run1.cases[caseId];
const case2 = run2.cases[caseId];

const grade1 = case1.grade !== undefined ? `${case1.grade}/10` : 'N/A';
const grade2 = case2.grade !== undefined ? `${case2.grade}/10` : 'N/A';
const result1Display = case1.result !== undefined
? (case1.result ? chalk.green('Pass') : chalk.red('Fail'))
: chalk.dim('N/A');
const result2Display = case2.result !== undefined
? (case2.result ? chalk.green('Pass') : chalk.red('Fail'))
: chalk.dim('N/A');
Comment thread
coderabbitai[bot] marked this conversation as resolved.

const tokens1 = case1.behaviorMetrics?.totalTokens || 0;
const tokens2 = case2.behaviorMetrics?.totalTokens || 0;
Expand All @@ -195,8 +199,8 @@ export async function compareCommand(run1Id: string, run2Id: string) {
const costDelta = formatMetricDelta(cost1, cost2, true);

console.log(
` ${caseId.padEnd(22)} ${grade1.padEnd(9)} ${grade2.padEnd(9)} ` +
`${tokens1.toLocaleString().padStart(6)} → ${tokens2.toLocaleString().padEnd(6)} ${tokenDelta.padEnd(12)} ` +
` ${caseId.padEnd(22)} ${padVisible(result1Display, 9)} ${padVisible(result2Display, 9)} ` +
`${tokens1.toLocaleString().padStart(6)} → ${tokens2.toLocaleString().padEnd(6)} ${padVisible(tokenDelta, 12)} ` +
`$${cost1.toFixed(4)} → $${cost2.toFixed(4)} ${costDelta}`
);

Expand All @@ -206,23 +210,27 @@ export async function compareCommand(run1Id: string, run2Id: string) {
totalCost1 += cost1;
totalCost2 += cost2;

if (case1.grade !== undefined && case2.grade !== undefined) {
totalGrade1 += case1.grade;
totalGrade2 += case2.grade;
gradedCount++;
if (case1.result !== undefined && case2.result !== undefined) {
if (case1.result) passCount1++;
if (case2.result) passCount2++;
resultCount++;
}
}

// Summary
console.log(chalk.dim('\n ' + '─'.repeat(75)));
console.log(chalk.bold('\n Aggregate Summary:\n'));

const avgGrade1 = gradedCount > 0 ? (totalGrade1 / gradedCount).toFixed(1) : 'N/A';
const avgGrade2 = gradedCount > 0 ? (totalGrade2 / gradedCount).toFixed(1) : 'N/A';
const passRate1 = resultCount > 0 ? Math.round(passCount1 / resultCount * 100) : null;
const passRate2 = resultCount > 0 ? Math.round(passCount2 / resultCount * 100) : null;

console.log(` Cases compared: ${commonCases.length}`);
if (gradedCount > 0) {
console.log(` Average grade: ${avgGrade1} → ${avgGrade2} ${formatMetricDelta(totalGrade1 / gradedCount, totalGrade2 / gradedCount, false)}`);
if (resultCount > 0) {
const passRateDelta = passRate2! - passRate1!;
const deltaSign = passRateDelta >= 0 ? '+' : '';
const deltaColor = passRateDelta > 0 ? chalk.green : passRateDelta < 0 ? chalk.red : chalk.dim;
const arrow = passRateDelta > 0 ? '↑' : passRateDelta < 0 ? '↓' : '→';
console.log(` Pass rate: ${passRate1}% → ${passRate2}% ${deltaColor(`${arrow} ${deltaSign}${passRateDelta}pp`)}`);
}
Comment thread
coderabbitai[bot] marked this conversation as resolved.
console.log(` Total tokens: ${totalTokens1.toLocaleString()} → ${totalTokens2.toLocaleString()} ${formatMetricDelta(totalTokens1, totalTokens2, true)}`);
console.log(` Total cost: $${totalCost1.toFixed(4)} → $${totalCost2.toFixed(4)} ${formatMetricDelta(totalCost1, totalCost2, true)}`);
Expand Down
Loading