Skip to content

Add guarded long-context benchmark tooling - #588

Closed
francescogiunta wants to merge 1 commit into
antirez:mainfrom
francescogiunta:gb10/efdadd4-baseline
Closed

Add guarded long-context benchmark tooling#588
francescogiunta wants to merge 1 commit into
antirez:mainfrom
francescogiunta:gb10/efdadd4-baseline

Conversation

@francescogiunta

Copy link
Copy Markdown

Summary

  • add a Linux process-group memory guard for model-backed benchmark runs
  • add an optional decode warmup window while preserving the existing CSV columns and meanings
  • add a deterministic fact-recall fixture sized for 65K and 100K context frontiers
  • add synthetic tests and usage documentation for the new tooling

Motivation

Long-context measurements on unified-memory systems need an explicit safety envelope and a reproducible workload. Decode throughput also benefits from reporting a post-warmup window without changing existing benchmark metrics.

Compatibility

The warmup defaults to zero. The original nine CSV columns remain in their existing order and keep their current meaning; the warmup and measured-window fields are appended. The canonical story fixture is not modified.

Validation

  • make test-memory-guard — 7 tests passed
  • make test-bench-metrics — C metrics and CLI tests passed
  • deterministic fixture regeneration produced SHA-256 31cb363cd5b5b7eaccffdf9c464ba491d6afea46b54b4d5baf3dde36299d7436
  • git diff --check passed

No model-backed performance result is claimed in this draft; those runs remain guarded campaign work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant