diff --git a/.agents/skills/precheck-pr/SKILL.md b/.agents/skills/precheck-pr/SKILL.md index b9cec776..0cada6b9 100644 --- a/.agents/skills/precheck-pr/SKILL.md +++ b/.agents/skills/precheck-pr/SKILL.md @@ -68,6 +68,12 @@ revision, hardware, configurations, baseline, raw results, and variability. Flag unsupported claims. Reviewing evidence does not authorize starting new benchmark campaigns; stay within the user's execution scope and budget. +For the PR's demo/evidence section, follow the +[self-review evidence guidance](../self-review/SKILL.md#pr-demoevidence). +Check provenance, measurement boundaries, asset rights/redaction, and whether +missing evidence is inapplicable or unverified. Do not require a video or GPU +campaign for a change that does not need one. + ## Report to the contributor Lead with actionable findings, ordered by severity, with file/line references, diff --git a/.agents/skills/self-review/SKILL.md b/.agents/skills/self-review/SKILL.md index e40bbf50..4a634785 100644 --- a/.agents/skills/self-review/SKILL.md +++ b/.agents/skills/self-review/SKILL.md @@ -1,6 +1,6 @@ --- name: self-review -description: Self-review System1-Omni changes before requesting maintainer review, including deciding whether output-parity checks or controlled A/B measurements are needed. Use for self-review and pre-submit checks in this repository. +description: Self-review System1-Omni changes before requesting maintainer review, including output-parity checks, controlled A/B measurements, and appropriate PR demo/evidence. Use for self-review and pre-submit checks in this repository. --- # System1-Omni self-review @@ -10,6 +10,11 @@ Review the full diff against the target branch for correctness, focused scope, architecture alignment and tests. Verify that documentation and PR claims match implemented behavior; report commands, outcomes and checks not run. +This skill prepares a local contributor report. It does not itself authorize edits, +commits, pushes, external posts, or review-status changes; follow separate user +instructions for those actions. Distinguish personally run checks, author-reported +results, and observed CI when reporting evidence. + Read the [architecture contracts](../../../docs/architecture.md). Check that processing and batch adapters remain separate from forward execution, shared scheduling owns runtime policy when implemented, and Rust host orchestration @@ -39,3 +44,55 @@ GPU access or a harness failure is unverified evidence, not a pass. Remove or qualify unsupported performance claims. Do not silently change tolerances or extend the run budget; fixes require regression coverage and a recorded revised protocol. This skill does not itself authorize external posts or paid execution. + +## PR demo/evidence + +Prepare a **Demo / evidence** section that shows what changed in inference or +serving, with enough provenance for accurate review and public updates. Select +evidence for the changed behavior; there is no asset quota or requirement to run +production-scale workloads. Nonvisual, docs-only, and test-only changes do not +require video or an expensive GPU campaign. Use `N/A` with a reason for inapplicable +items, and label relevant but unavailable measurements **unverified**, not passing. +Stay within separately authorized resources and the declared run budget. + +For a measured comparison, include: + +- Exact baseline/head commits (and any local patches), model/checkpoint/tokenizer + revisions, hardware and device count, driver/runtime/toolchain, precision, + workload/input manifest, input shapes/lengths, concurrency/batch settings, and + cache policy. Keep controls matched and explain necessary differences. +- Reproduction commands, dependency/setup requirements, and supported, unsupported, + and untested configurations. Link raw repeated-run results and the protocol; + report sample counts, variation, failures, and exclusions alongside summaries. +- Relevant latency, throughput, memory, and quality/output-parity evidence. State + metric units, denominators, timing boundaries, and memory measurement method. + Mark metrics not measured explicitly; do not imply all metrics improved. +- Separate cold startup/loading, first request/capture, and warmed runs. Keep + kernel/microbenchmark, native inference, and end-to-end HTTP measurements + distinct; a kernel speedup does not establish a user-visible serving speedup. + Distinguish historical/reference validation from tests of the integrated head. + +Use latency/throughput charts or memory comparisons when they clarify the raw +measurements, and simple architecture diagrams when they explain the change. +Label planned versus implemented components. A real video can help demonstrate +streaming responsiveness, startup, or an API interaction; show timing honestly, +including cuts, speed changes, stalls, failures, or human intervention. Video is +illustrative evidence of that run, not a substitute for benchmark or parity data. + +For each asset, provide a durable, reviewer-accessible link and caption with its +workload, source run/commit, and status as an actual run, recorded replay, or +explanatory illustration. Label replay speed and historical/upstream provenance; +do not present another project's trajectory as a demonstration of this integration. +Never fabricate screens or results, and remove or qualify unsupported claims. + +Before attaching publishable assets, check ownership, license/attribution, and +permission to share. Use safe sample data; redact credentials, private URLs, +personal/customer information, and sensitive screen/log content. Check the final +exports, captions, and metadata, retaining useful measurement context. Label +diagrams, mockups, and generated artwork as illustrations. If rights or safe +disclosure are unresolved, omit the asset and explain why. Preparing reusable +evidence does not authorize external posting. + +An optional shared demo running System1-Agents on Omni may connect the user task +to the serving change. Pin both repositories and keep task completion evidence +separate from inference performance and parity claims. diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index c0f2d18a..236829ca 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -8,6 +8,16 @@ ## Test Result +## Demo / evidence + + + ## Self-review Before marking this PR ready for review or requesting maintainer review, complete