Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .agents/skills/precheck-pr/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,12 @@ revision, hardware, configurations, baseline, raw results, and variability. Flag
unsupported claims. Reviewing evidence does not authorize starting new benchmark
campaigns; stay within the user's execution scope and budget.

For the PR's demo/evidence section, follow the
[self-review evidence guidance](../self-review/SKILL.md#pr-demoevidence).
Check provenance, measurement boundaries, asset rights/redaction, and whether
missing evidence is inapplicable or unverified. Do not require a video or GPU
campaign for a change that does not need one.

## Report to the contributor

Lead with actionable findings, ordered by severity, with file/line references,
Expand Down
59 changes: 58 additions & 1 deletion .agents/skills/self-review/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
name: self-review
description: Self-review System1-Omni changes before requesting maintainer review, including deciding whether output-parity checks or controlled A/B measurements are needed. Use for self-review and pre-submit checks in this repository.
description: Self-review System1-Omni changes before requesting maintainer review, including output-parity checks, controlled A/B measurements, and appropriate PR demo/evidence. Use for self-review and pre-submit checks in this repository.
---

# System1-Omni self-review
Expand All @@ -10,6 +10,11 @@ Review the full diff against the target branch for correctness, focused scope,
architecture alignment and tests. Verify that documentation and PR claims match
implemented behavior; report commands, outcomes and checks not run.

This skill prepares a local contributor report. It does not itself authorize edits,
commits, pushes, external posts, or review-status changes; follow separate user
instructions for those actions. Distinguish personally run checks, author-reported
results, and observed CI when reporting evidence.

Read the [architecture contracts](../../../docs/architecture.md). Check that
processing and batch adapters remain separate from forward execution, shared
scheduling owns runtime policy when implemented, and Rust host orchestration
Expand Down Expand Up @@ -39,3 +44,55 @@ GPU access or a harness failure is unverified evidence, not a pass. Remove or
qualify unsupported performance claims. Do not silently change tolerances or
extend the run budget; fixes require regression coverage and a recorded revised
protocol. This skill does not itself authorize external posts or paid execution.

## PR demo/evidence

Prepare a **Demo / evidence** section that shows what changed in inference or
serving, with enough provenance for accurate review and public updates. Select
evidence for the changed behavior; there is no asset quota or requirement to run
production-scale workloads. Nonvisual, docs-only, and test-only changes do not
require video or an expensive GPU campaign. Use `N/A` with a reason for inapplicable
items, and label relevant but unavailable measurements **unverified**, not passing.
Stay within separately authorized resources and the declared run budget.

For a measured comparison, include:

- Exact baseline/head commits (and any local patches), model/checkpoint/tokenizer
revisions, hardware and device count, driver/runtime/toolchain, precision,
workload/input manifest, input shapes/lengths, concurrency/batch settings, and
cache policy. Keep controls matched and explain necessary differences.
- Reproduction commands, dependency/setup requirements, and supported, unsupported,
and untested configurations. Link raw repeated-run results and the protocol;
report sample counts, variation, failures, and exclusions alongside summaries.
- Relevant latency, throughput, memory, and quality/output-parity evidence. State
metric units, denominators, timing boundaries, and memory measurement method.
Mark metrics not measured explicitly; do not imply all metrics improved.
- Separate cold startup/loading, first request/capture, and warmed runs. Keep
kernel/microbenchmark, native inference, and end-to-end HTTP measurements
distinct; a kernel speedup does not establish a user-visible serving speedup.
Distinguish historical/reference validation from tests of the integrated head.

Use latency/throughput charts or memory comparisons when they clarify the raw
measurements, and simple architecture diagrams when they explain the change.
Label planned versus implemented components. A real video can help demonstrate
streaming responsiveness, startup, or an API interaction; show timing honestly,
including cuts, speed changes, stalls, failures, or human intervention. Video is
illustrative evidence of that run, not a substitute for benchmark or parity data.

For each asset, provide a durable, reviewer-accessible link and caption with its
workload, source run/commit, and status as an actual run, recorded replay, or
explanatory illustration. Label replay speed and historical/upstream provenance;
do not present another project's trajectory as a demonstration of this integration.
Never fabricate screens or results, and remove or qualify unsupported claims.

Before attaching publishable assets, check ownership, license/attribution, and
permission to share. Use safe sample data; redact credentials, private URLs,
personal/customer information, and sensitive screen/log content. Check the final
exports, captions, and metadata, retaining useful measurement context. Label
diagrams, mockups, and generated artwork as illustrations. If rights or safe
disclosure are unresolved, omit the asset and explain why. Preparing reusable
evidence does not authorize external posting.

An optional shared demo running System1-Agents on Omni may connect the user task
to the serving change. Pin both repositories and keep task completion evidence
separate from inference performance and parity claims.
10 changes: 10 additions & 0 deletions .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,16 @@

## Test Result

## Demo / evidence

<!-- Follow .agents/skills/self-review/SKILL.md: exact baseline/head, model, hardware,
precision, workload/concurrency, reproduction commands, and relevant raw repeated-run
latency/throughput/memory/quality or parity evidence. Separate cold startup and warm
runs, and kernel/native/HTTP results. Include useful demo/figure links, source
revisions, limitations, asset rights and redaction. Label illustrations separately
from actual runs. Video is optional; use N/A with a reason where evidence does not
apply, and unverified for relevant measurements that could not be made. -->

## Self-review

Before marking this PR ready for review or requesting maintainer review, complete
Expand Down
Loading