Skip to content

P6.0-P6.2 runtime benchmark contract scaffold - #8

Merged
Harzva merged 1 commit into
main-devfrom
p5-real-callback-pilot
Jun 21, 2026
Merged

P6.0-P6.2 runtime benchmark contract scaffold#8
Harzva merged 1 commit into
main-devfrom
p5-real-callback-pilot

Conversation

@Harzva

@Harzva Harzva commented Jun 21, 2026

Copy link
Copy Markdown
Owner

Summary

  • Adds the P6.0 benchmark taxonomy contract for mobile task categories, runtime assertions, and score dimensions.
  • Adds the P6.1 communication substrate contract: MailboxMessage, EvidenceLedgerEntry, RuntimeEvent, MemoryCommitProposal, and role allowlists/budgets/input filters/return contracts.
  • Adds the P6.2 runtime verifier scaffold and a validator-compatible non-counted run package.

Non-counted Boundary

This is a P6.0-P6.2 contract/scaffold upgrade only. It is not a formal benchmark and must not be reported as counted strategy ablation evidence.

  • run_kind=strategy_pilot_not_counted
  • counts_as_experiment=false
  • counts_as_strategy_ablation_result=false
  • no model callback, provider claim, device action, raw transcript, or secret material is included

Evidence

  • Contract doc: docs/mobile-harness-benchmark/strategy-ablation/p60-p62-runtime-benchmark-contract.md
  • Run package: docs/mobile-harness-benchmark/strategy-ablation/runs/p60-p62-runtime-benchmark-upgrade/
  • Run JSON: docs/mobile-harness-benchmark/strategy-ablation/runs/p60-p62-runtime-benchmark-upgrade/run.json
  • Verifier JSON: docs/mobile-harness-benchmark/strategy-ablation/runs/p60-p62-runtime-benchmark-upgrade/runtime_benchmark_upgrade_verifier.json
  • Summary: docs/mobile-harness-benchmark/strategy-ablation/runs/p60-p62-runtime-benchmark-upgrade/summary.md

Verification

  • python3 -m py_compile scripts/run_mobile_harness_strategy_p60_p62_runtime_benchmark_upgrade.py scripts/validate_mobile_harness_strategy_ablation.py
  • python3 scripts/validate_mobile_harness_strategy_ablation.py --registry docs/mobile-harness-benchmark/strategy-ablation/strategy_registry.json --run docs/mobile-harness-benchmark/strategy-ablation/runs/p60-p62-runtime-benchmark-upgrade/run.json
    • PASS, 6 strategies, 18 results, strategy_pilot_not_counted
  • git diff HEAD^..HEAD --check
  • staged/commit secret and private path scan: no findings
  • boundary assertions passed for counts_as_experiment=false and counts_as_strategy_ablation_result=false

Review Focus

  • scripts/run_mobile_harness_strategy_p60_p62_runtime_benchmark_upgrade.py
  • docs/mobile-harness-benchmark/strategy-ablation/p60-p62-runtime-benchmark-contract.md
  • docs/mobile-harness-benchmark/strategy-ablation/runs/p60-p62-runtime-benchmark-upgrade/
  • docs/mobile-harness-roadmp/roadmp2026.6.20-p6-mobile-agent-coevolution.md
  • docs/mobile-harness-roadmp/roadmp2026.6.20.md

Next Step After Merge

Enter P6.3 Android real device lane: install the latest APK, grant or verify MobileCode Phone Use Accessibility service, run dry/action probes, and save screenshot, UI XML, logcat, focus state, and WebView state assertions. Keep that lane non-counted until repeated samples and promotion gates pass.

@Harzva
Harzva merged commit 82c4098 into main-dev Jun 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant