local cognition: add tool-calling and orchestration SLM portfolio

Goal

Add a measured local-model portfolio for tool calling/orchestration to z0intelligence rather than treating one model as a universal winner.

z0intelligence owns the cognition/model-selection plane. The runtime harnesses execute the selected action. AODL remains model-agnostic typed intent/IR, JEV/OpenJev remains the bounded fast decision lane, and deterministic capability/security policy remains authoritative.

Seed candidates

Runtime candidates:

nvidia/Nemotron-Orchestrator-8B — learned orchestration over heterogeneous tools/models; explicitly optimized for quality, latency/cost, and user preferences.
Qwen/Qwen3.5-9B — general local reasoning + native agent/tool use candidate.
Qwen/Qwen3.5-4B — smaller/faster general agent candidate; evaluate separately rather than assuming 9B dominates every control task.
MadeAgents/Hammer2.1-3b and optionally Hammer2.1-7b — dedicated multi-step/multi-turn function-calling candidates with irrelevant-tool rejection.
google/functiongemma-270m-it — tiny function-calling foundation intended for task-specific fine-tuning; treat as a specialist/action compiler candidate, not a general chat model.

Training method, not a runtime model entry:

Agent0 — zero-data/self-evolving curriculum + executor training with tool-integrated reasoning. Evolution Lab should own experiments based on this idea.

Primary references:

Model manifest

Introduce or extend a versioned model/capability manifest with fields such as:

model_id / revision
role_tags
parameter_count
license / gating notes
supported runtimes
quantization
measured_vram
context_limit_tested
tool_template / parser
structured-output support
single / parallel / multi-step / multi-turn calls
irrelevant-tool rejection
supports model-as-tool orchestration
cold_start_ms
ttft_ms
decode_tok_s
decision_latency_ms
confidence / abstention behavior
benchmark_receipt_ids

Do not encode benchmark marketing claims as runtime truth. Store source-reported results separately from reproduced local results.

Control-plane split

Target escalation path:

deterministic algorithm / state machine
  -> tiny learned specialist
  -> JEV/OpenJev bounded scorer
  -> local orchestration SLM
  -> local general SLM
  -> remote specialist/frontier fallback

The controller should escalate on uncertainty/risk and capability need, not merely prompt length or a static model tier.

The SLM should choose among already legal actions. Graph legality, dependencies, leases, queues, retries, budgets, permissions and destructive-action authority belong to deterministic runtime mechanisms.

First dogfood target

Use one reversible model/tool-routing lane already aligned with #14. Shadow-run candidate decisions against current behavior, preserve complete distributions/receipts where available, and fail open.

Acceptance

Related: #14.

Cross-repo implementation trackers created from this portfolio:

Evolution Lab tournament + Agent0 track: eval: tool-calling SLM tournament and Agent0-style self-evolution track evolution-lab#23
Tokenomics orchestration telemetry: telemetry: measure orchestration and tool-calling control quality tokenomics#4
Kerdoios local placement: allocation: resource-aware placement for local orchestration SLMs kerdoios#50
OMP backend-neutral SLM adapter: jev: add backend-neutral local orchestration SLM adapter lane oh-my-pi#83
frontier-kb evidence map: research: map small tool-calling models and orchestration methods frontier-kb#27
z0 generated registry/architecture view: registry: surface z0intelligence local cognition portfolio without duplicating it z0#5
Hermes JEV Skills has Issues disabled, so the dogfood scope was attached to PR TypeSafe-conformant local Judge (OpenJev 0.6B/4B) — protocol vs Pydantic/LangChain #1 instead: docs: ground JEV promotion in independent outcomes hermes-jev-skills#1

AODL intentionally gets no model-specific tracker. Its contract stays model-agnostic; learned controllers consume legal typed actions rather than becoming IR node kinds.

Free-capacity execution path now has a concrete DSH credential surface:

DSH fork branch: z0/bws-free-tier-providers
BWS-injected env only: GROQ_API_KEY, CEREBRAS_API_KEY
DSH uses built-in pi-ai Groq/Cerebras catalogs; no duplicate model registry
Kerdoios quota ledger / expiry-aware allocator: quota: per-model daily free-tier ledger for Groq and Cerebras kerdoios#51

z0intelligence should treat Kerdoios' remaining free capacity as an execution constraint/objective, not own provider quota accounting.

Desired behavior:

z0intelligence semantic suitability / uncertainty
        ↓ allowed candidates
Kerdoios capacity + free-quota pressure
        ↓ placement
DSH llm-pi-ai
        ↓
Groq / Cerebras

Important: route real work to compatible free capacity before it expires, but never lower verified quality just to burn quota. Free-only mode must fail closed before paid spill.

Hermes is now the next real dogfood harness for this portfolio: kvnloo/hermes-agent#319. Initial matrix is deterministic baseline + JEV reference where available + Laya + Julia-1 + Qwen 3B/current candidate, all shadow-only until outcome-grounded evidence exists.

results(summarize-faithfulness): v0 measured; groot qwen3-8b does not…

Native OMP trace/stats visibility — plugin-first ownership (2026-10-09)

Kevin correctly challenged placing the complete feature in the OMP repository. The product implementation stays here, using existing omp-extensions/local-cognition, src/z0int/cognition/shadow.py, cognition receipts and Tokenomics; no new z0 subsystem.

Minimal z0intelligence slice:

Observe one real OMP tool_call and a single configured shadow backend; preserve actual native session ID, tool-call ID and stable z0 receipt/workstream identity. Today buildShadowPayload supplies a tool-call ID inside facts, but the Python shadow JSONL receipt does not persist it; random shadow IDs alone are not a reliable join.
Emit one passive, bounded, namespaced session entry via OMP's existing pi.appendEntry without new model-facing messages, tool calls or independent store. Capture successful model invocation separately from request/send success, empty model set, abstain, timeout, invalid proposal, and unavailable.
Reuse the existing /z0int-cognition-status and add a targeted smoke fixture; no automatic activation, serving or model downloads. Preserve session ownership across late async result/session-switch/restart.
Once a native generic trace/stats extension point is qualified, publish compatible derived observations for native /trace/stats, while keeping original z0 receipts authoritative. Compute correctness and savings only from independently joined evaluation/outcome evidence.

The missing generic browser-native OMP consumer is separately isolated as oh-my-pi#148; that issue has been narrowed from the original overbroad product request. OMP's extension API has pi.appendEntry and custom-message renderers, but the current native trace parser skips other custom records and stats ingestion has no extension aggregator; hence no supported plugin-only path to its built-in browser dashboards was found on the inspected fork head. If the installed build exposes such an API, use it and close #148. No OMP-specific z0 policy logic.

Related existing identities/outcomes: #47, #54, #62; Tokenomics#4. This is a source-backed gap/ownership note, not a runtime qualification or claim that the native UI now displays z0. Preserve contributions from OMP's trace/extension maintainers and z0intelligence's local-cognition authors.

READY — ZI-20-S (Claude Code lane): local SLM shadow, session/tool correlation, coverage

Branch / SHA: factory/claude/20-shadow-identity @ d1a9f3a915efa16798913d7958b72e3f7f68d4da (three commits on 4ac01fc: e068f3e, c8da134, a90e3be, d1a9f3a). Not merged.
Paths: omp-extensions/local-cognition/{index.ts,index.test.ts,shadow.real-omp.test.ts,README.md}, src/z0int/cognition/shadow.py, tests/test_cognition_shadow.py. No reuse/bridge core file touched.

Against the plan above:

Native identity. Session id comes from the live handler context (environment variable only as fallback); tool-call id is carried through. The Python receipt now also persists tool_call_id and tool_name, so it joins without the session entry.
Passive session entry. One z0int-cognition-shadow entry per observation via pi.appendEntry; no model-facing message, tool input not copied. Outcomes distinguish model executed, compiler only (empty model set), unavailable, timeout, error and no transport; per model, selected, abstained, invalid call, unavailable, timeout. A model that never ran is not reported as an abstention.
Status and ownership. /z0int-cognition-status reports coverage by outcome. A result arriving after a session switch is not written into the new session; it stays in the status ring marked session_changed.
Native trace consumer: not attempted; oh-my-pi#148 stays design-only.

Tests. bun test omp-extensions/local-cognition/: 20 pass, 1 skip (the gated live test), exit 0. pytest tests/test_cognition_shadow.py tests/test_cognition_surface.py tests/test_cognition_server.py tests/test_bridge_worker.py: 72 passed. New tests were RED before each change.

Live qualification (Z0INT_COGNITION_LIVE_SLM=<model>, real OMP session and extension runtime, real bridge worker, scripted parent model): functiongemma_270m and hammer2.1_3b on the llama.cpp supervisor each gave 3 of 3 tool calls observed, settled as model executed, with session ids, tool-call ids, trace ids and receipt tool-call ids all matching, and zero model-facing messages.

Negative evidence.

The supervisor is not running on this host by default, so the deployed lane currently records every model as unavailable. It was started only for the tests and stopped.
The previous default asked two different models per tool call. Against the single-resident supervisor that never answers: 4 of 4 timeouts at a 4 s budget, 3 of 3 at 60 s. The default is now one model.
Cold load of hammer2.1_3b took about 11 s, above the lane's 4 s timeout; first observations after a load record as timeouts.
In an SDK-created session pi.appendEntry throws until the runtime is initialized; the live test binds it with OMP's own initializeExtensions.

Omitted: correctness or savings claims; nothing here measures whether the shadow model's choice was right. Consumer: Codex for integration; z0evals for any outcome join.

READY — shadow lane: one time budget for the extension and the worker (cold-load behavior)

Candidate: factory/claude/20-shadow-timeout-budget @ c7256f11aee74e1e9a2649e0ad7a224fa272d874; baseline: bd07c5d (tip of factory/claude/20-shadow-identity). Not merged.
Paths: omp-extensions/local-cognition/{index.ts,index.test.ts,README.md}, tests/test_cognition_shadow.py. No Python source change.

Defect. The extension gives up after 4 s but sent no budget; the worker defaulted to 30 s. On a slow model the session entry said timeout with no model detail while the worker kept waiting and later wrote a receipt saying the model answered. Two records of one tool call disagreed.

Fix. The extension sends 80% of its timeout as timeout_ms. Offline, with a worker stand-in and a model needing 250 ms against a 100 ms budget:

	Baseline	Candidate
Budget sent to worker	none (30 s default applies)	80 ms
Session entry	`timeout`, no model rows	`timeout`, model row `timeout`
Worker's conclusion	model answered, after the extension gave up	`timeout`, inside the budget
Records agree	no	yes

Tests. bun test omp-extensions/local-cognition/: 22 pass, 1 skip (gated live test), exit 0; the new extension test failed on the baseline. pytest tests/test_cognition_shadow.py tests/test_cognition_surface.py tests/test_cognition_server.py tests/test_bridge_worker.py: 74 passed. The two new Python tests pin existing worker behavior (it honors timeout_ms on a slow backend, previously untested); they are not RED→GREEN.

Not fixed, by design. A cold load (about 11 s measured for hammer2.1_3b) still exceeds a 4 s budget; those observations are now timeouts in both records. No model service was started for this change. Cost: no model calls.

READY (Claude lane): cold-load timeout behavior, bounded offline tests

Branch factory/claude/20-shadow-timeout-evidence @ f299b1a5706b21aa382b84a68d24176945944567, one commit on c7256f1 (factory/claude/20-shadow-timeout-budget). Files: src/z0int/cognition/shadow.py, tests/test_cognition_shadow.py. No model server was started; every test uses a fake backend.

Defect. A shadow call that hit its budget was written as error: shadow_timeout with latency_ms: 0.0 and nothing else. A model still loading was indistinguishable from an instant miss, and averaging latency over rows pulled the number down.

Change. Timeout rows now carry timed_out: true and waited_ms. latency_ms stays 0 because no inference completed. Rows that failed for another reason (unreachable, not served) carry neither field.

Tests

RED on c7256f1: 2 of the 3 new tests fail with KeyError: 'timed_out'.
GREEN on f299b1a: tests/test_cognition_shadow.py 27 passed; the timeout tests passed on 3 consecutive runs; all tests/test_cognition_*.py 193 passed; extension index.test.ts 22 pass.
New pinned behavior: a call that times out during a load is abandoned, not cancelled. The load finishes in the background and the next call answers. So the first cold call costs one missed observation, not a repeated miss.

Not run. ruff is not installed in this environment. Whole-suite collection has 2 pre-existing errors (tests/test_backend_bench.py, tests/test_tokenomics_bridge.py) unrelated to these files. No live model was used, so the 11 s cold load measured earlier for hammer2.1_3b was not re-measured.

Cost. About 6 tool calls, under 10 s of local test time, no model inference outside this session.

Independent verification of f299b1a, and a follow-up fix for what it found (Claude lane). Supports #20.

Verifier's result on factory/claude/20-shadow-timeout-evidence @ f299b1a: every stated claim reproduced (27 and 193 tests, RED at c7256f1, extension tests 22 pass, the abandoned call finishing its load). Over 900 natural runs at the deadline no mixed row appeared, and waited_ms was correct per row with two models. It found:

A race in my new lines. With a pause injected between the deadline check and building the row, a backend that failed on its own just afterwards produced a row carrying its own error together with timed_out: true. The extension would classify that row differently from the receipt.
Pre-existing: run_shadow promises not to raise, but a budget of 1e18 ms, infinity or a huge integer raised OverflowError, also through the environment override.
Pre-existing, not fixed: abandoned calls are never cancelled, so threads accumulate while a backend never returns (5 live threads before 200 timed-out calls, about 205 after).

Fix: factory/claude/20-shadow-timeout-race @ 9f482632c307f8b20d9a18d110cdb89cea6954f0, one commit on f299b1a.

The timeout fields are set only on rows whose failure is the timeout. A deterministic test reproduces the race: RED before (1 failed, 27 passed), GREEN after.
The wait is capped at one hour. Five new tests: RED before (5 failed), GREEN after. Resolved budgets for ordinary inputs are unchanged (0, negative, NaN and non-numbers still fall back to 30 s; 1 ms still clamps to 100 ms).
tests/test_cognition_shadow.py 33 passed; all tests/test_cognition_*.py 199 passed.
This fix has not had its own independent verification. Pushing started the repository's Core unit workflow, which runs on any push touching src or tests.

Cost. Verifier: 18 tool calls; 65,983 uncached input, 937,253 cache-read, 4,380 output tokens recorded. Fix: about 6 tool calls in the main session, which exposes no per-item token figure.

Keep every shadow budget as it resolved before the cap

Independent verification of 9f48263 found a behavior change I had not declared; fixed (Claude lane). Use factory/claude/20-shadow-timeout-race @ 79104c4765e9d81a5f3db3244980686d4f89c771 (one more commit on 9f48263). Supports #20.

What the verifier found on 9f48263. The race fix holds (removing the new gate makes the test fail). But its 293-row table of resolved budgets showed 81 differences from f299b1a, and two kinds were not intended:

A non-positive or NaN value in the environment override resolved to 30 s where it had been 0.1 s, a 300-fold longer wait. With a 0.5 s backend, a row that had been a timeout became an answer.
An unparsable explicit budget went from raising to a silent 0.1 s.

Fix. Each channel resolves exactly as before the cap existed. The only remaining change is that a value too large for the platform's timers becomes the one-hour cap. Eight new test cases fail on 9f48263 and pass now; 41 and 207 tests pass.

Still as the verifier noted: budgets above one hour that used to run with a larger finite value are now capped, which is the stated intent of the cap. This last commit has not been independently verified.

Cost: verifier 20 tool calls (73,289 uncached input / 1,503,668 cache-read / 5,154 output tokens recorded); fix about 3 tool calls in the main session.

Independent verification of 79104c4 (Claude lane): holds. Supports #20.

The verifier rebuilt the budget table across the three channels and compared f299b1a with 79104c4. Every difference is one of the two intended kinds: a wait above one hour is capped, and 202 inputs that used to raise OverflowError now resolve to the cap. The race fix still fails its test when the gate is removed. 41 and 207 tests pass.

One edge outside the stated claims: an explicit negative budget too large for a float used to raise and now resolves to the 0.1 s minimum. Nothing in the repository passes an explicit budget.

factory/claude/20-shadow-timeout-race @ 79104c4765e9d81a5f3db3244980686d4f89c771 is the head to use for this work.

Cost: 24 tool calls; 70,653 uncached input / 1,522,843 cache-read / 5,838 output tokens recorded.

Independent verification: holds. factory/claude/20-shadow-on-master @ 7914cd962ce7b19ccd1e7e01bd91002766268d4b is the shadow work as a small change on master. Supports #20.

The nine shadow commits were made on top of Codex's unmerged codex/multiharness-integration. This branch applies them directly onto master (5873723) so they can be reviewed on their own: 7 files, +911/-41, instead of about 15,800 lines.

What a separate verifier confirmed, from a fresh clone and a clean environment

Same change. All nine commits match their originals one for one (git range-diff and patch ids), with author, dates, messages and trailers kept. All seven files are byte-identical to the already-verified head 79104c4. Codex's 23 commits touch none of them.
No hidden dependency on Codex's branch. Every import resolves on master. The path from the extension to the worker exists on master: the cognition_shadow operation is listed, dispatched and run there. A request built by the extension and piped to the real bridge worker returned a correlated shadow receipt.
Tests: shadow tests 41 passed (21 on master); all cognition tests 207 passed (187 on master); extension tests 22 pass (12 on master), with the gated live test skipped.
Full offline suite: no test changes result against master; the only difference is the 20 new tests, all passing. One unrelated test (test_agentweb_bridge_overload) failed once while other jobs were running and passed in isolation and on rerun; it is the same flaky test seen in CI earlier today.
Merges with the other two candidates. Master plus this branch plus 08831a2 (feat(capacity): emit read-only capacity snapshot v1 #134) plus 66e1765 (Stop the e2e hot-path tax and sidestep 402/429 off Cursor #141) merge with no conflict and share no files.

Not run: the shadow decision path against a real served model (unit tests with fake backends only), and tests on the three-way merged tree.

Cost. Verifier: 27 tool calls. Building and pushing the branch: about 3 tool calls in the main session.

Bounded slice (2026-10-27): SLM portfolio routing table + tests on `master`

The portfolio branches (feat/slm-portfolio, feat/local-cognition-portfolio, exp/tool-calling-portfolio-v0, integrate/slm-methods-release-20261024) are 56–154 commits ahead of master and too large to land as one PR. On master @ 52aed5bb, manifests/capabilities.v1.json lists every text-worker entry (summarization, research, code_review, coding_implementation, cheap_bounded_worker, including the local qwen2.5:3b lane) with eligible: false, so route() can never pick a worker. Nobody can see that at a glance.

This slice makes the current portfolio state visible and testable. It changes no routing behavior.

Base / branch: base master, branch feat/zo6-slm-routing-table, Draft PR into master.

Allowed write paths (exclusive): src/z0int/routing_table.py (new), tests/test_routing_table.py (new).

Spec

build_table(registry: dict) -> list[dict]: one row per manifest entry with function, provider, model, route, local, eligible, evidenced (use z0int.automatic.evidenced(entry)), and blocker: the first of NOT_ELIGIBLE, NO_EVIDENCE, REMOTE_NEEDS_AUTH (non-local entries), or None.
summary(rows) -> dict: per function, counts of rows, eligible, evidenced, routable (blocker is None), and the list of local SLM candidates.
render_markdown(rows) -> str: a stable table sorted by function, then local first, then provider.
CLI: python -m z0int.routing_table [--manifest PATH] [--json]; default manifest is z0int.intelligence.REGISTRY.
Pure, no network.

Acceptance (tests)

On the committed manifest, summary reports 0 routable rows for each text-worker function, and each such row has blocker NOT_ELIGIBLE (this pins today's truth; the test must be updated on purpose when a lane is promoted).
On a synthetic manifest with one local entry eligible: true and evidence (note evidenced() hashes eval_source.path relative to the repo root; point it at an existing committed file and compute its sha256 in the test), that row has blocker is None and the function's routable == 1.
Non-local eligible+evidenced entries get REMOTE_NEEDS_AUTH.
Markdown output is byte-stable for the same input.
--json output parses.

Test command: pip install -e '.[test]' && pytest -q tests/test_routing_table.py

Next slice after this one (not in scope): promote one local SLM lane with recorded evidence and capture one real routed receipt.

N27-ZO6-ROUTINGTABLE
PR URL: #283 (Draft, base master)
Branch: feat/zo6-slm-routing-table
Head SHA: 7703048
Changed: Added src/z0int/routing_table.py with pure build_table/summary/stable Markdown/CLI over manifests/capabilities.v1.json, using z0int.automatic.evidenced, plus tests/test_routing_table.py covering zero statically routable text workers, synthetic hashed local evidence, remote authorization, blocker precedence, deterministic output, summary, and CLI JSON. No routing behavior change. Only the two allowed files changed.
Remains: execute pytest and review CI; actual local-SLM promotion and a real routed receipt are separate future work. tests NOT RUN - ChatGPT cannot execute code.
Test commands (repo root):
pip install -e '.[test]'
pytest -q tests/test_routing_table.py
python -m z0int.routing_table --json
python -m z0int.routing_table

Navigation Menu

Repository navigation

local cognition: add tool-calling and orchestration SLM portfolio #20

Goal

Seed candidates

Model manifest

Control-plane split

First dogfood target

Acceptance

Native OMP trace/stats visibility — plugin-first ownership (2026-10-09)

Bounded slice (2026-10-27): SLM portfolio routing table + tests on `master`

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

local cognition: add tool-calling and orchestration SLM portfolio #20

Description

Goal

Seed candidates

Model manifest

Control-plane split

First dogfood target

Acceptance

Activity

kvnloo commented on Sep 21, 2026

kvnloo commented on Sep 21, 2026

kvnloo commented on Sep 30, 2026

kvnloo commented on Oct 9, 2026

Native OMP trace/stats visibility — plugin-first ownership (2026-10-09)

kvnloo commented on Oct 10, 2026

kvnloo commented on Oct 10, 2026

kvnloo commented on Oct 10, 2026

kvnloo commented on Oct 10, 2026

kvnloo commented on Oct 10, 2026

kvnloo commented on Oct 10, 2026

kvnloo commented on Oct 10, 2026

kvnloo commented on Oct 27, 2026

Bounded slice (2026-10-27): SLM portfolio routing table + tests on master

kvnloo commented on Oct 28, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Bounded slice (2026-10-27): SLM portfolio routing table + tests on `master`