Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
63 commits
Select commit Hold shift + click to select a range
339eae5
ci(e2e): reuse prebuilt CLI and gateway artifacts (#2311)
elezar Jul 20, 2026
80987e9
docs: fix broken links and small inconsistencies (#2329)
mesutoezdil Jul 20, 2026
a2cd5f8
fix(gateway): honor tty flag for interactive exec (#2315)
emonq Jul 20, 2026
a9f7131
fix(ci): grant E2E permissions to release workflows (#2376)
pimlock Jul 20, 2026
f32c46d
chore(ci): pin pr gate action (#2368)
elezar Jul 20, 2026
2575585
chore(deps): bump actions/attest from 4.1.1 to 4.2.0 (#2357)
dependabot[bot] Jul 20, 2026
9377e0d
fix(providers): allow git clone/fetch via default GitHub provider (#2…
russellb Jul 20, 2026
745512e
fix(build): raise open-file limit for host musl cross-compile on macO…
purp Jul 20, 2026
ad29ab9
fix(supervisor-network): warn on unsupported L7 access presets (#2177)
lunarwhite Jul 21, 2026
5952a5a
feat(workspace): add workspace resource model with scoping, membershi…
derekwaynecarr Jul 21, 2026
f169084
fix(supervisor): tailor Landlock rights by inode type (#2380)
drew Jul 21, 2026
8d9502d
perf(build): share sccache across worktrees (#2379)
matthewgrossman Jul 21, 2026
e9ac0ee
chore(deps): bump actions/checkout from 7.0.0 to 7.0.1 (#2381)
dependabot[bot] Jul 21, 2026
744a65d
fix(driver-podman): avoid panic when HOME is unset on macOS (#2327)
mesutoezdil Jul 21, 2026
3ff15a1
fix(ci): fix mirror SHA detection in e2e-label-help workflow (#2236)
rhuss Jul 21, 2026
dae9261
docs(agents): keep project skills synchronized (#2349)
pimlock Jul 21, 2026
472e23f
fix(proxy): include OPA deny reason in CONNECT 403 response (#2363)
zanetworker Jul 21, 2026
bdd1ce8
fix(cli): respect CARGO_TARGET_DIR in openshell wrapper script (#2391)
jhjaggars Jul 21, 2026
396a3b7
docs(brand): add OpenShell brand assets (#2398)
drew Jul 21, 2026
d35d52d
fix(dco): Fix mismatched wording that breaks initial DCO checks (#2399)
matthewgrossman Jul 21, 2026
8b0e54b
docs(extensibility): add gateway interceptor guide (#2397)
drew Jul 21, 2026
2d5652b
docs(docker-compose): replace removed OpenClaw community sandbox with…
Schimuneck Jul 22, 2026
ac3d5c9
docs(prover): correct prove() exit code doc comment (#2395)
eviehoward Jul 22, 2026
cbdeb4d
fix(server): prevent unrelated sandbox deletes from blocking deletion…
pimlock Jul 22, 2026
fd1d3de
fix(e2e): detect gateway workload for health port-forward (#2400)
krishicks Jul 22, 2026
5432d01
feat(tui): add config key support to provider create/update forms (#2…
letv1nnn Jul 22, 2026
ca31805
fix(vm): reduce registry rootfs staging pressure (#2425)
pimlock Jul 22, 2026
cd9a0bf
fix(driver-k8s): add label selector to sandbox watch stream and list …
rhuss Jul 22, 2026
7b444bd
fix(agents): make baked payload readable (#2419)
elezar Jul 22, 2026
0674a00
refactor(cli): extract shared helpers into commands/common module (#2…
varshaprasad96 Jul 22, 2026
8a14b3a
fix(sandbox): skip read-only mounts during recursive chown of /sandbo…
varshaprasad96 Jul 22, 2026
541b97f
fix(mise): initialize Python dependencies in fresh worktrees (#2429)
matthewgrossman Jul 22, 2026
1d4ac70
fix(policy): keep internal allowed IP proposals pending (#2416)
alangou Jul 23, 2026
75d2468
fix(examples): add missing workspace fields to governance interceptor…
pavelanni Jul 23, 2026
59f7839
fix(auth): report gateway authentication status (#2435)
drew Jul 23, 2026
21da343
refactor(supervisor): pass agent proposal state explicitly (#2421)
elezar Jul 24, 2026
f7cd910
fix: eliminate parallel Rust test flakes (#2434)
pimlock Jul 24, 2026
b422b67
feat(cli): add --output json/yaml to sandbox get, status, and sandbox…
rhuss Jul 24, 2026
850bd42
refactor(tui): default create form state (#2458)
elezar Jul 24, 2026
77e5c32
feat(sandbox,gateway): route sandbox egress through corporate HTTP pr…
feloy Jul 24, 2026
01daf3a
chore(deps): bump astral-sh/setup-uv from 8.3.2 to 9.0.0 (#2440)
dependabot[bot] Jul 24, 2026
deced87
refactor(policy): extract shared L7 endpoint validation (#2389)
gracesmith6504 Jul 24, 2026
d4cd37b
chore(deps): bump docker/login-action from 4.4.0 to 4.5.1 (#2488)
dependabot[bot] Jul 27, 2026
afb462f
fix(router): strip unsupported Anthropic beta fields from Vertex rawP…
zanetworker Jul 27, 2026
516be60
fix: assorted byte-index slicing safety fixes (#2452)
andrewwhitecdw Jul 27, 2026
76a5397
fix: assorted arithmetic and indexing robustness fixes (#2451)
andrewwhitecdw Jul 27, 2026
2d10881
fix(policy): avoid panic truncating multi-byte UTF-8 paths for displa…
andrewwhitecdw Jul 27, 2026
39bf94e
fix(server): bind gateway listeners before sandbox resume (#2495)
elezar Jul 27, 2026
79bcf29
fix(proxy): retry with backoff on transient accept errors instead of …
politerealism Jul 27, 2026
2022d53
fix(tasks): scope pre-commit to lint checks (#2503)
drew Jul 27, 2026
0d5e5c5
fix(cli): write exec stderr events to stderr in interactive mode (#2447)
andrewwhitecdw Jul 27, 2026
52f9e9e
fix(cli): eliminate flaky subprocess integration tests (#2504)
rhuss Jul 27, 2026
24d491a
refactor(cli): extract gateway commands into commands/gateway module …
varshaprasad96 Jul 28, 2026
f00ad23
fix(podman): tolerate shutdown transport closes (#2498)
elezar Jul 28, 2026
7e9a7f5
fix(sandbox): gate Linux-only ordering import (#2513)
elezar Jul 28, 2026
b78c861
fix(server): stabilize flaky delete telemetry unit test (#2521)
derekwaynecarr Jul 28, 2026
efb2d9c
fix(e2e): bound podman stop timeout in tests (#2516)
elezar Jul 28, 2026
b1c7ff6
ci: add focused macOS Rust lint (#2515)
elezar Jul 28, 2026
2b7f04f
feat(examples): add supervisor middleware content guard (#2169)
pimlock Jul 28, 2026
7955c83
feat(k8s): support configuring workspace PVC storageClassName (#2463)
loveRhythm1990 Jul 29, 2026
bc14018
feat(sandbox): use policy-first OCI image identity (#2509)
matthewgrossman Jul 29, 2026
101cbc9
fix(cli): avoid panic on multi-byte UTF-8 in --since duration (#2446)
andrewwhitecdw Jul 29, 2026
7c06aec
fix(supervisor): quote nft log prefix in bypass rules
gracesmith6504 Jul 29, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
6 changes: 4 additions & 2 deletions .agents/skills/build-from-issue/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -377,7 +377,7 @@ Verification has two phases: unit tests + pre-commit, then E2E tests (if applica
On each attempt:

```bash
# Run pre-commit checks (includes unit tests, linting, formatting)
# Run pre-commit checks (linting, formatting, license headers)
mise run pre-commit
```

Expand Down Expand Up @@ -444,6 +444,8 @@ same branch. If the change affects user-facing compute-driver setup, also
update `docs/reference/sandbox-compute-drivers.mdx` or the relevant deployment
page.

Use the `sync-agent-infra` skill's maintenance map to identify related skill updates when the implementation changes behavior, commands, or development workflows. Run its full consistency check when the implementation adds, removes, or renames skills or crates; changes workflow relationships or skill coverage; modifies issue or PR templates; or changes agent cross-references. Fix any drift before committing.

### Step 12: Commit and Push

Commit all changes using conventional commit format. The `<type>` comes from the issue type in the plan:
Expand Down Expand Up @@ -638,7 +640,7 @@ If the `state:in-progress` label is present, the skill was previously started bu
| `gh pr list --state open --search "..."` | Search for open PRs |
| `gh pr create --title "..." --body "..."` | Create a pull request |
| `gh api user --jq '.login'` | Get current GitHub username |
| `mise run pre-commit` | Run pre-commit checks (includes unit tests, lint, format) |
| `mise run pre-commit` | Run pre-commit checks (lint, format, license headers) |
| `mise run e2e:docker` | Run smoke E2E against a standalone Docker-backed gateway |
| `mise run e2e:podman` | Run smoke E2E against a Podman-backed gateway |
| `mise run e2e:vm` | Run smoke E2E against the VM compute driver |
Expand Down
4 changes: 4 additions & 0 deletions .agents/skills/create-github-pr/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,10 @@ in the same branch. If the change affects user-facing compute-driver setup,
also update `docs/reference/sandbox-compute-drivers.mdx` or the relevant
deployment docs.

### Check Agent Infrastructure

Use the `sync-agent-infra` skill's maintenance map to identify related skill updates when the branch changes behavior, commands, or development workflows. Run its full consistency check when the branch adds, removes, or renames skills or crates; changes workflow relationships or skill coverage; modifies issue or PR templates; or changes agent cross-references. Resolve any drift before creating the PR.

### Run Pre-commit Checks

Run the local pre-commit task before opening a PR:
Expand Down
74 changes: 57 additions & 17 deletions .agents/skills/debug-inference/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
name: debug-inference
description: Debug why inference.local or external inference setup is failing. Use when the user cannot reach a local model server, has provider base URL issues, sees inference verification failures, hits protocol mismatches, or needs to diagnose inference on local vs remote gateways. Trigger keywords - debug inference, inference.local, local inference, ollama, vllm, sglang, trtllm, NIM, inference failing, model server unreachable, failed to verify inference endpoint, host.openshell.internal.
description: Debug why inference.local, direct external inference, or supervisor-only system inference is failing. Use when the user cannot reach a local model server, has provider base URL issues, sees inference verification failures, hits protocol mismatches, or needs to diagnose inference on local vs remote gateways. Trigger keywords - debug inference, inference.local, system inference, sandbox-system, local inference, ollama, vllm, sglang, trtllm, NIM, inference failing, model server unreachable, failed to verify inference endpoint, host.openshell.internal.
---

# Debug Inference
Expand All @@ -11,7 +11,7 @@ Use `openshell` CLI commands to inspect the active gateway, provider records, ma

## Overview

OpenShell supports two different inference paths. Diagnose the correct one first.
OpenShell supports three inference paths. Diagnose the correct one first.

1. **Managed inference** through `https://inference.local`
- Configured by `openshell inference set`
Expand All @@ -21,6 +21,10 @@ OpenShell supports two different inference paths. Diagnose the correct one first
- Controlled by `network_policies`
- Requires the application to call the external host directly
- Requires provider attachment and network access to be configured separately
3. **System inference** used by platform functions
- Configured by `openshell inference set --system`
- Uses the `sandbox-system` route
- Consumed in-process by the sandbox supervisor and not exposed to sandbox user code through `inference.local`

For local or self-hosted engines such as Ollama, vLLM, SGLang, TRT-LLM, and many NIM deployments, the most common managed inference pattern is an `openai` provider with `OPENAI_BASE_URL` pointing at a host the gateway can reach.

Expand All @@ -38,10 +42,13 @@ Use these commands first:
# Which gateway is active, and can the CLI reach it?
openshell status

# Show managed inference config for inference.local
# Show both the user-facing and system inference routes
openshell inference get

# Inspect the provider record referenced by inference.local
# Show only the supervisor-only system route
openshell inference get --system

# Inspect the provider record referenced by the relevant route
openshell provider get <provider-name>

# Inspect gateway topology details when remote/local confusion is suspected
Expand All @@ -59,9 +66,9 @@ When the user asks to debug inference, run diagnostics automatically in this ord

Establish these facts first:

1. Is the application calling `https://inference.local` or a direct external host?
1. Is sandbox code calling `https://inference.local`, is the application calling a direct external host, or is a platform function using system inference?
2. Which gateway is active, and is it local, remote, or cloud?
3. Which provider and model are configured for managed inference?
3. Which provider, model, and timeout are configured for the relevant route?
4. Is the upstream local to the gateway host, or somewhere else?

### Step 0: Check the Active Gateway
Expand All @@ -83,23 +90,31 @@ Common mistake:

- **Laptop-local model + remote gateway**: `host.openshell.internal` points to the remote gateway host, not your laptop. A laptop-local Ollama or vLLM server will not be reachable without a tunnel or shared reachable network path.

### Step 1: Check Whether Managed Inference Is Configured
### Step 1: Check Whether the Relevant Route Is Configured

Run:

```bash
openshell inference get
openshell inference get --system
```

Interpretation:

- **`Not configured`**: `inference.local` has no backend yet. Fix by configuring it:
- `openshell inference get` shows both the user-facing `inference.local` route and the system route. `--system` isolates the system route.
- **The `inference.local` route is `Not configured`**: managed inference has no backend. Configure it without `--system`:

```bash
openshell inference set --provider <name> --model <id>
```

- **Provider and model shown**: Continue to provider inspection.
- **System inference is `Not configured`**: platform functions have no system backend. Configure it separately:

```bash
openshell inference set --system --provider <name> --model <id>
```

- **Provider, model, and timeout shown**: Continue to provider inspection for the relevant route.

### Step 2: Inspect the Provider Record

Expand All @@ -111,10 +126,13 @@ openshell provider get <provider-name>

Check:

- Provider type matches the client API shape
- Provider type matches the client API shape and is supported for managed inference
- `openai` for OpenAI-compatible engines such as Ollama, vLLM, SGLang, TRT-LLM, and many NIM deployments
- `anthropic` for Anthropic Messages API
- `nvidia` for NVIDIA-hosted OpenAI-compatible endpoints
- `deepinfra` for DeepInfra's OpenAI-compatible endpoint
- `google-vertex-ai` for Vertex AI; Claude models use Anthropic Messages and other models use OpenAI Chat Completions
- `aws-bedrock` only through a configured Bedrock-compatible bridge today
- Required credential key exists
- `*_BASE_URL` override is correct when using a self-hosted endpoint

Expand All @@ -123,9 +141,11 @@ Fix examples:
```bash
openshell provider create --name ollama --type openai --credential OPENAI_API_KEY=empty --config OPENAI_BASE_URL=http://host.openshell.internal:11434/v1

openshell provider update ollama --type openai --credential OPENAI_API_KEY=empty --config OPENAI_BASE_URL=http://host.openshell.internal:11434/v1
openshell provider update ollama --credential OPENAI_API_KEY=empty --config OPENAI_BASE_URL=http://host.openshell.internal:11434/v1
```

`provider update` preserves the provider type and does not accept `--type`. Prefer bare credential keys, such as `--credential OPENAI_API_KEY`, when reading a real secret from the CLI environment.

### Step 3: Check Local Host Reachability

For host-backed local inference, confirm the upstream server:
Expand All @@ -142,21 +162,25 @@ Common mistakes:

### Step 4: Check Request Shape

Managed inference only works for `https://inference.local` and supported inference API paths.
User-facing managed inference only works for `https://inference.local` and supported inference API paths.

Supported patterns include:

- `POST /v1/chat/completions`
- `POST /v1/completions`
- `POST /v1/responses`
- `POST /v1/embeddings`
- `POST /v1/messages`
- `GET /v1/models`
- `GET /v1/models/*`
- `POST /model/{modelId}/invoke` for bridge-fronted `aws-bedrock`

Common mistakes:

- **Wrong scheme**: `http://inference.local` instead of `https://inference.local`
- **Unsupported path**: request does not match a known inference API
- **Protocol mismatch**: Anthropic client against an `openai` provider, or vice versa
- **Provider-specific mismatch**: Vertex Claude requests must use `/v1/messages`; other Vertex models currently use `/v1/chat/completions`; Bedrock uses its model-in-path invoke shape

Fix guidance:

Expand All @@ -166,6 +190,8 @@ Fix guidance:

### Step 5: Probe from a Sandbox

This probe validates the user-facing `inference.local` route. It does not exercise supervisor-only system inference.

Run a minimal request from inside a sandbox:

```bash
Expand All @@ -179,21 +205,28 @@ Interpretation:
- **`no compatible route`**: provider type and client API shape do not match
- **Connection refused / upstream unavailable / verification failures**: base URL, bind address, topology, or credentials are wrong

For system inference failures, inspect the platform function and sandbox supervisor/network logs after confirming `openshell inference get --system`. User code cannot call the `sandbox-system` route directly.

### Step 6: Reapply or Repair the Managed Route

After fixing the provider, repoint `inference.local`:
After fixing the provider, use `update` for a partial change or `set` to replace the route:

```bash
openshell inference set --provider <name> --model <id>
openshell inference update --provider <name>
openshell inference update --model <id>
openshell inference update --timeout 120
```

Add `--system` to target the system route. Without it, these commands target `inference.local`. A timeout of `0` uses the 60-second default; increase it for models with long reasoning or idle streaming phases.

If the endpoint is intentionally offline and you only want to save the config:

```bash
openshell inference set --provider <name> --model <id> --no-verify
```

Inference updates are hot-reloaded to all sandboxes on the active gateway within about 5 seconds by default.
Use `--no-verify` only when the endpoint is intentionally offline or the provider protocol cannot be verified, such as the current bridge-fronted Bedrock flow. Inference updates are hot-reloaded to running sandboxes within about 5 seconds by default.

### Step 7: Diagnose Direct External Inference

Expand All @@ -203,9 +236,11 @@ Check instead:

1. The application is configured to call the external hostname directly
2. A provider with the needed credentials exists
3. The sandbox is launched with that provider attached
3. The sandbox has that provider attached (`openshell sandbox provider list [name]`)
4. `network_policies` allow that host, port, and HTTP rules

Attach or detach a provider on an existing sandbox with `openshell sandbox provider attach <sandbox> <provider>` and `openshell sandbox provider detach <sandbox> <provider>`.

Use the `generate-sandbox-policy` skill when the user needs help authoring policy YAML.

## Fix: Local Host Inference Timeouts (Firewall)
Expand Down Expand Up @@ -305,11 +340,13 @@ Both commands should return the upstream model list.
| Symptom | Likely cause | Fix |
|---------|--------------|-----|
| `openshell inference get` shows `Not configured` | No managed inference route configured | `openshell inference set --provider <name> --model <id>` |
| System inference is `Not configured` | Platform-only route has no backend | `openshell inference set --system --provider <name> --model <id>` |
| `failed to verify inference endpoint` | Bad base URL, wrong credentials, wrong provider type, or upstream not reachable | Fix provider config, then rerun `openshell inference set`; use `--no-verify` only when the endpoint is intentionally offline |
| Base URL uses `127.0.0.1` | Loopback points at the wrong runtime | Use `host.openshell.internal` or another gateway-reachable host |
| Local engine works only when gateway is local | Gateway moved to remote host | Run the engine on the gateway host, add a tunnel, or use direct external access |
| `connection not allowed by policy` on `inference.local` | Unsupported path or method | Use a supported inference API path |
| `no compatible route` | Provider type does not match request shape | Switch provider type or change the client API |
| `no compatible route` | Provider type does not match request shape | Create or select a provider of the matching type, or change the client API |
| `inference.local` works but a platform function fails | User route is configured but `sandbox-system` is missing or wrong | `openshell inference get --system`; configure or update with `--system`; inspect supervisor logs |
| Direct call to external host is denied | Missing policy or provider attachment | Update `network_policies` and launch sandbox with the right provider |
| SDK fails on empty auth token | Client requires a non-empty API key even though OpenShell injects the real one | Use any placeholder token such as `test` |
| Upstream timeout from container to host-local backend | Host firewall or network config blocks container-to-host traffic | Allow the Docker bridge subnet to reach the inference port on the host gateway IP (see firewall fix section above) |
Expand All @@ -328,6 +365,9 @@ openshell gateway info
echo "=== Managed Inference ==="
openshell inference get

echo "=== System Inference Only ==="
openshell inference get --system

echo "=== Providers ==="
openshell provider list

Expand All @@ -340,7 +380,7 @@ openshell sandbox create -- curl https://inference.local/v1/chat/completions --j

When you report back, state:

1. Which inference path is failing (`inference.local` vs direct external)
1. Which inference path is failing (`inference.local`, direct external, or system inference)
2. Whether gateway topology is part of the problem
3. The most likely root cause
4. The exact fix commands the user should run
Loading
Loading