Docs (vibed) for serious: real documentation with a modern theme

[goodboy]

Add a real docs site: pydata theme, d2 diagrams, tested examples

Motivation

Our docs were years out of date — teaching a long-dead API
(tractor.run(), @stream, "arbiters") on a barebones theme — and
#157 had been asking for a proper modern docs site for ages, with
#175/#126/#123 piling on tutorial/testing/deploy asks.

This rebuilds the docs from the ground up: a 3-column
pydata-sphinx-theme site skinned black-and-white, d2-rendered
diagrams, and prose that teaches ONLY the current API — with every
code block literalinclude-d from examples/ so the docs can't rot
(what you read is what CI runs).

Src of research

Theme + tooling choices were research-backed (the #157 thread plus a
live survey of peer projects),

• pydata-sphinx-theme — numpy / ray / polars all use it today (ray
  migrated off sphinx-book-theme).
• d2 (terrastruct) for diagrams-as-code.
  • https://d2lang.com

Summary of changes

• Stand up the sphinx project on pydata-sphinx-theme 0.18 + sphinx
  9: new docs/conf.py, a docs dependency-group, a minimal b&w
  custom.css, and an opt-in nix develop .#docs shell carrying the
  d2 renderer.
• Add a local .. d2:: directive (docs/_ext/d2diagrams.py) that
  renders d2 sources at build time with a committed-SVG fallback
  (CI needs no d2 bin) + a vendored .. margin::, plus 7
  hand-authored diagrams.
• Write the content tree — landing, an "SC across processes"
  explainer + runtime architecture, 12 task guides (the flagship
  multi-process debugger, Context/streaming, cancellation,
  discovery, infected asyncio, typed msging, testing) and 10
  curated autodoc API pages — and fix the lib docstrings'
  informal-RST so the build is warning-free.
• Add/modernize the examples backing the guides: 5 new gap-filling
  scripts, the dated set moved to .wait_for_result(), and the
  stdlib primes baseline un-hidden — all CI-run.
• Redesign the landing hero as an inline-svg wireframe that flips
  with the light/dark toggle (fill: currentColor), carry it to the
  navbar + README, and brand the navbar polars-style.
• Wire up GitHub Pages: a build-on-PR / deploy-on-main workflow,
  sphinx.ext.githubpages (.nojekyll) + html_baseurl.
• Document the docs workflow itself (notes_to_self/howtodocs.md + a
  dev-tips section + an svgtool.py helper) and log AI provenance
  per NLNet policy.

TODOs before landing

• (c3e5ed1b, 274cab94) Rewrite the
  landing + README main example off ActorNursery.run_in_actor()
  onto @tractor.context / Portal.open_context(). run_in_actor
  is slated to become a hilevel wrapper (runtime/_supervise.py
  "TODO: DEPRECATE THIS"), so the showcase example shouldn't lead
  with it.
• (274cab94, 3753c671,
  1f84d5f5) README cleanup — simplify + strip most of the
  "example codez" section (leave ~just the one example the docs
  landing uses), and swap the |logo| to the transparent-faces
  wireframe so it shows through the GitHub repo-page theme instead of
  the old white-bg block.

Future follow up

• Remaining docs content refinements — prose filling/refining
  across the guides, plus the run_in_actor()-led showcase →
  Context sweep (guide/rpc.rst, guide/spawning.rst). Tracked in
  Docs content refinement pass (follow-up to #460) #472 so this PR lands now as the scaffold/middle-ground.

Links

Closes #157 (the "big boi docs" ask). Partially addresses #175
(tutorials/examples), #126 (testing-tips page) and #123 (gh-pages
deploy).

(this pr content was generated in some part by claude-code)

[Copilot]

Pull request overview

This PR introduces a major Sphinx documentation revamp for tractor, including a modern theme, new guide/API content, diagram support via d2, CI docs builds/deploy, and refreshed example scripts intended to be literalinclude’d by the docs.

Changes:

• Add a docs dependency group (Sphinx + theme + extensions) and a GitHub Actions workflow to build and deploy docs.
• Add a full new docs tree (start/explain/guide/api/project) with custom theme config, CSS, and local Sphinx extensions (.. d2::, .. margin::) plus new diagram sources.
• Add new runnable examples and update older examples to use wait_for_result().

Reviewed changes

Copilot reviewed 58 out of 65 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
pyproject.toml	Adds docs dependency group for building Sphinx docs with the chosen theme/extensions.
.github/workflows/docs.yml	Builds docs on PRs/pushes and deploys to GitHub Pages on main.
examples/uds_transport_actor_tree.py	New example demonstrating actor trees over UDS transport.
examples/typed_payloads.py	New example demonstrating @tractor.context(pld_spec=...) typed payload validation.
examples/streaming_broadcast_fanout.py	New example demonstrating MsgStream.subscribe() fan-out to multiple local tasks.
examples/service_daemon_discovery.py	New example demonstrating discovery + service-daemon pattern via registrar lookups.
examples/nested_actor_tree.py	New example showing a nested actor tree and fan-out RPC aggregation.
examples/parallelism/single_func.py	Updates example to use portal.wait_for_result() (API modernization).
examples/parallelism/concurrent_futures_primes.py	Makes the stdlib baseline script test-runner-friendly via a Trio shim and renames the sync entry function.
examples/actor_spawning_and_causality.py	Updates example to use portal.wait_for_result() (API modernization).
examples/a_trynamic_first_scene.py	Updates example to use wait_for_result() on actor run portals.
docs/conf.py	Reworks Sphinx configuration: pydata theme, extensions, intersphinx, opengraph, copybutton, local _ext/ setup.
docs/_static/css/custom.css	Adds custom CSS skin + margin aside/diagram styling for the new theme.
docs/_ext/marginalia.py	Adds local .. margin:: directive for right-margin asides.
docs/_ext/d2diagrams.py	Adds local .. d2:: directive for rendering/embedding d2 diagrams with SVG fallback.
docs/diagrams/actor_tree.d2	New d2 diagram source for actor supervision tree.
docs/diagrams/context_handshake.d2	New d2 diagram source for Context handshake flow.
docs/diagrams/debug_lock.d2	New d2 diagram source for debugger TTY lock sequencing.
docs/diagrams/error_propagation.d2	New d2 diagram source for one-cancels-all error propagation.
docs/diagrams/infected_aio.d2	New d2 diagram source for infected-asyncio architecture.
docs/diagrams/runtime_stack.d2	New d2 diagram source for runtime layering.
docs/diagrams/streaming_pipeline.d2	New d2 diagram source for streaming pipeline topology.
docs/start/index.rst	Adds “Getting started” landing and toctree.
docs/start/install.rst	Adds installation/platform guidance and uv-based workflows.
docs/start/quickstart.rst	Adds quickstart walkthrough referencing runnable examples and diagrams.
docs/explain/index.rst	Adds conceptual “Big ideas” index.
docs/explain/sc-distributed.rst	Adds conceptual explanation of structured concurrency across processes.
docs/explain/architecture.rst	Adds runtime architecture deep-dive with transport/spawn/registrar overview.
docs/guide/index.rst	Adds guides index and toctree for subsystem walkthroughs.
docs/guide/spawning.rst	Adds spawning/lifetime/supervision guide.
docs/guide/rpc.rst	Adds portal/RPC guide, including wait_for_result() semantics.
docs/guide/context.rst	Adds Context handshake/streaming/cancellation semantics guide.
docs/guide/streaming.rst	Adds streaming guide for one-way and context-based streaming + fan-out.
docs/guide/cancellation.rst	Adds cancellation and error propagation guide.
docs/guide/debugging.rst	Adds multi-process debugging guide (debug_mode, pause, crash REPL, stackscope).
docs/guide/discovery.rst	Adds registrar/discovery guide and service patterns.
docs/guide/clustering.rst	Adds higher-level cluster APIs guide.
docs/guide/parallelism.rst	Adds worker-pool/parallelism guide contrasting stdlib baseline with tractor approach.
docs/guide/asyncio.rst	Adds infected-asyncio guide and cross-loop task-linking semantics.
docs/guide/msging.rst	Adds typed messaging / msgspec protocol guide and validation behavior.
docs/guide/testing.rst	Adds test-suite usage and “examples are tests” conventions.
docs/project/index.rst	Adds project meta index (changelog/dev tips/roadmap pointers).
docs/project/dev-tips.rst	Adds contributor/dev tips (release flow, debugging, logging, monitoring).
docs/project/changelog.rst	Includes NEWS as the changelog source.
docs/api/index.rst	Adds API reference index and toctree.
docs/api/core.rst	Adds runtime/spawning API reference page.
docs/api/context.rst	Adds Context/MsgStream API reference page.
docs/api/ipc.rst	Adds IPC/logging API reference page.
docs/api/msg.rst	Adds typed messaging/codec API reference page.
docs/api/errors.rst	Adds errors/cancellation types API reference page.
docs/api/discovery.rst	Adds discovery/registrar API reference page.
docs/api/devx.rst	Adds debugging/devx API reference page.
docs/api/to_asyncio.rst	Adds asyncio interop API reference page.
docs/api/trionics.rst	Adds trionics helper API reference page.
docs/dev_tips.rst	Removes old dev tips doc (migrated to docs/project/dev-tips.rst).
ai/prompt-io/claude/20260611T175152Z_8526985c_prompt_io.raw.md	Adds AI prompt-io raw metadata snapshot for the docs generation pass.
ai/prompt-io/claude/20260611T175152Z_8526985c_prompt_io.md	Adds AI prompt-io summary snapshot for the docs generation pass.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[goodboy]

Review — xhigh multi-angle pass (correctness + reference integrity + prose accuracy)

Reviewed all 65 files / +7722. Bottom line: genuinely solid — a
CI-like sphinx-build exits 0 (the 24 warnings all trace to
pre-existing library docstrings + NEWS.rst, not this PR), every
literalinclude/:pyobject:/:doc:/toctree reference resolves, all
example scripts run green, and the autodoc API pages have zero broken
directives / failed imports / missing members. Findings below are the
residue; ✅ = fix prepared locally (incoming follow-up commit).

Correctness

1. ✅ examples/uds_transport_actor_tree.py — flagship example prints the wrong address (reproduced).
portal.chan.raddr.sockpath prints …/tractor/registry@1616.sock —
the root registrar's listener socket — not the child's. The UDS
transport derives both laddr/raddr from the same bound listener
path; only raddr.maybe_pid (via SO_PEERCRED) is child-specific.
So the "see the child's socket, no TCP port!" narrative shows a socket
named for the registrar; the child's real uds_child@<pid>.sock
only appears via the separate report_addr RPC. Fixed by relabeling
as the shared listener + leading with the peer pid.

2. ✅ .github/workflows/docs.yml — a PR run could cancel a production deploy.
Workflow-level concurrency: {group: 'pages', cancel-in-progress: true}
is shared by every run; a PR push while a main-merge deploy is mid
actions/deploy-pages cancels the in-flight production deploy. Fixed
by scoping the concurrency block to the deploy job with
cancel-in-progress: false (matches GitHub's Pages starter).

3. ✅ docs/explain/architecture.rst — documents a 'subint' backend not in this branch.
A full paragraph presents start_method='subint' as selectable, but
SpawnMethodKey = Literal['trio','mp_spawn','mp_forkserver'] here —
'subint' raises ValueError. Reworded to "in development / not yet
selectable on this release."

4. ✅ docs/guide/context.rst — tractor.StreamOverrun isn't exported.
StreamOverrun lives in tractor._exceptions, not the tractor
namespace ('StreamOverrun' in dir(tractor) → False); a reader
copying except tractor.StreamOverrun: gets AttributeError. Fixed
to reference the real module.

5. ✅ docs/guide/debugging.rst literalincludes 3 examples still using deprecated .result().
subactor_breakpoint.py, subactor_error.py, shielded_pause.py
(rendered into the flagship debugging page) still called
.result() — the deprecation purged elsewhere in this PR. Swept to
.wait_for_result(). ⚠️ these are pexpect-tested — please re-run
tests/devx/test_debugger.py for them.

6. ✅ docs/start/quickstart.rst — proc-title format wrong + self-contradictory.
Said _subactor[worker_0@<uuid>]; Aid.reprol() yields name@pid,
and architecture.rst already documents _subactor[<name>@<pid>].
Fixed <uuid> → <pid>.

Lower severity (not auto-applied — flagging for your call)

7. docs/_ext/d2diagrams.py — silent stale-SVG on render failure.
With a d2 binary present, a .d2 edit that fails to compile
(returncode != 0) falls back to the committed SVG and the build
still exits 0 (only a log.warning). Consider failing the build when
a render is attempted and errors (distinct from the no-binary
fallback, which is correctly graceful).

8. docs/_ext/d2diagrams.py — parallel_read_safe/parallel_write_safe=True overclaims.
run() writes the shared _diagrams/<stem>.svg during the read
phase; under sphinx-build -j + a d2 binary + the same diagram on two
pages, two workers can write the same file. Harmless in CI today.

9. docs/_ext/d2diagrams.py — output keyed only by src.stem.
diagrams/foo.d2 and other/foo.d2 would collide to one
_diagrams/foo.svg. Latent (all 7 stems unique today).

10. docs/api/context.rst (Context autoclass) renders ~6 docutils errors.
Sourced from tractor/_context.py docstrings (Behaviour: underline,
unexpected indentation, …) — silent in CI (nitpicky=False) but
visible on the page; needs a source-docstring cleanup (already flagged
by the conf.py TODO).

Cross-check note

If the copilot review flags examples/streaming_broadcast_fanout.py
cancel_actor() as "inside the open-context block" — that's a false
positive: the call sits at 8-space indent, outside the block
(canonical teardown). Verified by indentation.

---

🤖 review by Claude Code (/code-review, xhigh effort)

[goodboy]

Review nits #1–6 addressed in 37e7470:

• Glean what we can from aiomas #1 uds_transport_actor_tree.py — relabel the listener socket +
  lead with the per-child peer pid (the prior output printed the root
  registrar's socket as if it were the child's)
• Pump eet? #2 docs.yml — pages concurrency scoped to the deploy job w/
  cancel-in-progress: false (PR builds can no longer cancel an
  in-flight production deploy)
• Enhance the portal (close) API? #3 architecture.rst — 'subint' reframed as in-development (not
  a selectable start_method on this branch)
• standardize RPC multiplexing #4 context.rst — StreamOverrun now points at
  tractor._exceptions (it isn't re-exported from tractor)
• How to do (next gen, SC oriented) remote error serialization for cross host/process propagation?  #5 debugging/ — the 3 literalinclude'd examples swept off the
  deprecated .result() → .wait_for_result()
• Stop forking forkservers #6 quickstart.rst — proc-title corrected to name@pid

The lower-severity items (#7–10: d2-ext render-failure silence,
parallel_*_safe overclaim, stem-collision keying, and the Context
docstring rST cleanup) are left open for a follow-up.

⚠️ #5 touches the pexpect-driven debugger examples — worth a
tests/devx/test_debugger.py run before merge.

[goodboy]

Hmm flaky macos test (latency issues) which should be resolved by #468, so I'm likely just gonna 🤠 merge this.