Trying out sub-interpreters (subints), maybe `fork()` can be hacked now?'

[goodboy]

As per the new pub API dropping in 3.14,

• https://docs.python.org/3.14/library/concurrent.interpreters.html

  • https://docs.python.org/3.14/library/concurrent.interpreters.html#basic-usage
  • https://peps.python.org/pep-0734/

• https://docs.python.org/3.14/library/concurrent.futures.html#interpreterpoolexecutor

  Each worker’s interpreter is isolated from all the other
  interpreters. “Isolated” means each interpreter has its own
  runtime state and operates completely independently. For example,
  if you redirect sys.stdout in one interpreter, it will not be
  automatically redirected to any other interpreter. If you import
  a module in one interpreter, it is not automatically imported in
  any other. You would need to import the module separately in
  interpreter where you need it. In fact, each module imported in
  an interpreter is a completely separate object from the same
  module in a different interpreter, including sys, builtins, and
  even main.

The latter explanation more-or-less implies our existing runtime
should work OTB i think?

It'd sure be cool to get (close to) multi-threading like spawn
latency as well as see if we can work around the known limitations
with forking and trio due to runtime duplication and state
conflicts (see discussion below).

---

Existing trio issues,

• directly regarding fork()-ing,
  • Surviving fork() python-trio/trio#1614
  • Usecase for nested run (and blocking outside runer) python-trio/trio#608
  • Clear global THREAD_CACHE in child after fork() python-trio/trio#2764
  • We need a mechanism to crash Trio with a given exception; how should it work? python-trio/trio#1607

Pre-existing subint integrations (in the 3rd-party ecosys)

• trio-parallel patch adding support,

  • richardsheridan/trio-parallel@main...sint

• anyio's support since 3.13,

  • docs: https://anyio.readthedocs.io/en/stable/subinterpreters.html
  • question about async task in subint: How to run async task inside subinterpreter? agronholm/anyio#904
    • user's current approach to run web-server-threads:
      https://github.com/coreproject-moe/CoreProject/blob/d8eb3a7e212c17ec3534f7f9486f6040cb7a6f71/tracker/src/coreproject_tracker/cli.py#L28-L49

---

Our own thoughts, ideas for fork()-workaround/hacks..

I had previously schemed (that prior to going full gil-ess python)
we'd need to write our own fork-server (since the stdlib's has many
discovered issues) re #146 and the follow up comment
#146 (comment).

BUT, given the mem-state isolation of subints, i'm now thinking
besides just supporting multi-subints-per-proc (and thus multi-actor
procs) directly, we can also support a fork()-ing spawn method (on
*nix) using a potential approach of,

• in parent creating a subint which both calls trio.run()
  (presuming that's possible beforehand) and then waits to be forked
  (using one of the methods described in Surviving fork() python-trio/trio#1614),

• when the child comes up, terminating the "parent's" subint immediately and letting
  the child continue to either,

  • call a new trio.run() if it wasn't already possible in the parent.
  • continue executing in that (maybe new) trio.run() "child subint" and terminating
    the (now copy) of the parent's subint.

• when the parent is connected-back-to by the child, terminating the
  child's local subint instance if needed? (unless we can use it as
  a kind of trio.run()-fork-server; that would only work for the in
  the did-not-call-trio.run() yet case?)

---

But why support proc-forking if you already have subints??

Well there's all sorts of fancy feats you get from the OS with
multi-processing (namespaces, certain cgroups feats, multi-host
agnosticism, various potentional sec guarantees depending on OS..
etc.)

It's also not clear how many process isolation/ctl feats which can
be applied normally to shm-threads can also be used for subints..
that's going to be follow research for sure though!

---

Current deps-blockers for 3.14+ subints

• msgspec being a C-ext needs to port.

  • Port to new concurrent.interpreters support in cpython 3.14+ msgspec/msgspec#1026
    • Port to new concurrent.interpreters support in cpython 3.14+ msgspec/msgspec#1026 (comment)

• cancel-cascade hangs under heavy fork spawn contentions,

  • main_thread_forkserver: cancel-cascade occasionally hangs >9s under heavy fork-spawn contention #451

---

Follow up issues discovered during ongoing dev work

• general follow-thru from Spawner modules: split up subactor spawning backends #444, some already solved:

  • Follow-up: split spawn._spawn into per-backend submods #445

• leaky fds per Discovery-client leaks CLOSE_WAIT fds when registrar server-side closes #452