Mailbox impl

[kentbull]

This is a mailbox implementation that:

• supports the existing "multipart/form-data" format of kli mailbox add
• ensures /fwd messages are only supported for recipient AIDs where the mailbox hab has a supporting end role auth for that recipientAID with AuthorizedForwardHandler.
• mailbox add/remove on mailbox host with MailboxAddRemoveEnd HTTP endpoint, compatible with kli mailbox add
  • kli mailbox remove does not exist yet though when it does the MailboxAddRemoveEnd will be ready for it.
• delkel in kli mailbox add: Delegated AIDs can send mailbox add request with kli mailbox add
• Auto-make mailbox non-transferable AID on startup if the one identified by --name and --alias does not exist.
• Adds a test script showing how the mailbox works by facilitating challenge response.

AI Sourcing Note:
Codex assisted creation of this PR. I reviewed everything and adjusted things here and there, though it was drafted by Codex with my prompts.

[kentbull]

Comment from maintainers call on 5/5: healthKERI is potentially contributing their Courier service and we need to sync this with that.

[dhh1128]

Thanks for building this out, @kentbull — a standalone mailbox host is a real step forward, and the cooperative-doer structure is clear. I used an AI to review. After checking its findings, I think the feature is a good step forward, but I have some picky stuff to ask about since this component has security implications. Verbiage below is excerpted from my AI's findings, which is why it sounds so cocky/harsh.

---

All four blockers sit on the security/correctness boundary:

1. Silent TEL/ACDC drop. The host returns HTTP 204 for TEL/ACDC ilks, but no Tevery is wired into setupMailbox, so those events are appended to parser.ims and discarded — the 204 tells the sender "stored" when it wasn't. Wire a Tevery, or reject with a non-204 (mailboxing.py:263-270).
2. The authorization gate is untested. AuthorizedForwardHandler — the allow/reject logic that is the whole point of this PR — has no tests, so a regression in it reaches CI undetected (forwarding.py:559).
3. The OOBI mailbox-role branch is untested. OOBIEnd.on_get()'s new branch (how a recipient advertises its mailbox) has no coverage for the authorized, unauthorized, or AID-absent paths (ending.py:596-601).
4. Two different authz predicates, undocumented. Role activation is gated on end.allowed in one place and end.allowed or end.enabled in others, with no comment on which is correct where. Whoever normalizes these later will silently move a trust boundary. Please pick one and document why (mailboxing.py:187, forwarding.py:564, ending.py:598).

Two mediums are worth pulling forward: the unauthenticated admin on_post ingests unbounded attacker CESR into the shared LMDB before the 403 auth check (mailboxing.py:451) — I'd rate that one HIGH; and a validly-signed /fwd exn with a malformed q raises an uncaught KeyError (forwarding.py:562). Both are small fixes.

Two more have no line to anchor to: the mailbox CLI test dir (tests/app/cli/commands/mailbox/) is missing an __init__.py — every sibling command-test dir has one, and without it a same-named test module elsewhere can silently shadow one of these. And there's no benchmark baseline for what is keripy's first standalone service role; fine to defer, but the perf notes below can't be validated without one.

The rest are deferrable perf and cleanup nits: redundant startup DB reads (start.py:181); a fresh Parser plus full escrow sweep per admin on_post (mailboxing.py:144); unbounded Deck queues under stalled SSE consumers (mailboxing.py:317, matching the existing WitnessStart pattern); vestigial datetime/source fields in the startup dict (start.py:188).

[dhh1128]

[blocker] Silent TEL/ACDC discard — the 204 is a lie. HttpEnd returns HTTP 204 for TEL/ACDC ilks, but no Tevery is wired into setupMailbox, so these events land in parser.ims and are dropped. A sender that gets 204 believes its credential/registry event was stored when it wasn't. Either wire a Tevery here so they're actually processed, or return a non-2xx status for ilks the host doesn't handle so the sender knows.

[dhh1128]

_confirmRoleAuth short-circuits to HTTP 403 when a cut is requested and no EndpointRecord exists. That makes idempotent revocation fragile: a caller unsure whether a mailbox was ever added can't safely issue a cut. A first-ever/redundant cut should succeed (or no-op), not 403.

[dhh1128]

_roleEnabled is defined identically here and in cli/commands/mailbox/start.py:175. Since it encodes an authorization semantic, the duplicate will drift the moment that semantic changes. Make one canonical (here in mailboxing.py) and import it in start.py.

[dhh1128]

The cooperative-doer wiring (cueDo/msgDo/escrowDo) has no test in a running-controller context. The module docstring itself warns that broken cue wiring silently breaks SSE delivery while storage still appears to work — exactly the kind of failure a test should pin. Worth a small integration test that drives a cue through to an SSE response.

[dhh1128]

This deferred (in-function) import has no comment. It reads like circular-import protection, but with no note a future refactor will either delete it and break imports, or preserve it as cargo-cult. One line saying which cycle it breaks would save that.

[dhh1128]

[blocker] The new mailbox-role branch is untested. This branch is how a recipient advertises its mailbox to senders, i.e. part of the trust boundary. No test covers the authorized path, the unauthorized path, or the case where the mailbox AID isn't in hby.prefixes. A regression here would be invisible to CI.

[dhh1128]

[blocker] The authorization gate has no tests. AuthorizedForwardHandler is the reject-unauthorized-forwards logic that is the stated point of this PR, but nothing exercises the allow path or the reject path. Please add tests for both — this is the one thing most important to keep from regressing.

[dhh1128]

Minor/defer: handle() does one LMDB ends.get() per inbound /fwd message, though authorization state only changes on admin add/remove. A process-local dict cache keyed by recipient AID (invalidated from the admin endpoint) removes the per-message read — but establish a benchmark baseline first so the cache-invalidation complexity is justified by numbers.

[dhh1128]

Defer: ClientStub and WitnessPublisherStub are duplicated verbatim across the two tests. Fine as-is for now; worth lifting to module-level fixtures once a third mailbox-CLI test lands in this dir.

[dhh1128]

Kent:

I'm doing some stuff in a codebase that consumes keripy, and this PR caught my eye again because mailbox code might be useful there, so I thought I'd re-study this PR a bit.

Like my review process on your PR #1614, this is deeper water than I'm really qualified to swim in at the moment, so I did some AI analysis but didn't totally trust myself to adjudicate it, so part 2 of my process was to write some tests that proved or disproved the findings I was coming up with.

Bottom line is that I found some small stuff, plus one thing that I think is substantive that's worth discussing. And as with the other PR, I've raised a PR against your PR so you can have tests that demonstrate the issues I'm reporting. I don't necessarily think you should merge my PR, but maybe use it as a reference tool?

Here's what I found that seemed worth sharing back.

The host doesn't store forwarded messages

setupMailbox builds the ingress parser at version=hby.version (mailboxing.py:263), and _openMailboxHabery never sets a protocol version, so it's genus 2. At genus 2 the /fwd envelope that forwarding.Poster.forward puts on the wire fails attachment extraction — the parser logs "Parser msg extraction error", Exchanger.processEvent is never reached, and AuthorizedForwardHandler is never called.

I reproduced this in a harness that mirrors your wiring exactly (same Kevery(lax=True, local=False), Revery, Exchanger, parser):

sender genus	parser genus	handler invoked	stored
2	2 (as shipped)	0	0
1	2	0	0
1	1	1	1

The last row is the control — same message, same handler, stores fine.

I don't think this is your bug, and I don't think this PR should have to fix it. setupWitness has the same wiring (indirecting.py:88), kli witness start never passes version either (witness/start.py:128 — the -V/--version there is argparse's, not a protocol selector), and setupHby defaults to Version no matter how the keystore was initialized. So a default kli witness start at HEAD looks like it has the same broken /fwd ingress, and nothing covers the default path: tests/app/test_forwarding.py:502 pins Parser(version=Vrsn_1_0) by hand.

What does seem fair to raise here is that this is the first component whose entire value is /fwd storage, mailbox-local-challenge.sh exercises the broken path (and unlike every other script in scripts/demo/basic/ it pins no version), and setupMailbox doesn't accept the version kwarg that setupWitness accepts, so a library caller can't work around it. Test: test_mailbox_host_ingress_delivers_a_forwarded_message_to_the_handler.

aids doesn't reach the storage gate

setupMailbox takes aids and hands it only to Respondant (mailboxing.py:253); AuthorizedForwardHandler never sees it. The gate's only input is the ends. record, and that record is writable by any peer straight through the host's own / CESR ingress — the same Kevery(lax=True) + Revery mounted at mailboxing.py:273 — with no visit to /mailboxes.

I don't think this is a bypass of a check you intended: kli mailbox add is self-service by design and /mailboxes has no admission control either, so the front door does the same thing. But it does mean the docstring's claim at forwarding.py:566 — that this keeps the host from becoming "an open storage surface for unsolicited or malicious /fwd traffic" — holds against unsolicited traffic and not against malicious traffic, and an operator has no supported way to say which AIDs the host serves. Maybe we should either wire aids through to the handler or soften the docstring? Test: test_setup_mailbox_aids_restricts_which_recipients_get_forwarded_storage.

_confirmRoleAuth isn't tied to the reply that was submitted

mailboxing.py:128 re-reads ends. after ingest and asks whether the state now matches the request, not whether this reply produced it. So a controller who already authorized this host, and whose new reply is then rejected by BADA (stale datetime, post-rotation key mismatch), gets HTTP 200 and {"allowed": true} from a submission that changed nothing — and kli mailbox add reports "Mailbox … added" and publishes the end role to witnesses. Stripping the signature off a valid reply and reposting it does the same thing.

Nothing changes state, so it isn't an authorization bypass — it's a false success and an oracle for unauthenticated callers. .eans already holds the SAID of the reply that was accepted:

saider = hby.db.eans.get(keys=(cid, Roles.mailbox, mailboxAid))
if saider is None or saider.qb64 != serder.said:
    raise falcon.HTTPForbidden(description="Mailbox authorization reply was not accepted")

Test: test_mailbox_admin_rejects_a_reply_that_was_not_accepted.

Multipart fields that declare their real media type read as None

_readMultipart uses falcon's BodyPart.text, which returns None for any part whose Content-Type isn't text/plain (falcon/media/multipart.py:171). RFC 7578 defaults the per-part header to text/plain, which is why kli mailbox add works — but a client labelling the fields application/cesr gets 400 "missing kel" for a request that supplied one. delkel is the bad case: if parts.get("delkel") treats None as absent, so an honestly-labelled delegation replay is dropped silently and the delegated add fails with an opaque 403. part.data.decode(...) avoids all of it. Test: test_mailbox_admin_reads_parts_labeled_with_a_cesr_media_type.

The KEL replay is capped at 1 MiB

The app never configures MultipartFormHandler parse options, so max_body_part_buffer_size stays at falcon's 1 MiB default and a larger kel dies with "body part is too large", which says nothing about mailbox authorization. Ingesting a controller KEL replay is the endpoint's whole job; ~1200 interaction events with no witnesses is already 416 KB. Test: test_mailbox_admin_accepts_a_kel_replay_larger_than_falcons_default_part_buffer.

Related, and maybe the more interesting half: _ingestCesr runs a full parse plus processEscrows synchronously inside the Falcon handler, and it runs before _confirmRoleAuth. So a request that ends in a 403 has already persisted whatever KEL and delkel it sent into the live LMDB. Since hio serves WSGI on the same single-threaded Doist loop, that ingest also stalls every other client and every open SSE stream while it runs.

Mailboxer loses hby.base

mailboxing.py:250 is Mailboxer(name=alias, temp=hby.temp) where indirecting.py:60 is Mailboxer(name=alias, base=hby.base, temp=hby.temp). Two hosts on different --base values with the same alias share one message store while their Haberys stay isolated. Test: test_setup_mailbox_scopes_forwarded_message_storage_to_the_habery_base.

A malformed /fwd stops ingress

forwarding.py:588 indexes modifiers["pre"] unguarded, and Exchanger.processEvent catches only AttributeError around behavior.handle (peer/exchanging.py:188), so a KeyError propagates into the parser loop. It's inherited from ForwardHandler so it isn't a regression, but MailboxStart.msgDo is a single parsator generator behind an unauthenticated endpoint, so one bad exn stops mailbox ingress for the process. Test: test_authorized_forward_handler_survives_a_fwd_exn_without_a_recipient.

mailbox/start.py predates the witness-start hardening

This one is timing, not judgment. You wrote this file in April against the witness/start.py of the day. I later hit a set of problems in that file and fixed them in #1477, with backports in #1486 (v1.3.5) and #1487 (v1.2.14) — all merged in July and August, months after your commits. Since this branch is now rebased onto current main, the fixed witness/start.py is sitting in your own tree, and the new file still has the old shape.

Nothing here is a regression — nothing that worked stops working. But it does re-introduce bugs that were just fixed, and if mailbox support later gets backported to 1.2.x or 1.3.x it will carry them into branches those two PRs just cleaned. The gaps I see against the current file:

• start.py:54 — logging.getLevelName(args.loglevel) without .upper(), so --loglevel debug sets the level to the string "Level debug" and silently breaks filtering (witness/start.py:70).
• start.py:56 — ogler.headDirPath = args.logfile assigns a file path to a directory attribute, so --logfile /var/log/mailbox.log creates a directory by that name. Witness gained a --logdir flag and takes os.path.dirname() of the deprecated one (witness/start.py:72-74).
• logger is bound at import (start.py:50) and never rebound after ogler.level is set, so --loglevel doesn't reach it. witness/start.py:83 re-applies ogler.getLogger() for exactly this.
• _openMailboxHabery calls setupHby without noPrompt, so kli mailbox start under systemd with no passcode blocks on an interactive prompt rather than failing fast — which is the case you fixed yourself in 37eaa73.
• runMailbox wraps only _prepareMailboxHabitat (lines 94-102). setupMailbox and runController are outside the guard, so a failure in either leaves the Habery LMDB env open, which is the leftover-lock-blocks-the-next-start problem Log kli witness start failures and prevent startup stalls (#238) #1477 was chasing.

Only the first two are a straight lift: the whole launch() logging block at witness/start.py:68-84, plus the --logdir argument, ports verbatim, and your argument list is otherwise already aligned with witness — --no-prompt and --logdir are the only two it's missing. The other three are patterns rather than code, since runMailbox and the _mailbox* helpers have no witness counterpart and shouldn't grow one.

Smaller things

• KRAM is off. setupMailbox's Kevery omits cf=hby.cf, enableKram=True, which setupWitness passes (indirecting.py:73). Kevery.processMsg gates qry/rpy/exn on self.kramer when present (eventing.py:4739), so there's no replay mitigation on the surfaces this PR adds. Is this on purpose?
• Two authorization predicates. AuthorizedForwardHandler and _roleEnabled accept end.allowed or end.enabled; _confirmRoleAuth and the new OOBIEnd branch accept end.allowed alone. Inert today since updateEnd only ever writes allowed, but a cut clears allowed and leaves enabled untouched, so the two expressions disagree about revocation and the looser one guards storage.
• The read side is ungated. /fwd storage is gated by recipient authorization but the mbx query route pushes a stream cue for any AID in kevers (eventing.py:5651, with the witness-scoped check commented out just below). On a witness that was bounded by who you witness for; a standalone host has no such bound. This is pre-existing, but the symmetry might be worth a thought.
• start.py:105 — a non-numeric --http is swallowed by except ValueError and silently replaced by the URL's port; and if the stored URL has no explicit port, urlparse(...).port is None and reaches createHttpServer as httpPort=None.
• start.py:170 — _storedMailboxUrl raises when the mailbox advertises both an http and an https self-location, which is normal behind a TLS terminator. Was this intentional?
• test_setup_mailbox_requires_self_identity_state_before_boot binds port 9000 and fails when anything else holds it; httpPort=0 works.

What I liked

The post-ingest acceptance model feels right to me — accepting nothing on the strength of the envelope and deciding only from what survived normal processing. _requireMailboxIdentity refusing to boot on incomplete self-state is disciplined in exactly the way I prefer. The OOBIEnd branch correctly reuses replyToOobi, so what it serves is the controller-signed stored reply plus both KELs rather than something the mailbox minted. The delkel field and the urljoin fix in mailbox/add.py are both real bugs fixed. And the module docstrings are unusually good at saying why, which made this much easier to review.