sv2: ProposeTemplate — validate externally proposed templates (sv2-spec#239)

[average-gary]

Draft, because the wire format is still being discussed in sv2-spec discussion #239 (RFC: a TDP message so a JDS can validate a Custom Job through a Template Provider). This is the sv2-tp side of that proposal, opened so the discussion has a concrete Core-IPC implementation to look at. It now follows the shape the thread converged on (2026-10-08) and will track whatever moves from here.

What it does

A TDP client that sets SetupConnection.flags bit 0 (REQUIRES_JOB_VALIDATION, echoed in SetupConnection.Success) can send ProposeTemplate (0x77) carrying the DeclareMiningJob subset: version, coinbase_tx_prefix, coinbase_tx_suffix, wtxid_list, excess_data. There is no transaction_list; missing transactions use the JDP 6.4.7/6.4.8 layouts under TDP numbers. The TP:

1. on the networking thread, runs the checks that need no node call: list bounds, duplicate wtxids (duplicate-wtxid), coinbase prefix/suffix decode incl. a consensus-valid scriptSig length (bad-cb-decode), a request_id still in use by this client (duplicate-request-id), and the per-client in-flight bound;
2. hands the request to a worker thread (ThreadSv2ProposalHandler, one per TP, FIFO across clients). The worker answers job-validation-unavailable while isInitialBlockDownload() is true, before any lookup; otherwise resolves the wtxids with getTransactionsByWitnessID (Core v32) and, for anything unknown, replies ProvideMissingTransactions (0x78) with 0-indexed positions and keeps the request pending per client, keyed by request_id: at most MAX_PENDING_PROPOSALS (8, oldest evicted) for PENDING_PROPOSAL_TIMEOUT (30 s);
3. ProvideMissingTransactions.Success (0x7b) resumes a pending request. Each supplied transaction must hash to a wtxid at one of the requested positions and all requested positions must be covered (bad-missing-tx), checked before any node call; an unknown or expired request_id gets unknown-request-id;
4. with every transaction known, the worker fills the extranonce gap with zero bytes (scriptSig length from the prefix minus bytes present; assumes the extranonce is the scriptSig tail), assembles a CBlock on the tip, and calls checkBlock(check_merkle_root=false, check_pow=false);
5. on success tracks the block as a BlockTemplate in the existing template cache (same tip-change prune as our own templates) and replies ProposeTemplate.Success (0x79) with template_id, the tip prev_hash and fees, so a normal SubmitSolution later submits it via submitBlock with the miner's coinbase; otherwise ProposeTemplate.Error (0x7a) with Core's reject reason.

Replies from the worker are queued under the same locks SendWork/SubmitSolution use. Each client may have MAX_PROPOSALS_IN_FLIGHT (4) proposals queued or being validated; further ones get job-validation-unavailable. On a v31 node the flag is refused (no wtxid lookup, no submitBlock).

Commits

• ipc: add getTransactionsByWitnessID to the mining interface — capnp @9 as in Core v32, the null-transaction decode hook from Core's common-types.h, and MockMining support.
• sv2: add ProposeTemplate messages — structs, serializers (incl. 0x78/0x7b), Coinbase() assembly, message tests.
• sv2: negotiate REQUIRES_JOB_VALIDATION and forward job validation messages — Sv2Connman supported flags, Success echoes accepted flags, dispatch of 0x77 and 0x7b; RequestTransactionData no longer requires CoinbaseOutputConstraints on such a connection.
• sv2: validate proposed templates and track them for SubmitSolution — the synchronous prechecks, the worker thread, the pending map, the BlockTemplate adapter and the tests below.

Things Core IPC can't provide today, and how the patch copes

• A fee total for an externally proposed block: checkBlock returns only reason/debug/result, and the TxCollection.makeTemplate templates in mining: add TxCollection to bandwidth-efficiently validate external block templates bitcoin/bitcoin#35671 throw on getTxFees. ProposeTemplate.Success.fees is therefore sent as 0 (unknown) until Core exposes it; bad-cb-amount already bounds the coinbase at subsidy + fees.
• A header for "the current tip" without creating a template: the worker calls createNewBlock(useMempool=false, cooldown=false) and copies hashPrevBlock/nBits/nTime. That is an extra cs_main round trip per request; TxCollection.makeTemplate(prevhash, coinbase) would remove it, and the adapter is shape-compatible with what it returns.

Still open in #239

• The 0x7b number and whether to keep the JDP names for the missing-transaction legs.
• Whether the provide leg gets its own .Error, or keeps answering with ProposeTemplate.Error as here (unknown-request-id, bad-missing-tx).
• The per-connection cap on validated templates. Nothing is enforced beyond the existing tip-change prune; only a doc note.
• The reply for a client that exceeds the in-flight bound (job-validation-unavailable here) and the pending-map overflow rule (oldest evicted here).

Not done

The fuzz target and the functional scenario. Happy to add them before this leaves draft, or to park it until #35671 lands if you'd rather build on TxCollection.

Testing

Builds and ctest passes (7/7 suites, 10 new cases) on macOS with Homebrew cmake/capnp/boost. New coverage: message round-trips incl. 0x7b; missing → ProvideMissingTransactions → .Success → ProposeTemplate.Success → SubmitSolution with an extranonce that differs from the placeholder (submitted block carries the miner's coinbase and a recomputed merkle root); RequestTransactionData on a proposed id; duplicate-request-id, unknown-request-id, expiry and the 8-entry eviction via mock time; duplicate-wtxid; bad-missing-tx for a wrong transaction, one at an unrequested position and an incomplete set; bad-cb-decode through the handler; the IBD gate (no lookup, no createNewBlock); prune after a tip change and no premature prune for a client without templates; and a deterministic off-thread test using a MockMining latch on checkBlock (RequestTransactionData answered while a proposal is blocked in the node; bound, FIFO order and duplicate-request-id for a queued id). The SRI integration matrix was not run.

A native-TP implementation of the same messages lives in rbitcoin: https://github.com/average-gary/rbitcoin/tree/sv2/job-validation

🤖 Generated with Claude Code

[Sjors]

a TDP message so a JDS can validate a Custom Job through a Template Provide

That's an interesting idea. I'll study the spec proposal and your code here later, but I'm definitely open to adding support.

I'm also fine with an initial version that doesn't use TxCollection, since that's not merged yet into Bitcoin Core, and once it lands, we might still want to support older releases.

OTOH implementing a mirror mempool does sound tedious.

If you think anything is missing from bitcoin/bitcoin#35671, please leave a comment there as well.

[average-gary]

Thanks.

On the mirror mempool: this version doesn't need one. It resolves the declared wtxids with getTransactionsByWitnessID (Core v32) and asks the JDS for whatever the node lacks, so it only works against v32+ and refuses the setup flag on v31; a mirror was the v31 fallback I decided not to build. Once TxCollection lands the resolve/assemble/checkBlock sequence collapses onto collectTxs → addMissingTxs → makeTemplate, and the BlockTemplate adapter here is shape-compatible with what makeTemplate returns, so the switch should be local to the handler. One change is coming regardless, from your thread-1 comment on #239: the missing-transactions leg will become a request/provide pair keyed by request_id instead of repeating ProposeTemplate with a transaction_list, which also means holding the pending proposal (or the TxCollection) across the round trip. I'll keep this in draft and rebase it as the discussion converges.

Fable 5.1 assisted but hooman reviewed.