Gloas spec deviation PENDING head causes proposer to miss its slot

[gitToki]

This was identified by @jzh18

Description

In the consensus spec get_node_children (#fork-choice) unconditionally returns the same root EMPTY child and does not consult the filtered block tree for a PENDING node.

In lighthouse this doesn't always happen. When the chain is in extended non finality and the head reverts to the justified checkpoint, this leaves the head with an unresolved PayloadStatus::Pending, which later aborts block production. Every lighthouse proposer scheduled in that window gets a 400 and misses its slot, for as long as the condition holds.

Branch

glamsterdam-devnet-6

Present Behaviour

For a PENDING node, get_node_children synthesizes virtual EMPTY (and FULL) children that carry the parent's own proto_node_index (the justified node's index). The head walk then filters every child, virtual ones included against viable_nodes:

// consensus/proto_array/src/proto_array.rs — find_head_walk
loop {
    let children: Vec<_> = self
        .get_node_children(&head)?
        .into_iter()
        .filter(|(fc_node, _)| viable_nodes.contains(&fc_node.proto_node_index))
        .collect();

    if children.is_empty() {
        return Ok(head); // returns the PENDING seed verbatim
    }
    ...
}

In filter_block_tree, a node with children is added to viable only if at least one of its children is viable. So in deep non-finality where the justified node has descendants but the entire subtree is non-viable (stale voting sources, voting_source.epoch + 2 < current_epoch), the head reverts to the justified checkpoint start_index is not in viable_nodes. The virtual EMPTY child (whose index is start_index) is filtered out, children is empty, and the walk returns the seed node with its status still PayloadStatus::Pending.

find_head returns this status unchanged, and it's cached verbatim as head_payload_status in canonical_head.rs (the only head rejection there is for an invalid execution status, which doesn't match PENDING). On the block production path head_payload_status is forwarded as parent_payload_status (block_production/mod.rs) into should_build_on_full, which returns Err(InvalidPayloadStatus) for a PENDING status. The HTTP handler maps that to a 400 ("failed to fetch a block: ...") with no EMPTY fallback and no retry, so the scheduled proposer has nothing to sign and misses its slot.

Suggested fix

// consensus/proto_array/src/proto_array.rs
loop {
            let children: Vec<_> = if head.payload_status == PayloadStatus::Pending {
                self.get_node_children(&head)?
            } else {
                self.get_node_children(&head)?
                    .into_iter()
                    .filter(|(fc_node, _)| viable_nodes.contains(&fc_node.proto_node_index))
                    .collect()
            };

            if children.is_empty() {
                return Ok(head);
            }

[michaelsproul]

nice find, thanks