v0.15.5 Persistent "Send Queue full" in gossipsub despite low peer count and idle bandwidth

[553589912]

node-1 | {"timestamp":"2026-05-11T12:28:41.216035Z","level":"INFO","fields":{"message":"Inserted new unsafe block","hash":"0x7de2d7bec1c2c3895e58b750c74566092909117ae917edadcb31b67600cd4b7a","number":45856587,"total_duration":"121.982748ms","insert_duration":"117.902539ms"},"target":"engine"}
node-1 | {"timestamp":"2026-05-11T12:28:41.216125Z","level":"INFO","fields":{"message":"Safe head unchanged","safe_head":45856557},"target":"engine"}
node-1 | {"timestamp":"2026-05-11T12:28:41.556015Z","level":"WARN","fields":{"message":"Send Queue full. Could not send IHave(IHave { topic_hash: TopicHash { hash: "/optimism/8453/3/blocks" }, message_ids: [MessageId(45e05d477cd987fadb1271a5027861c2a108c194)] }).","peer":"16Uiu2HAkyTBnw64qWhp1szF4gbdapece5TLPKF4rgkbB7Mn5tFHv"},"target":"libp2p_gossipsub::behaviour"}
node-1 | {"timestamp":"2026-05-11T12:28:41.556070Z","level":"WARN","fields":{"message":"Send Queue full. Could not send IHave(IHave { topic_hash: TopicHash { hash: "/optimism/8453/3/blocks" }, message_ids: [MessageId(45e05d477cd987fadb1271a5027861c2a108c194)] }).","peer":"16Uiu2HAm98QUzAaJEedoUM2otzeDdjHTs7cTNNkBcMjmFyT9qmMd"},"target":"libp2p_gossipsub::behaviour"}
node-1 | {"timestamp":"2026-05-11T12:28:42.056912Z","level":"WARN","fields":{"message":"Send Queue full. Could not send IHave(IHave { topic_hash: TopicHash { hash: "/optimism/8453/3/blocks" }, message_ids: [MessageId(45e05d477cd987fadb1271a5027861c2a108c194)] }).","peer":"16Uiu2HAm98QUzAaJEedoUM2otzeDdjHTs7cTNNkBcMjmFyT9qmMd"},"target":"libp2p_gossipsub::behaviour"}
node-1 | {"timestamp":"2026-05-11T12:28:42.557737Z","level":"WARN","fields":{"message":"Send Queue full. Could not send IHave(IHave { topic_hash: TopicHash { hash: "/optimism/8453/3/blocks" }, message_ids: [MessageId(45e05d477cd987fadb1271a5027861c2a108c194)] }).","peer":"16Uiu2HAm98QUzAaJEedoUM2otzeDdjHTs7cTNNkBcMjmFyT9qmMd"},"target":"libp2p_gossipsub::behaviour"}

My base-node is constantly logging WARN messages: Send Queue full. Could not send IHave....
The weird part is that my network is not congested (verified via nload, TX is far below capacity), and the number of connected peers is very low (only 1-3 active peers).

[Kewe63]

Problem Analysis: "Send Queue full" in gossipsub

I investigated this issue. Here's a technical analysis:

---

Root Cause

This is a libp2p_gossipsub per-peer send queue issue, not a network congestion problem. The warning Send Queue full. Could not send IHave is emitted by the gossipsub behaviour when the outbound buffer for a specific peer is full.

Key factors:

1. Low peer count (1–3 active peers) — Every IHave message for every block must go to the same few peers. With more peers, the gossip load is distributed across more connections.

2. High block production rate — Base produces blocks roughly every 2 seconds. Each block generates at least one IHave announcement on the /optimism/8453/3/blocks topic. With 1–3 peers, each peer receives every IHave.

3. The per-peer send queue fills up — The default libp2p_gossipsub send queue size (~512 messages per peer) is adequate when there are many peers, but with only 1–3 peers, the queue fills faster than the peer can drain it (especially if the peer has high latency or its own backpressure).

---

Verify it's NOT a bandwidth issue

# Check for kernel-level packet drops
netstat -s | egrep 'packets discarded|overflow'

# Check peer count
curl -s http://localhost:8545 -X POST \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"net_peerCount","params":[],"id":1}'

---

Workarounds (in order of effectiveness)

1. Increase max outbound peers — The current --max-outbound-peers=100 targets 100 peers. If you're only connected to 1–3, check if your P2P port (9222 TCP/UDP) is reachable from outside. Use a port checking tool like [yougetsignal.com](https://www.yougetsignal.com). If the port is blocked, other nodes can't connect to you.

2. Add more bootstrap peers — The current bootnodes in .env.mainnet might not all be responsive. Consider adding more Base node bootnodes.

3. Restart the node — Sometimes the gossipsub mesh gets into a degraded state and a restart rebuilds the mesh with fresh connections.

---

Long-term Fix

The actual gossipsub send queue size is configured in the libp2p_gossipsub config, which is compiled into base-reth-node. A proper fix would be either:

• Adding a CLI flag to control the gossipsub send queue size in base/base
• Increasing the default queue size upstream

This is not a base/node Docker configuration issue — it's a reth/libp2p configuration parameter that needs to be exposed or increased in the base/base codebase.

[553589912]

netstat -s | egrep 'packets discarded|overflow'
0 incoming packets discarded
/data/app/base/node-0.15.5# curl -s http://localhost:8080 -X POST
-H "Content-Type: application/json"
-d '{"jsonrpc":"2.0","method":"net_peerCount","params":[],"id":1}'
{"jsonrpc":"2.0","id":1,"result":"0x1"}root@ec2-jp-wallet-prd-node-base-az1-fix:/data/app/base/node-0.15.5#
data/app/base/node-0.15.5#
/data/app/base/node-0.15.5# python3
Python 3.10.12 (main, Mar 3 2026, 11:56:32) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

0x1
1

cat /data/app/base/node-0.15.5/.env.mainnet | grep -iE "p2p|peer|boot|port"

P2P CONFIGURATION

OP_NODE_P2P_AGENT=base
OP_NODE_P2P_LISTEN_IP=0.0.0.0
OP_NODE_P2P_LISTEN_TCP_PORT=9222
OP_NODE_P2P_LISTEN_UDP_PORT=9222
OP_NODE_P2P_BOOTNODES=enr:-J24QNz9lbrKbN4iSmmjtnr7SjUMk4zB7f1krHZcTZx-JRKZd0kA2gjufUROD6T3sOWDVDnFJRvqBBo62zuF-hYCohOGAYiOoEyEgmlkgnY0gmlwhAPniryHb3BzdGFja4OFQgCJc2VjcDI1NmsxoQKNVFlCxh_B-716tTs-h1vMzZkSs1FTu_OYTNjgufplG4N0Y3CCJAaDdWRwgiQG,enr:-J24QH-f1wt99sfpHy4c0QJM-NfmsIfmlLAMMcgZCUEgKG_BBYFc6FwYgaMJMQN5dsRBJApIok0jFn-9CS842lGpLmqGAYiOoDRAgmlkgnY0gmlwhLhIgb2Hb3BzdGFja4OFQgCJc2VjcDI1NmsxoQJ9FTIv8B9myn1MWaC_2lJ-sMoeCDkusCsk4BYHjjCq04N0Y3CCJAaDdWRwgiQG,enr:-J24QDXyyxvQYsd0yfsN0cRr1lZ1N11zGTplMNlW4xNEc7LkPXh0NAJ9iSOVdRO95GPYAIc6xmyoCCG6_0JxdL3a0zaGAYiOoAjFgmlkgnY0gmlwhAPckbGHb3BzdGFja4OFQgCJc2VjcDI1NmsxoQJwoS7tzwxqXSyFL7g0JM-KWVbgvjfB8JA__T7yY_cYboN0Y3CCJAaDdWRwgiQG,enr:-J24QHmGyBwUZXIcsGYMaUqGGSl4CFdx9Tozu-vQCn5bHIQbR7On7dZbU61vYvfrJr30t0iahSqhc64J46MnUO2JvQaGAYiOoCKKgmlkgnY0gmlwhAPnCzSHb3BzdGFja4OFQgCJc2VjcDI1NmsxoQINc4fSijfbNIiGhcgvwjsjxVFJHUstK9L1T8OTKUjgloN0Y3CCJAaDdWRwgiQG,enr:-J24QG3ypT4xSu0gjb5PABCmVxZqBjVw9ca7pvsI8jl4KATYAnxBmfkaIuEqy9sKvDHKuNCsy57WwK9wTt2aQgcaDDyGAYiOoGAXgmlkgnY0gmlwhDbGmZaHb3BzdGFja4OFQgCJc2VjcDI1NmsxoQIeAK_--tcLEiu7HvoUlbV52MspE0uCocsx1f_rYvRenIN0Y3CCJAaDdWRwgiQG
BASE_NODE_P2P_LISTEN_IP=0.0.0.0
BASE_NODE_P2P_ADVERTISE_TCP_PORT=9222
BASE_NODE_P2P_ADVERTISE_UDP_PORT=9222
BASE_NODE_P2P_BOOTNODES=enr:-J24QNz9lbrKbN4iSmmjtnr7SjUMk4zB7f1krHZcTZx-JRKZd0kA2gjufUROD6T3sOWDVDnFJRvqBBo62zuF-hYCohOGAYiOoEyEgmlkgnY0gmlwhAPniryHb3BzdGFja4OFQgCJc2VjcDI1NmsxoQKNVFlCxh_B-716tTs-h1vMzZkSs1FTu_OYTNjgufplG4N0Y3CCJAaDdWRwgiQG,enr:-J24QH-f1wt99sfpHy4c0QJM-NfmsIfmlLAMMcgZCUEgKG_BBYFc6FwYgaMJMQN5dsRBJApIok0jFn-9CS842lGpLmqGAYiOoDRAgmlkgnY0gmlwhLhIgb2Hb3BzdGFja4OFQgCJc2VjcDI1NmsxoQJ9FTIv8B9myn1MWaC_2lJ-sMoeCDkusCsk4BYHjjCq04N0Y3CCJAaDdWRwgiQG,enr:-J24QDXyyxvQYsd0yfsN0cRr1lZ1N11zGTplMNlW4xNEc7LkPXh0NAJ9iSOVdRO95GPYAIc6xmyoCCG6_0JxdL3a0zaGAYiOoAjFgmlkgnY0gmlwhAPckbGHb3BzdGFja4OFQgCJc2VjcDI1NmsxoQJwoS7tzwxqXSyFL7g0JM-KWVbgvjfB8JA__T7yY_cYboN0Y3CCJAaDdWRwgiQG,enr:-J24QHmGyBwUZXIcsGYMaUqGGSl4CFdx9Tozu-vQCn5bHIQbR7On7dZbU61vYvfrJr30t0iahSqhc64J46MnUO2JvQaGAYiOoCKKgmlkgnY0gmlwhAPnCzSHb3BzdGFja4OFQgCJc2VjcDI1NmsxoQINc4fSijfbNIiGhcgvwjsjxVFJHUstK9L1T8OTKUjgloN0Y3CCJAaDdWRwgiQG,enr:-J24QG3ypT4xSu0gjb5PABCmVxZqBjVw9ca7pvsI8jl4KATYAnxBmfkaIuEqy9sKvDHKuNCsy57WwK9wTt2aQgcaDDyGAYiOoGAXgmlkgnY0gmlwhDbGmZaHb3BzdGFja4OFQgCJc2VjcDI1NmsxoQIeAK_--tcLEiu7HvoUlbV52MspE0uCocsx1f_rYvRenIN0Y3CCJAaDdWRwgiQG
OP_NODE_RPC_PORT=8545
OP_NODE_METRICS_PORT="7300"
BASE_NODE_METRICS_PORT="7300"

OP_GETH_ETH_STATS=nodename:secret@host:port

OP_GETH_BOOTNODES=enode://87a32fd13bd596b2ffca97020e31aef4ddcc1bbd4b95bb633d16c1329f654f34049ed240a36b449fda5e5225d70fe40bc667f53c304b71f8e68fc9d448690b51@3.231.138.188:30301,enode://ca21ea8f176adb2e229ce2d700830c844af0ea941a1d8152a9513b966fe525e809c3a6c73a2c18a12b74ed6ec4380edf91662778fe0b79f6a591236e49e176f9@184.72.129.189:30301,enode://acf4507a211ba7c1e52cdf4eef62cdc3c32e7c9c47998954f7ba024026f9a6b2150cd3f0b734d9c78e507ab70d59ba61dfe5c45e1078c7ad0775fb251d7735a2@3.220.145.177:30301,enode://8a5a5006159bf079d06a04e5eceab2a1ce6e0f721875b2a9c96905336219dbe14203d38f70f3754686a6324f786c2f9852d8c0dd3adac2d080f4db35efc678c5@3.231.11.52:30301,enode://cdadbe835308ad3557f9a1de8db411da1a260a98f8421d62da90e71da66e55e98aaa8e90aa7ce01b408a54e4bd2253d701218081ded3dbe5efbbc7b41d7cef79@54.198.153.150:30301

OP_NETHERMIND_BOOTNODES=enode://87a32fd13bd596b2ffca97020e31aef4ddcc1bbd4b95bb633d16c1329f654f34049ed240a36b449fda5e5225d70fe40bc667f53c304b71f8e68fc9d448690b51@3.231.138.188:30301,enode://ca21ea8f176adb2e229ce2d700830c844af0ea941a1d8152a9513b966fe525e809c3a6c73a2c18a12b74ed6ec4380edf91662778fe0b79f6a591236e49e176f9@184.72.129.189:30301,enode://acf4507a211ba7c1e52cdf4eef62cdc3c32e7c9c47998954f7ba024026f9a6b2150cd3f0b734d9c78e507ab70d59ba61dfe5c45e1078c7ad0775fb251d7735a2@3.220.145.177:30301,enode://8a5a5006159bf079d06a04e5eceab2a1ce6e0f721875b2a9c96905336219dbe14203d38f70f3754686a6324f786c2f9852d8c0dd3adac2d080f4db35efc678c5@3.231.11.52:30301,enode://cdadbe835308ad3557f9a1de8db411da1a260a98f8421d62da90e71da66e55e98aaa8e90aa7ce01b408a54e4bd2253d701218081ded3dbe5efbbc7b41d7cef79@54.198.153.150:30301

NOTE: The node type that was chosen when first running a node cannot be changed after the initial sync. Turning Archive into Pruned, or Pruned into Full is not supported source.

root@ : ss -tlnup | grep 9222
udp UNCONN 0 0 0.0.0.0:9222 0.0.0.0:* users:(("dockerd",pid=183305,fd=41))
tcp LISTEN 0 100001 0.0.0.0:9222 0.0.0.0:* users:(("dockerd",pid=183305,fd=40))

Thank you for your reply. I have investigated and confirmed that it is not a bandwidth issue. The configuration also appears to be following the official recommended settings."
Could you please help me check if there are any deficiencies in the configuration?

[553589912]

about "Long-term Fix
The actual gossipsub send queue size is configured in the libp2p_gossipsub config, which is compiled into base-reth-node. A proper fix would be either:

Adding a CLI flag to control the gossipsub send queue size in base/base
Increasing the default queue size upstream
This is not a base/node Docker configuration issue — it's a reth/libp2p configuration parameter that needs to be exposed or increased in the base/base codebase."

Does this require waiting for an official fix? The new architecture seems less stable than the old geth architecture and consumes higher I/O. At this point, we have not seen any advantages of the new architecture over the old one

[Kewe63]

Thanks for the detailed report! The root cause of the Send Queue full warnings is indeed the low peer count (net_peerCount returning 0x1). With only a single peer, all IHave messages are routed through the same connection, causing the per-peer send buffer to fill up immediately.

You don't need to wait for an upstream fix. Here’s how you can resolve it right now:

1. Open P2P Ports in AWS Security Group

Port 9222 (TCP & UDP) must be open for inbound traffic so other nodes can discover and connect to you:

• Go to AWS Console → EC2 → Security Groups → Inbound Rules
• Add the following rules:
  • Custom TCP, Port 9222, Source 0.0.0.0/0
  • Custom UDP, Port 9222, Source 0.0.0.0/0

2. (Optional) Switch Back to Geth

If you prefer stability while troubleshooting, you can immediately fall back to geth:

CLIENT=geth docker compose up --build

3. Increase the Peer Limit

Once port 9222 is open, you can increase the maximum number of outbound peers:

RETH_MAX_OUTBOUND_PEERS=250 docker compose up --build

Note: The Send Queue full warnings are non-fatal (messages are automatically retried), but the underlying issue is simply a lack of active peers. Once your node is properly discoverable and peers connect, the warnings will disappear.

Let me know if you run into any issues with these steps or need help verifying your peer count afterward!