Repository navigation
Description
Environment:
- pico-sdk 2.1.0 and 2.3.1 (checked both, code identical in the relevant paths)
- RP2040, custom board,
pico_enable_stdio_usb(1),pico_enable_stdio_uart(0)boards/metrotech_xerxes_rp2040.h - App: 1 ms main loop,
watchdog_enable(1000, 1),getchar_timeout_us(0)poll + ASCII replies (printf), ~1 kHz loop rate - Host: Debian 13, Docker app polling the CDC port 1–2 Hz (
GET/SETlines) - TinyUSB: bundled with SDK (device,
PICO_RP2040_USB_DEVICE_UFRAME_FIX=1)
Symptom:
Under sustained CDC traffic the board intermittently drops off the bus: host writes stop being answered for ~1–2 s, then Errno 5 (device gone), then the device re-enumerates (fresh tty, ~0.45 s). Frequency: between 1/min and 1/40 min under load; idle stretches of 10–20 min are clean. Kernel udev events show only a device remove/add (no hub over-current).
Evidence (on-hardware instrumentation):
Reset forensics that survive warm resets (.uninitialized_data + watchdog scratch registers) plus a custom isr_hardfault hook:
WATCHDOG->REASON= 1 (watchdog timer) on every reset — never 0 (no power/brownout)- worst main-loop iteration gap immediately before death: 8–14 ms — the loop was healthy and feeding the watchdog right up to the end
- death always lands in the command-processing /
getcharwindow of the loop (phase markers written toWATCHDOG->SCRATCH[0]); iteration counters 450k–2.2M (≈8–37 min of healthy looping) - one variant captured a hardfault: stacked
PC=0,LR=0x5001FDE4,CFSR=0x6AA8A5E1— the CFSR value is constant across separate resets and bit-pattern-wise not a plausible live fault status, i.e. the stack frame itself is suspect
Suspected mechanism:
-
The 1 ms background worker (
timer_task→low_priority_worker_irq) takesstdio_usb_mutexand callstud_task()inside IRQ context. We believetud_task()occasionally does not return (dcd/RP2040 USB wedge) while holding the mutex. -
The main loop's next
getchar_timeout_us(0)reachesstdio_usb_in_chars, sees data available, and waits on the mutex:if (!mutex_try_enter_block_until(&stdio_usb_mutex, make_timeout_time_ms(PICO_STDIO_DEADLOCK_TIMEOUT_MS))) { return PICO_ERROR_NO_DATA; // would deadlock otherwise }
-
PICO_STDIO_DEADLOCK_TIMEOUT_MSdefaults to 1000 ms — exactly the typical watchdog period. Agetchar_timeout_us(0)(documented as non-blocking!) can therefore block for the full watchdog window;watchdog_update()(called at loop top ~1 ms earlier) expires mid-block → reset,REASON=1. -
This explains all observations: loop gap stays small (the fatal call never returns, so it is never recorded), death phase stays in the
getchar/command window, resets only under traffic, USB comes back clean after the chip reboots.
Suggested fixes:
1. Don't let a non-blocking stdio poll wait 1000 ms on the mutex (contract fix):
// stdio_usb.c, stdio_usb_in_chars
int rc = PICO_ERROR_NO_DATA;
if (stdio_usb_connected() && tud_cdc_available()) {
- if (!mutex_try_enter_block_until(&stdio_usb_mutex, make_timeout_time_ms(PICO_STDIO_DEADLOCK_TIMEOUT_MS))) {
+ // in_chars is polled (often via getchar_timeout_us(0) from watchdog-fed
+ // loops); blocking here for PICO_STDIO_DEADLOCK_TIMEOUT_MS (default 1000
+ // ms == common watchdog period) starves the loop when the background
+ // tud_task worker wedges while holding the mutex. Losing this poll is
+ // harmless - it is retried on the next iteration.
+ if (!mutex_try_enter(&stdio_usb_mutex)) {
return PICO_ERROR_NO_DATA; // would deadlock otherwise
}(If a blocking variant is wanted for blocking-read users, gate it on the caller's deadline instead of a fixed 1000 ms.)
2. Escape the write loop in stdio_usb_out_chars when the driver accepts nothing (defensive):
if (n) {
int n2 = (int) tud_cdc_write(buf + i, (uint32_t)n);
tud_task();
tud_cdc_write_flush();
+ if (n2 == 0) {
+ // claimed space but accepted nothing; without this the loop spins
+ // forever with i never advancing (the 500 ms escape only exists in
+ // the n == 0 branch)
+ break;
+ }
i += n2;
last_avail_time = time_us_64();
}3. Document/config: call out that PICO_STDIO_DEADLOCK_TIMEOUT_MS (default 1000) interacts badly with watchdog-fed apps and suggest setting it well below the app watchdog period (verified: -DPICO_STDIO_DEADLOCK_TIMEOUT_MS=50 builds and is a viable mitigation).
Open question for maintainers:
We could not yet prove where tud_task() wedges — the hardfault frames we captured are themselves corrupted (PC=0, constant CFSR=0x6AA8A5E1 across resets), pointing to memory corruption during the wedge. Is a dcd_rp2040 hang under CDC control-transfer/buffer races known, even with PICO_RP2040_USB_DEVICE_UFRAME_FIX=1? We can provide instrumented builds (phase markers in watchdog scratch + a counter for mutex_try_enter_block_until timing out) if that helps.
Workaround we ship meanwhile:
Replies bypass stdio (tud_cdc_write directly under an IRQ-off critical section) so out_chars is unreachable, and the board treats the watchdog reset as self-recovery (~2 s outage; the USB device re-enumerates and the host application re-scans). It works, but the underlying wedge is real.
Activity
theMladyPan commentedon Sep 22, 2026
Root cause identified — correcting the diagnosis in the OP. The resets are
deterministic stack overflow on RP2040, not the stdio mutex wait. Evidence and
consequences for the suggested fixes below.
Mechanism (RP2040, pico-sdk 2.3.1, newlib / arm-none-eabi 15.2)
SCRATCH_Y(4 KB at 0x20041000) and is hard-cappedat 4 KB by the linker script.
PICO_STACK_SIZE=16384fails withsection .stack_dummy will not fit in region SCRATCH_Y/region SCRATCH_Y overflowed by 12288 bytes(= 16384−4096).SCRATCH_X(4 KB at 0x20040000) sits directly below.
flash_get_unique_id()runs an XIP-off SSI command sequence(
__no_inline_not_in_flash_func(flash_do_cmd_cs)) on the caller's stack.hardware_flash's docs cover the IRQ/XIP hazards ("entering a flash-residentinterrupt handler ... will be fatal") but say nothing about stack depth.
printf/snprintfframes) while thetud_taskalarm IRQ stacks onto the same MSP. WithPICO_USE_STACK_GUARDS=1the overflow trips the RP2040 PMSAv6-M MPU guard at
__StackLimit(32-bytesubregion granularity,
runtime_init_stack_guard.c); ARMv6-M escalatesMemManage to HardFault, the default handler cannot recover, and the watchdog
resets the chip one period later. Because M0+ exception entry pushes 8
registers (32 bytes) in one step against a 32-byte guard subregion, the
stacking write itself can cross the guard — which matches our otherwise
inexplicable fault frames (
PC=0, constantCFSRacross resets).Without the guard the same overflow silently corrupts adjacent state
(
SCRATCH_Xand beyond) — observed as a second mode where bulk endpointsstop transferring while the app loops normally.
Evidence (single-variable)
Repro: 1 ms loop,
watchdog_enable(1000, 1),getchar_timeout_us(0), ASCIIreplies over
stdio_usb. Per host session: burst of 1000 empty lines, burst of500 unknown-command lines, burst of 50
GETlines (each write newline-terminated, fenced with one
SET). TheGETburst resets the board 12/12 at1.0–1.2 s on every configuration below:
PICO_STDIO_DEADLOCK_TIMEOUT_MSprintfchain,flash_get_unique_id()per replyprintfchain, ID per replysnprintf+ directtud_cdc_write, ID per replyflash_get_unique_id()moved to bootRemoving the per-reply ROM call is the single change that fixes it. The 1.0–1.2 s
timing is invariant across all stdio parameters and equals one 1000 ms watchdog
period after burst onset — i.e. a fault/unrecoverable stop within the first
moments of the burst. Note
WATCHDOG_REASON(=TIMER) is ambiguous evidence: adefault
isr_hardfaultthat spins also ends in a watchdog reset.Consequences for the suggested fixes in this issue
PICO_STDIO_DEADLOCK_TIMEOUT_MS=50is measurably not a mitigation forthis class: 50 ms and 1000 ms give identical outcomes.
in_charsnon-blocking change and theout_charsloop escape remainworth landing as hardening (they close secondary routes), but they do not
affect the crash above.
PICO_USE_STACK_GUARDSis real on RP2040 (MPU present) and was enabled inthe failing builds — it converts silent corruption into a deterministic
HardFault, but the 32-byte stacking-vs-32-byte-subregion granularity limits
what it can catch, and nothing recovers from the fault.
Suggested SDK changes
hardware_flashROM wrappers (XIP-off coderuns on the caller's stack;
SCRATCH_Ycaps rp2040 stacks at 4 KB) — theexisting warnings cover IRQs and the second core only.
PICO_STACK_SIZE > 4096onrp2040 instead of a raw linker overflow.
exception-entry stacking) next to
PICO_USE_STACK_GUARDS; a canary word at__StackLimitwould complement it.flash_get_unique_id()costs an XIP-off sequence per call; a cached/lazyaccessor or a doc note would defuse its use in hot paths.
TinyUSB notes
ad7acc849("re-issue in-flight transfer on clear-halt toggle reset") is inno release tag — 0.19.0/0.20.0/0.21.0 lack it. (The
active+pendingstaterework
78d34d5b6/087ceb741did ship in 0.21.0.)PICO_RP2040_USB_DEVICE_UFRAME_FIXname but drives arewritten errata regime (
CFG_TUSB_RP2_ERRATA_E15), whose documentedsymptom — "USB Device controller will hang if certain bus errors occur during
an IN transfer" — matches our second failure mode. Against pico-sdk 2.3.1
glue, identical firmware wedges on 0.21.0 but is clean on 0.18.0 and 0.20.0.
A rename or
#warningwhen the old flag is consumed by the new regime wouldprevent glue-version traps (cf. pico-sdk#1260 on this flag's confusing
semantics).
peterharperuk commentedon Sep 23, 2026
Can you ask your LLM to summarise the issue and remove all irrelevant details - I don't want to read all that.
theMladyPan commentedon Sep 23, 2026
Sorry @peterharperuk
The issue was basically a combination of problems. The TLDR: the usb/serial interface +
printf(...)caused stack overflow when flooded with requests WHILE callingflash_get_unique_id()in each cycle, with stack basically limited to only 4kB and no clear warning on trying to enlarge stack (overflowing SCRATCH_Y bank i guess).These indirectly caused memory corruption, ghost USB hangs, starvation of watchdog and finally SW reset.
the details got indeed digged up by LLM, unfortunately I am not senior enough in this field to confirm all details, I provided them because they MIGHT be helpful.
Still a huge fan of community work, we have been using rp2040 in our sensors for 5 years now.
Thank you for understanding
Stanley
theMladyPan commentedon Sep 23, 2026
The most relevant paragraph to consider is :
Suggested SDK changes
runs on the caller's stack;
SCRATCH_Ycaps rp2040 stacks at 4 KB) — theexisting warnings cover IRQs and the second core only.
PICO_STACK_SIZE > 4096onrp2040 instead of a raw linker overflow.
exception-entry stacking) next to
PICO_USE_STACK_GUARDS; a canary word at__StackLimit would complement it.
flash_get_unique_id()costs an XIP-off sequence per call; a cached/lazyaccessor or a doc note would defuse its use in hot paths.
kvnloo commentedon Sep 29, 2026
I pulled one narrow actionable item out of this into #3216.
It documents that XIP-off
hardware_flashoperations still use the caller's stack and notes the default RP2040 core-0 stack lives in the 4 KiBSCRATCH_Yregion.No runtime change.