Skip to content

stdio_usb: getchar_timeout_us(0) can block for PICO_STDIO_DEADLOCK_TIMEOUT_MS (default 1000 ms) while the tud_task alarm-IRQ holds stdio_usb_mutex — starves app watchdog loops and turns USB stack wedges into watchdog resets #3201

Open

Description

@theMladyPan

Environment:

  • pico-sdk 2.1.0 and 2.3.1 (checked both, code identical in the relevant paths)
  • RP2040, custom board, pico_enable_stdio_usb(1), pico_enable_stdio_uart(0) boards/metrotech_xerxes_rp2040.h
  • App: 1 ms main loop, watchdog_enable(1000, 1), getchar_timeout_us(0) poll + ASCII replies (printf), ~1 kHz loop rate
  • Host: Debian 13, Docker app polling the CDC port 1–2 Hz (GET/SET lines)
  • TinyUSB: bundled with SDK (device, PICO_RP2040_USB_DEVICE_UFRAME_FIX=1)

Symptom:

Under sustained CDC traffic the board intermittently drops off the bus: host writes stop being answered for ~1–2 s, then Errno 5 (device gone), then the device re-enumerates (fresh tty, ~0.45 s). Frequency: between 1/min and 1/40 min under load; idle stretches of 10–20 min are clean. Kernel udev events show only a device remove/add (no hub over-current).

Evidence (on-hardware instrumentation):

Reset forensics that survive warm resets (.uninitialized_data + watchdog scratch registers) plus a custom isr_hardfault hook:

  • WATCHDOG->REASON = 1 (watchdog timer) on every reset — never 0 (no power/brownout)
  • worst main-loop iteration gap immediately before death: 8–14 ms — the loop was healthy and feeding the watchdog right up to the end
  • death always lands in the command-processing / getchar window of the loop (phase markers written to WATCHDOG->SCRATCH[0]); iteration counters 450k–2.2M (≈8–37 min of healthy looping)
  • one variant captured a hardfault: stacked PC=0, LR=0x5001FDE4, CFSR=0x6AA8A5E1 — the CFSR value is constant across separate resets and bit-pattern-wise not a plausible live fault status, i.e. the stack frame itself is suspect

Suspected mechanism:

  1. The 1 ms background worker (timer_task → low_priority_worker_irq) takes stdio_usb_mutex and calls tud_task() inside IRQ context. We believe tud_task() occasionally does not return (dcd/RP2040 USB wedge) while holding the mutex.

  2. The main loop's next getchar_timeout_us(0) reaches stdio_usb_in_chars, sees data available, and waits on the mutex:

    if (!mutex_try_enter_block_until(&stdio_usb_mutex, make_timeout_time_ms(PICO_STDIO_DEADLOCK_TIMEOUT_MS))) {
        return PICO_ERROR_NO_DATA; // would deadlock otherwise
    }
  3. PICO_STDIO_DEADLOCK_TIMEOUT_MS defaults to 1000 ms — exactly the typical watchdog period. A getchar_timeout_us(0) (documented as non-blocking!) can therefore block for the full watchdog window; watchdog_update() (called at loop top ~1 ms earlier) expires mid-block → reset, REASON=1.

  4. This explains all observations: loop gap stays small (the fatal call never returns, so it is never recorded), death phase stays in the getchar/command window, resets only under traffic, USB comes back clean after the chip reboots.

Suggested fixes:

1. Don't let a non-blocking stdio poll wait 1000 ms on the mutex (contract fix):

// stdio_usb.c, stdio_usb_in_chars
int rc = PICO_ERROR_NO_DATA;
if (stdio_usb_connected() && tud_cdc_available()) {
-    if (!mutex_try_enter_block_until(&stdio_usb_mutex, make_timeout_time_ms(PICO_STDIO_DEADLOCK_TIMEOUT_MS))) {
+    // in_chars is polled (often via getchar_timeout_us(0) from watchdog-fed
+    // loops); blocking here for PICO_STDIO_DEADLOCK_TIMEOUT_MS (default 1000
+    // ms == common watchdog period) starves the loop when the background
+    // tud_task worker wedges while holding the mutex. Losing this poll is
+    // harmless - it is retried on the next iteration.
+    if (!mutex_try_enter(&stdio_usb_mutex)) {
         return PICO_ERROR_NO_DATA; // would deadlock otherwise
     }

(If a blocking variant is wanted for blocking-read users, gate it on the caller's deadline instead of a fixed 1000 ms.)

2. Escape the write loop in stdio_usb_out_chars when the driver accepts nothing (defensive):

 if (n) {
     int n2 = (int) tud_cdc_write(buf + i, (uint32_t)n);
     tud_task();
     tud_cdc_write_flush();
+    if (n2 == 0) {
+        // claimed space but accepted nothing; without this the loop spins
+        // forever with i never advancing (the 500 ms escape only exists in
+        // the n == 0 branch)
+        break;
+    }
     i += n2;
     last_avail_time = time_us_64();
 }

3. Document/config: call out that PICO_STDIO_DEADLOCK_TIMEOUT_MS (default 1000) interacts badly with watchdog-fed apps and suggest setting it well below the app watchdog period (verified: -DPICO_STDIO_DEADLOCK_TIMEOUT_MS=50 builds and is a viable mitigation).

Open question for maintainers:

We could not yet prove where tud_task() wedges — the hardfault frames we captured are themselves corrupted (PC=0, constant CFSR=0x6AA8A5E1 across resets), pointing to memory corruption during the wedge. Is a dcd_rp2040 hang under CDC control-transfer/buffer races known, even with PICO_RP2040_USB_DEVICE_UFRAME_FIX=1? We can provide instrumented builds (phase markers in watchdog scratch + a counter for mutex_try_enter_block_until timing out) if that helps.

Workaround we ship meanwhile:

Replies bypass stdio (tud_cdc_write directly under an IRQ-off critical section) so out_chars is unreachable, and the board treats the watchdog reset as self-recovery (~2 s outage; the USB device re-enumerates and the host application re-scans). It works, but the underlying wedge is real.

Activity

  1. theMladyPan commented on Sep 22, 2026

    @theMladyPan
    ContributorAuthor

    Root cause identified — correcting the diagnosis in the OP. The resets are
    deterministic stack overflow on RP2040, not the stdio mutex wait. Evidence and
    consequences for the suggested fixes below.

    Mechanism (RP2040, pico-sdk 2.3.1, newlib / arm-none-eabi 15.2)

    1. The core-0 stack lives in SCRATCH_Y (4 KB at 0x20041000) and is hard-capped
      at 4 KB by the linker script. PICO_STACK_SIZE=16384 fails with
      section .stack_dummy will not fit in region SCRATCH_Y /
      region SCRATCH_Y overflowed by 12288 bytes (= 16384−4096). SCRATCH_X
      (4 KB at 0x20040000) sits directly below.
    2. flash_get_unique_id() runs an XIP-off SSI command sequence
      (__no_inline_not_in_flash_func(flash_do_cmd_cs)) on the caller's stack.
      hardware_flash's docs cover the IRQ/XIP hazards ("entering a flash-resident
      interrupt handler ... will be fatal") but say nothing about stack depth.
    3. Reply bursts deepen the stack (newlib printf/snprintf frames) while the
      tud_task alarm IRQ stacks onto the same MSP. With PICO_USE_STACK_GUARDS=1
      the overflow trips the RP2040 PMSAv6-M MPU guard at __StackLimit (32-byte
      subregion granularity, runtime_init_stack_guard.c); ARMv6-M escalates
      MemManage to HardFault, the default handler cannot recover, and the watchdog
      resets the chip one period later. Because M0+ exception entry pushes 8
      registers (32 bytes) in one step against a 32-byte guard subregion, the
      stacking write itself can cross the guard — which matches our otherwise
      inexplicable fault frames (PC=0, constant CFSR across resets).
      Without the guard the same overflow silently corrupts adjacent state
      (SCRATCH_X and beyond) — observed as a second mode where bulk endpoints
      stop transferring while the app loops normally.

    Evidence (single-variable)

    Repro: 1 ms loop, watchdog_enable(1000, 1), getchar_timeout_us(0), ASCII
    replies over stdio_usb. Per host session: burst of 1000 empty lines, burst of
    500 unknown-command lines, burst of 50 GET lines (each write newline-
    terminated, fenced with one SET). The GET burst resets the board 12/12 at
    1.0–1.2 s on every configuration below:

    reply path stack TinyUSB PICO_STDIO_DEADLOCK_TIMEOUT_MS result
    printf chain, flash_get_unique_id() per reply 2 KB 0.18.0 1000 12/12 resets
    printf chain, ID per reply 4 KB 0.18.0 1000 12/12 resets
    one snprintf + direct tud_cdc_write, ID per reply 4 KB 0.18.0 1000 12/12 resets
    same 4 KB master b80f1c107 1000 12/12 resets
    same 4 KB master b80f1c107 50 12/12 resets
    same, flash_get_unique_id() moved to boot 4 KB 0.18.0 / 0.20.0 / master 50 0/8 clean on all three

    Removing the per-reply ROM call is the single change that fixes it. The 1.0–1.2 s
    timing is invariant across all stdio parameters and equals one 1000 ms watchdog
    period after burst onset — i.e. a fault/unrecoverable stop within the first
    moments of the burst. Note WATCHDOG_REASON (=TIMER) is ambiguous evidence: a
    default isr_hardfault that spins also ends in a watchdog reset.

    Consequences for the suggested fixes in this issue

    • PICO_STDIO_DEADLOCK_TIMEOUT_MS=50 is measurably not a mitigation for
      this class: 50 ms and 1000 ms give identical outcomes.
    • The in_chars non-blocking change and the out_chars loop escape remain
      worth landing as hardening (they close secondary routes), but they do not
      affect the crash above.
    • PICO_USE_STACK_GUARDS is real on RP2040 (MPU present) and was enabled in
      the failing builds — it converts silent corruption into a deterministic
      HardFault, but the 32-byte stacking-vs-32-byte-subregion granularity limits
      what it can catch, and nothing recovers from the fault.

    Suggested SDK changes

    1. Document the stack dimension of hardware_flash ROM wrappers (XIP-off code
      runs on the caller's stack; SCRATCH_Y caps rp2040 stacks at 4 KB) — the
      existing warnings cover IRQs and the second core only.
    2. Fail configuration with a clear message for PICO_STACK_SIZE > 4096 on
      rp2040 instead of a raw linker overflow.
    3. Document the PMSAv6-M guard granularity limit (32-byte subregion vs 32-byte
      exception-entry stacking) next to PICO_USE_STACK_GUARDS; a canary word at
      __StackLimit would complement it.
    4. flash_get_unique_id() costs an XIP-off sequence per call; a cached/lazy
      accessor or a doc note would defuse its use in hot paths.

    TinyUSB notes

    • ad7acc849 ("re-issue in-flight transfer on clear-halt toggle reset") is in
      no release tag — 0.19.0/0.20.0/0.21.0 lack it. (The active+pending state
      rework 78d34d5b6/087ceb741 did ship in 0.21.0.)
    • 0.21.0 keeps the PICO_RP2040_USB_DEVICE_UFRAME_FIX name but drives a
      rewritten errata regime (CFG_TUSB_RP2_ERRATA_E15), whose documented
      symptom — "USB Device controller will hang if certain bus errors occur during
      an IN transfer" — matches our second failure mode. Against pico-sdk 2.3.1
      glue, identical firmware wedges on 0.21.0 but is clean on 0.18.0 and 0.20.0.
      A rename or #warning when the old flag is consumed by the new regime would
      prevent glue-version traps (cf. pico-sdk#1260 on this flag's confusing
      semantics).
  2. peterharperuk commented on Sep 23, 2026

    @peterharperuk
    Contributor

    Can you ask your LLM to summarise the issue and remove all irrelevant details - I don't want to read all that.

  3. theMladyPan commented on Sep 23, 2026

    @theMladyPan
    ContributorAuthor

    Sorry @peterharperuk

    The issue was basically a combination of problems. The TLDR: the usb/serial interface + printf(...) caused stack overflow when flooded with requests WHILE calling flash_get_unique_id() in each cycle, with stack basically limited to only 4kB and no clear warning on trying to enlarge stack (overflowing SCRATCH_Y bank i guess).

    These indirectly caused memory corruption, ghost USB hangs, starvation of watchdog and finally SW reset.

    the details got indeed digged up by LLM, unfortunately I am not senior enough in this field to confirm all details, I provided them because they MIGHT be helpful.

    Still a huge fan of community work, we have been using rp2040 in our sensors for 5 years now.
    Thank you for understanding
    Stanley

  4. theMladyPan commented on Sep 23, 2026

    @theMladyPan
    ContributorAuthor

    The most relevant paragraph to consider is :

    Suggested SDK changes

    • Document the stack dimension of hardware_flash ROM wrappers (XIP-off code
      runs on the caller's stack; SCRATCH_Y caps rp2040 stacks at 4 KB) — the
      existing warnings cover IRQs and the second core only.
    • Fail configuration with a clear message for PICO_STACK_SIZE > 4096 on
      rp2040 instead of a raw linker overflow.
    • Document the PMSAv6-M guard granularity limit (32-byte subregion vs 32-byte
      exception-entry stacking) next to PICO_USE_STACK_GUARDS; a canary word at
      __StackLimit would complement it.
    • flash_get_unique_id() costs an XIP-off sequence per call; a cached/lazy
      accessor or a doc note would defuse its use in hot paths.
  5. kvnloo commented on Sep 29, 2026

    @kvnloo

    I pulled one narrow actionable item out of this into #3216.

    It documents that XIP-off hardware_flash operations still use the caller's stack and notes the default RP2040 core-0 stack lives in the 4 KiB SCRATCH_Y region.

    No runtime change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions