Skip to content

Nested flash_safe_execute() leaves interrupts disabled; rom_explicit_buy() and rom_flash_op() nest it without saying so #3298

Open

Description

@gittest23

Summary

flash_safe_execute() cannot be nested. An inner call overwrites the saved interrupt state of
the calling core, so the outer call returns PICO_OK with interrupts still disabled; no error
is reported anywhere. Since #1835 (SDK 2.1.0), rom_explicit_buy() and rom_flash_op() call
flash_safe_execute() themselves. Code that wraps them in its own flash_safe_execute() hits
this silently. For rom_explicit_buy() that wrapping was correct in SDK 2.0.0, where it was a
bare ROM call, so such code broke with an SDK update.

Environment

  • pico-sdk 2.3.1. pico_flash/flash.c, pico_flash/include/pico/flash.h and
    pico_bootrom/include/pico/bootrom.h are unchanged on master and develop as of today.
  • RP2350 (Pico 2 W), Arm, no RTOS. pico_multicore with core 1 running and
    flash_safe_execute_core_init() called on it (the default multicore-lockout helper).

What we saw

Our firmware called rom_explicit_buy() inside its own flash_safe_execute(), to keep core 1
off the flash during the buy. The buy succeeded, but core 0 then ran on with PRIMASK set.
Nothing failed outright: in pico_cyw43_arch poll mode the radio's data then arrived only with
the driver's periodic check, so Bluetooth input came in 50 ms bursts. It took a while to trace
this back to the nesting.

Cause

In src/rp2_common/pico_flash/flash.c:

  • L105-106: static uint32_t irq_state[NUM_CORES];, one saved state per core
  • L203, default_enter_safe_zone_timeout_ms(): irq_state[get_core_num()] = save_and_disable_interrupts();
  • L210, default_exit_safe_zone_timeout_ms(): restore_interrupts_from_disabled(irq_state[get_core_num()]);

Nested, the outer entry saves "enabled", the inner entry overwrites it with "disabled", the
inner exit restores "disabled", and so does the outer exit.

There is a second effect. The inner exit also calls multicore_lockout_end_timeout_us(), which
releases the other core while the outer callback is still running. Anything the outer callback
does with the flash after the inner call returns is then unprotected.

https://github.com/raspberrypi/pico-sdk/blob/2.3.1/src/rp2_common/pico_flash/flash.c#L105-L106
https://github.com/raspberrypi/pico-sdk/blob/2.3.1/src/rp2_common/pico_flash/flash.c#L203
https://github.com/raspberrypi/pico-sdk/blob/2.3.1/src/rp2_common/pico_flash/flash.c#L210
https://github.com/raspberrypi/pico-sdk/blob/2.3.1/src/rp2_common/pico_bootrom/include/pico/bootrom.h#L985-L992

Minimal reproduction

With pico_multicore, core 1 running and flash_safe_execute_core_init() called there.
The result in the comment follows from the code above; we hit the same thing through
rom_explicit_buy().

static void inner(void *p) { (void)p; }
static void outer(void *p) { (void)p; flash_safe_execute(inner, NULL, 100); }

    // ... on core 0, with interrupts enabled:
    int rc = flash_safe_execute(outer, NULL, 100);   // PICO_OK
    uint32_t primask;
    __asm volatile ("mrs %0, primask" : "=r" (primask));
    printf("rc=%d PRIMASK=%lu\n", rc, (unsigned long)primask);   // PRIMASK=1: still disabled

Suggestions

  1. Detect nesting in flash_safe_execute() and refuse it, so the mistake shows up instead of
    leaving interrupts off:

    static volatile bool in_safe_zone[NUM_CORES];
    
    int flash_safe_execute(void (*func)(void *), void *param, uint32_t enter_exit_timeout_ms) {
        uint core = get_core_num();
        if (in_safe_zone[core]) {
    #if PICO_FLASH_ASSERT_ON_UNSAFE
            assert(false);   // nested flash_safe_execute
    #endif
            return PICO_ERROR_INVALID_STATE;
        }
        flash_safety_helper_t *helper = get_flash_safety_helper();
        if (!helper) return PICO_ERROR_NOT_PERMITTED;
        int rc = helper->enter_safe_zone_timeout_ms(enter_exit_timeout_ms);
        if (!rc) {
            in_safe_zone[core] = true;
            func(param);
            in_safe_zone[core] = false;
            rc = helper->exit_safe_zone_timeout_ms(enter_exit_timeout_ms);
        }
        return rc;
    }

    Interrupts are disabled on the core while its flag is set, and each core has its own slot,
    so the flag needs no lock. The alternative is a per-core depth counter, where only the
    outermost call enters and exits and inner calls just run their callback. That keeps
    existing nested code working, but it also hides the mistake and ignores the inner call's
    timeout.

  2. Document it. Add a note on flash_safe_execute() in flash.h saying it must not be called
    from within a function passed to it, directly or through rom_flash_op(),
    rom_explicit_buy() or the BTstack flash bank. Add a note on those two bootrom.h functions
    saying they run inside flash_safe_execute(). The other rom_flash_* wrappers
    (rom_flash_range_erase, rom_flash_range_program, rom_flash_exit_xip, ...) are bare and
    must be protected by the caller; nothing in the names shows which kind a function is.
    Documenting rom_helper_flash_op() and rom_helper_explicit_buy() as public API would also
    give callers a supported way to run several operations under one lockout.

  3. pico_btstack's flash bank ignores the return value of both its calls
    (btstack_flash_bank.c L82 and L170). The comment there relies on flash_safe_execute()
    asserting, but those asserts are compiled out in Release builds (NDEBUG), so a failed or
    timed-out call loses the TLV write silently (compare pico_multicore breaks BTstack TLV persistence #2725). With suggestion 1, a refused
    nested call would be lost the same way, so it should at least be logged, or panic.

Related

We now link with -Wl,--wrap=flash_safe_execute and a wrapper that refuses nested calls and
logs the caller's address.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions