LLVM LTO: Get true cross-language inlining (zero overhead)

[avrabe]

release-lto profile added (emit bitcode). Picolibc fork fix merged. Need to verify gale_ symbols are actually inlined by checking nm output. Current: +893 bytes (1.8%) with LLVM vs +3824 (7.6%) with GCC.

[avrabe]

Progress landed in `4cb7549`:

1. Fixed a silent size-capture shell bug in `.github/workflows/llvm-lto.yml` — `llvm-size ... || arm-zephyr-eabi-size ... | tee FILE` was being parsed as `(llvm-size) || (arm-zephyr-eabi-size | tee FILE)` due to `|` binding tighter than `||`. llvm-size succeeded, so tee-target files were never written, and earlier "LTO savings: -53 kB" summary lines were nonsense. Fixed via brace-group.

2. Added gale_ symbol count per build to the CI step + step summary. First 20 symbol names also dumped per build.

The actual LTO state as of 2026-04-24: LLVM and LLVM+LTO produce byte-identical ELFs (51,552 bytes each) with the same 10 gale_ external symbols. Cross-language inlining across the C↔Rust FFI boundary is currently a no-op.

Hypothesised root cause: hot-path pub extern "C" fn entries in ffi/src/lib.rs lack #[inline] annotations, so their LLVM bitcode carries external linkage and lld's LTO importer declines to clone the bodies into C callers.

Suggested next step (not in this commit): annotate the engine-bench hot-path FFI (gale_sem_count_give, gale_sem_count_take, gale_k_sem_give_decide, gale_k_sem_take_decide) with #[inline], re-run the symbol count, and — assuming the count drops — add a Renode LTO lane (engine-bench-renode-lto.yml) to produce the handoff delta number this issue is targeting.

Updated docs/research/optimization-story.md to reflect the actual state (compiles under LTO, no cross-language inlining yet, concrete next steps). FV-LTO-001 added to verification.yaml as a regression guard.

[avrabe]

Experiment #1 — #[inline] on 4 hot-path sem FFI fns: NEGATIVE result.

Commit `9460426` annotated `gale_sem_count_give`, `gale_sem_count_take`, `gale_k_sem_give_decide`, `gale_k_sem_take_decide` with `#[inline]`. LLVM LTO CI run 24912237109 on commit `4c94173` (which includes the annotation commit) reports:

Build	gale_ symbols
GCC + Gale	10
LLVM + Gale (no LTO)	10
LLVM + Gale + LTO	10 (target: << 10)

Still byte-identical to LLVM-no-LTO. The CMake block does fire (`-- Gale: cross-language LTO enabled`) and the `release-lto` profile with `lto = false` + `-Clinker-plugin-lto` + `-Cembed-bitcode=yes` is configured correctly — so the setup is passing its own sanity checks, but no inlining is happening at the output.

Three remaining suspects (most likely → least):

1. `#[inline]` may be a no-op on `pub extern "C" fn` + `#[no_mangle]`. `no_mangle` forces external linkage, which may override `#[inline]`'s per-CU emission. Next experiment: try `#[inline(always)]` — stronger signal to the compiler.
2. Zephyr C compile steps may not emit bitcode despite `CONFIG_LTO=y`. If C `.o` files are ARM machine code rather than LLVM bitcode, lld has nothing to inline against. Verify via `file build-/CMakeFiles/**/.o`.
3. rustc with `-Clinker-plugin-lto` + `lto = false` may not produce bitcode-only archives. Some rustc versions need `lto = "thin"` to emit bitcode. Verify via `file ffi/target/**/libgale_ffi.a` — expect LLVM bitcode, not ARM ELF object.

Parking for now. The published `-3.1%` handoff number at `-Os` GCC is unaffected — that measurement didn't involve LTO. Re-open this line of investigation when there's bandwidth for toolchain archaeology (est. 30–60 min local).

[avrabe]

Experiment #2 — `#[inline(always)]` on the same 4 sem hot-path fns: NEGATIVE.

LTO CI run 24913352772 on commit `99a1920`: still 10/10/10 gale_ symbols across GCC+Gale, LLVM no-LTO, LLVM+LTO. (The mutex matrix lane failed with exit code 4 on the apt.llvm.org install step — unrelated infrastructure transient on that single runner; semaphore + 4 other matrix lanes + the build job all passed.)

Rust-side inlining annotations are conclusively NOT the blocker — neither `#[inline]` nor `#[inline(always)]` moved the symbol count. Both annotations reverted in `20d97e4` so the source doesn't carry cosmetic no-ops; when the real bottleneck is fixed, inline hints can be re-added with evidence.

Updated suspect list

1. Rust annotations — ruled out.
2. Zephyr C compile steps not emitting bitcode despite `CONFIG_LTO=y` and `ZEPHYR_TOOLCHAIN_VARIANT=llvm`. If `.o` files are ARM ELF rather than LLVM bitcode, lld has nothing to inline against.
3. rustc with `-Clinker-plugin-lto` + `lto = false` in `ffi/Cargo.toml` `release-lto` profile may not produce bitcode-only archives. Some rustc versions need `lto = "thin"` to emit pure bitcode.

Quick local verification (Option B from the post-mortem)

```sh

Build the Zephyr LTO target locally

west build -b qemu_cortex_m3 -d /tmp/lto-debug \
-s zephyr/tests/kernel/semaphore/semaphore -- \
-DZEPHYR_TOOLCHAIN_VARIANT=llvm \
-DCMAKE_TRY_COMPILE_TARGET_TYPE=STATIC_LIBRARY \
-DZEPHYR_EXTRA_MODULES=$GALE_ROOT \
-DOVERLAY_CONFIG="$GALE_ROOT/zephyr/gale_overlay.conf;$GALE_ROOT/zephyr/gale_lto_overlay.conf"

Are C .o files bitcode?

find /tmp/lto-debug -name '*.o' | head -3 | xargs file

Expected for cross-LTO: "LLVM IR bitcode"

What we likely see: "ELF 32-bit LSB relocatable, ARM"

Is libgale_ffi.a bitcode?

file /tmp/lto-debug/modules/gale/libgale_ffi.a # path may differ
ar t /tmp/lto-debug/modules/gale/libgale_ffi.a | head -3 | \
while read f; do ar p ... | file -; done
```

If C .o files are ELF (not bitcode), the fix is to wire object-level `-flto` into Zephyr's clang invocations. If `libgale_ffi.a` archive members are ARM machine code (not bitcode), the rustc profile needs `lto = "thin"`.

Estimated 30–60 min of local toolchain archaeology to identify which one. Parking until someone has bandwidth — the published `-3.1%` claim is unaffected.

[avrabe]

Experiment #4 — corrected baseline + `-flto=thin` injection: still 10/10/10. Parking the investigation.

What landed today

• Root cause identified (`2500fbd`): Zephyr's CMake `optimization_lto` property is empty for the clang toolchain dir (only GCC overrides it). `-flto` was never reaching clang despite `CONFIG_LTO=y`.
• Module-side fix applied (`2500fbd`): `zephyr_compile_options(-flto=thin)` + `zephyr_ld_options(-flto=thin)` inside the `llvm AND CONFIG_LTO` guard in `zephyr/CMakeLists.txt`. CMake status message confirms it fires.
• Workflow design bug fixed (`2b7841b`): `Build 3 "LLVM + Gale (no LTO)"` was applying `gale_lto_overlay.conf` (which sets `CONFIG_LTO=y`) — making it identical to Build 4. So all prior 10/10/10 numbers were comparing Build 4 against itself. Now Build 3 only applies `gale_overlay.conf` with the lld + ISR-tables settings inlined, so the diff between builds is genuinely just `CONFIG_LTO`.

What still doesn't work

Run 24923771916 on `2b7841b`: Build 3 = 10 symbols, Build 4 = 10 symbols. Even with the correct comparison and our `-flto=thin` patch firing, no cross-language inlining is happening.

Two hypotheses we can't distinguish from CI alone:

1. Our `zephyr_compile_options` IS reaching clang but lld still doesn't inline across the C↔Rust boundary. May need explicit `-Wl,--lto-O` on the link line, or matching `-flto=thin` flavor between rustc bitcode emission and lld import.
2. Our `zephyr_compile_options` injection isn't propagating to the actual compile units (e.g., scoped to gale's TUs but not the kernel/app TUs that contain the C call sites).

Why we're parking

Both hypotheses need local archaeology that the hosted CI runner can't easily provide:

• `file build-/CMakeFiles/**/.o` — definitive: bitcode or ARM ELF?
• `file libgale_ffi.a` archive members — definitive: bitcode or machine code?
• `west build -v` to see the actual `clang -flto=thin ...` command line

Estimated 30–60 min in a host with the Zephyr SDK + ld.lld + ARM bare-metal sysroot to nail down. Out of scope for what can be iterated through CI cycles.

What's preserved

• The `zephyr/CMakeLists.txt` `-flto=thin` injection stays in place. It's a factually-correct fix for the documented Zephyr clang LTO hole even if it's not sufficient on its own; when the deeper blocker is identified, our flag is already there.
• The corrected workflow baseline (Build 3 ≠ Build 4) stays — future runs now produce a real comparison.
• FV-LTO-001 in `artifacts/verification.yaml` is the regression guard for the symbol count.

Public claim status

Unchanged: `-3.1%` handoff median / `-2.6%` max at `-Os` GCC, Renode, N=7750, p<1e-100. That measurement never went near LTO. The aspirational `-10%` to `-30%` LTO row in `docs/research/optimization-story.md` stays as "expected once cross-language inlining works" — we now know more about why it isn't yet.