Skip to content

test(firmware): pin what the firmware-update state machine refuses to do - #718

Merged
tylerkron merged 2 commits into
mainfrom
test/coverage-gap-9
Aug 31, 2026
Merged

tylerkron merged 2 commits into
mainfrom
test/coverage-gap-9

Conversation

@tylerkron

Copy link
Copy Markdown
Contributor

Produced by the coverage-gap maintenance routine (run 9). Tests only — no production code is touched, so it cannot change behaviour. Not merging — for review.

What was wrong

FirmwareUpdateService drives a state machine, and consumers see it through two pieces of public API: FirmwareUpdateState CurrentState and the StateChanged event. The machine's legal-transition table lives in an internal collaborator, and today it is only ever walked forwards by the two update flows (PIC32 and WiFi) along the transitions they already make. Nothing has ever asked it to say no.

That leaves the whole denial half of the contract unpinned. Someone could add an edge letting Programming jump straight to Complete — reporting a successful update that was never verified — or letting Failed go back to PreparingDevice so a retry resumes mid-flow instead of restarting, and every test in the suite would still be green. ResetIfTerminalState has the same shape: the flows only ever call it when the machine is already terminal, so its guard could be deleted entirely and nothing would notice, even though without it the call the service makes before starting work would silently rewind an in-flight update to Idle.

Separately, FirmwareUpdateStateChangedEventArgs.PreviousState is shipped public API and was asserted nowhere at all — the two states could have been reported the wrong way round and only a consumer's progress display would have shown it.

How it is fixed

One new test file, FirmwareUpdateStateMachineTests.cs (23 cases). Rather than restating the table row by row, it asserts properties of the machine — every phase can still reach Failed, and no state is stranded away from Idle so the service stays reusable — plus the specific edges that must not exist, the self-transition shortcut that renames the current operation without announcing a change, and a rejected transition leaving state, operation and subscribers untouched.

The graph the property tests reason over is discovered by probing a real context, not read out of the production dictionary, so a mutated table produces a mutated graph and the tests cannot quietly agree with it.

Mutation evidence

Every case was proved able to fail. Sixteen mutations were applied one at a time to FirmwareUpdateContext.cs from a pristine copy, each re-running the new file; all sixteen went red and the baseline was green again after each restore. The union of what they killed is all 23 cases.

They cover: dropping a phase's Failed edge; stranding Recovered with no route home; six added edges that skip a flash step (Programming/ErasingFlash → Complete, ErasingFlash → Verifying, Connecting → Programming, WaitingForBootloader → ErasingFlash, ReconnectingAfterFlash → CleaningUp); letting Failed and Complete resume mid-flow; removing or hollowing out the self-transition shortcut; reporting the transition backwards; mutating CurrentState before the legality guard; resetting unconditionally; and dropping Recovered from the terminal set.

Verification

Full solution suite under a test lock, -warnaserror: net9.0 and net10.0 both 4171 passed / 3 skipped / 0 failed, Daqifi.Mcp.Tests 217 passed on both. Zero warnings. git diff --stat against main touches nothing under src/Daqifi.Core/.

🤖 Generated with Claude Code

The transition table in FirmwareUpdateContext is only ever walked forwards by the
two update flows, so nothing today asks it to say no. An edge added to the table —
letting Programming report Complete without verifying, or letting Failed resume
mid-flow instead of restarting — breaks no existing test. ResetIfTerminalState has
the same shape: the flows only call it when the machine is already terminal, so its
guard could vanish unnoticed.

Adds direct tests that assert the machine's properties (every phase can still report
failure; no state is stranded away from Idle) and the specific edges that must not
exist, plus the self-transition shortcut, the rejection path leaving nothing
half-applied, and PreviousState on the public event args, which nothing asserted.

The graph is discovered by probing a real context rather than read out of the
production dictionary, so a mutated table produces a mutated graph.

Tests only; no production code changed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@tylerkron
tylerkron requested a review from a team as a code owner August 29, 2026 17:40
@tylerkron

Copy link
Copy Markdown
Contributor Author

/agentic_review

@qodo-code-review

Copy link
Copy Markdown

PR Summary by Qodo

Pin firmware update state machine rejection contracts

🧪 Tests 🕐 20-40 Minutes

Grey Divider

AI Description

• Add 23 direct tests for denied transitions, terminal resets, and failure/recovery reachability.
• Probe real contexts to discover transition graphs independently of production table internals.
• Pin event semantics while ensuring rejected and self-transitions preserve observable behavior.
Diagram

graph TD
  T["State Tests"] --> P["Probe Helpers"] --> C["Real Context"] --> R["Transition Rules"]
  P --> G["Discovered Graph"] --> A["Contract Assertions"]
  C --> E["State Events"] --> A
Loading
High-Level Assessment

The following are alternative approaches to this PR:

1. Enumerate the transition table explicitly
  • ➕ Simpler tests with straightforward expected edges
  • ➕ Failures identify the exact changed transition immediately
  • ➖ Duplicates the production table row by row
  • ➖ Creates brittle maintenance when valid workflows evolve
  • ➖ Risks tests being mechanically updated to match an incorrect table
2. Cover denials through update flows
  • ➕ Exercises the public service facade end to end
  • ➕ Avoids direct testing of an internal collaborator
  • ➖ Flows only traverse intended forward paths
  • ➖ Forcing arbitrary invalid transitions requires invasive seams
  • ➖ Graph-wide reachability and atomic rejection remain difficult to prove

Recommendation: Keep the probe-based property tests. They independently observe the real context, catch both missing and newly added edges, and complement focused forbidden-edge cases without mirroring production data. Explicit table snapshots and facade-only tests were considered but provide weaker mutation resistance or require unnecessary test seams.

Files changed (1) +321 / -0

Tests (1) +321 / -0
FirmwareUpdateStateMachineTests.csAdd direct firmware state-machine contract coverage +321/-0

Add direct firmware state-machine contract coverage

• Adds 23 cases covering graph reachability, forbidden flash-step shortcuts, terminal-state isolation, self-transition behavior, rejected-transition atomicity, reset guards, and state-change event payloads. Helper routines probe real FirmwareUpdateContext instances to derive paths and edges without reading the production transition dictionary.

src/Daqifi.Core.Tests/Firmware/FirmwareUpdateStateMachineTests.cs

@qodo-code-review

qodo-code-review Bot commented Aug 29, 2026 •

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📜 Skill insights (0)

Grey Divider


Remediation recommended

1. Unreachable states escape properties ✓ Resolved 🐞 Bug ⚙ Maintainability
Description
Both graph properties iterate paths.Keys, but DiscoverShortestPaths only inserts states already
reachable from Idle; disconnecting an enum state therefore removes it from the assertions instead
of failing them. For example, removing every inbound transition to JumpingToApp leaves that state
untested by these properties despite the tests claiming every phase/state is covered.
Code

src/Daqifi.Core.Tests/Firmware/FirmwareUpdateStateMachineTests.cs[R47-49]

+        var cannotFail = paths.Keys
+            .Where(state => !TerminalStates.Contains(state))
+            .Where(state => !CanTransition(paths[state], FirmwareUpdateState.Failed))
Relevance

●●● Strong

Recent accepted reviews strengthen incomplete exhaustive coverage and tests that miss cases through
partial enumeration.

PR-#660
PR-#680

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The test defines all enum values separately, but discovery starts with only Idle and adds a state
only after finding an accepted inbound edge; the assertions then iterate only the resulting partial
key set. The production enum includes JumpingToApp, and the finite transition table shows it can
be disconnected by losing its inbound entries.

src/Daqifi.Core.Tests/Firmware/FirmwareUpdateStateMachineTests.cs[28-29]
src/Daqifi.Core.Tests/Firmware/FirmwareUpdateStateMachineTests.cs[244-270]
src/Daqifi.Core/Firmware/FirmwareUpdateState.cs[46-49]
src/Daqifi.Core/Firmware/FirmwareUpdateContext.cs[47-60]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The graph property tests silently omit enum values that are unreachable from `Idle`, allowing disconnected firmware states to escape coverage.

## Issue Context
`DiscoverShortestPaths` contains only discovered states, while both broad properties iterate that partial dictionary. Assert that discovery covers `AllStates` before indexing paths or evaluating graph properties.

## Fix Focus Areas
- src/Daqifi.Core.Tests/Firmware/FirmwareUpdateStateMachineTests.cs[45-69]
- src/Daqifi.Core.Tests/Firmware/FirmwareUpdateStateMachineTests.cs[244-270]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


2. Reset guard skips active states ✓ Resolved 🐞 Bug ⚙ Maintainability
Description
The non-terminal reset theory omits Connecting, JumpingToApp, and ReconnectingAfterFlash, even
though all three are active update states. A regression that treats any one of these as terminal
would rewind an in-flight operation to Idle while this guard test remains green.
Code

src/Daqifi.Core.Tests/Firmware/FirmwareUpdateStateMachineTests.cs[R184-189]

+    [InlineData(FirmwareUpdateState.PreparingDevice)]
+    [InlineData(FirmwareUpdateState.WaitingForBootloader)]
+    [InlineData(FirmwareUpdateState.ErasingFlash)]
+    [InlineData(FirmwareUpdateState.Programming)]
+    [InlineData(FirmwareUpdateState.Verifying)]
+    [InlineData(FirmwareUpdateState.CleaningUp)]
Relevance

●●● Strong

Recent accepted reviews add omitted regression cases to exhaustive test populations; this is the
same state-coverage gap.

PR-#660
PR-#620

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The theory lists six non-terminal values, while the public enum documents Connecting, JumpingToApp,
and ReconnectingAfterFlash as lifecycle phases. The production transition table includes each as a
live state, and the service calls the reset method immediately before requiring the context to
remain Idle-only, so resetting one of them would bypass the intended in-flight rejection.

src/Daqifi.Core.Tests/Firmware/FirmwareUpdateStateMachineTests.cs[183-205]
src/Daqifi.Core/Firmware/FirmwareUpdateState.cs[23-26]
src/Daqifi.Core/Firmware/FirmwareUpdateState.cs[46-49]
src/Daqifi.Core/Firmware/FirmwareUpdateState.cs[79-91]
src/Daqifi.Core/Firmware/FirmwareUpdateService.cs[317-326]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`ResetIfTerminalState_LeavesAnUpdateThatIsStillRunningAlone` covers only six active states and misses three other in-flight states.

## Issue Context
Add `Connecting`, `JumpingToApp`, and `ReconnectingAfterFlash`, or generate cases from all enum values excluding the three terminal outcomes (and separately handle `Idle` if desired), so widening the terminal guard cannot escape coverage.

## Fix Focus Areas
- src/Daqifi.Core.Tests/Firmware/FirmwareUpdateStateMachineTests.cs[183-205]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

Context sources
Review mode: ⚖️ Balanced

Grey Divider

Tip of the day
💡 Did you know, you can type 'qodo, fix this' on a finding and the fix lands right on your PR

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Grey Divider

Qodo Logo

…erties

Qodo round 1, both findings conceded.

A state that lost every inbound edge fell out of the discovered graph, so both
graph properties stopped covering it instead of failing — the coverage vanished
with the state. The properties now walk the declared enum, and a new case asserts
every declared state is still reachable from Idle, so disconnecting one fails loudly.

The non-terminal reset theory was also missing Connecting, JumpingToApp and
ReconnectingAfterFlash, all live phases: treating any of them as terminal would
rewind an in-flight update to Idle with the guard test still green.

Mutations rerun (19, all red, baseline green after each restore). Three are new:
disconnecting JumpingToApp now kills 4 cases where it previously killed none, and
treating Connecting or ReconnectingAfterFlash as terminal each kill their new row.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@tylerkron

Copy link
Copy Markdown
Contributor Author

Both round-1 findings conceded and fixed in e5802a5.

1. Unreachable states escape properties — correct, and it was the more interesting of the two. DiscoverShortestPaths only contains states it found a route to, so removing every inbound edge to a state deleted it from paths.Keys and both properties quietly stopped asserting anything about it. The properties now iterate the declared enum and look paths up through a PathTo helper that fails with the state's name if it is missing, and a new case (EveryDeclaredState_IsReachableFromIdle) asserts the discovery covers every enum value. Proved with a new mutation that drops JumpingToApp's only two inbound edges: 0 cases before the fix, 4 after.

2. Reset guard skips active states — also correct. Connecting, JumpingToApp and ReconnectingAfterFlash are live phases and were missing from the non-terminal reset theory, which now has 9 rows. Two new mutations (treat Connecting as terminal; treat ReconnectingAfterFlash as terminal) each go red on exactly their new row.

Suite is 27 cases now. Mutation set rerun end to end: 19 mutations, all 19 red, baseline green after each restore, union of kills covering all 27 cases. Full solution suite under a test lock with -warnaserror: net9.0 and net10.0 both 4175 passed / 3 skipped / 0 failed, Daqifi.Mcp.Tests 217 passed on both, zero warnings.

/agentic_review

@tylerkron

Copy link
Copy Markdown
Contributor Author

/agentic_review

@qodo-code-review

Copy link
Copy Markdown

Code review by qodo was updated up to the latest commit e5802a5

@tylerkron

Copy link
Copy Markdown
Contributor Author

Qodo-clean, CI green — ready for review. (Round 2 on e5802a5: Bugs (0), both round-1 threads resolved; re-checked after a settle wait against the same head SHA. build + test on ubuntu/macos/windows all pass.)

@tylerkron
tylerkron added this pull request to the merge queue Aug 31, 2026
Merged via the queue into main with commit f009407 Aug 31, 2026
4 checks passed
@tylerkron
tylerkron deleted the test/coverage-gap-9 branch August 31, 2026 03:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant