feat(apply): implement deterministic reconciliation

[nisavid]

 

• 
  • src/cqmgr/adapters/google/quota_preference_writes.py
  • src/cqmgr/adapters/persistence/apply_records.py
  • src/cqmgr/adapters/persistence/audit.py
  • src/cqmgr/adapters/persistence/plans.py
  • src/cqmgr/application/operations/apply.py
  • src/cqmgr/application/operations/plans.py
  • src/cqmgr/application/ports/apply.py
  • src/cqmgr/application/ports/apply_records.py
  • src/cqmgr/application/ports/audit.py
  • src/cqmgr/application/ports/plans.py
  • src/cqmgr/application/ports/provider_writes.py
  • src/cqmgr/domain/apply_records.py
  • src/cqmgr/domain/audit.py
  • src/cqmgr/domain/plan_consumption.py
  • src/cqmgr/domain/plans.py
• 
  • .github/workflows/python.yml
  • scripts/verify_distribution.py
  • tests/adapters/test_google_quota_preference_writes.py
  • tests/application/test_apply_operations.py
  • tests/domain/test_apply_records.py
  • tests/plans/test_apply_record_repository.py
  • tests/plans/test_plan_codec.py
  • tests/plans/test_plan_consumption.py
  • tests/plans/test_plan_repository.py

Summary

• Implements Apply as ordered, non-atomic reconciliation over an immutable plan, with complete-bundle revalidation and a durable consumption barrier before the first provider write.
• Persists authenticated aggregate and per-child intent/outcome evidence, stops on failed or unknown children, preserves later children as unattempted, and recovers deterministically across crash windows without duplicate dispatch.
• Adds the official Cloud Quotas write/read-after-unknown adapter with explicit generic retries disabled and exact identity, etag, target, scope, and contact matching.

Review path

1. Start with the Apply state machine, preflight, recovery, and result contract in src/cqmgr/application/operations/apply.py.
2. Review explicit read/write ports, authenticated Apply records, and append-only unknown-resolution checkpoints in src/cqmgr/application/ports/apply.py, src/cqmgr/domain/apply_records.py, and src/cqmgr/adapters/persistence/apply_records.py.
3. Follow the recoverable plan consumption barrier in src/cqmgr/adapters/persistence/plans.py, src/cqmgr/domain/plan_consumption.py, and the related ports.
4. Finish with retry-free provider dispatch and reconciliation in src/cqmgr/adapters/google/quota_preference_writes.py, then the adversarial crash/retry tests.

Changes

• Revalidates every child, acquires the exclusive plan lease, writes an fsynced aggregate intent, and crosses the immutable consumption barrier before the first provider call.
• Binds Apply to the fresh principal/contact/evidence set and invalidates the whole plan on pre-dispatch drift or incomplete evidence.
• Classifies accepted, failed, unknown, critical-unknown, and unattempted children without flattening the aggregate result.
• Recovers expired and partial barriers fail-closed, resumes only safe undispatched children, and converts intent-without-outcome to unknown without redispatch.
• Records read-after-unknown resolution in a separate authenticated, atomic, append-only, single-assignment journal.
• Keeps timeout NotFound and nonmatching observations unresolved until rejection is proven, while retaining exact reconciliation identity even when containment persistence fails.
• Completes terminal accepted, failed, and provably no-write records safely after restart without redispatch or same-revision rewrites.
• Replays consumed terminal results only from authenticated immutable plan bytes after rechecking installation, principal, contact, scope, and acknowledgement bindings.
• Preserves quarantine identity and returns each retained critical audit record ID exactly once when terminal audit projection fails.
• Adds hermetic official-client tests proving retry=None, validate_only=False, allow_missing=False, current etag use, input-only contact semantics, and no generic retry loop.

Safety

• Verification uses repository fixtures and generated-client doubles only.
• No live quota preference create, update, delete, or provider validateOnly call was performed.
• No API enablement, capacity provisioning, workload creation, credential mutation, or keyring mutation was performed.

Verification

• uv run --locked ruff check . — passed.
• uv run --locked ruff format --check . — passed.
• uv run --locked pyrefly check — 0 errors.
• uv run --locked lint-imports — 5 contracts kept.
• uv run --locked pytest -q — 1,363 passed, 3 skipped; 91.78% branch coverage.
• Safety-critical Apply, provider-write, and persistence gate — 3,551 statements and 1,096 branches, 100%.
• uv build --clear --no-sources and distribution verification — passed.
• git diff --check origin/main..HEAD — passed.
• Independent adversarial review completed on every corrective head; every confirmed P1/P2 finding was regression-tested and resolved.

Closes #38

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 19a3977c-ad20-432b-a3fc-d8e8a2b909d4

📥 Commits

Reviewing files that changed from the base of the PR and between a0b913a and c9f6f49.

📒 Files selected for processing (4)

• src/cqmgr/adapters/persistence/plans.py
• src/cqmgr/application/operations/apply.py
• tests/application/test_apply_operations.py
• tests/plans/test_plan_repository.py

---

📝 Walkthrough

Walkthrough

The PR implements deterministic, durable Apply processing with ordered child dispatch, authenticated local records, Google quota preference writes, unknown-outcome reconciliation, plan invalidation and recovery, audit deduplication, expanded validation, and crash-safety coverage.

Changes

Apply lifecycle

Layer / File(s)	Summary
Apply contracts and domain state src/cqmgr/application/ports/*, src/cqmgr/domain/*	Adds typed Apply, provider-write, repository, revalidation, invalidation, and immutable child-transition contracts.
Google quota preference dispatch src/cqmgr/adapters/google/..., tests/adapters/...	Adds retry-free create/update dispatch, strict response matching, provider error mapping, and read-after-unknown reconciliation.
Authenticated Apply record persistence src/cqmgr/adapters/persistence/apply_records.py, tests/plans/test_apply_record_repository.py	Adds HMAC-authenticated, revision-checked, atomically published Apply records and append-only unknown-resolution evidence.
Plan recovery and audit durability src/cqmgr/adapters/persistence/plans.py, src/cqmgr/adapters/persistence/audit.py, tests/plans/*	Adds dispatched-plan recovery, durable invalidation, INVALIDATED ledger classification, idempotent quarantine, and audit deduplication.
Apply orchestration and recovery src/cqmgr/application/operations/apply.py, tests/application/test_apply_operations.py	Adds preflight revalidation, durable dispatch sequencing, ordered provider writes, unknown and critical-unknown handling, result construction, and restart recovery.
Validation and distribution coverage .github/workflows/python.yml, scripts/verify_distribution.py, tests/domain/test_apply_records.py	Expands critical coverage and distribution manifests and validates Apply state transitions and invariants.

Estimated code review effort: 5 (Critical) | ~120 minutes

Possibly related PRs

• nisavid/cqmgr#49 — The plan persistence and recovery changes build on the native plan and secret repository implementation.
• nisavid/cqmgr#50 — The audit API and filesystem journal behavior are directly extended with deduplication.
• nisavid/cqmgr#52 — Both changes update the shared package-file inventory in scripts/verify_distribution.py.

Poem

A rabbit stamps records, neat and bright,
Orders each quota write just right.
If clouds go quiet, reads hop near,
Durable trails keep outcomes clear.
Plans may pause, recover, or rest—
Apply hops onward, safety-blessed.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 57.19% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title is concise and directly describes the main change: deterministic reconciliation for Apply.
Linked Issues check	✅ Passed	The PR implements ordered Apply reconciliation, durable pre-dispatch persistence, deterministic per-child dispatch, and recovery/quarantine behavior required by #38.
Out of Scope Changes check	✅ Passed	The added modules and protocol changes all support Apply reconciliation, recovery, or its tests; no clearly unrelated feature work stands out.

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

[github-advanced-security]

CodeQL found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[nisavid]

@coderabbitai review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[greptile-apps]

Greptile Summary

Implements deterministic, crash-safe Apply reconciliation.

• Adds durable authenticated Apply records and plan-consumption state transitions.
• Adds ordered recovery, terminal-result reconstruction, and unknown-outcome resolution.
• Adds retry-free Google Cloud Quotas writes and extensive recovery tests.

Confidence Score: 5/5

The PR appears safe to merge because no blocking failures remain in the fixes associated with the previous review threads.

The failed-child recovery now terminalizes later children and the consumed-plan recovery reconstructs the durable terminal provider outcome after restart.

T-Rex Logs

What T-Rex did

• Validated the code-execution attempt against contract rules; the run captured the exact command, working directory, exit status 4, and elapsed time of 1.629 seconds, and execution stopped at the clear environment blocker as requested.

_{Ran code and verified through T-Rex}

Important Files Changed

Filename	Overview
src/cqmgr/application/operations/apply.py	Adds the ordered Apply state machine, durable recovery paths, terminal projection, and fail-closed reconciliation.
src/cqmgr/adapters/persistence/apply_records.py	Adds authenticated, atomic Apply-record and unknown-resolution persistence.
src/cqmgr/adapters/persistence/plans.py	Extends plan persistence with recoverable dispatch and consumption transitions.
src/cqmgr/adapters/google/quota_preference_writes.py	Adds deterministic Cloud Quotas writes and read-after-unknown reconciliation with generic retries disabled.
src/cqmgr/domain/apply_records.py	Defines Apply record state, child dispositions, transition invariants, and terminalization behavior.
tests/application/test_apply_operations.py	Exercises Apply ordering, persistence failures, restart recovery, and crash boundaries.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[Load immutable plan] --> B[Revalidate complete bundle]
B --> C[Acquire plan lease]
C --> D[Persist Apply intent]
D --> E[Mark plan dispatched]
E --> F[Dispatch children in order]
F --> G{Child outcome}
G -->|Accepted| F
G -->|Failed| H[Mark remaining children unattempted]
G -->|Unknown| I[Persist uncertainty and reconcile]
F -->|All accepted| J[Finalize accepted record]
H --> K[Finalize failed record]
I --> L[Finalize or retain unknown state]
J --> M[Consume plan]
K --> M
M --> N[Project durable terminal result on retry]

Loading

_{Reviews (3): Last reviewed commit: "fix(apply/recovery): return critical aud..." | Re-trigger Greptile}

[coderabbitai]

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

src/cqmgr/adapters/persistence/plans.py (1)

1000-1006: 🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win

Restore the parent == candidate guard in _ensure_private_directory. runtime_paths() still allows relative expanduser() results, so a missing working directory can leave Path(".") non-existent and the loop spinning indefinitely.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/cqmgr/adapters/persistence/plans.py` around lines 1000 - 1006, Restore
the parent-equality termination guard in _ensure_private_directory’s loop so
traversal stops when candidate.parent is candidate, including for relative paths
whose working directory is missing. Preserve the existing missing-path
collection and private mkdir behavior.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/domain/test_apply_records.py`:
- Around line 85-89: Update all three type-check suppressions on the replace
calls in the parameterized test to use mypy’s actual arg-type error code instead
of bad-argument-type, or remove them if those calls no longer produce errors.
Keep the test behavior and replace invocation unchanged.

---

Outside diff comments:
In `@src/cqmgr/adapters/persistence/plans.py`:
- Around line 1000-1006: Restore the parent-equality termination guard in
_ensure_private_directory’s loop so traversal stops when candidate.parent is
candidate, including for relative paths whose working directory is missing.
Preserve the existing missing-path collection and private mkdir behavior.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 7ef35569-21bc-4e7d-ab47-233685f46bb9

📥 Commits

Reviewing files that changed from the base of the PR and between 9942221 and a0b913a.

📒 Files selected for processing (24)

• .github/workflows/python.yml
• scripts/verify_distribution.py
• src/cqmgr/adapters/google/quota_preference_writes.py
• src/cqmgr/adapters/persistence/apply_records.py
• src/cqmgr/adapters/persistence/audit.py
• src/cqmgr/adapters/persistence/plans.py
• src/cqmgr/application/operations/apply.py
• src/cqmgr/application/operations/plans.py
• src/cqmgr/application/ports/apply.py
• src/cqmgr/application/ports/apply_records.py
• src/cqmgr/application/ports/audit.py
• src/cqmgr/application/ports/plans.py
• src/cqmgr/application/ports/provider_writes.py
• src/cqmgr/domain/apply_records.py
• src/cqmgr/domain/audit.py
• src/cqmgr/domain/plan_consumption.py
• src/cqmgr/domain/plans.py
• tests/adapters/test_google_quota_preference_writes.py
• tests/application/test_apply_operations.py
• tests/domain/test_apply_records.py
• tests/plans/test_apply_record_repository.py
• tests/plans/test_plan_codec.py
• tests/plans/test_plan_consumption.py
• tests/plans/test_plan_repository.py