Modernize the connected-account stack: BCM Data Exports, de-nesting, native CUR, single connection role, and terminology rename

[silvexis]

This PR started as "discover BCM Data Exports (CUR 2.0) buckets" and grew, by request, into a broader modernization of the customer-facing onboarding stack. All of it preserves the fixed reactor payload contract (the account-link-provisioned JSON keys are unchanged) and the existing-data identifiers (cz-cur-hourly-csv, cloudzero-cur-hourly-csv), so it is seamless from CloudZero's side.

What's in this PR

1. BCM Data Exports (CUR 2.0) discovery

Discovery now calls bcm-data-exports:ListExports/GetExport and grants the role read access to every export's S3 destination bucket (type-agnostic; ingest stays COST_AND_USAGE_REPORT-only). Per-export failures are isolated; pagination covered.

2. De-nesting: 6 nested stacks → 1

The parent inlines the Discovery and Notification Lambdas (native AWS::Lambda::Function + role) and provisions through one nested ConnectionResources stack. The notification Lambda no longer scrapes sibling stack outputs — it takes direct properties — and drops AWSCloudFormationReadOnlyAccess.

3. Native CUR (was a custom-resource Lambda)

Custom::CostAndUsageReport → native AWS::S3::Bucket + AWS::S3::BucketPolicy + AWS::CUR::ReportDefinition, gated to us-east-1 (the only region the CUR API/resource exists in) with DeletionPolicy: Retain. Billing connections needing a new CUR must launch in us-east-1; otherwise the role is still created and BillingCurStatus reports the skip.

4. Removed deprecated audit + cloudtrail-owner connection types

Templates, discovery detection, and cloudtrail:DescribeTrails removed. Their reactor payload keys remain and emit null/false (contract unchanged).

5. Terminology rename (internal only)

master payer → billing, resource owner → resource, the CloudZero "account" concept → connection. Reactor payload keys, AWS account identifiers (AccountId/ReactorAccountId/AWS::AccountId/MasterAccountId), and the connected_account.yaml launch filename are not renamed. terraform/ and policies/*.json are out of scope (follow-up).

6. Single fixed-name connection role

The resource-owner and master-payer roles collapse into one role cloudzero-connection (shared CommonAccessManagedPolicy + a billing-only inline CUR-bucket policy). The single ARN is reported into both the resource_owner and master_payer link slots. Fixed name ⇒ launch requires CAPABILITY_NAMED_IAM (Makefile + README updated).

7. Migration + guided cleanup

Discovery re-adopts an account's existing CUR (no duplicate) and detects previously-deployed CloudZero stacks (cloudformation:DescribeStacks), surfaced via the DetectedLegacyConnectionStacks output. A Custom::CleanupGuide writes CLOUDZERO_CLEANUP_GUIDE.md to the root of the connected CUR bucket with safe deletion steps. README documents the migration path.

8. Co-equal CloudZero principals

The reactor and connections accounts are presented as equally-important trust principals.

Customer-facing behavior changes

• Launch now requires CAPABILITY_NAMED_IAM.
• Billing connections that need a new CUR must be launched in us-east-1.
• "Migrating from an older CloudZero stack" section added to the README.

Verification

• cfn-lint clean on all templates.
• Discovery: 27 unit tests; Notification: 13 unit tests — including an explicit assert that the reactor payload key set is unchanged, plus legacy-detection and cleanup-guide tests.
• Grep guard: no stray old terminology outside the fixed payload keys; no AWS-identifier renames.
• aws cloudformation validate-template not run locally (no creds) — relying on cfn-lint + CI.

See docs/releases/1.0.102.md for the full release notes.

🤖 Generated with Claude Code

[khill2018]

approved, but I agree with greptile about the test additions. Let's add those in (happy to do it myself) and then merge! Edit to add I see Erik is on a plane going to make those changes myself and have somebody additionally review the PR

[khill2018]

@greptile

[greptile-apps]

Greptile Summary

This PR modernizes the CloudZero connected-account onboarding stack with several significant changes while preserving the fixed reactor payload contract and existing data identifiers.

• BCM Data Exports discovery: Discovery Lambda now calls ListExports/GetExport with per-page and per-export failure isolation; 6 nested stacks collapse to 1 (ConnectionResources), with Discovery and Notification inlined as native AWS::Lambda::Function resources in the parent template.
• Native CUR: Custom::CostAndUsageReport replaced by AWS::S3::Bucket + AWS::S3::BucketPolicy + AWS::CUR::ReportDefinition (us-east-1 gated, DeletionPolicy: Retain); a single fixed-name cloudzero-connection role replaces the separate resource-owner and master-payer roles.
• Migration tooling: detect_legacy_connection_stacks surfaces older CloudZero stacks; a new Custom::CleanupGuide writes CLOUDZERO_CLEANUP_GUIDE.md to the connected CUR bucket root.

Confidence Score: 5/5

The PR is safe to merge: the reactor payload contract is preserved, previously flagged issues (per-export isolation, mid-pagination failure resilience, pagination test coverage) are all addressed, and billing data is protected by DeletionPolicy:Retain throughout.

The architectural changes are well-scoped and the test suite explicitly guards the fixed external contract key set. The only open item is the pre-existing absence of pagination in list_cur_report_definitions, which is extremely unlikely to matter in practice but is not closed by this PR.

services/discovery/src/app.py — the list_cur_report_definitions function lacks pagination (pre-existing); all other changed files look correct.

Important Files Changed

Filename	Overview
services/discovery/src/app.py	Adds BCM Data Exports (CUR 2.0) discovery with per-export error isolation and per-page pagination failure resilience; legacy-stack detection added; pagination is handled for list_exports but not for describe_report_definitions
services/notification/src/app.py	Notification handler simplified to receive all values as direct CFN properties (no sibling-stack scraping); fixed reactor payload contract preserved with deprecated cloudtrail/audit keys emitting null
services/connected_account.yaml	De-nested from 6 stacks to 1; Discovery and Notification Lambdas inlined as native resources; ConnectionResources is the single remaining nested stack; parameter wiring is clean
services/account_type/account_resources.yaml	Single fixed-name connection role with CommonAccessManagedPolicy; native CUR resources with DeletionPolicy:Retain; cleanup guide custom resource; IAM conditions and resource ARNs are correctly structured
services/discovery/src/cleanup_guide.py	New cleanup guide Lambda: writes CLOUDZERO_CLEANUP_GUIDE.md to CUR bucket root; region-aware S3 client; best-effort (failures logged, never fail the stack)
services/discovery/tests/unit/test_app.py	27 unit tests covering BCM Data Exports pagination, per-export failure isolation, mid-pagination failure, legacy stack detection, and CUR format selection; previously flagged gaps now addressed
services/notification/tests/unit/test_app.py	13 unit tests including explicit guard on fixed reactor payload key set; billing/resource slot routing verified; deprovisioned message type tested

_{Reviews (10): Last reviewed commit: "Preserve collected export ARNs on mid-pa..." | Re-trigger Greptile}

[silvexis]

Thanks for the reviews. Addressed the feedback in 0f5f29c (on top of the stack-flattening work):

CodeQL — clear-text logging of sensitive data (py/clear-text-logging-sensitive-data) ✅
The discovery and notification handlers no longer log the full event / payload / output — those carry ExternalId and account-identifying fields. They now log only non-sensitive summary fields (RequestType, message_type, response status, the boolean classification flags).

@qiuz-cz — except ClientError too narrow (app.py:122) ✅
Broadened the BCM Data Exports list/get catches to (ClientError, BotoCoreError) so connectivity errors (e.g. EndpointConnectionError) degrade gracefully. On the import-time UnknownServiceError concern: requirements.txt pins boto3>=1.34.91, and bcm-data-exports has shipped in botocore since 1.34.0, so the client constructs fine on the python3.12 runtime.

@qiuz-cz — lazy %s logging (app.py:134) ✅ Switched the per-export GetExport warning to logger.warning('... %s', export_arn, ...).

@qiuz-cz — coverage gap for the if bucket: guard ✅ Added test_handler_skips_export_with_no_destination_bucket.

@qiuz-cz — hardcoded version/date in docs/releases/1.0.101.md — confirmed: release notes here are hand-written and committed (see 1.0.95.md, 1.0.100.md); the ${...} compare-URL is just the file's own convention, not semantic-release generation. Same pattern followed for the new 1.0.102.md.

Greptile P2s (per-export error isolation + pagination test) — these were resolved by @khill2018's 50c9b31 and are preserved through the refactor (the discovery handler was rewritten; the isolation + pagination behavior and tests carried over).

Also dropped the now-unused toolz dependency from both Lambda requirements.

Heads-up @khill2018: this PR now also contains the connected-account stack flattening (6 nested stacks → 1), native AWS::CUR::ReportDefinition, and removal of the deprecated audit / cloudtrail-owner account types — see docs/releases/1.0.102.md. Your 50c9b31 discovery change was preserved through the rewrite.

[silvexis]

Pushed a sizable follow-on in c9deb84 — terminology rename + single connection role + migration tooling. Flagging for a fresh look @qiuz-cz @khill2018.

Terminology (internal only): master payer → billing, resource owner → resource, the CloudZero "account" concept → connection. The reactor payload keys are unchanged (master_payer/resource_owner/is_master_payer_account/…), as are AWS account identifiers (AccountId/ReactorAccountId/AWS::AccountId/MasterAccountId) and the existing-data identifiers (cz-cur-hourly-csv, cloudzero-cur-hourly-csv).

Single role: the two roles collapse into one fixed-name role cloudzero-connection. The single ARN is reported into both the resource_owner and master_payer link slots (and legacy) to keep the link contract; master_payer stays null for a resource-only connection. Fixed name ⇒ launch now requires CAPABILITY_NAMED_IAM (added to the Makefile deploy/update targets and documented in the README).

Migration / cleanup: discovery re-adopts an existing CUR (no duplicate), and now detects prior CloudZero stacks via cloudformation:DescribeStacks, surfaced in the DetectedLegacyConnectionStacks output. A Custom::CleanupGuide writes CLOUDZERO_CLEANUP_GUIDE.md to the root of the connected CUR bucket with the safe deletion steps (handler services/discovery/src/cleanup_guide.py, unit-tested).

Also: reactor and connections accounts are now presented as co-equal trust principals.

Verification: cfn-lint clean on all templates; discovery 27 tests, notification 13 tests (incl. an explicit assert that the reactor payload key set is unchanged); grep guard confirms no stray old terminology outside the fixed payload keys and no AWS-identifier renames. aws cloudformation validate-template wasn't run here (no creds) — relying on cfn-lint + CI.

Note the README's "Migrating from an older CloudZero stack" section and the CAPABILITY_NAMED_IAM requirement are the two customer-facing behavior changes worth a close look.

[jordan-piscitelli-cloudzero]

Why would discovery do the report selection? Why not send all discovered reports to reactor and let reactor select the one correct one?