Skip to content

Report ARM ID in command and fault telemetry - #59

Merged
jmccormick7 merged 6 commits into
mainfrom
mccormickjo/report-armid-in-telemetry
Aug 12, 2026
Merged

jmccormick7 merged 6 commits into
mainfrom
mccormickjo/report-armid-in-telemetry

Conversation

@jmccormick7

Copy link
Copy Markdown

This PR updates telemetry calls to report the Arm Id up to ADX so we can better respond to frequently failing resource onboarding, as well as improve telemetry queries to understand repeat failures

Example in ADX:
image


This checklist is used to make sure that common guidelines for a pull request are followed.

Related command

General Guidelines

  • Have you run azdev style <YOUR_EXT> locally? (pip install azdev required)
  • Have you run python scripts/ci/test_index.py -q locally? (pip install wheel==0.30.0 required)
  • My extension version conforms to the Extension version schema

For new extensions:

About Extension Publish

There is a pipeline to automatically build, upload and publish extension wheels.
Once your pull request is merged into main branch, a new pull request will be created to update src/index.json automatically.
You only need to update the version information in file setup.py and historical information in file HISTORY.rst in your PR but do not modify src/index.json.

@github-actions

Copy link
Copy Markdown

The git hooks are available for azure-cli and azure-cli-extensions repos. They could help you run required checks before creating the PR.

Please sync the latest code with latest dev branch (for azure-cli) or main branch (for azure-cli-extensions).
After that please run the following commands to enable git hooks:

pip install azdev --upgrade
azdev setup -c <your azure-cli repo path> -r <your azure-cli-extensions repo path>

@jmccormick7
jmccormick7 force-pushed the mccormickjo/report-armid-in-telemetry branch from 00f299e to 7bb6097 Compare August 4, 2026 16:56
@jmccormick7
jmccormick7 requested a review from deeksha345 as a code owner August 4, 2026 16:56
@jmccormick7

jmccormick7 commented Aug 6, 2026 •

Copy link
Copy Markdown
Author
image
let _endTime = now();
let _startTime = datetime(2026-08-04T14:35:21Z);
cluster('ddazureclients').database('AzureCli').RawEventsAzCli
| where StartTime between (_startTime .. _endTime) or ingestion_time() between (_startTime .. _endTime)
| extend props = parse_json(Properties)
| extend RawCommandAny = coalesce(tostring(RawCommand), tostring(props["context.default.azurecli.rawcommand"]))
| where RawCommandAny has "connectedk8s" or Properties has "connectedk8s"
| where RawCommandAny contains "connect"
| where AzureSubscriptionId == "15c06b1b-01d6-407b-bb21-740b8617dea3"
| extend FaultType = tostring(props["context.default.azurecli.faulttype"])
| extend HelmTimeoutClassification = tostring(props["context.default.azurecli.helmtimeoutclassification"])
| extend DebugInfo = tostring(props["context.default.azurecli.debug_info"])
| extend ResourceId = extract(@"(?i)resourceid:\s*(/subscriptions/[^\s;]+)", 1, DebugInfo)
| order by StartTime desc
| where GeoCity == "Washington"
| project EventName, EventTimestamp, FaultType, HelmTimeoutClassification, RawCommandAny, ResourceId, ActionResult

Arm Ids populating to ADX

image Error formatting using Error code.

@jmccormick7
jmccormick7 force-pushed the mccormickjo/report-armid-in-telemetry branch from 8600c06 to a2dcf2e Compare August 6, 2026 15:15
@jmccormick7
jmccormick7 force-pushed the mccormickjo/report-armid-in-telemetry branch from e401a53 to 5a2af13 Compare August 11, 2026 21:00
@jmccormick7
jmccormick7 force-pushed the mccormickjo/report-armid-in-telemetry branch from 5a2af13 to cd180d1 Compare August 12, 2026 14:50
@jmccormick7
jmccormick7 merged commit 0ae724c into main Aug 12, 2026
4 of 8 checks passed
@jmccormick7
jmccormick7 deleted the mccormickjo/report-armid-in-telemetry branch August 12, 2026 16:55
ashnanze pushed a commit that referenced this pull request Oct 5, 2026
* add wrapper for telemetry that adds ArmId to payload

* add standardized error data type and methods

* update other telemetry call sites for errors to use centralized error
wrapper

* update helm get values failure to use new error reporting, and add
documentation

* update test and pass ruff checks

* fix argument error from rebase

---------

Co-authored-by: John McCormick <mccormickjo@microsoft.com>
ashnanze pushed a commit that referenced this pull request Oct 6, 2026
* add wrapper for telemetry that adds ArmId to payload

* add standardized error data type and methods

* update other telemetry call sites for errors to use centralized error
wrapper

* update helm get values failure to use new error reporting, and add
documentation

* update test and pass ruff checks

* fix argument error from rebase

---------

Co-authored-by: John McCormick <mccormickjo@microsoft.com>
github-actions Bot pushed a commit that referenced this pull request Oct 14, 2026
* Update onboarding flow to inject kubernetes secret without using helm (#47)

* Update onboarding flow to inject kubernetes secret without using helm
values

* add whl file for testing

* update whl file after rebase

---------

Co-authored-by: John McCormick <mccormickjo@microsoft.com>

* Report ARM ID in command and fault telemetry  (#59)

* add wrapper for telemetry that adds ArmId to payload

* add standardized error data type and methods

* update other telemetry call sites for errors to use centralized error
wrapper

* update helm get values failure to use new error reporting, and add
documentation

* update test and pass ruff checks

* fix argument error from rebase

---------

Co-authored-by: John McCormick <mccormickjo@microsoft.com>

* Update client proxy version (#72)

* Fix linting issues, and Update CI checks (#76)

* add a git workflow to update sync/upstream-main and add release docs

* fixing azdev pipelines and cli testing passthrough

* update azdev pipelines, and use federated credential for ado pipeline

* update CLI testing

* update cli testing to use github auth

* chore: retrigger ADO PR validation

* fix formatting issues for CI, and remove cli wide test that fails on a missing ref

* improve efficiency and fix version calculation

* ruff formatting

---------

Co-authored-by: John McCormick <mccormickjo@microsoft.com>

* Fix/connectedk8s prediagnostic job completion (#80)

* fix(connectedk8s): reconcile diagnostic job completion

* test(connectedk8s): cover diagnostic job outcomes

* fix(connectedk8s): extend diagnostic job timeout

* refactor(connectedk8s): rely on extended diagnostic timeout

* chore(connectedk8s): remove test attribution comment

* feat(connectedk8s): explain incomplete diagnostic jobs

* fix(connectedk8s): remove unused noqa directives

* chore(connectedk8s): retrigger validation pipeline

* fix(connectedk8s): resolve ruff style failures

* fix(connectedk8s): suppress delayed import lint

* fix(connectedk8s): suppress diagnostic helper return lint

* Adding AZK8S0600-0699 pre-onboarding diagnostic error codes (#81)

* feat(connectedk8s): add AZK8S0600-0699 pre-onboarding diagnostic error codes

Add structured error code infrastructure and wire the 0600-0699 range
(pre-onboarding diagnostics) so users see actionable codes in errors and
warnings instead of raw messages.

New file:
- _error_codes.py: ERROR_CATALOG dict (full range 0001-0805) plus
  raise_with_code() and warn_with_code() helpers that prepend the code
  to the message and emit telemetry in a single call.

_constants.py:
- Added AZK8S0600-AZK8S0607 error code constants.

_precheckutils.py:
- Import raise_with_code, warn_with_code from _error_codes.
- AZK8S0604: job not scheduled -> warn_with_code (non-fatal).
- AZK8S0606: log save failed (OSError + ValueError/TypeError) -> warn_with_code (non-fatal).
- AZK8S0605: job not complete -> warn_with_code (non-fatal).
- AZK8S0601: outer except in executing_cluster_diagnostic_checks_job -> raise_with_code(CLIInternalError).
- AZK8S0607: helm install of diagnostic checks chart failed -> raise_with_code(CLIInternalError).

custom.py:
- Import raise_with_code from _error_codes.
- AZK8S0600: outer except wrapping fetch_diagnostic_checks_results -> raise_with_code(CLIInternalError).
- AZK8S0603: Diagnostic_Check_Incomplete -> raise_with_code(ValidationError).
- AZK8S0602: Diagnostic_Check_Failed -> raise_with_code(ValidationError).

* fix(connectedk8s): preserve prediagnostic error classifications

* docs(connectedk8s): clarify error classification

* Update csp version (#85)

* Implement Helm Error Code Standardization (AZK8S05XX) (#73)

* add helm errors

* update precheck utils to raise the helm errors for helm client failures

* introduce generic Helm client error and fix cmd typing and error supression

* update tests

* remove whl

* fixes

* add arm id to prechecks + troubleshooting telemetry + remove helm url

helm url could contain credentials so removing from telemetry

* fix param issue in delete_arc_agents

* ruff format and simplify if statment

* fix unittest

* update test inputs

* fixes

* update stub to use kwargs as well

* update helm checks and sanitize telemetry payload

* move logging, and raise in place

---------

Co-authored-by: John McCormick <mccormickjo@microsoft.com>

* Proxy Bypass for Arc Service endpoints and Container Insights. (#86)

* Add Container Insights proxy bypass constants and utils

* Add --proxy-bypass and --clear-proxy-bypass parameters

* Apply Container Insights proxy bypass during connect

* Apply Container Insights proxy bypass only when its keyword is specified

* Remove Arc keyword from --proxy-skip-range

* Add Arc keyword to proxy bypass constants

* Rename --proxy-bypass to --add-proxy-bypass and validate keywords

* Add proxy bypass parameters to update

* Append Arc endpoints to noProxy during connect

* Preserve Arc proxy bypass across update

* Remove Arc endpoints from noProxy during update

* Add Container Insights proxy bypass into update

* Update comment for Container Insights

* Set Container Insights proxy bypass to false on clear

* Remove Container Insights proxy bypass on delete

* Clear stale Container Insights proxy bypass on re-onboard

* Allow clearing the Arc proxy bypass with a new skip range

* Add proxy bypass keywords in help

* Add tests for the proxy bypass keywords

* To address proxy bypass naming, messages and tests

* Add telemetry to Container Insights ConfigMap failures.

* Fix missing cmd argument in helm values call and apply ruff format

---------

Co-authored-by: Sai Tareesh Reddy Eppeti <seppeti@microsoft.com>

* Standardize custom locations errors (AZK8S0700–AZK8S0702) (#90)

* feat(connectedk8s): standardize custom locations errors

* test(connectedk8s): verify custom locations telemetry

* Standardize network and connectivity errors (AZK8S0300–AZK8S0309) (#87)

* feat(connectedk8s): standardize network connectivity errors

* fix(connectedk8s): suppress outbound check local count lint

* docs(connectedk8s): clarify network error handling

* fix(connectedk8s): consolidate network diagnostic telemetry

* Add AGC Endpoint Configuration for Agent installation and Client-Side Proxy Operation (#68)

Add AGC Endpoint Configuration for Agent installation and Client-Side Proxy Operation

Added the AGC Helm override for MsiAdapterArtifactImageRegistry
Passing CloudConfig for Client-Side Proxy Scenario in AGC since default cloud name isn't supported.

* Implement kubectl/ Cluster Access Error Standardization (AZK8S02XX) (#74)

* Standardize network and connectivity errors (AZK8S0300–AZK8S0309) (#87)

* feat(connectedk8s): standardize network connectivity errors

* fix(connectedk8s): suppress outbound check local count lint

* docs(connectedk8s): clarify network error handling

* fix(connectedk8s): consolidate network diagnostic telemetry

* Add AGC Endpoint Configuration for Agent installation and Client-Side Proxy Operation (#68)

Add AGC Endpoint Configuration for Agent installation and Client-Side Proxy Operation

Added the AGC Helm override for MsiAdapterArtifactImageRegistry
Passing CloudConfig for Client-Side Proxy Scenario in AGC since default cloud name isn't supported.

* add error type for failed to inject private key

* initial error implementation

* standardize cmd type and add logger.warning to connectedk8s warning telemetry function

* ruff format

* fixes to telemetry

* update az error classes for kubectl errors

* restore no raising error to namespace cleanup on delete

* update errors to pass in the underlying exception type

* ruff format

* test(connectedk8s): remove misplaced proxy error tests

Keep the proxy error coverage on the proxy-errors branch while correcting the unrelated Kubernetes exception expectations and namespace cleanup clock mock.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* ensure only one telemetry event and no duplicates

* remove duplicate faults

* update telemetry to strip apostrophes for AZ telemetry

* ruff

* linting fixes

* fix(connectedk8s): remove duplicate command context arguments

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix duplicate telemetry

* ruff format

* fix indentation

* chore: retrigger mergability+pr checks

---------

Co-authored-by: Ashlee Nanze <ashleenanzework@gmail.com>
Co-authored-by: junw98 <junweihe@microsoft.com>
Co-authored-by: John McCormick <mccormickjo@microsoft.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* ARM operation Error Handling/Telemetry (AZK8S04XX) (#82)

* add in ARM operation error handling/reporting

* ruff format

* fix(connectedk8s): sanitize telemetry payloads

Apply the shared redaction helpers to standardized errors and extension event values while preserving ARM resource IDs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(connectedk8s): preserve telemetry exception types

Keep sanitized exception messages without collapsing fault classification, and cover command and ARM boundaries against duplicate fault emission.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(connectedk8s): type sanitized exception

Bind the dynamically created telemetry exception to BaseException so strict mypy does not infer an Any return.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* refactor(connectedk8s): consolidate ARM helpers

Parameterize connected-cluster PUT and GET helpers by operation while preserving operation-specific telemetry and error definitions.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fixes to telemetry sanitization

* dont block onboarding for couldn't check provider registration

* fix(connectedk8s): harden telemetry field redaction

Redact case-insensitive JSON and key-value credential fields while preserving quoting and avoiding similarly named non-sensitive fields.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: John McCormick <mccormickjo@microsoft.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add Proxy Pester Tests and restrict Arc bypass on reconnect. (#94)

* Announce the Arc proxy bypass only where it reaches the agents

* Refuse the Arc proxy bypass while reconnecting

* Only treat a complete Arc endpoint set as the proxy bypass

* Add end-to-end tests for the proxy bypass

* Refuse clearing the Arc proxy bypass while the skip range lists it

---------

Co-authored-by: Sai Tareesh Reddy Eppeti <seppeti@microsoft.com>

* Update dns result parsing to ignore transient issues (#93)

* update dns result parsing to ignore transient issues

* ruff

* reject invalid IP and take last error

---------

Co-authored-by: John McCormick <mccormickjo@microsoft.com>

* Client Proxy Error Code Reporting/Telemetry (AZK8S08XX) (#83)

* add client proxy error reporting

* test(connectedk8s): add client proxy error coverage

Cover client proxy error classification, legacy fault aliases, and preserved ARM credential error handling on the proxy-errors branch.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(connectedk8s): sanitize telemetry error payloads

Move recommendation handling to the raising reporter, sanitize event properties and exception messages, and cover apostrophe-bearing proxy failures.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: John McCormick <mccormickjo@microsoft.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add helm override for kube state metrics repository (#95)

Add helm override for kube-state-metrics repository

* Standardize configuration input errors (AZK8S0100-AZK8S0106) (#92)

* Standardize configuration input errors (AZK8S0100-AZK8S0106)

* Remove generated test attribution comments

* Fix gateway resource ID type checking

* Fix connectedk8s unhandled error telemetry coverage (#91)

* fix(connectedk8s): standardize unhandled error telemetry

* fix(connectedk8s): preserve classified proxy errors in tests

* Add ownership tracking for Arc proxy bypass. (#96)

* Track Arc proxy bypass ownership

* Fix unit tests and ruff formatting

---------

Co-authored-by: Sai Tareesh Reddy Eppeti <seppeti@microsoft.com>

* chore(connectedk8s): prepare 1.11.4 release

* Fix connectedk8s 1.11.4 release review findings (#102)

* Sanitize prediagnostic log telemetry

* Fix prediagnostic telemetry command context

* Preserve proxy bypass compatibility

* Patch existing onboarding private key secret

* Format proxy bypass compatibility test

* Remove legacy Arc proxy skip range alias

* Reject removed Arc proxy skip range alias

* Revert "Reject removed Arc proxy skip range alias"

This reverts commit 6701f1e.

* Fix connectedk8s test secret scan (#103)

* Fix MCR precheck to use HTTPS (#104)

* Revert manual connectedk8s history update

* Restore connectedk8s 1.11.4 release history

---------

Co-authored-by: john mccormick <123213439+jmccormick7@users.noreply.github.com>
Co-authored-by: John McCormick <mccormickjo@microsoft.com>
Co-authored-by: shlokpatel57 <shlokpatel@microsoft.com>
Co-authored-by: Tareesh <saitareeshreddy12@gmail.com>
Co-authored-by: Sai Tareesh Reddy Eppeti <seppeti@microsoft.com>
Co-authored-by: junw98 <junweihe@microsoft.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants