Add wf_reported_problems_set and wf_reported_problems_cleared Metrics - #11346
Draft
ytaben wants to merge 1 commit into
Draft
Add wf_reported_problems_set and wf_reported_problems_cleared Metrics#11346ytaben wants to merge 1 commit into
wf_reported_problems_set and wf_reported_problems_cleared Metrics#11346ytaben wants to merge 1 commit into
Conversation
Add wf_reported_problems_set and wf_reported_problems_cleared counters, emitted when the TemporalReportedProblems search attribute is set/updated after consecutive workflow task problems and when it is cleared on the next successful workflow task. This makes server-side "stuck workflow" detection observable/alertable without polling the visibility store per namespace. Counters are emitted at the SA state transitions (after the existing early-returns) so unchanged/no-op paths do not over-count, and only on the live paths (not history replay). Tags: namespace, cause (set only), and workflow_type (unconditional, matching existing workflow-lifecycle metrics such as WorkflowSuccessCount and PausedActivitiesCounter). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed?
This PR adds
wf_reported_problems_setandwf_reported_problems_clearedmetrics.These are counter metrics that are incremented exactly when the corresponding search attribute is set or cleared (
TemporalReportedProblems)Why?
We are looking for a way to help our users with reliably identifying and alerting on situations when workflows in their namespaces are not making progress for any reason.
At present - there are several ways to do this but none appear to be quite as reliable.
For example - client side workflow task failures do not trigger on server side issues. We had a case where users tried cross-namespace signalling. Workflow task 'succeeds' on the SDK side but only the server side sees WFT failure and increments a metric. Afaik something like payload size can also be a reason for server side rejection though maybe SDKs check it themselves now. This should also help with s2s timeouts where workflows are stuck because workers are dead.
Conversely - a single transient workflow task failure that is immediately retried afterwards should not be a reason to page someone. In other words - either client or server side WFT failure metrics don't show if the workflow is persistently stuck.
Notes
We added counters for both addition and removal of the search attribute for completeness but are ok with only having addition.
We are also ok with not having WF type tag though it looks to be canonically present throughout the repo.
We are also ok to close this if there was already a better way to do this that we did not think of.
How did you test it?
Pending test: deploy a cherry-pick on top of our current Temporal version in a forkUPD:
Deployed this commit to our sandbox cluster and forced WFT failure (workflow type not registered)
Found
wf_reported_problems_setmetric reported with correct 'namespace', 'workflowtype' tags and 'cause:workflowtaskfailedcauseunspecified'(we use datadog so it may be mangling casing)
Potential risks
This technically adds metric costs