Skip to content

feat: add the Postgres query regression watch template for Cloud SQL - #18

Open
mayankpande88 wants to merge 1 commit into
mainfrom
feat/postgres-query-regression-watch
Open

mayankpande88 wants to merge 1 commit into
mainfrom
feat/postgres-query-regression-watch

Conversation

@mayankpande88

Copy link
Copy Markdown
Contributor

Adds a template that watches a Cloud SQL for PostgreSQL instance for queries that got slower and stayed slower, and opens one ticket per episode with the numbers and a likely cause.

What it does, every 3 hours

  • A fixed script reads Query Insights history and finds statements whose time per call is well above the same hours of the previous week, for 12 hours, on two checks in a row. A statement reported in the last 24 hours is not reported again.
  • When blocks per call double on several tables at once, it files one "shared cause" note (vacuum held back, bloat, stale statistics) instead of a ticket per query.
  • For a new episode it adds instance operations, table statistics and long-open transactions (when a Postgres connection is set), and an AI investigation of the cause and fix. Then it opens the ticket. It changes nothing.

Look at

  • scripts/postgres_query_regression_watch/pqf.py: the rule (detect, check). The template embeds this file; build.py --check in CI fails if the template is stale.
  • The if: conditions in template.yaml.in: everything after the script runs only when it reports something new.

How to verify

  1. Instantiate the template on a GCP account whose Cloud SQL instance has Query Insights on, with a ticket integration.
  2. Run it by hand with "Check at" set to an hour you know was slow. A ticket opens with the statement, before/now numbers and the investigation.
  3. Run it with "Check at" empty on a quiet instance. The script step reports new: false and the later steps are skipped.
Engineering detail
  • The detection runs as a script step on the account's credentials (read-only calls: Cloud Monitoring time series, Cloud Trace). It prints one JSON object on stdout.
  • Rule: hourly mean per normalised statement (parameter lists collapsed). Baseline is the same hour on each of the previous 7 days, ≥ 8 points. Slow means above max(2× median, p90, median + 20 ms), in all but one hour per six of a 12-hour window, with ≥ 20 calls/h and ≥ 60 s/h of extra database time. If most of the busiest statements are slow at once, it reports the instance, not a query.
  • Classification: blocks per call up means plan or access path; rows per call up means code or data; neither means waits, IO or CPU (not reported). Sampled plans before vs now: plan changed / same plan / mix shift / too few.
  • No memory between runs: the script repeats the check at earlier run times to confirm a finding and to tell whether it is new.
  • With "Check at" set, the database steps are skipped. Table statistics show only the present, and in a test they led the investigation to the wrong conclusion about a past hour.
  • Validation: 60 days replayed on two instances, every 3 hours. 0 query tickets, and 1 shared-cause note for a real episode (a long-open transaction holding back vacuum). With slowdowns injected into real series, 10× was caught in 74-91% of cases and 3× in 25-60%.

Local e2e: created the workflow from this template (manual trigger) on a development environment, and ran it against a Cloud SQL instance with "Check at" set to a known incident hour.

  • Script step: 16 s, new: true, shared cause on 6 tables.
  • Instance operations step: none in the span.
  • Database steps: returned table statistics through the Postgres connection on its own account.
  • Investigation: ran; on the run without database data it found the vacuum horizon pinned by an old transaction, from autovacuum logs.
  • Ticket: opened in a sandbox GitLab repository.

After review fixes (plan errors no longer stop detection, empty thresholds use defaults, the database filter also scopes the shared-cause check), the script was re-run locally against the same instance and hour: same result.

Unit tests: 20 (python3 -m unittest discover -s scripts/postgres_query_regression_watch -p 'test_*.py').

Every three hours, finds queries that got slower and stayed slower against the same
hours of the previous week, from Cloud SQL Query Insights, and opens one ticket per
episode with the numbers, sampled plans, instance operations, optional table statistics
from a Postgres connection, and an AI investigation of the cause.

The detection is a fixed script with unit tests under scripts/; build.py embeds it in
the template and CI checks the template is current.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants