Skip to content

Fix deletion handling in aggregations - #730

Merged
vish-cs merged 1 commit into
datacommonsorg:masterfrom
vish-cs:bq
Aug 20, 2026
Merged

Fix deletion handling in aggregations#730
vish-cs merged 1 commit into
datacommonsorg:masterfrom
vish-cs:bq

Conversation

@vish-cs

@vish-cs vish-cs commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Updated deletion query in aggregation which timed out for Place import.

  • Removed the redundant AND predicate IN ('linkedContainedInPlace', 'linkedMemberOf', 'linkedMember') filter. This slows the deletion as there is no index on predicates
  • Increased the RPC timeout to 6 hours instead of 1 hour

@codacy-production

codacy-production Bot commented Aug 20, 2026

Copy link
Copy Markdown

Not up to standards ⛔

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 0 complexity

Metric Results
Complexity 0

View in Codacy

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a configurable timeout for Spanner deletion operations to prevent 1-hour timeouts during partitioned DML executions, and simplifies the SQL query used to delete linked edges. The review feedback highlights two key improvements: avoiding fragile access to private Spanner client attributes by utilizing the public timeout parameter in execute_partitioned_dml, and simplifying the argument retrieval in main.py by directly accessing args.deletion_timeout instead of using getattr.

Comment thread pipeline/workflow/aggregation-helper/aggregation/deleter.py Outdated
Comment thread pipeline/workflow/aggregation-helper/main.py Outdated
@vish-cs
vish-cs force-pushed the bq branch 2 times, most recently from 35fe39a to 5e38ecf Compare August 20, 2026 05:14
@vish-cs
vish-cs enabled auto-merge (squash) August 20, 2026 08:20
@vish-cs
vish-cs merged commit 9cb0e12 into datacommonsorg:master Aug 20, 2026
10 of 11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants