Skip to content

fix(charts): Limit retrieval on bigquery queries from Superset - #36387

Closed
ethan-l-geotab wants to merge 2 commits into
apache:masterfrom
ethan-l-geotab:os-partial-retvl
Closed

fix(charts): Limit retrieval on bigquery queries from Superset#36387
ethan-l-geotab wants to merge 2 commits into
apache:masterfrom
ethan-l-geotab:os-partial-retvl

Conversation

@ethan-l-geotab

@ethan-l-geotab ethan-l-geotab commented Dec 2, 2025

Copy link
Copy Markdown
Contributor

SUMMARY

Fixes: #36385

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

before:
image

after
image

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

  • Has associated issue:
  • Required feature flags:
  • Changes UI
  • Includes DB Migration (follow approval process in SIP-59)
    • Migration is atomic, supports rollback & is backwards-compatible
    • Confirm DB migration upgrade and downgrade tested
    • Runtime estimates and downtime expectations provided
  • Introduces new feature or API
  • Removes existing feature or API

@codecov

codecov Bot commented Dec 2, 2025

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 18.51852% with 44 lines in your changes missing coverage. Please review.
✅ Project coverage is 67.94%. Comparing base (92986c2) to head (21ea43e).
⚠️ Report is 1307 commits behind head on master.

Files with missing lines Patch % Lines
superset/db_engine_specs/bigquery.py 15.21% 38 Missing and 1 partial ⚠️
superset/common/query_context_processor.py 28.57% 4 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           master   #36387       +/-   ##
===========================================
+ Coverage        0   67.94%   +67.94%     
===========================================
  Files           0      637      +637     
  Lines           0    46913    +46913     
  Branches        0     5096     +5096     
===========================================
+ Hits            0    31873    +31873     
- Misses          0    13758    +13758     
- Partials        0     1282     +1282     
Flag Coverage Δ
hive 43.70% <9.25%> (?)
mysql 67.00% <18.51%> (?)
postgres 67.05% <18.51%> (?)
presto 47.33% <11.11%> (?)
python 67.90% <18.51%> (?)
sqlite 66.67% <18.51%> (?)
unit 100.00% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@JZ6

JZ6 commented Dec 3, 2025

Copy link
Copy Markdown
Contributor

we keep running into this problem, this fix would be great!

rusackas pushed a commit that referenced this pull request Mar 11, 2026
Implement memory-aware progressive fetching in BigQuery's fetch_data
method. Large result sets (950+ MB) previously crashed Chrome by loading
everything into memory at once. The fix samples an initial batch to
estimate row size, then fetches only as many rows as fit within the
BQ_FETCH_MAX_MB config limit (default 200 MB). A warning toast is shown
to users when results are truncated.

This is always-on with no feature flag -- operators control the budget
via the BQ_FETCH_MAX_MB config constant.

Originally by @ethan-l-geotab in #36387.

Co-authored-by: ethan-l-geotab <ethanliong@geotab.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@rusackas

Copy link
Copy Markdown
Member

Closing since #36385 has since been resolved as completed. If the BigQuery limit behavior still misbehaves on current master, @ethan-l-geotab, reopen with a quick repro and we'll revisit. Thanks!

@rusackas rusackas closed this Jun 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Large chart data doesn't return a helpful error message

3 participants