feat(streaming): Streaming CSV uploads for over 100k records for constant memory usage

[amaannawab923]

CSV_STREAMING_ROW_THRESHOLD=100000

🔧 Summary

This PR introduces streaming CSV export functionality for SQL Lab, enabling efficient exports of large query result sets with real-time progress tracking. Users can now export millions of rows without timeouts or memory errors, while seeing live progress updates during the download.

Screen.Recording.2025-10-03.at.2.06.36.PM.mov

🐛 The Issue

When exporting large SQL Lab query results or Charts to CSV, users face several critical problems:

• Memory crashes: Exporting 500K+ rows loads the entire dataset into memory at once, often crashing the browser tab
• Timeout failures: Large exports take 2-3 minutes with no feedback, frequently timing out before completion
• No progress indication: Users have no idea if the export is working or stuck, leading to multiple retry attempts
• Unreliable for production use: Data analysts avoid CSV export for large datasets, resorting to manual workarounds

For example, a data analyst trying to export a 500,000-row financial report would experience a frozen browser, no progress bar, and likely a timeout error after waiting several minutes.

🧠 Root Cause

The current CSV export loads all rows into memory at once before generating the file. This means:

• A 500,000-row export needs 500+ MB of memory loaded simultaneously
• The browser tab is blocked until the entire file is ready
• Network timeouts kill the request before large exports finish
• There's no way to show progress since nothing streams to the user until completion

The existing architecture simply cannot scale beyond a few thousand rows reliably.

✅ The Fix

This PR implements a complete streaming CSV export system that:

Backend Improvements

• New streaming endpoint (/api/v1/sqllab/export_streaming/) that sends data in chunks rather than all at once
• Server-side cursors fetch rows from the database in batches of 1,000 instead of loading everything upfront
• Progressive response streaming sends 64KB chunks to the browser as they're generated, keeping the connection alive
• Smart session management prevents database connection issues during long-running exports

Frontend Enhancements

• Streaming Export Modal shows real-time progress with row count, file size, and download speed
• Automatic threshold detection using the CSV_STREAMING_ROW_THRESHOLD config (default: 1,000 rows)
• Small exports use traditional method (instant download) while large exports automatically stream with progress
• Cancel button lets users abort long-running exports without closing the browser tab

Configuration

Administrators can control when streaming activates by setting CSV_STREAMING_ROW_THRESHOLD in superset_config.py. This same threshold is used for both Chart and SQL Lab exports, providing consistent behavior across the platform.

🧪 How It Works

1. User clicks "Download to CSV" in SQL Lab
2. Frontend checks if the result set is larger than CSV_STREAMING_ROW_THRESHOLD (default 100k rows)
3. For small exports: Works exactly as before - instant download
4. For large exports:
   • Progress modal opens immediately
   • Backend starts streaming CSV data in chunks

The key difference is that data flows continuously from database → backend → browser in manageable chunks, rather than accumulating in memory all at once.

🖼️ UX Enhancements

After (with streaming):

• Click "Download to CSV" on large result set
• Progress modal appears instantly showing export has started
• Live progress bar with percentage
• No browser freezing, no timeouts, no crashes

Users now have full visibility and control over large exports, with the confidence that multi-million row downloads will complete successfully.

🎯 Impact

Memory Usage

• Before: Entire dataset loaded in memory (500K rows = 500+ MB)
• After: Constant 64KB buffer regardless of dataset size

User Experience

• Before: No feedback, frequent failures, manual workarounds required
• After: Clear progress, reliable completion, professional export experience

[korbit-ai]

Based on your review schedule, I'll hold off on reviewing this PR until it's marked as ready for review. If you'd like me to take a look now, comment /korbit-review.

Your admin can change your review schedule in the Korbit Console

[codecov]

Codecov Report

❌ Patch coverage is 31.14754% with 168 lines in your changes missing coverage. Please review.
✅ Project coverage is 68.09%. Comparing base (43e9e1e) to head (dfc23fb).
⚠️ Report is 14 commits behind head on master.

Files with missing lines	Patch %	Lines
superset/commands/streaming_export/base.py	26.19%	62 Missing ⚠️
superset/charts/data/api.py	30.30%	38 Missing and 8 partials ⚠️
...erset/commands/sql_lab/streaming_export_command.py	34.09%	29 Missing ⚠️
superset/sqllab/api.py	30.30%	23 Missing ⚠️
...et/commands/chart/data/streaming_export_command.py	50.00%	8 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           master   #35478       +/-   ##
===========================================
+ Coverage        0   68.09%   +68.09%     
===========================================
  Files           0      632      +632     
  Lines           0    46445    +46445     
  Branches        0     5031     +5031     
===========================================
+ Hits            0    31626    +31626     
- Misses          0    13558    +13558     
- Partials        0     1261     +1261     

Flag	Coverage Δ
hive	43.80% <25.40%> (?)
mysql	67.21% <31.14%> (?)
postgres	67.26% <31.14%> (?)
presto	47.42% <31.14%> (?)
python	68.06% <31.14%> (?)
sqlite	66.88% <31.14%> (?)
unit	100.00% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

🎪 Showtime is building environment on GHA for b5393a4

[github-actions]

🎪 Showtime is building environment on GHA for b5393a4

[github-actions]

🎪 Showtime is building environment on GHA for b5393a4

[github-actions]

🎪 Showtime deployed environment on GHA for b5393a4

• Environment: http://34.220.88.207:8080 (admin/admin)
• Lifetime: 48h auto-cleanup
• Updates: New commits create fresh environments automatically

[github-actions]

🎪 Showtime is building environment on GHA for ed5c1e3

[github-actions]

🎪 Showtime is building environment on GHA for ed5c1e3

[github-actions]

🎪 Showtime is building environment on GHA for ed5c1e3

[github-actions]

🎪 Showtime deployed environment on GHA for ed5c1e3

• Environment: http://35.92.165.125:8080 (admin/admin)
• Lifetime: 48h auto-cleanup
• Updates: New commits create fresh environments automatically

[betodealmeida]

This is looking great! left a few comments.

[EnxDev]

LGTM!

[msyavuz]

LGTM as well. Do you want to take another look @betodealmeida?

[betodealmeida]

Awesome!