[artcashin]

Problem

Opened with their defaults, 25 EIA dataset widgets fail. With no filters and no limit, those datasets match more rows than MAX_AUTO_ROWS (500,000), so api_query refuses the request by design:

The unfiltered request matches N rows, above the automatic pagination ceiling of 500000. Narrow the request with facet filters or dates, or set an explicit limit.

The refusal is right for the API, but the widgets offer no default that avoids it, so every one of them fails on first open. They include AEO, IEO, International, SEDS, three electricity datasets, four electricity grid datasets, two nuclear outage datasets and twelve petroleum datasets.

Change

Only the widget metadata changes, in eia_router.py:

• _widget_config takes newest_page. When it's set, the widget gets limit 5000 and a hidden sort of desc, so it opens on the newest 5,000 rows.
• The 21 catalog datasets that need it are listed in _NEWEST_PAGE, and the four flat commands (AEO, IEO, International, SEDS) pass newest_page=True.
• The REST defaults are unchanged: limit stays None (full pagination) and sort stays asc. The widget's limit stays visible, so a user can raise it or add filters.

Tests

New tests/test_widget_newest_page.py, which reads the app's OpenAPI:

• the 25 listed routes carry limit 5000 and hidden sort desc in widget_config;
• no other EIA route carries either;
• each listed route's API defaults are still limit=None, sort="asc".

The first test fails without the change; with it, all three pass. The provider suite passes: 2,097 tests (Python 3.12).

Results from a deployment

We run this as an overlay on the 2.0.0 wheel in OpenBQ's Docker image. Before the change, a sweep of every widget at its defaults got the "too broad" refusal above on these 25 widgets. Our overlay test checks the same 25 widget defaults through openbb-platform-api's widgets.json builder, and that the other 194 EIA widgets are unchanged.

Not included

The list is static. If EIA grows another dataset past the ceiling, its widget needs adding.

🤖 Generated with Claude Code

[dlee-bq]

Thanks for the PR, @artcashin.

I agree that some endpoints require practical default values, the bulk data sources and spreadsheet files are more appropriate for mining the full dataset.

These types of considerations should get baked in to the generate_catalog and api_query utilities layer instead of patching at the router level.