Repository navigation
fix(storage): download_filings(filings=) downloads only the requested filings (GH #1608) - #1610
Merged
Merged
Conversation
… filings (GH #1608) With filings= and no filing_date, download_filings() turned the filings into one min:max date range and downloaded every daily feed file in it. The only use of the requested accessions before downloading was a skip nested inside `if not date_filtered_filings.empty`, so a date with no requested filing was never skipped. Snowflake's six 10-Ks (2021-03-31 to 2026-03-20) cost 1,298 feed files against a stubbed quarterly listing, to keep 6 filings. Two more defects on the same path: - An existing date folder was skipped before the requested filings were checked, so a second call with other filings on a shared date downloaded nothing. The "already stored" check under it only ran with overwrite_existing=True, which is why Filings.download() passed True. - Even on the right dates, a filing cost its day's whole feed file: 2,892,246,544 bytes for 2026-03-20 against 17,816,574 for Snowflake's 10-K submission (sizes from SEC). download_filings is split into three paths: - dates only: unchanged, every feed file in the range. - filings=, source='submission' (default): each filing's full submission .txt, the file Filing.sgml() fetches from SEC, stored at <dir>/<YYYYMMDD of filing date>/<accession>.nc(.gz), where the local readers look first. The SGML parser detects the format from content, so a <SEC-DOCUMENT> file reads like a <SUBMISSION> feed bundle. A 404 is reported and the rest continue. - filings=, source='feed': only feed days holding a pending filing. A feed file is keyed by dissemination date, which can differ from the filing date (edgartools-a3ej), so the filing date is tried first and only filings still missing move to adjacent days, nearest first, within the read side's 4-day window. The issue's suggested fix (skip any feed day that is not a requested filing date) would have lost those filings silently. Filings already stored under data_directory, in their filing-date folder or an adjacent one, are skipped unless overwrite_existing. filing_date with filings= now narrows the filings to that range. Filings.download() passes overwrite_existing=False and gains source. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 of 6 tasks
4 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
download_filings(filings=...)downloads only the requested filings. It used to download SEC's daily feed file for every business day between the first and last filing date and filter only when saving: 1,298 feed files for Snowflake's six 10-Ks. Now:source="submission"(default): each filing's full submission file, the same fileFiling.sgml()fetches from SEC, stored at<dir>/<YYYYMMDD>/<accession>.nc(.gz)where the local readers look first.source="feed": feed files, but only the days that hold a requested filing. Each filing's own date is tried first; a filing not there is looked for in the adjacent days, nearest first, because a feed file is keyed by dissemination date (edgartools-a3ej). The issue's suggested fix (skip every day that isn't a filing date) would have dropped those filings without a warning.Filings.download()passesoverwrite_existing=False(it passedTrueonly to get past that folder skip) and takessource.filing_datetogether withfilings=narrows the filings to that range.Fixes #1608
For the reviewer
filings=changes from feed files to per-filing submission files. Filings land in the same folders, and the SGML parser detects<SEC-DOCUMENT>vs<SUBMISSION>from content, so readers are unaffected. It's what the docstring already promised, so this is titled a fix, with no upgrade-guide entry.Verification
download_filings(filings=...)stored it as a 2.4 MB.nc.gz, and reading it back with no network gives period of report 2026-01-31 and primary documentsnow-20260131.htm. The six Snowflake 10-K accessions in the test come from SEC's submissions JSON for CIK 1640147.tests/issues/regression/test_issue_1608.py, 11 offline tests (feed listing, feed download and submission request stubbed); 10 fail on main. Against the same stubbed listing, main downloads 1,298 feed files for the six 10-Ks; this branch downloads 6. Covers the next-day feed case (0000950129-95-001652 from a3ej), a shared date already stored, a 404 (reported, the rest continue), and reading a stored Apple 10-K submission (0000320193-24-000123) throughFiling.sgml()with the network blocked.sourceraisesValidationError; filings SEC doesn't have are logged by accession number.docs/guides/local-storage.md"Filtering Downloads" describes both sources with the measured sizes.changelog.d/1608.fixed.mdtests/storage+ a3ej + new tests: 174 passed, 4 skipped.hatch run test-fast: 42,548 passed, 1 failed (the raw-ValueError ratchet on the newsourcecheck, since changed toValidationError; ratchet and new tests then pass). Regression provenance OK. Lint: no new findings.AI assistance
🤖 Generated with Claude Code