Repository navigation
5.62.1 backports: certifi floor, proxy rate-limit fix, and 13 wrong-value fixes - #1673
Merged
Merged
Conversation
…#1509) _parse_investment_identifier types any identifier ending in a bracketed percent as Unknown, a rule written for HTGC's category roll-ups ("Debt Investments Software (52.80%)"), and from_xbrl drops Unknown rows. PhenixFIN ends a real loan with its coupon ("... Delayed Draw Term Loan (14.00%)", 10-Q 0001213900-26-085667), so it left the portfolio with no warning. A suffixed row that states a principal amount is now re-parsed without the suffix and keeps the type and company that parse gives. HTGC's roll-ups state cost but never principal: measured over 46 filings from 27 BDCs, the only row this moves is PhenixFIN's loan. Keying on cost instead re-admits 51 HTGC roll-ups (+$18,335.9M on the FY2024 10-K); stripping and re-parsing alone re-admits 48 (+$10,979.9M), since "Debt Investments Software" parses as Unclassified. Co-authored-by: Matthew Mitchell <matthewprestonmitchell@gmail.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> (cherry picked from commit f307d6a)
* fix(markdown): keep the delimiter row for headerless tables (#1484) The pipe renderer wrote the header row and the separator only when the table had headers, so a headerless table came out as loose rows. Write a blank header row and a delimiter row for it. * Promote headerless table data into Markdown headers * Preserve literal promoted Markdown table cell contents (cherry picked from commit 9600182)
…1546) #1524 promotes the first non-empty row of a headerless table to the header so the table gets a delimiter row. A table with only one non-empty row (page footers, single footnotes) then became a header with a lone `| --- |` and no body: 182 such tables in the bac 10-K fixture, 117 in abbv. Those now keep the plain row they had before #1524; the count of header-only tables per fixture is back to its pre-#1524 value (bac 2, abbv 1). The promoted header now collapses whitespace with ' '.join(cell.split()), the same rule #1533 applies to body cells. The changelog fragment is rewritten to describe the change against the last release (headerless tables had no delimiter row) instead of against the PR's first draft. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit df56141)
…#1533) * fix(markdown): keep multi-line body cells inside their pipe-table row Body cells were joined into the pipe row with their internal line breaks (from <br> or several paragraphs), so the row ended early and Markdown parsers put the rest of the cell and the following cells outside the table. Join a body cell's lines with a space before emitting the row. Fixes #1486 * fix(markdown): collapse every whitespace run in pipe-table body cells Review: normalise each body cell with ' '.join(cell.split()) so \r, tabs and other whitespace cannot break the row either, drop the markdown_it import from the regression test, and rename it to the test_issue_ prefix. (cherry picked from commit 49d0da6)
…s label (#1549) * fix(xbrl): a decrease row was folded into the increase row sharing its label (#1471) Statement.get_raw_data() applies _merge_complementary_rows() on top of XBRL.get_statement(), to join the rows a company leaves behind when it renames a concept between years. It matched on label, level and dimension status, and required the period values to be complementary. Oracle's FY2026 10-K (0001193125-26-277521) labels both its Increases and Decreases foreign-currency translation concepts "Cumulative translation adjustments and other, net", files the increase for FY2026 and the decrease for FY2025 and FY2024, and so satisfied every test. The merge copied the decrease values into the increase row and deleted the decrease row, which carried the only -1 preferred_sign. to_dataframe(presentation=False) then exported the $11M and $8M decreases as increases with preferred_sign 1. Two changes, both in _merge_complementary_rows: - Rows whose display signs differ are no longer merged (_display_signs, beside _period_kinds). The assumption is that a rename keeps the direction of the line item it renames; the fixture corpus holds no cross-concept rename of a non-dimensional row to test it against, and Oracle is the only cross-concept merge known. A skipped merge leaves two rows with one label and loses no value. - When rows do merge, the second row's preferred_signs are now copied across with its values (a union; the first row's entries win), as units and period_types already were. Before, a negated row kept its -1 only for the periods that came from the first row. The visible change reaches the rendered statement and the default to_dataframe() as well: Oracle's note used to render one row, "22 11 8", and now renders two rows with the same label, "22" and "11 8". Measured over the fixture corpus (39 fixture filings, 2,544 statements) before and after: the same 2 merges, both of one concept with itself, both sign +1. Tests (7): the Oracle replay, the synthetic increase/decrease pair, the same pair with an empty preferred_signs on the increase row and in reverse order, a same-direction rename control (sign +1 and -1, which also checks the sign union), and the helper. Against origin/main's merge function, 6 fail and the helper test passes. A blanket refusal to merge across concepts fails the two controls, and so does dropping the sign union. Cassette: tests/cassettes/test_oracle_keeps_its_increase_and_decrease_rows_apart.yaml was recorded from live SEC on 2026-10-15 against unpatched main and not edited. It holds the two XBRL documents of the filing (orcl-20260531.xsd and orcl-20260531_htm.xml, 9.5 MB; Oracle embeds its linkbases in the schema) rather than the 29 MB full submission, and loads the same 91 roles and 2,195 facts as filing.xbrl(). Ground truth is the filed table in the income taxes note, in millions: 22, (11), (8). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(changelog): note the rendered and default-export change for #1471 The fragment named only get_raw_data() and to_dataframe(presentation=False), but the merge also feeds the rendered statement and the default to_dataframe(): the Oracle row renders as two rows with the same label now, where it was one. Say so, and shorten the rest to stay under the 500-character limit (457 now). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit a7d6336)
…t a component (#1567) * fix(financials): get_revenue() returns a bank's total net revenue, not a component (edgartools-u9yc9) Banks report total revenue as us-gaap:RevenuesNetOfInterestExpense, which the standardization's 'Revenue' mapping does not carry, so get_revenue() fell to the next thing that matched. Measured on FY2025 10-Ks against data.sec.gov companyconcept: - JPM: no mapped concept on the face statement, so the label patterns ran and `Revenue$` took "Noninterest revenue": 87.0B of 182.4B - WFC: fee lines are tagged contract revenue, which was searched before any bank total: "Fee income", 16.1B of 83.7B - AXP: 37.4B of 72.2B - USB: no total row at all; the label patterns took "Card revenue", 1.7B GS and MS were right only because a label pattern happened to hit "Net revenues". Search RevenuesNetOfInterestExpense after Revenue and before contract revenue. On a bank's statement (a us-gaap:NoninterestIncome row), skip the label patterns, since every "... revenue" label there is a component, so USB reports None rather than a wrong number (follow-up edgartools-hmj0j reads the filed us-gaap:Revenues fact instead). Add the concept to Statement.REVENUE_CONCEPTS so the ratio and trend code agrees. Before/after over 23 filers: JPM, WFC, AXP corrected; USB 1.7B -> None; BAC, C, GS, MS, PNC, SCHW, MET, AIG, ALL, COF (None before and after), AAPL, PFE, XOM, BRK.A, KO, WMT, TSLA, UNH unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(financials): correct a pin that snapshotted JPM's noninterest revenue as revenue (edgartools-u9yc9) test_y4yp4_metrics_render_reuse pinned get_financial_metrics()['revenue'] for the JPM FY2023 fixture at 68,837M, the value get_revenue() returned before this fix. That is us-gaap:NoninterestIncome; the bank's total net revenue in the same filing (0000019617-24-000225) is 158,104M. The pin was a behaviour-preservation snapshot for the render-reuse refactor, not a value checked against SEC. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 2b7953b)
…ce statement has none (edgartools-hmj0j) (#1580) After u9yc9, a bank-layout income statement with no total-revenue row returned None rather than a component. U.S. Bancorp and Capital One are that shape, but both file the total off the face statement as a non-dimensional us-gaap:Revenues fact. get_revenue() now reads it for the period that period_offset selects among the statement's columns. Instrumented first: U.S. Bancorp's instance (0000036104-26-000011) has 18 us-gaap:Revenues facts. 15 are segment values (is_dimensioned=True, e.g. OperatingSegmentsMember x a business segment: 12,083M, 7,607M, ...), and exactly one per year is non-dimensional: 28,656M / 27,455M / 28,144M for 2025/2024/2023. Capital One (0000927628-26-000024): 53,434M / 39,112M / 36,787M. No answer when the filing has no such fact or more than one distinct value. Summing the face rows would be wrong: U.S. Bancorp's NII plus noninterest income is 28,540M. The period column naming in _order_period_columns moves to _period_column_name so the new lookup maps a column back to its period by the same rule. EntityFacts already returns both totals; no change there. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 8cd5799)
An 8-K whose item heading puts a dash after "Item" was read as having no items. Liaoning Shuiyun Qinghe Rice Industry's 8-K 0000710782-24-000005 (filed 2024-12-20) heads its item "Item - 5.02 Departure of Directors of Principal Officers; Election of Directors; Appointment of Principal Officers." On 8cd5799, CurrentReport.items was [], content_type was "other", and report["5.02"] raised SectionNotFoundError, so the MCP reader answered "No items detected in this 8-K." Root cause: the text-based item patterns in edgar/company_reports/current_report.py required whitespace between "Item" and the number (Item\s+). Three regexes did: the item finder (_extract_items_from_text), the start of an item's text (_extract_item_content_from_text) and the end of an item, at the next item heading. Fix: one shared label, _ITEM_LABEL = r'Item\s*[-–—:]?\s*', used by all three, so a dash, en dash, em dash or colon is accepted between "Item" and the number. The end rule must change too: with only the other two widened, a dash-headed item runs on into the next dash heading. Measured on the saved primary document of 0000710782-24-000005: items == ['Item 5.02'], content_type == 'director_change', and report["5.02"] is 382 characters, from the heading to "List of shareholder:\nRu Fan", before the SIGNATURE block. The same change gives ['Item 5.02'] for 0000710782-24-000003 ("Item - 5.02") and ['Item 8.01'] for 0000710782-24-000004 ("Item - 8.01"). Mid-sentence references stay unmatched, because the line-start or horizontal-rule anchor is unchanged, and a lookup for an item the filing lacks still raises SectionNotFoundError. Regression test: tests/issues/regression/test_issue_1589.py, with the saved primary document in tests/fixtures/issues/regression/issue_1589/. On 8cd5799, 9 of its 12 tests fail; with this change all 12 pass. The changelog fragment is changelog.d/1589.fixed.md. Co-authored-by: Claude Haiku 5.5 <noreply@anthropic.com> (cherry picked from commit b6e0014)
…ount (GH #1585) (#1594) Item C.12 states a No as an element (<isCashCollateral>N</isCashCollateral>) and a Yes as attributes of another element (<cashCollateralCondition isCashCollateral="Y" cashCollateralVal="..."/>); non-cash collateral and loans by the fund take the same two shapes (SECURITY_LENDING_GROUP in eis_NPORT_Filer.xsd). SecurityLending.from_xml read only the elements, so every Yes read None and the amount was lost. SGOV NPORT-P 0002071691-26-016719: cash collateral Y on 1 of 24 holdings ($441,347,500), loan by fund Y on 2 of 24. IVV 0002071691-26-019760: 1 and 59 of 508. The flags now fall back to the condition element, and the amounts are three new optional fields: cash_collateral_value, non_cash_collateral_value, loan_value. Co-authored-by: mmitc <266342770+mmitc@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit c27f094)
) _render_table_pipe escaped pipes and backslashes only in grouped (multi-row) headers (#1474) and in a promoted header row (#1484). A flat header and every body cell went out raw, so a literal pipe opened an extra column and a lone backslash escaped the character after it. The issue's two-column header emitted three cells over a two-column delimiter, and markdown-it then read the whole table as a paragraph. "A | B" in a body cell split the same way. One helper, _escape_pipe_cell, now escapes every header and body cell, backslashes first. Over the 85 tracked HTML fixtures one filing changes: AbbVie's FY2024 10-K (0001551152-25-000020), whose page-footer cells read "| 2024 Form 10-K" beside the page number, about 114 lines, all footers. The #1474 suite pinned the old flat-header bytes by design ("does not assert Markdown escape correctness"); its flat-existing-escape-policy case now pins the escaped bytes instead. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 888acb0)
…ies the range (edgartools-329yu) (#1566) EFTS applies dateRange=custom only when it receives both startdt and enddt; with one it silently searches every year. search_filings sent only the bound it was given, so search_filings("tariff", forms="10-K", start_date="2025-01-01") returned 10,000+ hits led by 2010 and 2003 filings (raw EFTS, 2026-10-16); with enddt the same query has 3,841, all 2025-26. The docstring's own date-range example passed start_date alone. Fill the open bound: start -> 2001-01-01 (EFTS coverage), end -> tomorrow (EFTS dates are US Eastern; tomorrow keeps a caller west of it, or a slightly slow clock, from cutting off SEC's current day). Pagination reuses the stored params, so next()/fetch_more() are covered. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 446d447)
… revenue (edgartools-jrqvs) (#1630) Financials.get_revenue() searches us-gaap:Revenues first since gh #1294. The EntityFacts side still ranked RevenueFromContractWithCustomerExcludingAssessedTax first and only gave way to a total at least 2x larger (edgartools-fdye). That caught MetLife (31.6x) and American Tower (11.4x) but nothing within 2x. Contract revenue can be one face line of the total (GM "Automotive" $167,971M of $185,019M) or a note-only figure (COP $51,824M vs the face $58,944M). It can also exceed the total (Williams $14,899M vs $11,950M), so "take the larger" is not the rule either. Measured over 106 large filers' FY2025 10-Ks: 15 tag both concepts for the same year, 3 with equal values. MET and AMT were already right. The other 10 (CVX COP OXY KMI WMB D AEP SRE WMT GM) now match their face income statement and Financials.get_revenue() through get_revenue(), get_revenue_detailed(), the statement's Total Revenue and get_ttm_revenue(). Revenues now comes first in every facts-side list: _REVENUE_CONCEPTS, the gross-profit fallbacks, the display order, get_ttm_revenue() and the 'revenue' synonym group. get_revenue() and get_revenue_detailed() now share one _REVENUE_VARIANTS constant, and get_revenue_detailed() gains the fdye consolidated-total rule it was missing. Recency still settles which year answers (#1149, NVIDIA's tag migration); the order only breaks same-filing ties. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 2b92e2a)
…statement honours period_filter (GH #1469) (#1633) An unclassified note role (statement_type None) fell through select_periods to the duration selector, so a note holding only instant facts got annual columns with every cell blank. IBM's long-term debt components note (0000051143-25-000015) showed three empty FY columns instead of the filed $824M / $838M unamortized discount. XBRL.render_statement also called select_periods directly and dropped period_filter, so an explicit instant filter still produced duration headers. select_periods now takes the statement's rows. When every value sits in an instant period, and the type is not already an instant or equity type, it picks the statement's own instants that carry at least half as many rows as the densest one, newest first. render_statement and Statement.to_dataframe route through determine_periods_to_display and pass the rows; render_statement now forwards period_filter. period_view stays unwired: IBM's "Three Recent Periods" view names quarterly keys absent from reporting_periods and would cut three columns to one (edgartools-vsmia). Across 2,754 roles in the 31 XBRL fixtures, 519 change, every one from duration to instant columns on a statement with no duration facts; none of the new columns is empty and 93% of their cells hold a value (0% before). Balance sheet and income statement selections are unchanged. The #1305 test pinned the old behaviour for an uncatalogued Apple balance sheet parenthetical ("stays untyped, including its generic periods"); it still stays untyped, and now shows its instant share-count dates. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 7ce2a58)
…ache (edgartools-kd2ap) configure_http put the proxy into HTTP_MGR.httpx_params. httpx2 turns a client-level proxy into an all:// mount that takes precedence over the transport passed in, so after the call every request went to a bare HTTPTransport through the proxy: no CachingTransport, no RateLimitingTransport, in the shared and the async client alike, while get_http_config() looked normal. The documented corporate-proxy recipe therefore sent unthrottled traffic to SEC. Set HTTP_MGR.proxy instead; httpxthrottlecache applies it on the innermost transport, under the cache and the rate limiter, for the shared, async and uncached clients. get_http_config() and diagnose_ssl report HTTP_MGR.proxy, which also reflects HTTPS_PROXY (it read None before). tests/core/test_httprequests.py::test_configure_http_proxy asserted the old placement (httpx_params["proxy"]); it now asserts the proxy is on the manager and not a client param. The regression test compares each client's transport chain with and without a proxy: identical, except the connection pool at the bottom is the proxy. All 5 fail against main. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 6e6be36) Backport to 5.62.x: httpx instead of httpx2 in the comment and test; the uncached-client case is dropped because 5.62.x has no uncached_http_client; get_http_config() keeps its 5.x shape and reports HTTP_MGR.proxy.
…Sign root (edgartools-a9bak) On 5.x, httpx verifies SEC connections against certifi's bundle unless EDGAR_USE_SYSTEM_CERTS is set, so an installed certifi older than 2026.10.19 kept trusting "emSign ECC Root CA - G3" (eMudhra emSign CAs with compromised EC private keys, advisory 2026-10-20). certifi was only an indirect dependency through httpx; declare it with the fixed floor so upgrading edgartools upgrades it. 5.62.x only: on main (6.0) httpx2 verifies against the OS store and never reads certifi; the exposure there is a hand-built SSL_CERT_FILE bundle, handled in PR #1647. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On 5.x a missing item returns None with a FutureWarning and raises only under EDGARTOOLS_STRICT_ERRORS=1; the main version asserts the 6.0 raise. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…test tests/conftest.py swaps in a fresh httpclient.HTTP_MGR, so a reference imported at module load can be stale. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dgunning
changed the title
…0 lacks 5.x still supports Python 3.10 (requires-python >=3.10), where tomllib does not exist, so the test module failed to import and test-fast (3.10) went red on #1673. The floor is now checked with a regex on pyproject.toml. A skip would hide the check from exactly the 3.10 users 5.x exists for, and skips are banned under tests/issues/regression/. Python 3.10.20 clean uv venv (httpx only): pytest -m fast 8,623 passed, 0 failed; lowering the floor to 2026.7.22 makes the test fail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
Backports to the 5.62.x hotfix branch (scope: security, plus wrong values with no workaround), for a 5.62.1 release. Each commit is a cherry-pick (
-x) of a main commit unless marked otherwise.Security
httpxverifies against certifi's bundle unlessEDGAR_USE_SYSTEM_CERTSis set. An installed certifi older than 2026.10.19 therefore still trusted the compromised "emSign ECC Root CA - G3" (advisory 2026-10-20). certifi had been only an indirect dependency, through httpx. Main doesn't need this, because httpx2 uses the OS store there (fix(ssl): diagnose_ssl() flags a CA bundle that still trusts the compromised emSign root (edgartools-a9bak) #1647 covers the main side).configure_http(proxy=...)keeps the rate limiter and the cache (fix(http): configure_http(proxy=...) keeps the rate limiter and the cache (edgartools-kd2ap) #1654, edgartools-kd2ap). The bug is the same on 5.x: the proxy test fails 4/4 against 5.62.x'shttpclient.py. Adapted for 5.x: the comment and test sayhttpx, the uncached-client case is dropped because that client doesn't exist on 5.x, andget_http_config()keeps its 5.x shape.Wrong values with no workaround
render_statementhonoursperiod_filter.get_revenue()returns a bank's total net revenue.SecurityLending.from_xmlreads a "Yes" cash-collateral answer as None, losing the collateral amount #1585): N-PORT "Yes" securities-lending answer and its amount.PortfolioInvestments.from_xbrl#1504): a BDC position whose identifier ends in its coupon is kept. Its sibling BDC:from_xbrldrops a costed loan as the "total" of a sibling with the same fair value (ARCC Proofpoint second lien, $50.5M) #1505 is already on 5.62.x.search_filingswith onlystart_dateorend_dateapplies the range.|and\; fix(markdown): keep multi-line body cells inside their pipe-table row #1533 keeps multi-line cells in their row; fix(markdown): promote the first row in headerless pipe tables #1524 and fix(markdown): leave a one-row headerless table unpromoted (#1484) #1546 (GH Document.to_markdown() omits the delimiter for tables without headers #1484) handle headerless-table promotion.EDGARTOOLS_STRICT_ERRORS=1.Left out, because they depend on 6.0-only work
docs/upgrade/6.0.md(5.62.x's copy kept). fix(xbrl): an instant-only note renders its own instants, and render_statement honours period_filter (GH #1469) #1633 conflicted only on fix(xbrl): preserve selected statement roles and parenthetical periods (#1305) #1481's test, which isn't on 5.62.x. fix(search): search_filings with only start_date or end_date now applies the range #1566 conflicted on an import line.Not in this PR
xmltools. It will follow as its own PR.Verification
Run in a clean env built with uv against this branch: httpx 0.28.1, no httpx2, and certifi resolved to 2026.10.19 by the new floor. The shared dev env has both httpx and httpx2, and httpxthrottlecache picks httpx2 whenever it can import it, so that env can't validate 5.x.
pytest -m fast: 5.62.x unchanged gives 8,556 passed, 0 failed, the same count as the 5.62.0 release. This branch gives 8,624 passed, 0 failed.httpclient.pyand passes with the fix.86:a1:ec:ba:…:6b:be:6b, from that bundle's own header.🤖 Generated with Claude Code