Repository navigation
fix(8-k): detect items headed "Item - 5.02" with a dash (#1589) - #1590
Merged
Merged
Conversation
An 8-K whose item heading puts a dash after "Item" was read as having no items. Liaoning Shuiyun Qinghe Rice Industry's 8-K 0000710782-24-000005 (filed 2024-12-20) heads its item "Item - 5.02 Departure of Directors of Principal Officers; Election of Directors; Appointment of Principal Officers." On 8cd5799, CurrentReport.items was [], content_type was "other", and report["5.02"] raised SectionNotFoundError, so the MCP reader answered "No items detected in this 8-K." Root cause: the text-based item patterns in edgar/company_reports/current_report.py required whitespace between "Item" and the number (Item\s+). Three regexes did: the item finder (_extract_items_from_text), the start of an item's text (_extract_item_content_from_text) and the end of an item, at the next item heading. Fix: one shared label, _ITEM_LABEL = r'Item\s*[-–—:]?\s*', used by all three, so a dash, en dash, em dash or colon is accepted between "Item" and the number. The end rule must change too: with only the other two widened, a dash-headed item runs on into the next dash heading. Measured on the saved primary document of 0000710782-24-000005: items == ['Item 5.02'], content_type == 'director_change', and report["5.02"] is 382 characters, from the heading to "List of shareholder:\nRu Fan", before the SIGNATURE block. The same change gives ['Item 5.02'] for 0000710782-24-000003 ("Item - 5.02") and ['Item 8.01'] for 0000710782-24-000004 ("Item - 8.01"). Mid-sentence references stay unmatched, because the line-start or horizontal-rule anchor is unchanged, and a lookup for an item the filing lacks still raises SectionNotFoundError. Regression test: tests/issues/regression/test_issue_1589.py, with the saved primary document in tests/fixtures/issues/regression/issue_1589/. On 8cd5799, 9 of its 12 tests fail; with this change all 12 pass. The changelog fragment is changelog.d/1589.fixed.md. Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com> Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Owner
|
Thanks for this. It's a clean, well-scoped fix. One shared I checked the fixture against SEC's copy of Small things, none blocking:
Happy to merge as is. |
Contributor
Author
|
Will do. |
2 of 9 tasks
Contributor
Author
|
Done in #1616: the fixture docstring now says the saved page keeps the bot-defense script tag that www.sec.gov adds, and why the test is unaffected. The fixture itself is unchanged. Made with Claude Code: Claude Opus 5.5 (extra-high reasoning effort) wrote the change; GPT-6.1 Sol in Codex (ultra reasoning effort) recommended keeping the fixture as served and reviewed the change adversarially. |
dgunning
pushed a commit
that referenced
this pull request
Oct 18, 2026
…1589) (#1616) The fixture docstring said the saved 8-K page is byte for byte what the SEC served, which is true, but the file also carries the bot-defense script tag www.sec.gov adds to every response (its src changes per request), which is not part of the filing. The docstring now says so, and why the test is unaffected: the HTML parser removes script elements before building the text, and the assertions concern the filed item headings and text. Docstring only; the fixture and the assertions are unchanged. Follow-up to the review on #1590. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
4 tasks done
This was referenced Oct 22, 2026
dgunning
added a commit
that referenced
this pull request
Oct 22, 2026
…alue fixes (#1673) * fix(bdc): keep a position whose identifier ends in its coupon (#1504) (#1509) _parse_investment_identifier types any identifier ending in a bracketed percent as Unknown, a rule written for HTGC's category roll-ups ("Debt Investments Software (52.80%)"), and from_xbrl drops Unknown rows. PhenixFIN ends a real loan with its coupon ("... Delayed Draw Term Loan (14.00%)", 10-Q 0001213900-26-085667), so it left the portfolio with no warning. A suffixed row that states a principal amount is now re-parsed without the suffix and keeps the type and company that parse gives. HTGC's roll-ups state cost but never principal: measured over 46 filings from 27 BDCs, the only row this moves is PhenixFIN's loan. Keying on cost instead re-admits 51 HTGC roll-ups (+$18,335.9M on the FY2024 10-K); stripping and re-parsing alone re-admits 48 (+$10,979.9M), since "Debt Investments Software" parses as Unclassified. Co-authored-by: Matthew Mitchell <matthewprestonmitchell@gmail.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> (cherry picked from commit f307d6a) * fix(markdown): promote the first row in headerless pipe tables (#1524) * fix(markdown): keep the delimiter row for headerless tables (#1484) The pipe renderer wrote the header row and the separator only when the table had headers, so a headerless table came out as loose rows. Write a blank header row and a delimiter row for it. * Promote headerless table data into Markdown headers * Preserve literal promoted Markdown table cell contents (cherry picked from commit 9600182) * fix(markdown): leave a one-row headerless table unpromoted (#1484) (#1546) #1524 promotes the first non-empty row of a headerless table to the header so the table gets a delimiter row. A table with only one non-empty row (page footers, single footnotes) then became a header with a lone `| --- |` and no body: 182 such tables in the bac 10-K fixture, 117 in abbv. Those now keep the plain row they had before #1524; the count of header-only tables per fixture is back to its pre-#1524 value (bac 2, abbv 1). The promoted header now collapses whitespace with ' '.join(cell.split()), the same rule #1533 applies to body cells. The changelog fragment is rewritten to describe the change against the last release (headerless tables had no delimiter row) instead of against the PR's first draft. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit df56141) * fix(markdown): keep multi-line body cells inside their pipe-table row (#1533) * fix(markdown): keep multi-line body cells inside their pipe-table row Body cells were joined into the pipe row with their internal line breaks (from <br> or several paragraphs), so the row ended early and Markdown parsers put the rest of the cell and the following cells outside the table. Join a body cell's lines with a space before emitting the row. Fixes #1486 * fix(markdown): collapse every whitespace run in pipe-table body cells Review: normalise each body cell with ' '.join(cell.split()) so \r, tabs and other whitespace cannot break the row either, drop the markdown_it import from the regression test, and rename it to the test_issue_ prefix. (cherry picked from commit 49d0da6) * fix(xbrl): a decrease row was folded into the increase row sharing its label (#1549) * fix(xbrl): a decrease row was folded into the increase row sharing its label (#1471) Statement.get_raw_data() applies _merge_complementary_rows() on top of XBRL.get_statement(), to join the rows a company leaves behind when it renames a concept between years. It matched on label, level and dimension status, and required the period values to be complementary. Oracle's FY2026 10-K (0001193125-26-277521) labels both its Increases and Decreases foreign-currency translation concepts "Cumulative translation adjustments and other, net", files the increase for FY2026 and the decrease for FY2025 and FY2024, and so satisfied every test. The merge copied the decrease values into the increase row and deleted the decrease row, which carried the only -1 preferred_sign. to_dataframe(presentation=False) then exported the $11M and $8M decreases as increases with preferred_sign 1. Two changes, both in _merge_complementary_rows: - Rows whose display signs differ are no longer merged (_display_signs, beside _period_kinds). The assumption is that a rename keeps the direction of the line item it renames; the fixture corpus holds no cross-concept rename of a non-dimensional row to test it against, and Oracle is the only cross-concept merge known. A skipped merge leaves two rows with one label and loses no value. - When rows do merge, the second row's preferred_signs are now copied across with its values (a union; the first row's entries win), as units and period_types already were. Before, a negated row kept its -1 only for the periods that came from the first row. The visible change reaches the rendered statement and the default to_dataframe() as well: Oracle's note used to render one row, "22 11 8", and now renders two rows with the same label, "22" and "11 8". Measured over the fixture corpus (39 fixture filings, 2,544 statements) before and after: the same 2 merges, both of one concept with itself, both sign +1. Tests (7): the Oracle replay, the synthetic increase/decrease pair, the same pair with an empty preferred_signs on the increase row and in reverse order, a same-direction rename control (sign +1 and -1, which also checks the sign union), and the helper. Against origin/main's merge function, 6 fail and the helper test passes. A blanket refusal to merge across concepts fails the two controls, and so does dropping the sign union. Cassette: tests/cassettes/test_oracle_keeps_its_increase_and_decrease_rows_apart.yaml was recorded from live SEC on 2026-10-15 against unpatched main and not edited. It holds the two XBRL documents of the filing (orcl-20260531.xsd and orcl-20260531_htm.xml, 9.5 MB; Oracle embeds its linkbases in the schema) rather than the 29 MB full submission, and loads the same 91 roles and 2,195 facts as filing.xbrl(). Ground truth is the filed table in the income taxes note, in millions: 22, (11), (8). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(changelog): note the rendered and default-export change for #1471 The fragment named only get_raw_data() and to_dataframe(presentation=False), but the merge also feeds the rendered statement and the default to_dataframe(): the Oracle row renders as two rows with the same label now, where it was one. Say so, and shorten the rest to stay under the 500-character limit (457 now). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit a7d6336) * fix(financials): get_revenue() returns a bank's total net revenue, not a component (#1567) * fix(financials): get_revenue() returns a bank's total net revenue, not a component (edgartools-u9yc9) Banks report total revenue as us-gaap:RevenuesNetOfInterestExpense, which the standardization's 'Revenue' mapping does not carry, so get_revenue() fell to the next thing that matched. Measured on FY2025 10-Ks against data.sec.gov companyconcept: - JPM: no mapped concept on the face statement, so the label patterns ran and `Revenue$` took "Noninterest revenue": 87.0B of 182.4B - WFC: fee lines are tagged contract revenue, which was searched before any bank total: "Fee income", 16.1B of 83.7B - AXP: 37.4B of 72.2B - USB: no total row at all; the label patterns took "Card revenue", 1.7B GS and MS were right only because a label pattern happened to hit "Net revenues". Search RevenuesNetOfInterestExpense after Revenue and before contract revenue. On a bank's statement (a us-gaap:NoninterestIncome row), skip the label patterns, since every "... revenue" label there is a component, so USB reports None rather than a wrong number (follow-up edgartools-hmj0j reads the filed us-gaap:Revenues fact instead). Add the concept to Statement.REVENUE_CONCEPTS so the ratio and trend code agrees. Before/after over 23 filers: JPM, WFC, AXP corrected; USB 1.7B -> None; BAC, C, GS, MS, PNC, SCHW, MET, AIG, ALL, COF (None before and after), AAPL, PFE, XOM, BRK.A, KO, WMT, TSLA, UNH unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(financials): correct a pin that snapshotted JPM's noninterest revenue as revenue (edgartools-u9yc9) test_y4yp4_metrics_render_reuse pinned get_financial_metrics()['revenue'] for the JPM FY2023 fixture at 68,837M, the value get_revenue() returned before this fix. That is us-gaap:NoninterestIncome; the bank's total net revenue in the same filing (0000019617-24-000225) is 158,104M. The pin was a behaviour-preservation snapshot for the render-reuse refactor, not a value checked against SEC. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 2b7953b) * fix(financials): get_revenue() reads a bank's filed total when the face statement has none (edgartools-hmj0j) (#1580) After u9yc9, a bank-layout income statement with no total-revenue row returned None rather than a component. U.S. Bancorp and Capital One are that shape, but both file the total off the face statement as a non-dimensional us-gaap:Revenues fact. get_revenue() now reads it for the period that period_offset selects among the statement's columns. Instrumented first: U.S. Bancorp's instance (0000036104-26-000011) has 18 us-gaap:Revenues facts. 15 are segment values (is_dimensioned=True, e.g. OperatingSegmentsMember x a business segment: 12,083M, 7,607M, ...), and exactly one per year is non-dimensional: 28,656M / 27,455M / 28,144M for 2025/2024/2023. Capital One (0000927628-26-000024): 53,434M / 39,112M / 36,787M. No answer when the filing has no such fact or more than one distinct value. Summing the face rows would be wrong: U.S. Bancorp's NII plus noninterest income is 28,540M. The period column naming in _order_period_columns moves to _period_column_name so the new lookup maps a column back to its period by the same rule. EntityFacts already returns both totals; no change there. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 8cd5799) * fix(8-k): detect items headed "Item - 5.02" with a dash (#1590) An 8-K whose item heading puts a dash after "Item" was read as having no items. Liaoning Shuiyun Qinghe Rice Industry's 8-K 0000710782-24-000005 (filed 2024-12-20) heads its item "Item - 5.02 Departure of Directors of Principal Officers; Election of Directors; Appointment of Principal Officers." On 8cd5799, CurrentReport.items was [], content_type was "other", and report["5.02"] raised SectionNotFoundError, so the MCP reader answered "No items detected in this 8-K." Root cause: the text-based item patterns in edgar/company_reports/current_report.py required whitespace between "Item" and the number (Item\s+). Three regexes did: the item finder (_extract_items_from_text), the start of an item's text (_extract_item_content_from_text) and the end of an item, at the next item heading. Fix: one shared label, _ITEM_LABEL = r'Item\s*[-–—:]?\s*', used by all three, so a dash, en dash, em dash or colon is accepted between "Item" and the number. The end rule must change too: with only the other two widened, a dash-headed item runs on into the next dash heading. Measured on the saved primary document of 0000710782-24-000005: items == ['Item 5.02'], content_type == 'director_change', and report["5.02"] is 382 characters, from the heading to "List of shareholder:\nRu Fan", before the SIGNATURE block. The same change gives ['Item 5.02'] for 0000710782-24-000003 ("Item - 5.02") and ['Item 8.01'] for 0000710782-24-000004 ("Item - 8.01"). Mid-sentence references stay unmatched, because the line-start or horizontal-rule anchor is unchanged, and a lookup for an item the filing lacks still raises SectionNotFoundError. Regression test: tests/issues/regression/test_issue_1589.py, with the saved primary document in tests/fixtures/issues/regression/issue_1589/. On 8cd5799, 9 of its 12 tests fail; with this change all 12 pass. The changelog fragment is changelog.d/1589.fixed.md. Co-authored-by: Claude Haiku 5.5 <noreply@anthropic.com> (cherry picked from commit b6e0014) * fix(funds): read an N-PORT "Yes" securities-lending answer and its amount (GH #1585) (#1594) Item C.12 states a No as an element (<isCashCollateral>N</isCashCollateral>) and a Yes as attributes of another element (<cashCollateralCondition isCashCollateral="Y" cashCollateralVal="..."/>); non-cash collateral and loans by the fund take the same two shapes (SECURITY_LENDING_GROUP in eis_NPORT_Filer.xsd). SecurityLending.from_xml read only the elements, so every Yes read None and the amount was lost. SGOV NPORT-P 0002071691-26-016719: cash collateral Y on 1 of 24 holdings ($441,347,500), loan by fund Y on 2 of 24. IVV 0002071691-26-019760: 1 and 59 of 508. The flags now fall back to the condition element, and the amounts are three new optional fields: cash_collateral_value, non_cash_collateral_value, loan_value. Co-authored-by: mmitc <266342770+mmitc@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit c27f094) * fix(markdown): escape | and \ in every pipe-table cell (GH #1483) (#1632) _render_table_pipe escaped pipes and backslashes only in grouped (multi-row) headers (#1474) and in a promoted header row (#1484). A flat header and every body cell went out raw, so a literal pipe opened an extra column and a lone backslash escaped the character after it. The issue's two-column header emitted three cells over a two-column delimiter, and markdown-it then read the whole table as a paragraph. "A | B" in a body cell split the same way. One helper, _escape_pipe_cell, now escapes every header and body cell, backslashes first. Over the 85 tracked HTML fixtures one filing changes: AbbVie's FY2024 10-K (0001551152-25-000020), whose page-footer cells read "| 2024 Form 10-K" beside the page number, about 114 lines, all footers. The #1474 suite pinned the old flat-header bytes by design ("does not assert Markdown escape correctness"); its flat-existing-escape-policy case now pins the escaped bytes instead. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 888acb0) * fix(search): search_filings with only start_date or end_date now applies the range (edgartools-329yu) (#1566) EFTS applies dateRange=custom only when it receives both startdt and enddt; with one it silently searches every year. search_filings sent only the bound it was given, so search_filings("tariff", forms="10-K", start_date="2025-01-01") returned 10,000+ hits led by 2010 and 2003 filings (raw EFTS, 2026-10-16); with enddt the same query has 3,841, all 2025-26. The docstring's own date-range example passed start_date alone. Fill the open bound: start -> 2001-01-01 (EFTS coverage), end -> tomorrow (EFTS dates are US Eastern; tomorrow keeps a caller west of it, or a slightly slow clock, from cutting off SEC's current day). Pagination reuses the stored params, so next()/fetch_more() are covered. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 446d447) * fix(facts): EntityFacts revenue reads the filed total before contract revenue (edgartools-jrqvs) (#1630) Financials.get_revenue() searches us-gaap:Revenues first since gh #1294. The EntityFacts side still ranked RevenueFromContractWithCustomerExcludingAssessedTax first and only gave way to a total at least 2x larger (edgartools-fdye). That caught MetLife (31.6x) and American Tower (11.4x) but nothing within 2x. Contract revenue can be one face line of the total (GM "Automotive" $167,971M of $185,019M) or a note-only figure (COP $51,824M vs the face $58,944M). It can also exceed the total (Williams $14,899M vs $11,950M), so "take the larger" is not the rule either. Measured over 106 large filers' FY2025 10-Ks: 15 tag both concepts for the same year, 3 with equal values. MET and AMT were already right. The other 10 (CVX COP OXY KMI WMB D AEP SRE WMT GM) now match their face income statement and Financials.get_revenue() through get_revenue(), get_revenue_detailed(), the statement's Total Revenue and get_ttm_revenue(). Revenues now comes first in every facts-side list: _REVENUE_CONCEPTS, the gross-profit fallbacks, the display order, get_ttm_revenue() and the 'revenue' synonym group. get_revenue() and get_revenue_detailed() now share one _REVENUE_VARIANTS constant, and get_revenue_detailed() gains the fdye consolidated-total rule it was missing. Recency still settles which year answers (#1149, NVIDIA's tag migration); the order only breaks same-filing ties. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 2b92e2a) * fix(xbrl): an instant-only note renders its own instants, and render_statement honours period_filter (GH #1469) (#1633) An unclassified note role (statement_type None) fell through select_periods to the duration selector, so a note holding only instant facts got annual columns with every cell blank. IBM's long-term debt components note (0000051143-25-000015) showed three empty FY columns instead of the filed $824M / $838M unamortized discount. XBRL.render_statement also called select_periods directly and dropped period_filter, so an explicit instant filter still produced duration headers. select_periods now takes the statement's rows. When every value sits in an instant period, and the type is not already an instant or equity type, it picks the statement's own instants that carry at least half as many rows as the densest one, newest first. render_statement and Statement.to_dataframe route through determine_periods_to_display and pass the rows; render_statement now forwards period_filter. period_view stays unwired: IBM's "Three Recent Periods" view names quarterly keys absent from reporting_periods and would cut three columns to one (edgartools-vsmia). Across 2,754 roles in the 31 XBRL fixtures, 519 change, every one from duration to instant columns on a statement with no duration facts; none of the new columns is empty and 93% of their cells hold a value (0% before). Balance sheet and income statement selections are unchanged. The #1305 test pinned the old behaviour for an uncatalogued Apple balance sheet parenthetical ("stays untyped, including its generic periods"); it still stays untyped, and now shows its instant share-count dates. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 7ce2a58) * fix(http): configure_http(proxy=...) keeps the rate limiter and the cache (edgartools-kd2ap) configure_http put the proxy into HTTP_MGR.httpx_params. httpx2 turns a client-level proxy into an all:// mount that takes precedence over the transport passed in, so after the call every request went to a bare HTTPTransport through the proxy: no CachingTransport, no RateLimitingTransport, in the shared and the async client alike, while get_http_config() looked normal. The documented corporate-proxy recipe therefore sent unthrottled traffic to SEC. Set HTTP_MGR.proxy instead; httpxthrottlecache applies it on the innermost transport, under the cache and the rate limiter, for the shared, async and uncached clients. get_http_config() and diagnose_ssl report HTTP_MGR.proxy, which also reflects HTTPS_PROXY (it read None before). tests/core/test_httprequests.py::test_configure_http_proxy asserted the old placement (httpx_params["proxy"]); it now asserts the proxy is on the manager and not a client param. The regression test compares each client's transport chain with and without a proxy: identical, except the connection pool at the bottom is the proxy. All 5 fail against main. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 6e6be36) Backport to 5.62.x: httpx instead of httpx2 in the comment and test; the uncached-client case is dropped because 5.62.x has no uncached_http_client; get_http_config() keeps its 5.x shape and reports HTTP_MGR.proxy. * fix(ssl): floor certifi at 2026.10.19, which drops the compromised emSign root (edgartools-a9bak) On 5.x, httpx verifies SEC connections against certifi's bundle unless EDGAR_USE_SYSTEM_CERTS is set, so an installed certifi older than 2026.10.19 kept trusting "emSign ECC Root CA - G3" (eMudhra emSign CAs with compromised EC private keys, advisory 2026-10-20). certifi was only an indirect dependency through httpx; declare it with the fixed floor so upgrading edgartools upgrades it. 5.62.x only: on main (6.0) httpx2 verifies against the OS store and never reads certifi; the exposure there is a hand-built SSL_CERT_FILE bundle, handled in PR #1647. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(8-k): #1590's missing-item test asserts 5.x's contract on 5.62.x On 5.x a missing item returns None with a FutureWarning and raises only under EDGARTOOLS_STRICT_ERRORS=1; the main version asserts the 6.0 raise. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(http): read httpclient.HTTP_MGR through the module in the proxy test tests/conftest.py swaps in a fresh httpclient.HTTP_MGR, so a reference imported at module load can be stale. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(a9bak): read the certifi floor without tomllib, which Python 3.10 lacks 5.x still supports Python 3.10 (requires-python >=3.10), where tomllib does not exist, so the test module failed to import and test-fast (3.10) went red on #1673. The floor is now checked with a regex on pyproject.toml. A skip would hide the check from exactly the 3.10 users 5.x exists for, and skips are banned under tests/issues/regression/. Python 3.10.20 clean uv venv (httpx only): pytest -m fast 8,623 passed, 0 failed; lowering the floor to 2026.7.22 makes the test fail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Matthew Mitchell <realmatthewmitchell@gmail.com> Co-authored-by: Matthew Mitchell <matthewprestonmitchell@gmail.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: Michael <51033404+theluckystrike@users.noreply.github.com> Co-authored-by: Cristian <80774326+cristianchiriac@users.noreply.github.com> Co-authored-by: Max Haro <122497486+MaxHaro@users.noreply.github.com> Co-authored-by: Brian Stevenson @ synfonia <brian@synfonia.io> Co-authored-by: mmitc <266342770+mmitc@users.noreply.github.com>
dgunning
added a commit
that referenced
this pull request
Oct 23, 2026
… (GH #1773) Root cause: parsed_items kept its own item regex, (?:ITEM|Item)\s+(\d+...), which needs whitespace then the number. #1590 fixed the dash heading only in CurrentReport, so the two parsers had drifted. Change: parsed_items builds its pattern from CurrentReport's _ITEM_LABEL and _ITEM_NUMBER and normalizes the number; the local is_valid_8k_item check is gone because _ITEM_NUMBER only matches real item numbers. Verification: 8-K 0000710782-24-000005 (CIK 710782), SEC metadata '5.02': parsed_items now '5.02' (was ''), offline from the #1589 fixture and live via cassette. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dgunning
added a commit
that referenced
this pull request
Oct 23, 2026
… issues) (#1794) * fix(earnings): get_key_metrics returned a revenue component instead of total revenues (GH #1778) Root cause: the revenue loop took the first income-statement row whose label contained 'sales' or 'revenue', so Alta's leading "New and used equipment sales" row won over "Total revenues". Change: rank revenue labels (total row > bare aggregate label such as "Net sales" > any other match); the best rank wins, first row within a rank. "Total cost of ..." and "Total ... expense" rows are not treated as totals. Verification: Alta Equipment 8-K 0000950170-25-105056 now gives 481.2M (was 265.6M); Caterpillar 0000018230-26-000040 gives 20,543M (total sales and revenues, was 19,581M machinery component). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 1e40700d97220e84ffc299d9b0d1488ef48fbbdc) * fix(8k): item detection no longer reports "ITEM 2024" as an item or ends the item before it (GH #1772) Root cause: item detection, item start and item end in current_report.py matched any number after the item label (\d+\.?\s*\d*), so a body line starting "ITEM 2024 ..." became item '2024' and ended Item 5.02. Change: one shared _ITEM_NUMBER pattern matches only a whole 8-K item number ("2.02", "9. 01", legacy "1"-"9"). Item start now compares the whole normalized number, so "Item 5" no longer starts at "Item 5.02". Verification: the issue's synthetic body now gives ['5.02', '9.01'] (was ['2024', '5.02', '9.01']) and Item 5.02 keeps the bonus line. Real filing 0000710782-24-000005 (fixture from #1589) still gives ['5.02'] with the "Item - 5.02 Departure of Directors of Principal Officers" heading. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(8k): EntityFiling.parsed_items returns items headed "Item - 5.02" (GH #1773) Root cause: parsed_items kept its own item regex, (?:ITEM|Item)\s+(\d+...), which needs whitespace then the number. #1590 fixed the dash heading only in CurrentReport, so the two parsers had drifted. Change: parsed_items builds its pattern from CurrentReport's _ITEM_LABEL and _ITEM_NUMBER and normalizes the number; the local is_valid_8k_item check is gone because _ITEM_NUMBER only matches real item numbers. Verification: 8-K 0000710782-24-000005 (CIK 710782), SEC metadata '5.02': parsed_items now '5.02' (was ''), offline from the #1589 fixture and live via cassette. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(earnings): 8-K earnings lookup finds EX-99 exhibits by document type (GH #1776) Root cause: find_earnings_exhibits matched "EX-99" in the description and "ex-99"/"ex99" in the filename but never read attachment.document_type, so a release with a plain filename and a prose description was skipped. The sort key read only the description, so JPMorgan's EX-99.1 release (no ex99 in its name) was missed while the EX-99.2 supplement was returned. Change: accept document_type starting with EX-99, and sort by the number in the document type (then the description). Verification: Accenture 8-K 0001467373-25-000221: earnings was None, now q1fy26earnings8-kexhibit.htm with revenue 18,742,125 thousand. Exxon 0000034088-26-000065 now finds its EX-99.1; JPMorgan 0001628280-26-065582 now returns a3q26erfexhibit991narrative.htm ahead of the EX-99.2 supplement. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(earnings): 8-K earnings values no longer shift under the next period header (GH #1775) Root cause: _extract_clean_dataframe compacted header labels and value cells separately and zipped them left to right. The two-row header branch kept a scale caption over the row labels ("In millions, except per share amounts", "(IN THOUSANDS, ...", "($ In Millions, ...") as a column (its skip test was case-sensitive "(In"/"(Dollars" only), so every value sat one column early and the current-period header returned the prior period. A "%" printed in its own cell shifted the rest of its row the same way. Change: when a table has header rows, each value cell is placed under the lowest header cell above it by grid column (_align_by_position), and the cells of one value ("$", "38,198", "%") are joined. Scale captions are never columns (_is_scale_caption). Header columns that receive no value are dropped. If any value cell does not fall under exactly one header column, the old left-to-right layout is kept. Verification (current quarter, before -> after): - Graphic Packaging 0001408075-25-000020 net income 190M -> 104M - Certara 0001827090-25-000034 revenue 93,313K -> 104,570K, net loss -1,968K - Harte Hanks 0000045919-24-000010 revenue 54,778K -> 49,491K, EPS -0.27 - Five Star Bancorp 0001628280-25-046534 net income 14,508K -> 16,344K - Home Depot 0000354950-26-000026 net sales 39,704M -> 38,198M, EPS 2.58 - Intel 0000050863-26-000009 net income -126M -> -591M On a random sample of 26 earnings 8-Ks (2024-2026), 35 tables changed; none lost a (row, value) pair, and those inspected were caption or "%" shifts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(earnings): 8-K earnings tables with three header rows keep every period column (GH #1777) Root cause: ASGN's results table has group, month and year header rows, and the empty cell over the row labels has rowspan="2". The header builder read only two rows and ignored rowspan, so the month row was misaligned and three labels covered five value columns; the nine-month columns were cut. The repeated "September 30," label then made df.iloc[pos][col] return several cells, so get_key_metrics() and to_facts_dataframe() skipped them. Change, on top of the grid-position layout from #1775: - _place_header_cells honours rowspan, so every header row lines up; - get_key_metrics() and FinancialTable.to_facts_dataframe() read the chosen column by position; - _parse_period_header reads stacked labels joined with " - " ("September 30, - 2025"), so facts get a period end; - a ")" or "%" in the spacer cell after "(636" joins its number, and a "$" or "(" in a spacer cell joins the next number. Verification: ASGN 8-K 0000890564-25-000058: revenue/net income/diluted EPS None/None/None -> 1,011.4M/38.1M/0.87; revenue row now has all five filed columns (1,011.4; 1,031.0; 1,020.6; 3,000.3; 3,114.7) with period ends. Ring Energy 0001384195-25-000022: None -> revenue 78,601,336. Citizens 0000024090-25-000126 (two columns headed 2025): None -> total revenues 62,808K, net income 2,417K. NexGel 0001641172-25-023188: the ")" cells are gone; loss from operations -636 / -1,054 / -1,413 / -1,925. Random sample of 26 earnings 8-Ks: 35 tables changed, none lost a (row, value) pair. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(earnings): "% Change" values no longer sit under the year-ago period (GH #1780) Root cause: Home Bancorp's tables have no detected header rows, so _extract_date_headers_from_rows supplied the labels and kept only cells holding a date. The two "% Change" cells were dropped from the labels but their values stayed in the rows, so 3/31/2024 got the percent change and the year-ago value landed in Col_3. The caption row was also kept as data, because its skip test was the case-sensitive "(In"/"(Dollars". Change: _find_date_header_rows returns the row(s) the dates came from; that row is the header row for the grid-position layout from #1775 (so "% Change" is its own column) and is skipped as data. Scale captions are matched with _is_scale_caption. FinancialTable.to_facts_dataframe() skips percent and change columns, detected from any part of the header path ("% Change", "2025 vs 2024", "... - 2025 - %"), so percentages are not reported as USD amounts even when the label carries a year. Verification: Home Bancorp 8-K 0001436425-25-000016: the 3/31/2024 column of total interest income -1.0 -> 44,126 (filed), net income 13.0 -> 9,199; Total Assets on the balance sheet 1.0 -> 3,357,604. Net income facts went from five (including 13000 and 19000 USD) to the three filed amounts. Random sample of 26 earnings 8-Ks: 5 tables changed, none lost a (row, value) pair. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(earnings): thousands tables in a wrapper div are no longer scaled as millions (GH #1779) Root cause: step 6 of _detect_table_scale read only the table's own siblings. When the table is the only child of a wrapper <div> (every table in the Cardlytics release, the Gartner income statement), the caption sits one level up and was never read, so the document-level fallback applied. For Cardlytics that fallback is millions, from one guidance sentence. Change: climb through one-child wrappers before reading preceding siblings, and read the nearest sibling first. Verification: - Cardlytics 0001666071-25-000124 ("(Amounts in thousands, except per share amounts)"): revenue 63,249,000,000 -> 63,249,000; net loss -9,283,000,000 -> -9,283,000. - Gartner 0000749251-25-000044 ("in millions"): revenue 1,686.5 -> 1,686,500,000. - Core Natural Resources 0001710366-25-000008, Duolingo 0001562088-24-000048, Kulicke & Soffa 0000056978-25-000078: thousands. - Random sample of 26 earnings 8-Ks: 12 tables changed scale, each now matching the caption printed above it (e.g. Summit Materials 0001628280-24-019631 units -> thousands, "$ in thousands"). Not fixed: Snail 0001641172-25-024808 prints whole dollars with no caption and still inherits the document-level "millions"; that fallback needs a design decision. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(earnings): income_statement picks the statement of operations over segment tables (GH #1781) Root cause: EarningsRelease.income_statement returned the INCOME_STATEMENT table with the most rows. Segment tables (Mueller: one period, a column per segment), reconciliations (Graphic Packaging's Adjusted EBITDA table, Intel's GAAP-to-non-GAAP table) and metric tables (Groupon's supplemental metrics) are longer than the statement. Separately, the caption row "(in millions, except per share amounts)" became the parent context of the rows below it, so "Net sales" was classified per-share and returned unscaled. Change: candidates are ranked by _income_statement_rank: revenue and net income rows present; not a reconciliation (a quarter or more of labels say adjusted, non-GAAP, EBITDA or start with GAAP, so a statement ending in two adjusted lines still counts); not one period split by segment (two or more columns parse to the same single period; unparseable headers such as "2025" are unknown, not split); then row count. Scale captions are no longer parent contexts. Verification (before -> after): - Mueller 0001350593-25-000054: FY2024 segment table, EPS 0.74, revenue 755.5 (unscaled) -> consolidated statement, EPS 0.33 (year 1.22 in the same table), revenue 380.8M, net income 52.6M. - Groupon 0001490281-25-000040: revenue 86,889K (North America Q3 2024) -> 125,702K, net income 20,593K, diluted EPS 0.46. Q3 2025 and Q4 2025 releases now give 122,825K and 132,708K from the statement. - Graphic Packaging 0001408075-25-000020: revenue 15.3 (a margin) -> 2,204M, diluted EPS 0.42 (adjusted) -> 0.34 (GAAP). - Intel 0000050863-26-000009: revenue None -> 13,674M. Random sample of 26 earnings 8-Ks: the chosen table is the same as on origin/main for every release. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
Before: an 8-K whose item heading puts a dash after "Item" (for example "Item - 5.02 Departure of Directors ...") reports no items.
EightK.itemsis empty,content_typeisother, andeightk["5.02"]raisesSectionNotFoundError. After: a dash, en dash, em dash or colon is accepted between "Item" and the number. The same filing then reports['Item 5.02'],content_typeisdirector_change, andeightk["5.02"]returns the item from its heading to the end of the item, before the signature block.Fixes #1589
Verification
items == ['Item 5.02'],content_type == 'director_change', the exact heading text and the item's last two lines, "List of shareholder:" and "Ru Fan".tests/issues/regression/, and its docstring links the issue or bead.tests/issues/regression/test_issue_1589.py, with the fixture intests/fixtures/issues/regression/issue_1589/.None. A lookup for an item the filing does not contain still raisesSectionNotFoundError, and the test asserts it.changelog.d/<id>.<section>.mdfragment, not an edit toCHANGELOG.md.6.0
docs/upgrade/6.0.mdsays what users see now and what to do insteadCI fails a PR that stages a 6.0 change without touching the upgrade guide. If it flags something that isn't a 6.0 change, add the
no-upgrade-guidelabel, say why here, and re-run the job.AI assistance
Working context (optional)
Plan: one shared item label,
Item\s*[-–—:]?\s*, used where the text pass finds an item, where an item's text starts and where it ends. Those three places hadItem\s+, which needs whitespace and so never matched a dash.Evidence checked:
items == [],content_type == 'other',report["5.02"]raisesSectionNotFoundError,report.get("5.02")returnsNone.Filing.text()does, renders to 2,821 characters. The heading "Item - 5.02 Departure of Directors of Principal Officers; Election of Directors; Appointment of Principal Officers." is on its own line.items == ['Item 5.02'],content_type == 'director_change',report["5.02"]is 382 characters and ends with "List of shareholder:\nRu Fan".['Item 5.02'],['Item 5.02']and['Item 8.01'].report.sections(andreport.document.sections) still finds 0 sections for this filing. That comes from the separate section parser inedgar/documents, which this change leaves alone; widening itsitem_502pattern on its own did not change the result either (the cause was not investigated).items,content_typeandreport["5.02"]do not depend on it.Item\s+makestest_dash_heading_ends_the_previous_itemfail, because a dash-headed item then runs into the next dash heading. This is why the end-of-item pattern uses the same label.pytest -m fast -n auto, whole tree, on this branch, run twice: 42,346 passed, 52 failed, 147 errors, 3 skipped, 1 xfailed; and 42,345 passed, 53 failed, 147 errors, 3 skipped, 1 xfailed. Every failing test and error was run again on unchanged 8cd5799 in the same environment. 198 of the 200 fail or error there too: all 147 errors areUnicodeDecodeError('charmap' codec, the Windows default codepage) from reading UTF-8 files, and 51 are failures whose causes were not diagnosed. Run again withPYTHONUTF8=1, the same 200 tests give 26 failures on this branch and the same 26 on 8cd5799. The other two,test_parsing_speed_thresholdandtest_large_document_performance, are timing checks that pass or fail from run to run under load (the first failed on 8cd5799 in one run and passed in another); both pass alone on this branch and on 8cd5799.ruff check edgar: the only changed file,edgar/company_reports/current_report.py, has the same 7 findings as its base version (S110,try-except-pass), and the change adds none. The tree as a whole reports findings in files this change does not touch.edgar_readwithsections=["items"]on 0000710782-24-000005 returnsNo items detected in this 8-K.on the base commit and the Item 5.02 text on this branch. Also run, one file at a time: the 8-K tests that read real filings (tests/company_reports/test_eightK.py, test_eightk_improvements.py and nine regression files, 139 tests), all passing. An 80-filing sample of other 8-Ks (2005, 2015, 2024, 2025) gives identical items and item text on base and on this branch. The documentation was not changed and its examples were not run.items,content_type, item text andedgar_readoutput. The one that changes is 0000710782-24-000005, as described above; none of the other 65 heads an item with a dash or a colon. The new label also accepts "Item5.02" with no space at all; no filing in either sample has such a heading, so that case is untested.