Repository navigation
Conversation
Retain each document table column's group and leaf labels using its header rowspan/colspan positions, including legal zero row spans. Escape header pipe delimiters and literal backslashes without changing original table cells. Exclude actual header rows from the existing numeric data-alignment heuristic while preserving its body behavior. Cover the filed Apple duration/date labels, equity components, Rich call order, and synthetic header boundaries. Fixes dgunning#1474
Assert that a body wider than its headers retains every amount and an empty header, and that a narrower body keeps its header columns and empty padding. The wider-body case fails when the header-matrix bounds guard is removed. Production renderer and changelog content remain unchanged. Refs dgunning#1474
Use explicit non-None checks to narrow parsed tables and expanded matrix cells in the grouped Markdown header regression. The 31 test identities, parameters and expected values remain unchanged. Refs dgunning#1474
Apply grouped header paths only to tables with at least two header rows. Keep flat and headerless table output compatible and add span and whitespace controls. Record the existing Apple equity header-classification limitation in exact regression assertions. (GH dgunning#1474)
|
Read the diff, did not run the suite. The Apple 10-Q case ($90,753 million under the wrong date) is a real misattribution and projecting
|
Add matching-year and three-level period/duration/date cases. Clarify span identity, flat-header escaping compatibility and the header-only matrix boundary without changing rendering behavior. (GH dgunning#1474)
Keep the original header-row geometry when table parsing separates spacer rows from detected headers. Bound row spans to their source row groups and leave parsed cell spans, body data, and flat-table rendering unchanged. Add exact Eaton equity-table fixtures and public Markdown regression cases for missing rows, blank-origin spans, header replacement, and text rendering before Markdown.
|
Thanks for checking these details. I added two regression cases and comments covering the boundaries you identified.
The final branch passes all 60 focused regression cases with sockets blocked and retries disabled. Its fresh fast suite passes with 8,135 passed, 9 skipped and 1 expected failure, with no failures or reruns. The fresh base passes 8,075 cases with the same skips and expected failure; all 8,085 common test outcomes are unchanged. On the same 60-case regression comparison, the base has 44 failures and 16 passes. The Apple R2/R3 filing tests remain the real-world check: the $90,753 million comparative net-sales amount stays under The synthetic examples below were captured at the published A spanning label appears onceTest: Input (synthetic HTML): <table>
<tr><th rowspan="2">Business</th><th colspan="2">Revenue</th></tr>
<tr><th>2026</th><th>2025</th></tr>
<tr><td>Cloud</td><td>10</td><td>8</td></tr>
<tr><td>Other</td><td>3</td><td>4</td></tr>
</table>Rendered Markdown:
Separate cells with the same text keep both levelsTest: Input (synthetic HTML): <table>
<tr><th rowspan="2">Business</th><th colspan="2">2024</th></tr>
<tr><th>2024</th><th>2023</th></tr>
<tr><td>Cloud</td><td>10</td><td>8</td></tr>
</table>Rendered Markdown:
Quarterly and annual columns keep their duration and dateTest: Input (synthetic HTML): <table>
<tr><th rowspan="3">Metric</th><th colspan="2">Current year</th><th colspan="2">Prior year</th></tr>
<tr><th>Three Months Ended</th><th>Year Ended</th><th>Three Months Ended</th><th>Year Ended</th></tr>
<tr><th>Dec. 31, 2025</th><th>Dec. 31, 2025</th><th>Dec. 31, 2024</th><th>Dec. 31, 2024</th></tr>
<tr><td>Sales</td><td>10</td><td>40</td><td>8</td><td>30</td></tr>
</table>Rendered Markdown:
Tested beyond the fixtures. Scanned 650 offline 10-K and 10-Q primary documents (50,892 HTML
The first run caught one regression affecting three Eaton equity tables: an empty header row crossed by rowspans put Real filingsComparison method and scope. Inventory scanning used HTML markup to select complex shapes and controls; it did not render all 650 documents. The retained-selection rerun compared The source check rebuilds colspan/rowspan grids, matches parsed tables by text, and matches body rows to source rows. Of 3,535 multi-header tables, 3,256 could be scored: 132 had no source-table signature match and 147 had no usable body-row match. Another 225 unmatched body rows were skipped within the scoring. The 77,413 placements are matched non-empty body cells, not independently verified financial facts. These percentages describe agreement with the source grid; they do not prove that the filer's grid, parser classifications or financial interpretation are correct. The judge's whole-blank-column check agrees in 104 of the 105 spacer cases; the SBUX exception contains a pre-existing multiline body cell (#1486). Against the first The first four examples were captured from tracked fixtures through Flat header and 33-row balance sheet. Accession Before = after, including all 33 body rows: Header-only excerpt; body rows omitted from display.
Exact captured Markdown| CONDENSED CONSOLIDATED BALANCE SHEETS (Unaudited) - USD ($) $ in Millions | Mar. 29, 2025 | Sep. 28, 2024 |
| --- | --- | --- |Duration/date groups and row-spanning stub. Accession Before: Header-only excerpt; body rows omitted from display.
Exact captured Markdown| CONDENSED CONSOLIDATED STATEMENTS OF OPERATIONS (Unaudited) - USD ($) shares in Thousands, $ in Millions | 3 Months Ended | Mar. 29, 2025 | 6 Months Ended | |
| --- | --- | --- | --- | --- |After: Header-only excerpt; body rows omitted from display.
Exact captured Markdown| CONDENSED CONSOLIDATED STATEMENTS OF OPERATIONS (Unaudited) - USD ($) shares in Thousands, $ in Millions | 3 Months Ended Mar. 29, 2025 | 3 Months Ended Mar. 30, 2024 | 6 Months Ended Mar. 29, 2025 | 6 Months Ended Mar. 30, 2024 |
| --- | --- | --- | --- | --- |The statement title spans two header rows. Each duration now stays with both filed dates; the 24 body rows are identical. Three header rows and a wide table. Accession Before: Header-only excerpt; body rows omitted from display.
Exact captured Markdown| June 30, 2024 | Less than 12 Months | | | Unrealized Losses | | 12 Months or Greater | | Unrealized Losses | | | Total Fair Value | | Total Unrealized Losses | | |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |After: Header-only excerpt; body rows omitted from display.
Exact captured Markdown| (In millions) June 30, 2024 | Less than 12 Months Fair Value | Less than 12 Months Fair Value | Less than 12 Months | Less than 12 Months Unrealized Losses | | 12 Months or Greater Fair Value | 12 Months or Greater Fair Value | 12 Months or Greater Unrealized Losses | 12 Months or Greater Unrealized Losses | | Total Fair Value | Total Fair Value | Total Unrealized Losses | Total Unrealized Losses | |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |This table has 16 rendered columns and six body rows. Its third header row supplies the date in the stub; the numeric columns have two non-empty label levels, rather than three. All body rows are identical. Known R1 limitation (#1477). Accession Before: Header-only excerpt; body rows omitted from display.
Exact captured Markdown| Cover Page - shares shares in Thousands | 6 Months Ended | |
| --- | --- | --- |After: Header-only excerpt; body rows omitted from display.
Exact captured Markdown| Cover Page - shares shares in Thousands Entity Information [Line Items] Document Type Document Quarterly Report Document Period End Date | 6 Months Ended Mar. 29, 2025 10-Q true Mar. 29, 2025 | Apr. 18, 2025 |
| --- | --- | --- |The parser still incorrectly promotes Additional complex filings. A separate comparison captured the JPMorgan, Berkshire and GSBD tables through JPMorgan deposits: balances, rates and periods. Q2 2026 10-Q, accession
Berkshire Note 24: period and insurance segment. Q2 2026 10-Q, accession
The base lost American Airlines: quarter versus year-to-date. Accession Before: Header-only excerpt; body rows omitted from display. The retained capture contains only the header line; the separator below is added for this preview.
Exact captured Markdown| | 2025 | 2024 | 2025 | 2024 | | Increase (Decrease) |After: Header-only excerpt; body rows omitted from display. The retained capture contains only the header line; the separator below is added for this preview.
Exact captured Markdown| | Three Months Ended June 30, 2025 | Three Months Ended June 30, 2024 | Increase (Decrease) | Six Months Ended June 30, 2025 | Six Months Ended June 30, 2024 | Increase (Decrease) |Intel: fair-value hierarchy. Accession Eaton correction checked through the public API. Accession Limits shown by those captures:
|
dgunning
left a comment
There was a problem hiding this comment.
Thanks. The fix holds up well. Its 60 new tests pass, they fail on main (the Apple R2 case first), and the 712 offline document and Markdown tests pass on the branch. To check for side effects, I compared output on main and on this branch across the 79 real filings under tests/fixtures/html/ (12,933 tables). In the Markdown, no body value moves or changes order, and the restored header paths are right where I checked them, for example CAT's "Level 1" fair-value column and CVX's "Upstream U.S." segment columns.
One claim in the description doesn't match what I measured. to_dataframe() and text() do change. 9 DataFrames and 8 text() outputs differ, all in equity-statement tables:
adbe/10k/adbe-10-k-2025-01-13.html, table 33 (the4,756row)so/10k/so-10-k-2026-02-19.html, tables 96 and 102 (e.g.2,080,2,297,(1,855))gbdc/10k/gbdc-10-k-2024-11-19.html, tables 172, 173 and 223, andgbdc/10q/gbdc-10-q-2025-08-04.html, tables 125 and 126c/10k/c-10-k-2025-02-21.html, table 252
The cause is the row_idx >= self.header_row_count change in table_matrix.py, which every TableMatrix user shares, not just the Markdown renderer. The new output is the correct one: on main these rows sit one column right of where every other row in the same table puts that column's values, and on this branch they line up. So it's an extra fix, but this ships in a 5.x minor release, so it should be stated rather than described as unchanged.
Could you:
- Add a sentence to
changelog.d/1474.fixed.mdand the PR description saying thatto_dataframe()andtext()column placement is also corrected in some equity-statement tables. - Add one regression assertion pinning it, e.g. the Southern Company
2,080/2,297rows landing in the same column as that table's other rows. The SO and ADBE filings are already tracked fixtures.
Two non-blocking notes for the record:
- Markdown bytes change in 76 of the 79 filings. That's expected for this fix, but RAG users will see their indexed text change, so it's worth one clause in the changelog.
- Where header detection promotes body rows (#1477, #1485), header cells now concatenate every promoted value. That produces 38 of 6,613 tables with a phrase repeated inside one cell (cover-page checkboxes, for example) and 192 cells over 120 characters. That's no worse than
main, which kept only one value and dropped the rest, and it belongs to those issues, not this PR.
With 1 and 2 in, this is good to merge for 5.62.0. To make it, it needs to land by 10-11.
|
Thanks, Dwight. Both changes are in
One correction to "line up". These transaction rows are rows that header detection promotes (#1477, #1485). On this branch they line up with the other promoted rows and with their component's header path. The closing balances in the body still sit one slot to the right within the same component, even though the filing starts |
…het (#1508) test-slow has failed on main since 2026-10-08: PR #1480 (GH #1474) kept grouped table header paths, and eleven tracked filings stopped losing numbers (133 in total), but they were not moved from BASELINE_NUMBER_LOSS to CLEAN_FILINGS in that commit, so test_a_repaired_filing_is_banked fails. Moved: 20-F 0001062993-16-008650 (42), aapl/10q (10), pg/10q (24), tsla/10q (4), abbv/10k (4), hd/10k (32), ma/10k (2), nke/10k (7), pfe/10k (3), rf/10k (4), xom/10k (1). Measured on the full local corpus (text_boundary_corpus included), the same eleven as CI's 61-fixture view; the ratchet's 7 tests pass after the move. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…het (#1508) (#1510) test-slow has failed on main since 2026-10-08: PR #1480 (GH #1474) kept grouped table header paths, and eleven tracked filings stopped losing numbers (133 in total), but they were not moved from BASELINE_NUMBER_LOSS to CLEAN_FILINGS in that commit, so test_a_repaired_filing_is_banked fails. Moved: 20-F 0001062993-16-008650 (42), aapl/10q (10), pg/10q (24), tsla/10q (4), abbv/10k (4), hd/10k (32), ma/10k (2), nke/10k (7), pfe/10k (3), rf/10k (4), xom/10k (1). Measured on the full local corpus (text_boundary_corpus included), the same eleven as CI's 61-fixture view; the ratchet's 7 tests pass after the move. (cherry picked from commit 2250052) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
What this changes
Document.to_markdown()now keeps each grouped table column's complete header path, from group to leaf. Tables with zero or one header row retain their previous Markdown bytes. In Markdown, non-empty body values and their order are preserved.Upgrading: Grouped-table Markdown includes detected group, duration and date headers, and
to_dataframe()andtext()correct some equity-component column placements; filed amounts stay unchanged.In Apple’s Q2 FY2025 10-Q (
0000320193-25-000057), comparative net sales of $90,753 million appeared beneathMar. 29, 2025. It now appears beneath3 Months Ended Mar. 30, 2024, alongside the other filed duration/date headers and amounts.Empty HTML rows still count when a header rowspan crosses them. In Eaton's equity tables (
0001551182-25-000006and0001551182-26-000013), this keeps the filed value398.8beneathOrdinary shares Sharesand4beneathOrdinary shares Dollars. Column placement is also corrected into_dataframe()andtext()for some equity-statement tables. Amounts align with their logical equity components; the filed amounts themselves stay unchanged.Dwight’s maintainer review found changed Markdown bytes in 76 of 79 filings, which can change text stored in RAG indexes.
Fixes #1474
Verification
The original implementation comparison used base
a5a8fb3fand published revision34b5372f:Both full suites have 9 skips, 1 expected failure and no reruns. All 8,085 common base test identities retain their outcomes. The focused regression comparison uses the same 60 cases on the base and branch with sockets blocked and retries disabled. It covers full header paths, rowspans across empty rows, row-group boundaries, Rich rendering and public table edits.
Review follow-up. The new Georgia Power DataFrame assertion checks filed transaction origins and closing balances within their logical equity-component groups. The same 61 header cases produce 45 failures and 16 passes with the original base’s production code, and all 61 pass with this PR. Removing only the shared placement guard makes the new assertion fail. A fresh comparison passes 8,135 fast tests at published
34b5372fand 8,136 with this update; all 8,145 existing outcomes are identical, with nine skips, one expected failure and no reruns in both. Six gates and the changed test’s Ruff, formatting and Pyright checks pass. These checks ran before commit;4fe543b4contains their exact tested bytes, with production and fixtures unchanged.Original filing survey. Scanned 650 offline 10-K and 10-Q primary documents (50,892 HTML table elements), then compared all 8,406 parsed tables in 88 selected documents. All 4,871 zero/one-header tables render byte-identically. All non-empty Markdown body values are unchanged and remain in the same order. All 161 checked Eaton placements recover complete header paths with no misplaced labels. The first run identified the Eaton regression; the corrected rerun found no new regression against that version.
Filing comparison method and remaining limits
The inventory selected complex shapes and controls; it did not render all 650 documents. All tables in the 88 selected documents received table-level comparisons; 64 documents also received full
Document.to_markdown()comparisons. The rerun uses production bytes identical to this branch. Source-grid scoring covers 3,256 of 3,535 grouped tables: 76,858 of 77,413 matched placements (99.28%) have complete paths, versus 4,607 (5.95%) on base. The judge excludes 279 tables and skips 225 unmatched body rows. It reports 24 missing or misplaced labels in seven cases unchanged from the firstaf956560version, from 16,321 checks. Source-grid agreement does not independently validate financial interpretation; the INTU properties-table improvement was reviewed manually, with #1477 still present.Header detection can still promote cover-page or abstract content (#1477), stop before later header rows (#1485), or miss part of a grouped header (#1487). Multiline body cells (#1486) and flat-header escaping (#1483) are separate renderer limitations.
For that original implementation, cassette, regression provenance, offline, upgrade and changelog checks pass. Its regression files pass formatting, Ruff and Pyright; production code adds no new lint or type diagnostics. The same three production files already fail the baseline formatter check.
0000320193-25-000057. Eaton assertions use three byte-exact SEC table fragments and check ordinary-share counts and dollars. The Georgia Power assertion checks component spans and 2023 balances in the tracked combined Southern Company filing.tests/issues/regression/, and its docstring links Document.to_markdown() drops grouped duration and date labels and mislabels amount columns #1474.changelog.d/1474.fixed.mdrecords the wrong/right comparative net-sales header and accession, the Eaton rowspan correction, the equity-table column correction and the indexed-Markdown impact.Merge order: Independent; see the merge-order discussion.
🤖 Generated with Claude Code