Skip to content

fix(markdown): preserve grouped table header paths (#1474) - #1480

Merged
dgunning merged 8 commits into
dgunning:mainfrom
synfonia-llc:codex/1474-preserve-markdown-header-paths
Oct 8, 2026
Merged

dgunning merged 8 commits into
dgunning:mainfrom
synfonia-llc:codex/1474-preserve-markdown-header-paths

Conversation

@synfonia-llc

@synfonia-llc synfonia-llc commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

What this changes

Document.to_markdown() now keeps each grouped table column's complete header path, from group to leaf. Tables with zero or one header row retain their previous Markdown bytes. In Markdown, non-empty body values and their order are preserved.

Upgrading: Grouped-table Markdown includes detected group, duration and date headers, and to_dataframe() and text() correct some equity-component column placements; filed amounts stay unchanged.

In Apple’s Q2 FY2025 10-Q (0000320193-25-000057), comparative net sales of $90,753 million appeared beneath Mar. 29, 2025. It now appears beneath 3 Months Ended Mar. 30, 2024, alongside the other filed duration/date headers and amounts.

Empty HTML rows still count when a header rowspan crosses them. In Eaton's equity tables (0001551182-25-000006 and 0001551182-26-000013), this keeps the filed value 398.8 beneath Ordinary shares Shares and 4 beneath Ordinary shares Dollars. Column placement is also corrected in to_dataframe() and text() for some equity-statement tables. Amounts align with their logical equity components; the filed amounts themselves stay unchanged.

Dwight’s maintainer review found changed Markdown bytes in 76 of 79 filings, which can change text stored in RAG indexes.

Fixes #1474

Verification

The original implementation comparison used base a5a8fb3f and published revision 34b5372f:

Check Original base Published revision
Fast suite 8,075 passed 8,135 passed
Header regressions 44 failed, 16 passed 60 passed

Both full suites have 9 skips, 1 expected failure and no reruns. All 8,085 common base test identities retain their outcomes. The focused regression comparison uses the same 60 cases on the base and branch with sockets blocked and retries disabled. It covers full header paths, rowspans across empty rows, row-group boundaries, Rich rendering and public table edits.

Review follow-up. The new Georgia Power DataFrame assertion checks filed transaction origins and closing balances within their logical equity-component groups. The same 61 header cases produce 45 failures and 16 passes with the original base’s production code, and all 61 pass with this PR. Removing only the shared placement guard makes the new assertion fail. A fresh comparison passes 8,135 fast tests at published 34b5372f and 8,136 with this update; all 8,145 existing outcomes are identical, with nine skips, one expected failure and no reruns in both. Six gates and the changed test’s Ruff, formatting and Pyright checks pass. These checks ran before commit; 4fe543b4 contains their exact tested bytes, with production and fixtures unchanged.

Original filing survey. Scanned 650 offline 10-K and 10-Q primary documents (50,892 HTML table elements), then compared all 8,406 parsed tables in 88 selected documents. All 4,871 zero/one-header tables render byte-identically. All non-empty Markdown body values are unchanged and remain in the same order. All 161 checked Eaton placements recover complete header paths with no misplaced labels. The first run identified the Eaton regression; the corrected rerun found no new regression against that version.

Filing comparison method and remaining limits

The inventory selected complex shapes and controls; it did not render all 650 documents. All tables in the 88 selected documents received table-level comparisons; 64 documents also received full Document.to_markdown() comparisons. The rerun uses production bytes identical to this branch. Source-grid scoring covers 3,256 of 3,535 grouped tables: 76,858 of 77,413 matched placements (99.28%) have complete paths, versus 4,607 (5.95%) on base. The judge excludes 279 tables and skips 225 unmatched body rows. It reports 24 missing or misplaced labels in seven cases unchanged from the first af956560 version, from 16,321 checks. Source-grid agreement does not independently validate financial interpretation; the INTU properties-table improvement was reviewed manually, with #1477 still present.

Header detection can still promote cover-page or abstract content (#1477), stop before later header rows (#1485), or miss part of a grouped header (#1487). Multiline body cells (#1486) and flat-header escaping (#1483) are separate renderer limitations.

For that original implementation, cassette, regression provenance, offline, upgrade and changelog checks pass. Its regression files pass formatting, Ruff and Pyright; production code adds no new lint or type diagnostics. The same three production files already fail the baseline formatter check.

  • Ground truth: Apple R2/R3 assertions compare filed duration/date headers and amounts; R6 checks equity-component labels and values. Filing accession: 0000320193-25-000057. Eaton assertions use three byte-exact SEC table fragments and check ordinary-share counts and dollars. The Georgia Power assertion checks component spans and 2023 balances in the tracked combined Southern Company filing.
  • Regression test for a bug is in tests/issues/regression/, and its docstring links Document.to_markdown() drops grouped duration and date labels and mislabels amount columns #1474.
  • Changelog: changelog.d/1474.fixed.md records the wrong/right comparative net-sales header and accession, the Eaton rowspan correction, the equity-table column correction and the indexed-Markdown impact.

Merge order: Independent; see the merge-order discussion.

🤖 Generated with Claude Code

Retain each document table column's group and leaf labels using its header
rowspan/colspan positions, including legal zero row spans. Escape header pipe
delimiters and literal backslashes without changing original table cells.

Exclude actual header rows from the existing numeric data-alignment heuristic
while preserving its body behavior. Cover the filed Apple duration/date labels,
equity components, Rich call order, and synthetic header boundaries.

Fixes dgunning#1474
Assert that a body wider than its headers retains every amount and an empty
header, and that a narrower body keeps its header columns and empty padding.

The wider-body case fails when the header-matrix bounds guard is removed.
Production renderer and changelog content remain unchanged.

Refs dgunning#1474
Use explicit non-None checks to narrow parsed tables and expanded matrix
cells in the grouped Markdown header regression. The 31 test identities,
parameters and expected values remain unchanged.

Refs dgunning#1474
Apply grouped header paths only to tables with at least two header rows. Keep flat and headerless table output compatible and add span and whitespace controls.

Record the existing Apple equity header-classification limitation in exact regression assertions. (GH dgunning#1474)
@arhancanli

Copy link
Copy Markdown

Read the diff, did not run the suite. The Apple 10-Q case ($90,753 million under the wrong date) is a real misattribution and projecting rowspan=0 to the end of the header group is the right fix. Notes on the code:

  1. _combine_multi_row_headers now joins every non-empty level with a space. The old code collapsed to the date when one was present. Two things to check: (a) when the same text sits at two levels of one column (a vertically spanned cell that the matrix repeats, or a group and leaf both reading "2024"), the result is "2024 2024". entry.row_origin == row_index guards the vertical case in _expand_table_structure, but _combine_multi_row_headers takes plain strings, so a consecutive-duplicate drop there would be cheap insurance. (b) the 3-level case (period group, then "Year Ended", then the date) now yields "Three Months Ended Year Ended ..."-style labels if a filing stacks them; a test on one 3-row header fixture would show whether that reads well.
  2. The \\ and | escaping is applied only when len(node.headers) >= 2, so a one-row header containing | still breaks the pipe table while the same text in a two-row header is escaped. That keeps the byte-identical promise for flat tables, but the inconsistency deserves a comment, or an escape in both branches if the flat-table bytes only change for headers that were already broken.
  3. row_idx >= self.header_row_count in _place_cells depends on header_row_count being set before placement; it is (set from len(header_rows) at the start of build_from_rows), but TableMatrix().build_from_rows(header_rows, []) here means the data-row branch of that heuristic is never exercised from this renderer. Worth a one-line comment so nobody reorders it later.

Add matching-year and three-level period/duration/date cases. Clarify span identity, flat-header escaping compatibility and the header-only matrix boundary without changing rendering behavior. (GH dgunning#1474)
Keep the original header-row geometry when table parsing separates spacer
rows from detected headers. Bound row spans to their source row groups and
leave parsed cell spans, body data, and flat-table rendering unchanged.

Add exact Eaton equity-table fixtures and public Markdown regression cases
for missing rows, blank-origin spans, header replacement, and text rendering
before Markdown.
@synfonia-llc

synfonia-llc commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks for checking these details. I added two regression cases and comments covering the boundaries you identified.

  1. A vertically spanned cell contributes its label only at its source row. Separate group and leaf cells retain both labels even when their text matches. Dropping consecutive equal strings would remove a real level from 2024 2024 while leaving both levels in the sibling 2024 2023, making header-path depth inconsistent across sibling columns. The added 2024/2024 case checks this explicitly. The three-level example uses current/prior year → quarterly/annual duration → date, so quarterly and annual columns sharing an end date keep distinct complete labels and correctly aligned amounts.
  2. Added a comment beside the escaping guard documenting the existing flat-header limitation. This preserves the flat-table compatibility described in the PR; the existing regression explicitly records that policy without treating it as correct escaping. The flat-header escaping problem is tracked in #1483.
  3. Added comments explaining that the Markdown path builds a header-only matrix and expands body rows separately, and that the header count must be set before placement. The direct matrix tests also exercise numeric body spans with zero through four header rows.

The final branch passes all 60 focused regression cases with sockets blocked and retries disabled. Its fresh fast suite passes with 8,135 passed, 9 skipped and 1 expected failure, with no failures or reruns. The fresh base passes 8,075 cases with the same skips and expected failure; all 8,085 common test outcomes are unchanged. On the same 60-case regression comparison, the base has 44 failures and 16 passes.

The Apple R2/R3 filing tests remain the real-world check: the $90,753 million comparative net-sales amount stays under 3 Months Ended Mar. 30, 2024.

The synthetic examples below were captured at the published 5077fc0 review follow-up; their tests also pass on final 34b5372f. Each test parses the Markdown back into a table and checks every header and body value.

A spanning label appears once

Test: original-business-revenue.

Input (synthetic HTML):

<table>
<tr><th rowspan="2">Business</th><th colspan="2">Revenue</th></tr>
<tr><th>2026</th><th>2025</th></tr>
<tr><td>Cloud</td><td>10</td><td>8</td></tr>
<tr><td>Other</td><td>3</td><td>4</td></tr>
</table>

Rendered Markdown:

Business Revenue 2026 Revenue 2025
Cloud 10 8
Other 3 4
Separate cells with the same text keep both levels

Test: distinct-matching-year-labels.

Input (synthetic HTML):

<table>
<tr><th rowspan="2">Business</th><th colspan="2">2024</th></tr>
<tr><th>2024</th><th>2023</th></tr>
<tr><td>Cloud</td><td>10</td><td>8</td></tr>
</table>

Rendered Markdown:

Business 2024 2024 2024 2023
Cloud 10 8
Quarterly and annual columns keep their duration and date

Test: three-level-period-duration-date.

Input (synthetic HTML):

<table>
<tr><th rowspan="3">Metric</th><th colspan="2">Current year</th><th colspan="2">Prior year</th></tr>
<tr><th>Three Months Ended</th><th>Year Ended</th><th>Three Months Ended</th><th>Year Ended</th></tr>
<tr><th>Dec. 31, 2025</th><th>Dec. 31, 2025</th><th>Dec. 31, 2024</th><th>Dec. 31, 2024</th></tr>
<tr><td>Sales</td><td>10</td><td>40</td><td>8</td><td>30</td></tr>
</table>

Rendered Markdown:

Metric Current year Three Months Ended Dec. 31, 2025 Current year Year Ended Dec. 31, 2025 Prior year Three Months Ended Dec. 31, 2024 Prior year Year Ended Dec. 31, 2024
Sales 10 40 8 30

Tested beyond the fixtures. Scanned 650 offline 10-K and 10-Q primary documents (50,892 HTML <table> elements), then compared all 8,406 parsed tables in 88 selected documents on the base and correction:

  • 4,871 of 4,871 zero/one-header tables byte-identical. All non-empty body values are unchanged and remain in the same order; 105 grouped tables differ only in empty spacer placement. No base header text was lost in the 3,535 grouped tables.
  • 76,858 of 77,413 checked body-cell placements (99.28%) retain their complete source-grid header path, versus 4,607 (5.95%) on the base.
  • 24 missing or misplaced labels in 7 tables, of 16,321 labels checked, versus 8,803 in 3,060 tables on the base. The seven remaining cases are unchanged from the first af956560 version.

The first run caught one regression affecting three Eaton equity tables: an empty header row crossed by rowspans put Shares and Dollars over the wrong columns. The correction restores complete paths for all 161 checked Eaton placements, with zero misplaced labels. The rerun identified no new regression against that first version; its fourth changed table, INTU's properties table, was reviewed separately and still has the existing #1477 header-classification limitation.

Real filings

Comparison method and scope. Inventory scanning used HTML markup to select complex shapes and controls; it did not render all 650 documents. The retained-selection rerun compared a5a8fb3f with 511f94bb; the correction's production bytes are identical to final 34b5372f. All 88 documents received table-level comparisons with no parse or render error. Of these, 64 documents under 3 MB also received full Document.to_markdown() comparisons; the other 24 did not. No changed line in those 64 full outputs was outside a table.

The source check rebuilds colspan/rowspan grids, matches parsed tables by text, and matches body rows to source rows. Of 3,535 multi-header tables, 3,256 could be scored: 132 had no source-table signature match and 147 had no usable body-row match. Another 225 unmatched body rows were skipped within the scoring. The 77,413 placements are matched non-empty body cells, not independently verified financial facts. These percentages describe agreement with the source grid; they do not prove that the filer's grid, parser classifications or financial interpretation are correct. The judge's whole-blank-column check agrees in 104 of the 105 spacer cases; the SBUX exception contains a pre-existing multiline body cell (#1486).

Against the first af956560 version, 8,402 tables are byte-identical and four change: the three Eaton tables and INTU accession 0000896878-26-000037, zero-based table [62]. INTU is outside the automatic source match and was reviewed manually: its labels return to their own source columns and its body matches the base, while data rows still appear in its header (#1477). The two Microsoft rowspan controls, accession 0000950170-25-100235, [34]/[35], remain byte-identical in this corpus rerun.

The first four examples were captured from tracked fixtures through HTMLParser(ParserConfig(enable_parallel=False, max_workers=1)).parse(original_bytes) and public Document.to_markdown(); other parser options use their defaults. Before: a5a8fb3; after: 5077fc0. Those four table numbers are 1-based document.tables positions, and their rendered body rows are identical.

Flat header and 33-row balance sheet. Accession 0000320193-25-000057, R4.htm, table 1.

Before = after, including all 33 body rows:

Header-only excerpt; body rows omitted from display.

CONDENSED CONSOLIDATED BALANCE SHEETS (Unaudited) - USD ($) $ in Millions Mar. 29, 2025 Sep. 28, 2024
Exact captured Markdown
| CONDENSED CONSOLIDATED BALANCE SHEETS (Unaudited) - USD ($)   $ in Millions | Mar. 29, 2025 | Sep. 28, 2024 |
| --- | --- | --- |

Duration/date groups and row-spanning stub. Accession 0000320193-25-000057, R2.htm, table 1.

Before:

Header-only excerpt; body rows omitted from display.

CONDENSED CONSOLIDATED STATEMENTS OF OPERATIONS (Unaudited) - USD ($) shares in Thousands, $ in Millions 3 Months Ended Mar. 29, 2025 6 Months Ended
Exact captured Markdown
| CONDENSED CONSOLIDATED STATEMENTS OF OPERATIONS (Unaudited) - USD ($)   shares in Thousands, $ in Millions | 3 Months Ended | Mar. 29, 2025 | 6 Months Ended |  |
| --- | --- | --- | --- | --- |

After:

Header-only excerpt; body rows omitted from display.

CONDENSED CONSOLIDATED STATEMENTS OF OPERATIONS (Unaudited) - USD ($) shares in Thousands, $ in Millions 3 Months Ended Mar. 29, 2025 3 Months Ended Mar. 30, 2024 6 Months Ended Mar. 29, 2025 6 Months Ended Mar. 30, 2024
Exact captured Markdown
| CONDENSED CONSOLIDATED STATEMENTS OF OPERATIONS (Unaudited) - USD ($) shares in Thousands, $ in Millions | 3 Months Ended Mar. 29, 2025 | 3 Months Ended Mar. 30, 2024 | 6 Months Ended Mar. 29, 2025 | 6 Months Ended Mar. 30, 2024 |
| --- | --- | --- | --- | --- |

The statement title spans two header rows. Each duration now stays with both filed dates; the 24 body rows are identical.

Three header rows and a wide table. Accession 0000950170-24-087843, msft-20240630.htm, table 38.

Before:

Header-only excerpt; body rows omitted from display.

June 30, 2024 Less than 12 Months Unrealized Losses 12 Months or Greater Unrealized Losses Total Fair Value Total Unrealized Losses
Exact captured Markdown
| June 30, 2024 | Less than 12 Months |  |  | Unrealized   Losses |  | 12 Months or Greater |  | Unrealized   Losses |  |  | Total   Fair Value |  | Total   Unrealized   Losses |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |

After:

Header-only excerpt; body rows omitted from display.

(In millions) June 30, 2024 Less than 12 Months Fair Value Less than 12 Months Fair Value Less than 12 Months Less than 12 Months Unrealized Losses 12 Months or Greater Fair Value 12 Months or Greater Fair Value 12 Months or Greater Unrealized Losses 12 Months or Greater Unrealized Losses Total Fair Value Total Fair Value Total Unrealized Losses Total Unrealized Losses
Exact captured Markdown
| (In millions) June 30, 2024 | Less than 12 Months Fair Value | Less than 12 Months Fair Value | Less than 12 Months | Less than 12 Months Unrealized Losses |  | 12 Months or Greater Fair Value | 12 Months or Greater Fair Value | 12 Months or Greater Unrealized Losses | 12 Months or Greater Unrealized Losses |  | Total Fair Value | Total Fair Value | Total Unrealized Losses | Total Unrealized Losses |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |

This table has 16 rendered columns and six body rows. Its third header row supplies the date in the stub; the numeric columns have two non-empty label levels, rather than three. All body rows are identical.

Known R1 limitation (#1477). Accession 0000320193-25-000057, R1.htm, table 1.

Before:

Header-only excerpt; body rows omitted from display.

Cover Page - shares shares in Thousands 6 Months Ended
Exact captured Markdown
| Cover Page - shares   shares in Thousands | 6 Months Ended |  |
| --- | --- | --- |

After:

Header-only excerpt; body rows omitted from display.

Cover Page - shares shares in Thousands Entity Information [Line Items] Document Type Document Quarterly Report Document Period End Date 6 Months Ended Mar. 29, 2025 10-Q true Mar. 29, 2025 Apr. 18, 2025
Exact captured Markdown
| Cover Page - shares shares in Thousands Entity Information [Line Items] Document Type Document Quarterly Report Document Period End Date | 6 Months Ended Mar. 29, 2025 10-Q true Mar. 29, 2025 | Apr. 18, 2025 |
| --- | --- | --- |

The parser still incorrectly promotes Document Type, 10-Q, true, and the period-end date into six header rows, tracked in #1477. Flattening changes the rendered header above; it does not correct that classification. The remaining 68 body rows are identical.

Additional complex filings. A separate comparison captured the JPMorgan, Berkshire and GSBD tables through HTMLParser(ParserConfig()) and MarkdownRenderer()._render_table_pipe, comparing a5a8fb3f with af956560. The published 5077fc0 review follow-up shared that production code; these captures precede the Eaton correction. These observations concern the selected captured tables; these indices are zero-based. The comparisons below show selected header cells; “blank” means an empty header, not a missing amount.

JPMorgan deposits: balances, rates and periods. Q2 2026 10-Q, accession 0001628280-26-054343, document.tables[78]. All 16 columns after the stub now retain their balance/rate and period labels. Selected cells in the Demand(a) row:

Captured value Before header After header
342,568 (balance, millions) June 30, 2025 Average balances Three months ended June 30, 2026
321,907 (balance, millions) blank Average balances Three months ended June 30, 2025
3.37% (rate) blank Average interest rates Six months ended June 30, 2025

Berkshire Note 24: period and insurance segment. Q2 2026 10-Q, accession 0001193125-26-341032, document.tables[52]. Selected amount cells in Premiums earned and investment income:

Captured value Before header After header
11,291 (GEICO) blank Second Quarter 2026 GEICO
26,176 (total) blank Second Quarter 2026 Total

The base lost GEICO; the updated header keeps it with the table's own period. Body rows are identical in this table.

American Airlines: quarter versus year-to-date. Accession 0000006201-25-000038, zero-based table [91]. Exact header lines reported by that historical source-grid review:

Before:

Header-only excerpt; body rows omitted from display. The retained capture contains only the header line; the separator below is added for this preview.

2025 2024 2025 2024 Increase (Decrease)
Exact captured Markdown
|  | 2025 | 2024 | 2025 | 2024 |  | Increase   (Decrease) |

After:

Header-only excerpt; body rows omitted from display. The retained capture contains only the header line; the separator below is added for this preview.

Three Months Ended June 30, 2025 Three Months Ended June 30, 2024 Increase (Decrease) Six Months Ended June 30, 2025 Six Months Ended June 30, 2024 Increase (Decrease)
Exact captured Markdown
|  | Three Months Ended June 30, 2025 | Three Months Ended June 30, 2024 | Increase (Decrease) | Six Months Ended June 30, 2025 | Six Months Ended June 30, 2024 | Increase (Decrease) |

Intel: fair-value hierarchy. Accession 0000050863-25-000009, zero-based table [208]. The base lost Level 1. The reported updated labels include December 28, 2024 Fair Value Measurements Level 1 and December 30, 2023 Fair Value Measurements Level 1, retaining both date and hierarchy level.

Eaton correction checked through the public API. Accession 0001551182-25-000006, [28], and 0001551182-26-000013, [22]/[23]. The original comparison found that an empty header row had been removed while rowspans crossing it remained unchanged, pushing Shares and Dollars onto blank columns. A separate raw-document replay through public Document.to_markdown() on the corrected branch now preserves those physical rows: each header has 18 columns rather than 20. In the filed Balance at January 1, 2022 row, 398.8 is under Ordinary shares Shares and $4 is under Ordinary shares Dollars; the quarterly 387.9 / 392.9 share counts and corresponding $4 are aligned too. Public table dictionaries, DataFrames and plain text match the parent, and Markdown after Rich rendering matches Markdown before it. The retained-corpus source check independently finds 100/100, 28/28 and 33/33 complete matched placements, with 0 misplaced labels in all three; their bodies match the base bytes. The quarterly tables also have unmatched body rows, so those path scores cover the matched rows. Existing multiline body cells (#1486) still require separate care when counting parsed Markdown rows.

Limits shown by those captures:

  • JPMorgan's document.tables[44], [97], [98], [206] and [207] have zero detected header rows. Their actual headers remain in the body; this renderer change cannot repair header detection. The premature stopping is tracked in #1485.
  • JPMorgan's FTE reconciliation, [24], has group rows spanning 42 grid columns against 45 for the leaf row/body. The 8,113 amount still has Three months ended June 30, Managed basis, missing 2026; 1,817 has only Managed basis, missing its 2025 period. This pre-existing alignment problem is improved but unresolved. Source-coverage diagnostics are requested in #1487, without guessing missing years.
  • Berkshire's repurchase table, [83], promotes April's data and May's label into headers, the classification problem tracked in #1477. The base drops those labels; the updated header reads Period April May. Neither matches the filed layout.
  • GSBD's FY2025 10-K, accession 0001193125-26-077458, has the same classification issue in [81]: As of December 31, 2025 is promoted into a second header row. The update restores Level 3 Instruments but joins it to that section text. All 36 body lines are identical.
  • The source-grid review also identified header/body offsets in TMUS accession 0001283699-25-000012, [23]/[25], and SBUX accumulated-other-comprehensive-income tables, including body-rowspan complications. Preserving header spans does not establish that their labels align with the intended body amounts. Existing multiline body cells can also split Markdown rows, as in the SBUX scan case; that separate limitation is tracked in #1486. Where body, cover-page or signature rows become headers, flattening also retains that text in long paths; the classification remains a #1477 concern.
  • Empty spacer columns collapse: JPMorgan [78] goes from 20 to 17 columns, [196] from 29 to 28, and Berkshire's BNSF table [75] loses one all-empty column. These comparisons found every non-empty body value unchanged and in the same order; this does not imply identical body lines where spacers changed.

@dgunning dgunning left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. The fix holds up well. Its 60 new tests pass, they fail on main (the Apple R2 case first), and the 712 offline document and Markdown tests pass on the branch. To check for side effects, I compared output on main and on this branch across the 79 real filings under tests/fixtures/html/ (12,933 tables). In the Markdown, no body value moves or changes order, and the restored header paths are right where I checked them, for example CAT's "Level 1" fair-value column and CVX's "Upstream U.S." segment columns.

One claim in the description doesn't match what I measured. to_dataframe() and text() do change. 9 DataFrames and 8 text() outputs differ, all in equity-statement tables:

  • adbe/10k/adbe-10-k-2025-01-13.html, table 33 (the 4,756 row)
  • so/10k/so-10-k-2026-02-19.html, tables 96 and 102 (e.g. 2,080, 2,297, (1,855))
  • gbdc/10k/gbdc-10-k-2024-11-19.html, tables 172, 173 and 223, and gbdc/10q/gbdc-10-q-2025-08-04.html, tables 125 and 126
  • c/10k/c-10-k-2025-02-21.html, table 252

The cause is the row_idx >= self.header_row_count change in table_matrix.py, which every TableMatrix user shares, not just the Markdown renderer. The new output is the correct one: on main these rows sit one column right of where every other row in the same table puts that column's values, and on this branch they line up. So it's an extra fix, but this ships in a 5.x minor release, so it should be stated rather than described as unchanged.

Could you:

  1. Add a sentence to changelog.d/1474.fixed.md and the PR description saying that to_dataframe() and text() column placement is also corrected in some equity-statement tables.
  2. Add one regression assertion pinning it, e.g. the Southern Company 2,080 / 2,297 rows landing in the same column as that table's other rows. The SO and ADBE filings are already tracked fixtures.

Two non-blocking notes for the record:

  • Markdown bytes change in 76 of the 79 filings. That's expected for this fix, but RAG users will see their indexed text change, so it's worth one clause in the changelog.
  • Where header detection promotes body rows (#1477, #1485), header cells now concatenate every promoted value. That produces 38 of 6,613 tables with a phrase repeated inside one cell (cover-page checkboxes, for example) and 192 cells over 120 characters. That's no worse than main, which kept only one value and dropped the rest, and it belongs to those issues, not this PR.

With 1 and 2 in, this is good to merge for 5.62.0. To make it, it needs to land by 10-11.

@synfonia-llc

Copy link
Copy Markdown
Contributor Author

Thanks, Dwight. Both changes are in 4fe543b4:

  1. Changelog and description. changelog.d/1474.fixed.md and the description now say that to_dataframe() and text() column placement is also corrected in some equity-statement tables. The changelog also notes your finding that Markdown changes in 76 of the 79 filings, which affects text stored in RAG indexes.
  2. Regression assertion. test_1480_markdown_header_rowspans.py adds a Georgia Power case on the tracked Southern Company 10-K. It finds each equity component's columns from the header path rather than from fixed indexes. It then pins the 2,080 and 2,297 rows and the 2023 closing balances. Removing only the row_idx >= self.header_row_count guard makes it fail at 2,297.

One correction to "line up". These transaction rows are rows that header detection promotes (#1477, #1485). On this branch they line up with the other promoted rows and with their component's header path. The closing balances in the body still sit one slot to the right within the same component, even though the filing starts 2,297 and the 2023 balance of 17,923 in the same column. The difference is that body rows still get the numeric-alignment shift and promoted rows no longer do. ADBE's 4,756 behaves the same way. The assertion pins today's placement, so when #1477/#1485 stop promoting these rows it will flag the move and should be updated with that fix. Filed amounts are unchanged throughout.

@dgunning
dgunning merged commit 2d3fd2a into dgunning:main Oct 8, 2026
11 checks passed
dgunning added a commit that referenced this pull request Oct 11, 2026
…het (#1508)

test-slow has failed on main since 2026-10-08: PR #1480 (GH #1474) kept
grouped table header paths, and eleven tracked filings stopped losing
numbers (133 in total), but they were not moved from BASELINE_NUMBER_LOSS
to CLEAN_FILINGS in that commit, so test_a_repaired_filing_is_banked fails.

Moved: 20-F 0001062993-16-008650 (42), aapl/10q (10), pg/10q (24),
tsla/10q (4), abbv/10k (4), hd/10k (32), ma/10k (2), nke/10k (7),
pfe/10k (3), rf/10k (4), xom/10k (1). Measured on the full local corpus
(text_boundary_corpus included), the same eleven as CI's 61-fixture view;
the ratchet's 7 tests pass after the move.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
dgunning added a commit that referenced this pull request Oct 11, 2026
…het (#1508) (#1510)

test-slow has failed on main since 2026-10-08: PR #1480 (GH #1474) kept
grouped table header paths, and eleven tracked filings stopped losing
numbers (133 in total), but they were not moved from BASELINE_NUMBER_LOSS
to CLEAN_FILINGS in that commit, so test_a_repaired_filing_is_banked fails.

Moved: 20-F 0001062993-16-008650 (42), aapl/10q (10), pg/10q (24),
tsla/10q (4), abbv/10k (4), hd/10k (32), ma/10k (2), nke/10k (7),
pfe/10k (3), rf/10k (4), xom/10k (1). Measured on the full local corpus
(text_boundary_corpus included), the same eleven as CI's 61-fixture view;
the ratchet's 7 tests pass after the move.


(cherry picked from commit 2250052)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Document.to_markdown() drops grouped duration and date labels and mislabels amount columns

3 participants