fix: keep a section's own heading when its TOC anchor is nested inside it (#1369)

[marianogarciamelo]

What this changes

On filings the TOC anchor is nested inside the heading it marks (<p><b><a name="item2"></a>ITEM 2. PROPERTIES</b></p>), so Section.markdown() returned each TOC-detected section without its own heading. Collection now starts at the outermost block that begins with the start anchor, so the heading is kept. Filings whose anchors stand on their own are unchanged.

Fixes #1369

Verification

• Regression test for a bug is in tests/issues/regression/, and its docstring links the issue or bead
• Changelog: a changelog.d/<id>.<section>.md fragment, not an edit to CHANGELOG.md

The regression test uses inline HTML and runs offline, so I have not ticked the ground-truth box. I checked the real filing by hand instead; see below.

6.0

• This PR adds a deprecation, a 6.0 warning, or a "removed in 6.0" note, and docs/upgrade/6.0.md says what users see now and what to do instead

Working context (optional)

I worked on this with an AI assistant (Claude).

Plan. Mirror _truncate_at on the start side, as the issue suggests. _start_block climbs from the start anchor while it is the first content of each enclosing element (no text and no element before it), stopping short of <body>, and returns the outermost such element. collect_range_elements uses it as start_element when the caller passed none, so the existing start-at-element path does the rest.

Evidence.

• Google Inc. FY2004 10-K (0001193125-05-065298), HTMLParser(ParserConfig(form="10-K")), first line of each TOC-detected section's markdown(): 0 of 18 start with their "ITEM N." heading on main, 18 of 18 on this branch. Item 15, split across two table cells, now returns both cells.
• New test fails on main and passes here. tests/documents/test_section_slicer.py and tests/issues/regression/test_issue_826.py pass: 13 passed.
• ruff check edgar/documents/utils/section_slicer.py and check_regression_provenance.py pass.

Ruled out. _start_block returns None, leaving collection unchanged, when the anchor is a sibling of its heading, when text precedes the anchor inside its block, and when another element precedes it.

Not included. The end side still leaves an empty <p><b></b></p> where the next heading was truncated, as the issue notes. Happy to handle that here or in a follow-up.

[dgunning]

Thanks, this is a careful PR and the Google FY2004 result reproduces exactly (0/18 → 18/18, Item 15 gets both cells). Two things before it can merge:

1. The one CI failure is the fix working. test_issue_1345_shared_page_anchor.py::test_prospectus_trailing_section_markdown_stops_at_financial_statements now gets 1982 instead of 1934; the extra 48 chars are the section's own **Where You Can Find Additional Information ** heading. Please re-pin it with a comment saying so.

2. The climb reaches modern filings too. I compared Section.markdown() for every section in tests/fixtures/html (82 filings, 1,457 sections) on main vs this branch: 53 sections only gained their heading, as intended, but 34 sections in 12 modern filings changed in other ways (MSFT 10-K/10-Q, ORCL, TALO, HUBS, GS, MS, SO, ABNB). For example:

• MSFT 10-K Item 6: **ITEM 6. [R** / **ESERVED]** / PART II on separate lines → **ITEM 6\. \[R** **ESERVED\]** PART II on one
• a 2010 20-F (CIK 1464165): bold **Item 4.** / **Information on the Company** → plain Item 4. Information on the Company
• page-footer and running-header lines disappearing from the middle of GS Item 1A and MS Item 5

In these layouts the anchor is the first child of a wrapper that also holds the following content, so _start_block climbs to that wrapper, and the whole block is cloned as one top-level element that the markdown renderer lays out differently. Could you narrow the climb (e.g. only through inline wrappers like b/font/span/a/u/i plus the one block that holds the heading; Item 15 still needs to reach the <table>) and include a before/after over tests/fixtures/html where every changed section is a pure heading prefix?

Minor, optional: on that ABNB section markdown() now includes the heading while text() (1,910 chars both before and after) does not, so it may be worth deciding whether the two should agree.

The trailing empty <p><b></b></p> on the end side is fine as a follow-up.

[marianogarciamelo]

Thanks for the careful review and for running the full fixture comparison. Understood on both points. I'll re-pin the ABNB test with a comment, narrow the climb to inline wrappers plus the one block that holds the heading (keeping the <table> case for Item 15), and include a before/after over tests/fixtures/html showing every changed section is a pure heading prefix.

[marianogarciamelo]

Thanks for the review. Both points are addressed in the latest commits, and I've merged current main.

1. ABNB test re-pinned. test_prospectus_trailing_section_markdown_stops_at_financial_statements now expects 1982, with a comment and an assertion that the markdown starts with **Where You Can Find Additional Information **.

2. Climb narrowed. _start_block now:

• returns None unless there is loose text directly inside the anchor or right after it, since that is the only text collection drops;
• climbs only through inline wrappers (a, b, font, i, span, u, strong, em, ...) and stops at the first block;
• climbs through table cells and rows to their <table>, so Item 15 keeps both cells.

Before/after over tests/fixtures/html (Section.markdown() for every section, main vs this branch; 82 filings, 1,457 sections):

OTHER abnb/424b4/abnb-424b4-2020-12-09.html tax_considerations
   main  : 'Non\\-U\\.S\\.\n\n Holders \n\nThe following discussion is a summary of the m'
   branch: '**Material U\\.S\\. Federal Income Tax Consequences to  Non\\-U\\.S\\.  Hol'
OTHER abnb/s1/abnb-s1-2020-11-16.html tax_considerations
   main  : 'Non\\-U\\.S\\.\n\n Holders \n\nThe following discussion is a summary of the m'
   branch: '**Material U\\.S\\. Federal Income Tax Consequences to  Non\\-U\\.S\\.  Hol'

unchanged: 1402   gained a heading prefix only: 53   changed otherwise: 2

The two "changed otherwise" are ABNB's tax_considerations. On main that section starts with Non-U.S. Holders: the first part of the heading, "Material U.S. Federal Income Tax Consequences to", is loose text after the anchor and was dropped. It now starts with the full heading. It isn't a pure prefix because the old half-heading is absorbed into the restored one. Happy to handle it differently if you'd prefer.

Google FY2004 10-K still gives 18 of 18, and Item 15 is | | ITEM 15. EXHIBITS | AND FINANCIAL STATEMENT SCHEDULES |.

Tests. Added three cases to the regression file: a <font>-wrapped heading, a heading split across table cells, and a modern layout where the heading is already in its own tags and must be left alone. hatch run test-fast: 7939 passed.

This isn't limited to pre-2010 filings (ABNB's 2020 filings have the same layout), so I've reworded the changelog fragment. On text() vs markdown() for the ABNB section: I've left text() alone here and can look at it as a follow-up with the trailing empty <p><b></b></p>.