TOC: a keyword-matched row can claim an item key ahead of the explicit "Item N." row #1344

Hello @dgunning - back again with a few parser findings. This is one of five edge cases being reported tonight. The edgartools-parse-defect-sweep gist from last time is updated for reference.

I haven't started on any of these yet, but if/when I get the chance to spend some cycles on one I'll comment on it to let you know.

Details below (AI Generated)

dgunning/edgartools. Verified on 5.58.0 (Python 3.12).*

Summary

TOCAnalyzer resolves a TOC row to an item either from an explicit "Item N"
label or, when the row carries only a title, from the form's keyword vocabulary
(FormSchema.match_text, the fallback added for GH #837). Rows are consumed in
document order and the first row to claim a section key keeps it, so a
sub-heading row that merely matches an item's keywords takes the key before
the filing's own "Item 1A. Risk Factors" row is reached. Two title families do
this on real filings:

Both consequences are silent, and the section the user asked for is returned at
confidence 0.95:

• the stolen item returns the summary — a few thousand characters far too
  early in the document;
• the preceding item's end boundary moves to that early anchor, so it
  over-captures. That is the visible symptom.

In our corpus this one mechanism accounts for 11 of the 23 sections that still
mis-parse across 7 filers, and it reproduces on the generic, Workiva and
Donnelley paths — it is in the shared text→item fallback, not in one agent
parser.

Reproduction 1 — Hertz 10-K, Item 1A stolen by a cautionary-note row

from edgar import find, set_identity
set_identity("Your Name your@email")
obj = find("0001657853-25-000015").obj()      # HERTZ GLOBAL HOLDINGS 10-K, FY2024

len(obj["Item 1"])    # 159,382  — ends deep inside Item 1A
len(obj["Item 1A"])   #   8,524  — the "summary of risk factors", not Item 1A
obj["Item 1"].find("ITEM\xa01A. RISK FACTORS")   # 59,307 (37% into Item 1)
obj["Item 1A"][:60]
# 'Table of Contents\n\nHERTZ GLOBAL HOLDINGS,\xa0INC. AND SUBSIDIARIES\n\nTHE H'

The TOC rows, in the order the parser consumes them (link index, text, target):

   3  part_i_item_1a   yes 'CAUTIONARY NOTE REGARDING FORWARD-LOOKING STATEM…' -> #…_16   <- claims the key
   5  part_i_item_1    yes 'ITEM 1.'                                          -> #…_22   <- claims the key
   7  part_i_item_1a   yes 'ITEM 1A.'                                         -> #…_64   DROPPED (key already claimed by #3)
   8  part_i_item_1a   yes label='ITEM 1A' 'RISK FACTORS'                     -> #…_64   DROPPED (key already claimed by #3)

Row #3's full text is 'CAUTIONARY NOTE REGARDING FORWARD-LOOKING STATEMENTS AND SUMMARY OF RISK FACTORS', and:

from edgar.documents.utils.toc_analyzer import TOCAnalyzer
from lxml import html as lxml_html

html = find("0001657853-25-000015").html()
tree = lxml_html.fromstring(html.encode("utf-8"))
an = TOCAnalyzer(form="10-K")

an._parse_item_from_text(text)      # 'Item 1A'
an.schema.match_text(text.lower())  # 'Item 1A'

So part_i_item_1a is claimed by a row that precedes the "PART I" row, while
the filing's own two Item 1A rows (#7 label, #8 title, both pointing at the
correct …_64) are dropped as duplicates.

Why the existing duplicate arbitration does not save it

_build_section_mapping already arbitrates duplicates: if the new anchor's
target carries the item's heading and the existing one does not, it replaces.
On this filing both checks return False, so the branch never fires:

an._anchor_matches_heading(tree, "id…_16", "Item 1A")   # False  (the thief)
an._anchor_matches_heading(tree, "id…_64", "Item 1A")   # False  (the real anchor)

_anchor_matches_heading looks at following::*[…][position() <= 3], and after
either anchor those three elements are the running page header:

['Table of Contents', 'HERTZ GLOBAL HOLDINGS, INC. AND SUBSIDIARIES',
 'THE HERTZ CORPORATION AND SUBSIDIARIES']

The item heading is further down the page, so the lookahead window can't see it
and document order decides. Widening that window would be one fix; ranking the
rows would be another (below).

Reproduction 2 — PPL 10-K, Item 8 claimed by the notes index

obj = find("0000922224-24-000008").obj()      # PPL Corp 10-K, FY2023

len(obj["Item 7A"])   #  58,912 — runs past "ITEM 8." at 46%
len(obj["Item 8"])    # 297,733 — starts at the notes index, not at Item 8
obj["Item 8"][:60]
# 'Table of Contents\n\nCOMBINED NOTES TO FINANCIAL STATEMENTS\n\nIndex to Co'

Here only one row claims part_ii_item_8 — the TOC's own
'Index to Combined Notes to Consolidated Financial Statements' row, pointing
at the notes index rather than at the "ITEM 8." heading:

 215  part_ii_item_8   yes 'Index to Combined Notes to Consolidated Financia…' -> #…_190   <- claims the key

PPL's genuine Item 8 row carries no link at all, so there is no explicit row
to prefer: on this flavour only the vocabulary can help. Two checks we ran that
don't discriminate, in case they look tempting: the claiming row sits inside
the main TOC region (element order 16726, alongside the other TOC rows at
15900–17183), and the resulting anchors are still monotonic in document order,
so neither "is it in the TOC table" nor an ordering sanity check flags it.

Scope

Seven filers, three TOC paths, all verified live on 5.58.0:

A signal already exists in the output, on the wrong section: the robbed item
lands with confidence=0.5 and the warning "Part I Item 1A content is 8,103
chars, below the expected minimum of 15,978 … extraction likely truncated". The
item that is actually damaged is the one before it.

Fix directions

1. Rank the rows instead of letting document order decide. Within one TOC,
   a key claimed by an explicit "Item N" label should outrank a keyword-only
   match for the same key, whatever their order. That is a two-pass resolution
   (collect candidates per key, then prefer a labelled one) and needs no new
   vocabulary. It fixes the five Item 1A filers above. It does not fix PPL/ET,
   whose explicit row is unlinked.
2. Exclusions for the two title families. Add "summary" to the Item 1A
   rule's excluded and "index" to Item 8's. That fixes all seven filers with
   a two-line change. The honest risk: a filer whose only Item 8 row reads
   "Index to Financial Statements" would lose the key and fall back to the body
   scan — we have no such filing in our corpus, but the exclusion is a blunter
   instrument than the ranking.
3. Optionally, widen _anchor_matches_heading's three-element lookahead (a
   running page header eats the whole window on any filing with one), which
   would let the existing arbitration resolve flavour 1 on its own.

(1) and (3) are the structural half; (2) is the one that also covers PPL. The
keyword fallback itself must stay — it is what maps Workiva's bare "Business"
and "Risk Factors" rows (GH #837), and on PPL those same keyword rows map items
1 and 1A correctly.

Environment

edgartools 5.58.0, Python 3.12, all filings fetched live from EDGAR.