Skip to content

fix(mapper): use the Lexbor parser in clean_html_mapper - #1085

Merged
cmgzn merged 1 commit into
datajuicer:mainfrom
cristianchiriac:fix/clean-html-selectolax-lexbor
Oct 8, 2026
Merged

cmgzn merged 1 commit into
datajuicer:mainfrom
cristianchiriac:fix/clean-html-selectolax-lexbor

Conversation

@cristianchiriac

Copy link
Copy Markdown
Contributor

selectolax 1.0.0 (released 2026-10-03) removed the Modest backend, so import selectolax.parser now raises ImportError. clean_html_mapper calls selectolax.parser.HTMLParser through LazyLoader, which turns that into:

AttributeError: module 'selectolax' has no attribute 'parser'

pyproject.toml lists selectolax without a version bound, so a fresh pip install gets 1.0 and the operator fails on every sample. uv.lock still pins 0.3.30, which is why uv sync environments and CI haven't hit it yet.

This switches the operator to selectolax.lexbor.LexborHTMLParser. It has been available since selectolax 0.3, and LazyLoader resolves the lexbor submodule the same way it did parser.

Testing (pytest tests/ops/mapper/test_clean_html_mapper.py, Python 3.12):

selectolax before after
1.0.0 all tests fail with the AttributeError above 8 passed
0.3.30 (uv.lock) 8 passed 8 passed

I also compared the Modest and Lexbor output directly on all 26 text/target samples in that test file. Both backends return exactly the expected text for every sample, so the cleaned output doesn't change.

I didn't run the rest of the suite. On Windows the editable install needs MSVC, so I ran this test file against the dependencies from pyproject.toml with PYTHONPATH=..

selectolax 1.0 (2026-10-03) removed the Modest backend, so importing
selectolax.parser raises ImportError, which LazyLoader reports as
"module 'selectolax' has no attribute 'parser'". pyproject.toml allows any
selectolax version, so a fresh pip install breaks clean_html_mapper. Use
selectolax.lexbor.LexborHTMLParser, available since selectolax 0.3, which
produces the same text for all test cases.

@cmgzn cmgzn left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@cmgzn
cmgzn merged commit a567bc6 into datajuicer:main Oct 8, 2026
5 checks passed

This branch was previously deployed

1 inactive deployment
Testing — 5b7f7771 Deployed Oct 6, 2026 by cristianchiriac via unittest-single #1780
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants