Repository navigation
Document re.Scanner #106338
Description
Documentation
re.Scanner is currently marked as "experimental" but was added to the standard library 24 years ago[0].
People seem to think this is an interesting feature[1][2] and a github code search for re.Scanner( gives >500 hits[3],
suggesting people have found practical uses for the class.
Proposal
I suggest we add a few lines to the docs describing what it is and perhaps a short docstring with an example.
Some previously collated discussions:
#91308 (comment)
[0].
Line 327 in dbefa88
[1]. https://news.ycombinator.com/item?id=36517749
[2]. https://www.reddit.com/r/ProgrammerTIL/comments/4tpt03/python_til_about_rescanner_which_is_useful_for/
[3]. https://github.com/search?q=re.Scanner%28+language%3APython&type=code
Activity
AlexWaygood commentedon Jul 2, 2023
@vstinner put together a list of places where this has previously been discussed here: #91308 (comment)
Cc. @serhiy-storchaka as well, for interest
rhettinger commentedon Jul 3, 2023
I believe that Scanner was intentionally left undocumented and only left in because someone might have found it and relied on it. The tool was deemed to be a dead end.
serhiy-storchaka commentedon Jul 3, 2023
One of problems is that it may conflict with the name of internal class
_sre.SRE_Scannerif we make it public like we did with_sre.SRE_Patternand_sre.SRE_Match(now exposed asre.Patternandre.Matchfor documentation purposes)._sre.SRE_Scanneris used in implementation ofre.Pattern.finditer(), but it also the type of the result of undocumentedre.Pattern.scanner(). If we decide to documentre.Pattern.scanner(), we will have a problem.encukou commentedon Sep 8, 2026
Any new code is better off starting with the example tokenizer in the docs, which doesn't include
re.Scanner's quirks (like theif callable()or skippingNone), but lets you add any quirks you need. It also adds line/column information to tokens, for better error messages.I think we should link these docs from re.Scanner source, and leave it at that.
fry69 commentedon Oct 19, 2026
FYI: That link got mangled (you need explicit
https://for outside links in Markdown, or it get prefixed with the current domain).Here is where it wants to point to -> https://docs.python.org/3/library/re.html#writing-a-tokenizer
zeyadrabie commentedon Nov 5, 2026
Hey. Can I be assigned to this? What I would do is what @encukou suggested: add a comment on
re.ScannerinLib/re/__init__.pythat links to https://docs.python.org/3/library/re.html#writing-a-tokenizer, and stop there, soScanneritself stays undocumented.encukou commentedon Nov 5, 2026
You can't be assigned (GitHub's UI doesn't allow it), but it looks like no one else is working on this, feel free to send the PR.
gh-106338: Link the tokenizer docs from the re.Scanner source (GH-161595