Skip to content

feat: speak LaTeX formulas in English for VoiceOver and TalkBack - #979

Open
maksg wants to merge 2 commits into
mainfrom
rn-latex-voiceover
Open

maksg wants to merge 2 commits into
mainfrom
rn-latex-voiceover

Conversation

@maksg

@maksg maksg commented Oct 12, 2026

Copy link
Copy Markdown
Collaborator

What/Why?

Testing

PR Checklist

  • Code compiles and runs on iOS
  • Code compiles and runs on Android
  • Updated documentation/README if applicable
  • Ran example app to verify changes
  • E2E tests are passing
  • Required E2E tests have been added (if applicable)

@maksg
maksg requested a review from jnowakow October 26, 2026 10:52
maksg and others added 2 commits October 26, 2026 12:02
Port the iOS package's LaTeXSpeech parser to the React Native package so
VoiceOver reads formulas as words ("x squared over 2", "integral from 0
to 1 of ...") instead of raw LaTeX. The math.equation label gains a
{speech} placeholder and defaults to "Math: {speech}"; {latex} still
reads the source. Inline formulas, previously a silent object
replacement character inside the paragraph element, become their own
accessibility elements, discovered through a mathLatex attribute stamped
on the attachment run. Android substitutes {speech} with the source
until a Kotlin port lands.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Port the LaTeX speech parser to Kotlin so TalkBack reads formulas as
words, resolving the {speech} placeholder of the math.equation label
for $$ blocks and for inline formulas. Inline formulas, previously a
silent object replacement character inside the paragraph node, become
TalkBack nodes of their own through a SpokenSpan interface the math
span implements, which keeps the optional math source set out of main.
Headings are now paragraph context for the accessibility helper rather
than a span it segments around, so formulas, links and images inside a
heading get their own nodes as they already do on iOS.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@maksg
maksg force-pushed the rn-latex-voiceover branch from cfab412 to 9a91fd8 Compare October 26, 2026 13:28

@jnowakow jnowakow left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

During the review I found a few issues that are worth looking at:

  1. Some formulas sound identical when they mean different things. Argument.isSimple is just itemCount == 1, but one item can be a whole fraction, root or power, so nesting disappears from the speech. Actual outputs:
    • e^{\frac{x}{2}} and \frac{e^x}{2} both give "e to the power x over 2"
    • \sqrt{\frac{a}{b}} and \frac{\sqrt{a}}{b} both give "square root of a over b"

This affects ENRMLaTeXSpeech.swift, LatexSpeech.kt and the upstream enriched-markdown-ios LaTeXSpeech.swift.

  1. Math inside a link is read wrong on both platforms. md4c nests the math inside the link ([see $x^2$ here](u) gives Link > [Text, LatexMathInline, Text]).

    • iOS: segmentedElementsForParagraph sets segmentStart = NSMaxRange(mathRange), which moves backwards. VoiceOver reads the link "see  here", then "Math: x squared", then "here" again as plain text.
    • Android: if (span.start < segmentPos) continue skips the math span, so the formula is silent.
  2. Multi-digit numbers are read as one number, but LaTeX takes only one digit.

    • \frac12 gives "fraction 12 over end fraction" and the denominator is lost.
    • x^23 renders as x²3 but is spoken "x to the power 23".
    • x_12 gives "x sub 12".
  3. Common constructs are misread:

    • \sum\limits_{i=1}^n gives "sum of limits sub i equals 1 end sub to the power n". \int\limits has the same problem.
    • \operatorname*{argmax} gives "times argmax".
    • In pmatrix/cases, the environment names are read aloud and & and \\ are silent, so rows run together.
  4. The new speech tests break builds with math turned off:

    • Android: src/test/.../math/LatexSpeechTest.kt imports LatexSpeech, which only joins the source set when enableMath is on (build.gradle). With math off, the whole unit-test source set fails to compile.
    • iOS: ENRMLaTeXSpeechTests.mm would fail because {speech} stays as literal text.
    • CI never builds with math off, so this is currently hidden.

I would also consider implementing this functionality as a shared C++ code. Current solution works but any future change has to be applied in two places.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants