Skip to content

Fix position-based highlighting when a text contains ampersand and other reserved HTML entities - #20595

Merged
mhkuu merged 30 commits into
feature/html-parserfrom
179-fix-position-based-highlighting-when-a-text-contains-ampersand-and-other-reserved-htlm-entities
Sep 25, 2023
Merged

mhkuu merged 30 commits into
feature/html-parserfrom
179-fix-position-based-highlighting-when-a-text-contains-ampersand-and-other-reserved-htlm-entities

Conversation

@FAMarfuaty

@FAMarfuaty FAMarfuaty commented Aug 24, 2023 •

Copy link
Copy Markdown
Contributor

Context

  • In Classic editor, when a text contained an ampersand, the ampersand was transformed into & in the editor's HTML. This in turn messed up the position-based highlighting feature since when creating the Mark object (the object that represent the highlighting) in yoastseo , we calculate the ampersand length as 1 (referring to &), instead of 5. This means that for Classic editor, we need to calculate the ampersand length as 5, instead of 1.
  • Also important to note that in Gutenberg, we apply annotation to the rich text of a block, where the rich text represents the ampersand as & instead of &. This means that when creating the Mark object for Gutenberg, we should still calculate the ampersand length as 1, instead of 5.
  • This PR fixes this for some of the most commonly used HTML entities, like &lt; for <, &copy; for ©, as they all start with an ampersand.

Summary

This PR can be summarized in the following changelog entry:

  • Fixes an unreleased bug where position-based highlighting in Classic editor would become wonky when a text contained a common reserved HTML entity.
  • [shopify-seo] Fixes an unreleased bug where position-based highlighting in Classic editor would become wonky when a text contained a common reserved HTML entity.

Relevant technical choices:

  • The summary of the approach:
    Yoastseo

    • Make it so that reserved characters like& enters the paper as html entities (e.g. &amp;)
      • These are the complete list of the html entities we're processing:
      • &amp; => &
      • &lt; => <
      • &gt; => >
      • &quot; => "
      • &apos; => '
      • &ndash; => –
      • &mdash; => —
      • &copy; => ©
      • &reg; => ®
      • &trade; => ™
      • &pound; => £
      • &yen; => ¥
      • &euro; => €
      • &dollar; => $
      • &deg; => °
      • &asymp; => ≈
      • &ne; => ≠
    • Turn the ampersand & in the html entity into #, e.g. &amp; -> #amp; before building the tree. This is to avoid early conversion of the html entity into its reserved character version. This also means that when the tree is build, we're calculating the length of the html entity instead of the reserved character, so & will be 5 character long
    • When tokenizing the text into sentences, change the hashed html entity #amp; back to its reserved character &, while keeping the length information as 5 instead of 1. This is done so that we would still be able to match word "J&T" matches "J&T"

    gutenberg.js

    • As mentioned above, the annotation in Gutenberg is applied to the rich text of a block where the reserved character is used instead of its html entity. This means that before creating the annotation object for this editor, we need to adjust the position information of the Mark object when there is an html entity in the html.
    • In adjustMarkOffsets of packages/js/src/decorator/helpers/positionBasedAnnotationHelper.js, we run a check if the block html contains html entities
    • If the html entities are found we adjust the block start and end offset of the Mark object so that reserved character like & has 1 length instead of 5

Test instructions

Test instructions for the acceptance test before the PR gets merged

This PR can be acceptance tested by following these steps:

Scenario 1: Testing in Classic editor

  • Activate Classic editor
  • Create a new Post/product
  • Use this text (make sure to paste it with no formatting)

LINAK has a concept enabling you to halve the time it takes to assemble a height adjustable desk. It is called "Kick & Click™" and allows you to assemble desks without the use of screws or tools.

Halve the time spent on desk assembly

Traditionally, it takes 12 screws to mount the two columns of a typical height-adjustable office desk to the top frame. But with the LINAK Kick & Click™ mounting solution, the only screws you need are the ones attaching the tabletop. The rest is just ‘kick’ gently and ‘click’ – your desk is ready!

Using Kick & Click™— it's easy and very intuitive. There are pictograms with visuals showing the product parts, and it is obvious where you need to ‘kick’ (or just push). The ‘click’ sound indicates that assembly has been done correctly. It costs only 4899 so buy K&C now! This is K&C, aka as Kick & Click!

  • Set the keyphrase to Kick & Click™
    • Confirm the Keyphrase density assessment gives the following feedback: 🍏 The keyphrase was found 3 times. This is great!
    • Confirm that when clicking on the eye icon 👁️ of the assessment, the keyphrase is highlighted 3 times in the text (the & character is NOT highlighted, the ™️ IS highlighted)
  • Add an HTML entity to the keyphrase in the text, which is enclosed in quotation marks
    • Switch to the Text editor (or HTML editor in Block editor)
    • Remove the ™️ from the end of the Kick & Click™ in the first paragraph, and instead add its HTML entity, &trade;, so that it says Click&trade;
    • Switch back to the Visual editor
    • Confirm the Keyphrase density assessment gives the same feedback as above, and that the highlighted words stay the same
  • Add an ®️ character as an HTML entity to the text in the same sentence as the keyphrase
    • Switch to the Text editor (or HTML editor in Block editor)
    • Add &reg; to the word LINAK in the third paragraph so that it's LINAK&reg;
    • Switch back to the Visual editor
    • Confirm the Keyphrase density assessment gives the same feedback as above, and that the highlighted words stay the same
  • Save the post
  • Confirm the Keyphrase density assessment gives the same feedback as above, and that the highlighted words stay the same
  • Set the keyphrase to K&C
  • Confirm the Keyphrase density assessment gives the following feedback: 🍏 The keyphrase was found 2 times. This is great!
  • Confirm that when clicking on the eye icon 👁️ of the assessment, the keyphrase is highlighted 2 times in the text (the & character is ALSO highlighted)
  • Add a pound character as an HTML entity to the text in the same sentence as the keyphrase
    • Switch to the Text editor (or HTML editor in Block editor)
    • Add &pound; to the number 4899 in the last paragraph so that it becomes 4899&pound;
    • Switch back to the Visual editor
    • Confirm the Keyphrase density assessment gives the same feedback as above, and that the highlighted words stay the same
  • Add quotation marks as HTML entities to the keyphrase in the text
    • Switch to the Text editor (or HTML editor in Block editor)
    • Add &quot; to the beginning and end to the second K&C in the last paragraph so that it becomes &quot;K&amp;C&quot;
    • Switch back to the Visual editor
    • Confirm the Keyphrase density assessment gives the same feedback as above, and that the highlighted words stay the same
  • Style the focus keyphrase in the text, e.g. make it bold, italic or add a link
    • Be wild for this step 😉 Try to come up with different styling, for example for the keyphrase Kick & Click™, make the word Kick bold and the word Click italic, or vice versa. Try also make one of the word both bold and italic, etc. This is to make sure that the solution works for when both html tags and entities are found in the text.
  • Confirm the Keyphrase density assessment gives the same feedback as above, and that the highlighted words stay the same
  • Save the post (this post can be used for testing Scenario 3)

Scenario 2: Testing in Block editor

  • Repeat the steps mentioned in Scenario 1 above (but create the post in Block editor) and confirm that you get the same result

Scenario 3: Testing content, created in Classic editor, in Block editor

  • Deactivate Classic editor
  • Open the post (created in Classic editor as specified in Scenario 1) in Block editor
  • Set the keyphrase to Kick & Click™ (don't convert the content to blocks yet)
    • Confirm the Keyphrase density assessment gives the following feedback: 🍏 The keyphrase was found 3 times. This is great!
    • Confirm that when clicking on the eye icon 👁️ of the assessment, the keyphrase is highlighted 3 times in the text (the & character is NOT highlighted, the ™️ IS highlighted)
  • Convert the content to blocks
    • Confirm the Keyphrase density assessment still gives the following feedback: 🍏 The keyphrase was found 3 times. This is great!
    • Confirm that when clicking on the eye icon 👁️ of the assessment, the keyphrase is highlighted 3 times in the text (the & character is NOT highlighted, the ™️ IS highlighted)

Scenario 4: Testing content, created in Block editor, in Classic editor

  • Activate Classic editor
  • Open the post (created in Block editor as specified in Scenario 2) in Classic editor
  • Set the keyphrase to K&C
  • Confirm the Keyphrase density assessment gives the following feedback: 🍏 The keyphrase was found 2 times. This is great!
  • Confirm that when clicking on the eye icon 👁️ of the assessment, the keyphrase is highlighted 2 times in the text (the & character is ALSO highlighted)

Scenario 4: Testing content in Elementor

  • Install and activate Elementor
  • Create a new Post in Elementor
  • Use this text (make sure to paste it with no formatting)

LINAK has a concept enabling you to halve the time it takes to assemble a height adjustable desk. It is called "Kick & Click™" and allows you to assemble desks without the use of screws or tools.

Halve the time spent on desk assembly

Traditionally, it takes 12 screws to mount the two columns of a typical height-adjustable office desk to the top frame. But with the LINAK Kick & Click™ mounting solution, the only screws you need are the ones attaching the tabletop. The rest is just ‘kick’ gently and ‘click’ – your desk is ready!

Using Kick & Click™— it's easy and very intuitive. There are pictograms with visuals showing the product parts, and it is obvious where you need to ‘kick’ (or just push). The ‘click’ sound indicates that assembly has been done correctly. It costs only 4899 so buy K&C now! This is K&C, aka as Kick & Click!

  • Set the keyphrase to Kick & Click™
    • Confirm the Keyphrase density assessment gives the following feedback: 🍏 The keyphrase was found 3 times. This is great!
  • Add an HTML entity to the keyphrase in the text, which is enclosed in quotation marks
    • Switch to the Text editor
    • Remove the ™️ from the end of the Kick & Click™ in the first paragraph, and instead add its HTML entity, &trade;, so that it says Click&trade;
    • Switch back to the Visual editor
    • Confirm the Keyphrase density assessment gives the same feedback as above
  • Add an ®️ character as an HTML entity to the text in the same sentence as the keyphrase
    • Switch to the Text editor (or HTML editor in Block editor)
    • Add &reg; to the word LINAK in the third paragraph so that it's LINAK&reg;
    • Switch back to the Visual editor
    • Confirm the Keyphrase density assessment gives the same feedback as above
  • Save the post
  • Confirm the Keyphrase density assessment gives the same feedback as above
  • Set the keyphrase to K&C
  • Confirm the Keyphrase density assessment gives the following feedback: 🍏 The keyphrase was found 2 times. This is great!
  • Add a pound character as an HTML entity to the text in the same sentence as the keyphrase
    • Switch to the Text editor
    • Add &pound; to the number 4899 in the last paragraph so that it becomes 4899&pound;
    • Switch back to the Visual editor
    • Confirm the Keyphrase density assessment gives the same feedback as above
  • Add quotation marks as HTML entities to the keyphrase in the text
    • Switch to the Text editor (or HTML editor in Block editor)
    • Add &quot; to the beginning and end to the second K&C in the last paragraph so that it becomes &quot;K&amp;C&quot;
    • Switch back to the Visual editor
    • Confirm the Keyphrase density assessment gives the same feedback as above
  • Style the focus keyphrase in the text, e.g. make it bold, italic or add a link
  • Confirm the Keyphrase density assessment gives the same feedback as above

Scenario 5:Test in Shopify

  • Repeat the testing steps in Scenario 1 and confirm that you get the same result

Relevant test scenarios

  • Changes should be tested with the browser console open
  • Changes should be tested on different posts/pages/taxonomies/custom post types/custom taxonomies
  • Changes should be tested on different editors (Block/Classic/Elementor/other)
  • Changes should be tested on different browsers
  • Changes should be tested on multisite

Test instructions for QA when the code is in the RC

  • QA should use the same steps as above.

QA can test this PR by following these steps:

Impact check

This PR affects the following parts of the plugin, which may require extra testing:

  • The solution introduced in this PR also touches the highlighting feature for a text that contains html tags, for which I've added a test step in the testing instructions above.

UI changes

  • This PR changes the UI in the plugin. I have added the 'UI change' label to this PR.

Other environments

  • This PR also affects Shopify. I have added a changelog entry starting with [shopify-seo], added test instructions for Shopify and attached the Shopify label to this PR.

Documentation

  • I have written documentation for this change.

Quality assurance

  • I have tested this code to the best of my abilities.
  • During testing, I had activated all plugins that Yoast SEO provides integrations for.
  • I have added unit tests to verify the code works as intended.
  • If any part of the code is behind a feature flag, my test instructions also cover cases where the feature flag is switched off.
  • I have written this PR in accordance with my team's definition of done.
  • I have checked that the base branch is correctly set.

Innovation

  • No innovation project is applicable for this PR.
  • This PR falls under an innovation project. I have attached the innovation label.
  • I have added my hours to the WBSO document.

Fixes #179

@FAMarfuaty
FAMarfuaty changed the base branch from feature/html-parser to trunk August 24, 2023 12:07
@FAMarfuaty
FAMarfuaty changed the base branch from trunk to feature/html-parser August 24, 2023 12:08
@coveralls

Copy link
Copy Markdown

Pull Request Test Coverage Report for Build 5963441257

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall first build on 179-fix-position-based-highlighting-when-a-text-contains-ampersand-and-other-reserved-htlm-entities at 47.152%

Totals Coverage Status
Change from base Build 5963459713: 47.2%
Covered Lines: 12507
Relevant Lines: 26525

💛 - Coveralls

@marinakoleva marinakoleva changed the title 179 fix position based highlighting when a text contains ampersand and other reserved htlm entities Fix position-based highlighting when a text contains ampersand and other reserved HTML entities Sep 14, 2023
@marinakoleva marinakoleva added this to the feature/HTML-parser milestone Sep 14, 2023
@marinakoleva marinakoleva added the changelog: non-user-facing Needs to be included in the 'Non-userfacing' category in the changelog label Sep 14, 2023
@mhkuu

mhkuu commented Sep 19, 2023

Copy link
Copy Markdown
Contributor

This solution currently does not work for the block editor, because there we do not take HTML entities into account when we remove HTML tags in packages/js/src/decorator/helpers/positionBasedAnnotationHelper.js. In adjustMarkOffsets we should add handling of those -- we should recognize them and then deduct their length (-1 for the "simple" character). Possibly related: https://github.com/Yoast/lingo-other-tasks/issues/217

@FAMarfuaty FAMarfuaty added the Shopify This PR impacts Shopify. label Sep 21, 2023
@FAMarfuaty
FAMarfuaty marked this pull request as ready for review September 22, 2023 06:27
@mhkuu

mhkuu commented Sep 22, 2023

Copy link
Copy Markdown
Contributor

CR: looks great, just some small fixes for readability 🥇

@mhkuu

mhkuu commented Sep 25, 2023

Copy link
Copy Markdown
Contributor

ACC: During acceptance I found two bugs unrelated to this PR, for which I will create issues:

  • If the keyphrase is formatted for only a part (e.g., <strong>te</strong>st, where the keyphrase is test), the keyphrase is not highlighted completely.
  • The highlighting in Shopify does not work when the keyphrase is formatted. Specifically, it seems that if you add highlighting within a sentence, that breaks the highlighting. If it overarches the sentence, the highlighting works.

Otherwise, this PR is good to go! 🚀

@mhkuu
mhkuu merged commit c275fb1 into feature/html-parser Sep 25, 2023
@mhkuu
mhkuu deleted the 179-fix-position-based-highlighting-when-a-text-contains-ampersand-and-other-reserved-htlm-entities branch September 25, 2023 07:26
@mhkuu mhkuu modified the milestones: feature/HTML-parser, 21.4 Oct 3, 2023
@mhkuu mhkuu added the innovation Innovative issue. Relating to performance, memory or data-flow. label Nov 1, 2023
@FAMarfuaty FAMarfuaty mentioned this pull request Dec 21, 2023
10 of 18 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog: non-user-facing Needs to be included in the 'Non-userfacing' category in the changelog innovation Innovative issue. Relating to performance, memory or data-flow. Shopify This PR impacts Shopify.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants