Feature: Read a code fence's info string as CommonMark does: language after spaces, and on an unclosed fence #9406

Lexical version: 0.52.0 (main at 98e936d gives the same results)

Steps to reproduce

Import each document below with $convertFromMarkdownString(markdown, TRANSFORMERS), then export it with $convertToMarkdownString(TRANSFORMERS):

Current behaviour

CommonMark says everything after an opening fence, with leading and trailing whitespace stripped, is the info string, and its first word is usually the language. That holds whether or not the block is closed: an unclosed fence runs to the end of the document. Lexical reads the language only when it comes right after the backticks, and keeps the rest of the line out of the code only when the block is closed. Otherwise the info string becomes the first line of the code and stays there through every save.

``` js is how some editors and formatters write fences, and GitHub, CommonMark and markdown-it all read it as a js block.

Cause

In packages/lexical-markdown/src/MarkdownTransformers.ts:

• CODE_START_REGEX is /^([ \t]*{3,})([\w-]+)?[ \t]?/`, so a space before the language ends the match.
• CODE.handleImportAfterStartMatch() treats the rest of the line as metadata only on the closed-fence path. The unclosed path unshifts currentLine.slice(startMatch[0].length) into the code.

This is deliberate and covered by tests: CodeBlockInfoString.test.ts ("keeps content that follows a fence with no language", "leaves an unterminated fence unchanged"), CodeBlockLeadingBlankLine.test.ts ("still treats text after the opening fence as content"), and the '```javascript Incomplete tag' cases in LexicalMarkdown.test.ts. I'm asking whether you'd accept following CommonMark here. ``` code and ``` js can't be told apart, so supporting the second means changing the first.

Suggested change

--- a/packages/lexical-markdown/src/MarkdownTransformers.ts
+++ b/packages/lexical-markdown/src/MarkdownTransformers.ts
@@ -222,7 +222,7 @@ const UNORDERED_LIST_REGEX = /^(\s*)[-*+]\s/;
 const CHECK_LIST_REGEX = /^(\s*)(?:[-*+]\s)?\s?(\[(\s|x)?\])\s/i;
 const HEADING_REGEX = /^(#{1,6})\s/;
 const QUOTE_REGEX = /^>\s/;
-const CODE_START_REGEX = /^([ \t]*`{3,})([\w-]+)?[ \t]?/;
+const CODE_START_REGEX = /^([ \t]*`{3,})[ \t]*([\w-]+)?[ \t]?/;
 const CODE_END_REGEX = /^[ \t]*`{3,}$/;
@@ -956,10 +956,16 @@ export const CODE: MultilineElementTransformer = {
       }
     }
 
+    // An unclosed fence runs to the end of the document, and its info string
+    // is read the same way as a closed fence's.
+    const meta = startMatch[2]
+      ? currentLine.slice(startMatch[0].length).trim()
+      : '';
     const linesInBetween = lines.slice(startLineIndex + 1);
-    linesInBetween.unshift(currentLine.slice(startMatch[0].length));
+    linesInBetween.unshift(meta ? '' : currentLine.slice(startMatch[0].length));
 
     CODE.replace(rootNode, null, startMatch, null, linesInBetween, true);
+    $setCodeMeta(rootNode, meta);
     return [true, lines.length - 1];

With this change, the first three rows import as CommonMark does and are exported as ```js (with title=x kept as metadata). These are unchanged: a closed ```js title="x" fence, ```js with nothing after it, a bare ``` fence, the single-line ```Single line Code``` form, and an indented ``` js fence (now read as js). ``` code / more / ``` becomes language code, code more, as in CommonMark.

Tested with headless Lexical against main at 98e936d. Exactly the 5 tests listed above fail, all asserting the current behaviour, and the other unit tests in lexical-markdown, lexical-mdast and lexical-playground/__tests__/unit pass. tsc, ESLint and Prettier pass. The shortcut uses the same expression, and typing ``` plus a space still matches, but I didn't try it in a browser.

The single-line form (```code```) isn't changed here; see #9404.

Related: PR #9055 and the closed PR #9020 (info string metadata on closed fences), #1456. @lexical/mdast reads these as CommonMark does.