Feature request: automatic matching pairs

[docwhat]

Most languages have a common set of matching pairs that can be filled in by the editor automatically (e.g., ( & ), " & ") but most languages have additional matching pairs and some even have quirks.

This seems like something well suited to language servers.

Some examples:

# Ruby
# Example pair: "if" "end" on different lines
if thing_is_true
  do_something
end

" viml (AKA VimScript)
" comments are double quotes.
" but in other cases are pairs.
" another example: "function" "endfunction" on different lines
function thing(value)
  echon "string" a:value
endif

// Javascript
// example: "{" "}" on different lines (surrounding blocks)
(value) -> {
  // expect a newline here
}

# Python doesn't have many end markers
if thing
  something()

[mickaelistria]

If the pairs are purely syntaxical and statically defined in the language, then you can consider using the "language-configuration.json" files pseudo-standardized as part of VSCode AFAIK which allow to describe such pairs, so editors can use them to produce their usual features.
This format is support by VSCode and Eclipse IDE at least.

[docwhat]

Pairs work best when we understand their context.

A simple example is ' in a comment or a string. You shouldn't create it as a pair since it is probably a contraction (i.e., "don't").

Another example is given a starting point of the following (▐ is where the cursor is):

const myfun = (a, b) -> {▐}

After enter is pressed:

const myfun = (a, b) -> {
    ▐
}

[matklad]

👍

In general, it is impossible to understand precisely if something is a pair or not, without a deep understanding of the language. C++'s < and > are a prime example. I need this feature for Rust as well (currently implemented as a custom extension).

On a more philosophical note, I'm all for moving everything language-configuration.json can do into a language server: real parser will always be more precise than a textmate grammar.

[kdvolder]

On a more philosophical note, I'm all for moving everything language-configuration.json can do into a language server: real parser will always be more precise than a textmate grammar.

+1 to that from me.

[matklad]

cc #999

[mikesamuel]

Here're some ways the SemanticTokensLegend might extend to help differentiate between bracket and non-bracket tokens. I also tried to collect some use cases and corner cases to spur discussion.

Perhaps something like

 export interface SemanticTokensLegend {
     /**
      * The token types a server uses.
      */
      tokenTypes: string[];

     /**
       * The token modifiers a server uses.
       */
      tokenModifiers: string[];
+
+    bracketPairs?: BracketPair[];
 }

where a BracketPair explains how to pair brackets in a left to right scan over tokens.

interface BracketPair {
    opener: TokenFilter;
    closeer: TokenFilter; 
}

interface TokenFilter {
    /**
     * If non-null, a member of the SemanticTokenTypes tokenTypes list.
     * This allows using token type information to distinguish tokens that
     * may not be brackets in some contexts, for example `<` which may
     * be a bracket for generic type parameters or a comparison operator.
     */
    tokenType?: string | null;
    /**
     * If non-null, a member of the SemanticTokenTypes tokenModifiers list.
     * This similarly allows distinguish ambiguous brackets based on a modifier.
     */
    tokenModifier?: string | null;
    /**
     * A pattern that matches bracket tokens.
     * It's a regular expression, so use `[(]` to match open parenthesis.
     */
    tokenTextPattern: string;
}

There is precedent for regular expressions: interface LinkedEditingRanges's wordPattern.

I think this provides enough flexibility to use existing metadata to:

Deal with ambiguous brackets

// java
// Non-bracket < and > in an actual parameter list
int x = f(a < b, c > d);
// Angle bracket < and > in a type in a formal parameter list
int f(a<b, c> d) {}

Deal with multiple closing contexts

// javascript
{ // curly for block
  let x: { /* curly for type */ str:string} = { // curly for object
    str: `chars ${ /* curly for string template hole */ expr } chars`
  };
}

where maybe a semantic tokenizer uses different tokenTypes or modifiers to distinguish statement {...}, expression {...}, and string ${...}

Deal with complex closers

A language that wants to pair an open set of brackets can do so. For example:

BracketPair({
    opener: TokenFilter({
        tokenTextPattern: "<[a-zA-Z][\w-:]*",
    }),
    closer: TokenFilter({
        tokenTextPattern: "/?>",
    })
})

might pair HTML-style tag boundaries without enumerating all the possible tag names.

Not supported in this scheme: Matching part of a token

If the tokenizer does not split angle brackets out, this scheme will not allow matching brackets within a larger token.

// typescript
let x: List<List<List<T>>>= [];

If the semantic tokenizer does not split >>>= (for example, because it's a valid compound shift operator) into separate >, >, >, and = tokens, then interface BracketPair would not separately allow each of the < to consume a different portion of the >>>= token.

Deal with missing brackets

Maybe this is out of scope for this issue, but BracketPair could be abused to flag mismatched brackets with an extension that allows adding a modifier.

Consider

f(
 { x: 1, y: 2 /* Missing closing curly bracket */
)

Sometimes error tolerant parsing requires inserting tokens. I don't know whether clients would need to opt-into zero width token support.

Absent that, we could allow a pair matcher to specify what tokens cannot appear between the opener and closer

 interface BracketPair {
     opener: TokenFilter;
     closer: TokenFilter; 
+    stopAt?: TokenFilter | null;
 }

So for example,

BracketPair({
    opener: TokenFilter({
        tokenTextPattern: "\\[",
    }),
    closer: TokenFilter({
        tokenTextPattern: "\\]",
    }),
    stopAt: TokenFilter({
        tokenTextPattern: "[;\)\}],
    }),
})

Might allow popping an unmatched [ from the token stack when we see a ;, ) or } token that is not part of or nested in a bracket group.

Indent/dedent matching

# python
def f():
    body

could lex to include tokens that use tokenType or tokenModifiers to distinguish indent and dedent:

1. def, f as simple tokens
2. The normal bracket pair: (, )
3. A simple token :
4. An token with a tokenModifier of "indent" corresponding to the newline and/or space
5. A simple token body
6. A token with a tokenModifier of "dedent"

Pairing could use BracketPairs that ignore the textual content and rely only on the other fields.

Supporting dedent tokens might require support for zero-width tokens because a single newline or the end-of-file might signal the end of an arbitrary number of nested blocks.

Corner case: Opener/closer ambiguity

Whether openers or closers are specified as regular expressions or as strings, it's possible for a token to match both as an opener or a closer.

This sometimes occurs as in fenced markdown blocks where the same sequence is used to start and end

```
fenced
```
not fenced

Those are arguably string delimiters, not brackets, but the same substring is used both to mark the beginning and the end.

I don't know the perfect scheme for classifying a token that could be both an opener and a closer except to be greedy: if it closes something it's a closer, else it's an opener.

Not supported in this scheme: complex relationships between opener and closers

Going back to the markdown fenced block example, if you like markdown in your markdown you can do

````md
A fenced block looks like
```
fenced
```
not fenced
````

For n > 3, a sequence of n backticks is closed by a sequence of the same number.

That relationship is not expressible with regular expressions, except possibly by abusing back references.

Self-closing tokens

If a language lexes () as a single token, as may happen in languages where it corresponds to a literal Unit value, it should probably either be treated as neither an opener nor a closer, or as both. It seems simple enough to ensure it's treated as neither; I don't know why one would want the latter, but mention it in case it jogs someone's memory.

Self-closing SGML tags may also fall in this category.