Fix lexing of comments

[matklad]

Lexer does not support block-style comments at all. This should be fixed :)

[CAD97]

Block comments can be unterminated. How should the lexer handle this? The simple solution is to just emit a COMMENT token to the end of the file, and I'm not sure what other options are available with this lexer design.

[matklad]

You mean block comments, right?

Yeah, I think an unclosed /* should just consume the whole file until the end, and than, on the currently non-existent validators layer we should produce an error message about an unclosed comment.

[CAD97]

(Yes, that's what I meant)

[matklad]

Curiously, this is an example of hard (unsolvable?) problem for incremental parsing: if the user types /*, will have to through away the previous tokens completely. That means that parsing after modification would be O(N) no matter how hard we try to make it incremental.