String and char literals

[matklad]

String and char literals are tricky, b/c of escape sequences. Lexer should not interpret escape sequences unless absolutely necessary: it is the work of the validator to check them and to convert them to a real literal.

• fix the lexer itself
• create a validator
• tests

[mystor]

You might be interested in, for example, the basic string validators found in proc-macro2 (https://github.com/alexcrichton/proc-macro2/blob/master/src/stable.rs#L739-L788).

They don't actually parse the string into a value, they just validate that it is well formed. We can then layer an actual interpreter on later.

[matklad]

Oh, exellent! This is almost exactly what I wanted for the validaors API!

Ideally, I think validators should be able to simultaneously produce a value of the literal and error messages if the literal is invalid. For strings specifically, validator should also produce the offsets of escape sequences, so tha IDE can highlight escapes, and show particular invalid escape. Basically, validator for strings should be a sublexer.

[mystor]

You might also be interested in syn's literal parsing code which does most of what you want (doesn't do the offset stuff but that shouldn't be hard to add, and I think it assumes the input is somewhat valid? Can't remember exactly)

https://github.com/dtolnay/syn/blob/441a7aada6d16555923edf407cd52f5cae24ec66/src/lit.rs#L323-L969

It was based off of an old rust lexer I had written which had more complete error checking (I think https://github.com/mystor/rustlex? There are a bunch of bugs in that crate though).

[aochagavia]

@matklad we now have validators and tests. Does something still need to be fixed in the lexer?

Missing validators are:

• Raw string
• Raw byte string
• Byte string
• Byte

[matklad]

Yeah, I think this should be closed!

@aochagavia could you open a separate issue for byte strings and bytes, with pointers to existing infra?

Off the top of my head, I don't think we need any validation for raw strings: they are raw, so anything goes.

[matklad]

oh, you already did, that's #223 :)

[aochagavia]

Regarding raw strings... We should at least check that they are properly closed, right?

[matklad]

oh, you are right, that one we must check!