Right now this is done while lexing:
|
if old_ch_is_newline { |
|
if self.save_new_lines_and_multibyte { |
|
self.filemap.next_line(self.pos); |
|
} |
|
self.col = CharPos(0); |
|
} else { |
|
self.col = self.col + CharPos(1); |
|
} |
|
if new_ch_len > 1 { |
|
if self.save_new_lines_and_multibyte { |
|
self.filemap.record_multibyte_char(self.pos, new_ch_len); |
|
} |
|
} |
While that may seem like it avoids going twice over the file contents, lexing is slow, while newline and non-width-1 (which is also improperly computed by means of multi-byte characters, instead of the unicode-width library) character finding can be optimized one way or another.
Cleaning this up would get rid of hacks such as the save_new_lines_and_multibyte flag in the code linked above and having to reimplement this in anything else using CodeMap outside of libsyntax.
Right now this is done while lexing:
rust/src/libsyntax/parse/lexer/mod.rs
Lines 423 to 435 in f861b6e
While that may seem like it avoids going twice over the file contents, lexing is slow, while newline and non-width-1 (which is also improperly computed by means of multi-byte characters, instead of the
unicode-widthlibrary) character finding can be optimized one way or another.Cleaning this up would get rid of hacks such as the
save_new_lines_and_multibyteflag in the code linked above and having to reimplement this in anything else usingCodeMapoutside of libsyntax.