Span issues with FromStr in unstable proc_macro

While writing improved tests for proc_macro2, I noticed some surprising behaviour of the FromStr impl for proc_macro::TokenStream.

When parsing a string with FromStr, a token parsed from the very beginning of the string can have a different source_file() than a token parsed elsewhere. For example:

let tokens = "aaa\nbbb".parse::<proc_macro::TokenStream>().unwrap().into_iter().collect::<Vec<_>>();
println!("span: {:?}", tokens[0].span);
println!("source_file: {:?}", tokens[0].span.source_file());
println!("");
println!("span: {:?}", tokens[1].span);
println!("source_file: {:?}", tokens[1].span.source_file());

Will produce the output:

span: Span(Span { lo: BytePos(6734829), hi: BytePos(6734832), ctxt: #0 })
source_file: SourceFile { path: ProcMacroSourceCode, is_real: false }

span: Span(Span { lo: BytePos(50), hi: BytePos(58), ctxt: #25 })
source_file: SourceFile { path: Real("pm_test.rs"), is_real: true }

I would've expected both tokens to have the same context, and similar BytePos-es. Specifically I would have expected both of these tokens to have a span located in a fake ProcMacroSourceCode file, but set to resolve in the context of the enclosing scope (as if span.resolved_at(Span::call_site()) was called).

I should note that adding a ' ' (space) character to the beginning of the string causes all tokens to have spans matching the second one.

I think the cause of the first-span-being-different issue is

rust/src/libsyntax/parse/lexer/mod.rs

Lines 202 to 209 in 178becd

    
           pub fn new(sess: &'a ParseSess, filemap: Lrc<syntax_pos::FileMap>) -> Self { 
        
               let mut sr = StringReader::new_raw(sess, filemap); 
        
               if sr.advance_token().is_err() { 
        
                   sr.emit_fatal_errors(); 
        
                   FatalError.raise(); 
        
               } 
        
               sr 
        
           }

, which causes the first token to be advanced-to and peeked at before the override_span is set here:

rust/src/libsyntax/parse/mod.rs

Line 230 in 604d4ce

srdr.override_span = override_span;

The cause of the issue where all parsed tokens have the line and column of the input file instead of lines and columns into a fake proc-macro object seems to be caused by the way that override_span is used - namely how it overrides the entire span instead of just the expansion context.

@rustbot

@rustbot label +E-needs-test

Does not reproduce anymore. A bisect reveals that it was fixed by #50924.

The fix didn't add any tests.

@rustbot

@rustbot claim

@KaiTomotake

@KaiTomotake you still on it? happy to push if you are not on it

Rollup merge of rust-lang#163094 - zakrad:regr-test-48944, r=davidtwco

Unrolled build for #163094 in rollup 163227

Navigation Menu

Repository navigation

`Span` issues with `FromStr` in unstable `proc_macro` #48944

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	pub fn new(sess: &'a ParseSess, filemap: Lrc<syntax_pos::FileMap>) -> Self {
	let mut sr = StringReader::new_raw(sess, filemap);
	if sr.advance_token().is_err() {
	sr.emit_fatal_errors();
	FatalError.raise();
	}
	sr
	}

Span issues with FromStr in unstable proc_macro #48944

Description

Activity

mystor commented on Mar 12, 2018

mystor commented on Mar 12, 2018

oskgo commented on Jan 25, 2024

KaiTomotake commented on Feb 1, 2026

zakrad commented on Sep 15, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

`Span` issues with `FromStr` in unstable `proc_macro` #48944