ICE: attribute-related parser bug when consuming generated code

[mzhang28]

I was trying to add Cap'n Proto to the list of languages that Tokei is able to recognize. I didn't realize that the build script took this list and generated rust code directly and blindly typed "Cap'n Proto" as the language name, since that's what it is.

The error seems to come from attr_wrapper.rs from within the parser.

Code

// Check https://git.sr.ht/~mzhang/rust-ice-tokei-quote

I haven't been able to minimize the full program yet, since it's hard to determine which of the symbols is actually causing the error, but the error comes from the parser so I removed all the other languages to make the generated file smaller. Check the repository for full source.

Meta

rustc --version --verbose:

rustc 1.56.1 (59eed8a2a 2021-11-01)
binary: rustc
commit-hash: 59eed8a2aac0230a8b53e89d4e99d55912ba6b35
commit-date: 2021-11-01
host: x86_64-unknown-linux-gnu
release: 1.56.1
LLVM version: 13.0.0

Error output

error: expected one of `(`, `,`, `=`, `{`, or `}`, found `'n`
  --> /home/michael/Downloads/tokei/target/debug/build/tokei-1c3ae2d72a936ed1/out/language_type.rs:12:31
   |
12 |     #[allow(missing_docs)] Cap'n Proto,
   |                               ^^ expected one of `(`, `,`, `=`, `{`, or `}`

error: expected item, found `'n`
  --> /home/michael/Downloads/tokei/target/debug/build/tokei-1c3ae2d72a936ed1/out/language_type.rs:12:31
   |
12 |     #[allow(missing_docs)] Cap'n Proto,
   |                               ^^

thread 'rustc' panicked at 'Stack should be empty: final_buf=FrameData { open: /home/michael/Downloads/tokei/target/debug/build/tokei-1c3ae2d72a936ed1/out/language_type.rs:11:23: 11:24 (#0), open_delim: Brace, inner: [(Token(Token { kind: Pound, span: /home/michael/Downloads/tokei/target/debug/build/tokei-1c3ae2d72a936ed1/out/language_type.rs:12:5: 12:6 (#0) }), Alone), (Delimited(DelimSpan { open: /home/michael/Downloads/tokei/target/debug/build/tokei-1c3ae2d72a936ed1/out/language_type.rs:12:6: 12:7 (#0), close: /home/michael/Downloads/tokei/target/debug/build/tokei-1c3ae2d72a936ed1/out/language_type.rs:12:26: 12:27 (#0) }, Bracket, AttrAnnotatedTokenStream([(Token(Token { kind: Ident("allow", false), span: /home/michael/Downloads/tokei/target/debug/build/tokei-1c3ae2d72a936ed1/out/language_type.rs:12:7: 12:12 (#0) }), Alone), (Delimited(DelimSpan { open: /home/michael/Downloads/tokei/target/debug/build/tokei-1c3ae2d72a936ed1/out/language_type.rs:12:12: 12:13 (#0), close: /home/michael/Downloads/tokei/target/debug/build/tokei-1c3ae2d72a936ed1/out/language_type.rs:12:25: 12:26 (#0) }, Paren, AttrAnnotatedTokenStream([(Token(Token { kind: Ident("missing_docs", false), span: /home/michael/Downloads/tokei/target/debug/build/tokei-1c3ae2d72a936ed1/out/language_type.rs:12:13: 12:25 (#0) }), Alone)])), Alone)])), Alone), (Token(Token { kind: Ident("Cap", false), span: /home/michael/Downloads/tokei/target/debug/build/tokei-1c3ae2d72a936ed1/out/language_type.rs:12:28: 12:31 (#0) }), Alone)] } stack=[FrameData { open: no-location (#0), open_delim: NoDelim, inner: [(Token(Token { kind: Ident("pub", false), span: /home/michael/Downloads/tokei/target/debug/build/tokei-1c3ae2d72a936ed1/out/language_type.rs:11:1: 11:4 (#0) }), Alone), (Token(Token { kind: Ident("enum", false), span: /home/michael/Downloads/tokei/target/debug/build/tokei-1c3ae2d72a936ed1/out/language_type.rs:11:5: 11:9 (#0) }), Alone), (Token(Token { kind: Ident("LanguageType", false), span: /home/michael/Downloads/tokei/target/debug/build/tokei-1c3ae2d72a936ed1/out/language_type.rs:11:10: 11:22 (#0) }), Alone)] }]', compiler/rustc_parse/src/parser/attr_wrapper.rs:501:5


error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md

note: rustc 1.56.1 (59eed8a2a 2021-11-01) running on x86_64-unknown-linux-gnu

note: compiler flags: -C embed-bitcode=no -C debuginfo=2 -C incremental --crate-type lib

note: some of the compiler flags provided by cargo are hidden

query stack during panic:
end of query stack
error: could not compile `tokei` due to 2 previous errors

Backtrace

stack backtrace:
   0:     0x7f916ce9551c - std::backtrace_rs::backtrace::libunwind::trace::h3fea1eb2e0ba2ac9
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/../../backtrace/src/backtrace/libunwind.rs:90:5
   1:     0x7f916ce9551c - std::backtrace_rs::backtrace::trace_unsynchronized::h849d83492cbc0d59
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x7f916ce9551c - std::sys_common::backtrace::_print_fmt::he3179d37290f23d3
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/sys_common/backtrace.rs:67:5
   3:     0x7f916ce9551c - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h140f6925cad14324
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/sys_common/backtrace.rs:46:22
   4:     0x7f916cef322c - core::fmt::write::h31b9cd1bedd7ea38
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/fmt/mod.rs:1150:17
   5:     0x7f916ce865a5 - std::io::Write::write_fmt::h1fdf66f83f70913e
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/io/mod.rs:1667:15
   6:     0x7f916ce98a80 - std::sys_common::backtrace::_print::he7ac492cd19c3189
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/sys_common/backtrace.rs:49:5
   7:     0x7f916ce98a80 - std::sys_common::backtrace::print::hba20f8920229d8e8
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/sys_common/backtrace.rs:36:9
   8:     0x7f916ce98a80 - std::panicking::default_hook::{{closure}}::h714d63979ae18678
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/panicking.rs:210:50
   9:     0x7f916ce98637 - std::panicking::default_hook::hf1afb64e69563ca8
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/panicking.rs:227:9
  10:     0x7f916d66db01 - rustc_driver::DEFAULT_HOOK::{{closure}}::{{closure}}::h045e68b20488b66e
  11:     0x7f915f755633 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h8bfb4d924224caf9
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/alloc/src/boxed.rs:1650:9
  12:     0x7f915f7b925d - proc_macro::bridge::client::<impl proc_macro::bridge::Bridge>::enter::{{closure}}::{{closure}}::h24358b8652b5dd4f
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/proc_macro/src/bridge/client.rs:320:21
  13:     0x7f916ce99299 - std::panicking::rust_panic_with_hook::h02231a501e274a13
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/panicking.rs:628:17
  14:     0x7f916ce98d50 - std::panicking::begin_panic_handler::{{closure}}::h5b5e738edf361af7
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/panicking.rs:521:13
  15:     0x7f916ce959e4 - std::sys_common::backtrace::__rust_end_short_backtrace::h601a115def7987b1
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/sys_common/backtrace.rs:141:18
  16:     0x7f916ce98cb9 - rust_begin_unwind
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/panicking.rs:517:5
  17:     0x7f916ce5d83b - std::panicking::begin_panic_fmt::hc15cd62c02c70986
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/panicking.rs:460:5
  18:     0x7f916f9f8061 - <rustc_parse::parser::attr_wrapper::LazyTokenStreamImpl as rustc_ast::tokenstream::CreateTokenStream>::create_token_stream::h2c601a6a30e365a4
  19:     0x7f916fbf3fd0 - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &mut F>::call_once::hcb8fda303ecf5383
  20:     0x7f916f144b4a - <core::iter::adapters::flatten::FlatMap<I,U,F> as core::iter::traits::iterator::Iterator>::next::h7c63b5fc52f3a45f
  21:     0x7f916fbf3387 - <alloc::vec::Vec<T> as alloc::vec::spec_from_iter::SpecFromIter<T,I>>::from_iter::h4adb4cb438f035c3
  22:     0x7f916fbf4a48 - rustc_ast::tokenstream::AttrAnnotatedTokenStream::to_tokenstream::h584678373637b993
  23:     0x7f916f9df8c8 - rustc_parse::prepend_attrs::h1364fa75f41f56d0
  24:     0x7f916f9dee96 - rustc_parse::nt_to_tokenstream::h1aa6ce7dfdc60874
  25:     0x7f916f96958f - <rustc_expand::proc_macro::ProcMacroDerive as rustc_expand::base::MultiItemModifier>::expand::he8ebb7500431b2c0
  26:     0x7f916ee90204 - rustc_expand::expand::MacroExpander::fully_expand_fragment::he5c8c9a230b73262
  27:     0x7f916f965b94 - rustc_expand::expand::MacroExpander::expand_crate::h5d2700cd2ff2185f
  28:     0x7f916f1cc3b3 - rustc_session::utils::<impl rustc_session::session::Session>::time::h135c8694209b661d
  29:     0x7f916f1a345e - rustc_interface::passes::configure_and_expand::h725af0cc897b2c37
  30:     0x7f916f1a6aea - rustc_interface::queries::Queries::expansion::h8182cea4fc8c15c8
  31:     0x7f916f17fb9a - rustc_interface::queries::<impl rustc_interface::interface::Compiler>::enter::hff7e36ca6c2085f0
  32:     0x7f916f16d02c - rustc_span::with_source_map::he4ccdc3d435ea73b
  33:     0x7f916f17f5cc - scoped_tls::ScopedKey<T>::set::hf1cc160e90acdba2
  34:     0x7f916f16e73b - std::sys_common::backtrace::__rust_begin_short_backtrace::h0b3064789b6c03d4
  35:     0x7f916f16b905 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h3907954af9fcbd87
  36:     0x7f916cea58e3 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::hf2a5e508372e2387
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/alloc/src/boxed.rs:1636:9
  37:     0x7f916cea58e3 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h46642f5b75478456
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/alloc/src/boxed.rs:1636:9
  38:     0x7f916cea58e3 - std::sys::unix::thread::Thread::new::thread_start::h1fbdb50adbde1cab
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/sys/unix/thread.rs:106:17
  39:     0x7f916cda5259 - start_thread
  40:     0x7f916ccba5e3 - __GI___clone
  41:                0x0 - <unknown>

[Badel2]

I got the same error while fuzzing, I think it's tracked here: #81920