Stdlib: token_get_all() null byte after $ — T_VARIABLE not literal dollar + T_BAD_CHARACTER (ext/tokenizer/tokenizer.c) #13896

Category

stdlib · php-src-strict · lexer/tokenizer parity

Problem

For source <?php $\0 = 1;, token_get_all() must lex the bytes after <?php as a literal '$' token followed by T_BAD_CHARACTER for the embedded NUL. This compiler’s VM returns T_VARIABLE with value '$' plus T_BAD_CHARACTER, diverging from Zend’s token stream.

Downstream tools that diff token streams against php-src (static analysis, highlight pipelines) see a false variable name.

php-src reference

• ext/tokenizer/tokenizer.c — token_get_all delegates to Zend scanner
• Zend/zend_language_scanner.l — bad-character handling after $

Repro (verified 2026-06-30)

test/repro/maintainer_gap_token_get_all_null_byte.php

php test/repro/maintainer_gap_token_get_all_null_byte.php
php bin/vm.php test/repro/maintainer_gap_token_get_all_null_byte.php

Scope (PHP-in-PHP first)

Avoid new C in runtime/ unless thin ABI for scanner hook is unavoidable.

Done when (php-src-strict)

• Token stream matches Zend for NUL-after-$ sample
• Repro exits 0 on VM
• No regressions on normal T_VARIABLE names

Verification

php bin/vm.php test/repro/maintainer_gap_token_get_all_null_byte.php

Links

• Epic Epic: Self-host critical path — compiler compiles itself (M3→M5) #1492 · related Stdlib: token_name() — map T_* id to canonical name (ext/tokenizer/tokenizer.c parity) #4710 (token_name) · Stdlib: PhpToken built-in class missing — token_get_all() companion (ext/tokenizer/tokenizer.c) #6077 (PhpToken class)