Bison parser generates different zendtokentype constants for each patch version since php 8.3.

[vtsykun]

Description

Hi, I noticed that since PHP 8.3 the value of T_* constants changes for each patch version. For example, if I'll use token_get_all and cache or serialize returning result I can got unexpected behavior.

https://3v4l.org/8L2eX

<?php

$text = <<<TXT
<?php

namespace AAA;

require_once '/tmp/dev2/t4.php';
class AAA {
}

\$class = new AAA();
TXT;

$tokens = token_get_all($text);

foreach ($tokens as $token) {
    echo json_encode($token) . "\n";
}

For example
T_STRING constant is 313 for 8.3.0, 8.3.2, 8.3.4, 8.3.7, 8.3.9 versions.
T_STRING constant is 262 for 8.3.1, 8.3.3, 8.3.5 - 8.3.6, 8.3.8 versions.

However for 8.1, 8.2 there are not changes between php patch versions.

PHP Version

8.3

Operating System

No response

[damianwadley]

Correct. You should avoid relying on the exact values of constants, on principle, but that's especially true for the T_* parser tokens as they are totally an implementation detail within PHP and are subject to change.

[cmb69]

And since that is even explicitly documented, we can close this ticket as invalid.

[hashar]

I have tracked it down to different versions of Bison being used. https://phabricator.wikimedia.org/T396312#10903133

The PHP tarballs includes source files compiled using Bison. When the tarballs are made by different persons each having different versions of Bison, the token values would differ. I have looked at the tarball for php 8.3.20 and 8.3.21, the token values can be found in Zend/zend_language_parser.h and Bison adds a comment at the top indicating its version:

Header file:

$ grep -P '(A Bison parser|T_CLASS_C = )' php-8.3.{20,21}/Zend/zend_language_parser.h
php-8.3.20/Zend/zend_language_parser.h:/* A Bison parser, made by GNU Bison 3.0.4.  */
php-8.3.20/Zend/zend_language_parser.h:    T_CLASS_C = 382,
php-8.3.21/Zend/zend_language_parser.h:/* A Bison parser, made by GNU Bison 3.8.2.  */
php-8.3.21/Zend/zend_language_parser.h:    T_CLASS_C = 346,               /* "'__CLASS__'"  */

And in the C source we can see a more recent version of bison uses a different constant internally:

$ grep -P '(A Bison parser|T_CLASS_C = )' php-8.3.{20,21}/Zend/zend_language_parser.c
php-8.3.20/Zend/zend_language_parser.c:/* A Bison parser, made by GNU Bison 3.0.4.  */
php-8.3.20/Zend/zend_language_parser.c:    T_CLASS_C = 382,
php-8.3.21/Zend/zend_language_parser.c:/* A Bison parser, made by GNU Bison 3.8.2.  */
php-8.3.21/Zend/zend_language_parser.c:  YYSYMBOL_T_CLASS_C = 107,                /* "'__CLASS__'"  */

For the PHP 8.3 series of release, all releases made by Jakub Zelenka were done with Bison 3.0.4 and have the same values. Other releases were cut by other people using a more recent version of Bison and thus differ.

As mentioned above at 14937#issuecomment-2226224571: the token values must NOT be used. The token names should be used instead.

[nikic]

Correct. You should avoid relying on the exact values of constants, on principle, but that's especially true for the T_* parser tokens as they are totally an implementation detail within PHP and are subject to change.

This is only correct for the userland constants exported by the tokenizer extension. For the values defined in Zend headers, this is an ABI break. Changing the value of enums in public headers is a big no-no.

zend_language_parser.y should probably be explicitly specifying the numerical value of tokens (see https://www.gnu.org/software/bison/manual/html_node/Token-Decl.html) to ensure these are stable across bison versions.