lx: Distinguish between unexpected EOF and EOF in ignored zones (broken by #509) #510
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
PR #509 introduced a bug: It didn't distinguish between an unexpected end of input and an end of input in a zone that matches but ignores its input. This caused several lxpos tests to fail due to getting a TOK_UNKNOWN rather than a TOK_EOF when the input has trailing whitespace, but I didn't notice until after merging because the normal build doesn't regenerate the code for src/lx/lexer.lx or src/libfsm/lexer.lx. (I had ensured all the libre dialect lexers and parsers were regenerated, but missed those.) Instead of always printing TOK_UNKNOWN, this this inspects the zone mappings to determine whether the current end ID represents a dead end for the zone. If not, it should instead print TOK_EOF.
silentbicycle
commented
Aug 26, 2025
| * should stay small enough that linear search is fine. If this becomes | ||
| * prohibitively expensive, then build a bitset of dead-end IDs upfront | ||
| * in one pass. */ | ||
| for (struct ast_zone *z = ast->zl; z != NULL; z = z->next) { |
There was a problem hiding this comment.
I set up a variety of scenarios and they all behaved consistently this being how lx identifies dead ends internally, but please let me know if I'm misunderstanding something or there's a more direct way to check this. I didn't see a way to tell what zone the accept_c callback is running inside of, but the linear scan across all zones should be small in practice.
katef
approved these changes
Aug 29, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR #509 introduced a bug: It didn't distinguish between an unexpected end of input and an end of input in a zone that matches but ignores its input. This caused several lxpos tests to fail due to getting a
TOK_UNKNOWNrather than aTOK_EOFwhen the input has trailing whitespace, but I didn't notice until after merging because the normal build doesn't regenerate the code forsrc/lx/lexer.lxorsrc/libfsm/lexer.lx. (I had ensured all the libre dialect lexers and parsers were regenerated, but missed those.)Only
src/lx/print/c.chas code changes, the other files are all generated code updates.Instead of always generating
TOK_UNKNOWN, this this inspects the zone mappings to determine whether the current end ID represents a dead end for the zone. If not, it should instead generate TOK_EOF.