Why in the Java version cpp lexer, the tokens obtained by "getTokens()” do not include whitespace characters?

[q1anz1]

Hello

I downloaded the file about parsing C++ syntax from https://github.com/antlr/grammars-v4/tree/master/cpp in Java version,
and about parsing C# from https://github.com/antlr/grammars-v4/tree/master/csharp in Java version too.

And I found that things went different with the same code:
code for parsing cpp:

CharStream input = CharStreams.fromString(myCppCode);

CPP14Lexer cpp14Lexer = new CPP14Lexer(input);

CommonTokenStream commonTokenStream = new CommonTokenStream(cpp14Lexer);

List<Token> allTokens = commonTokenStream.getTokens();

code for parsing c#:

CharStream input = CharStreams.fromString(myCsharpCode);

CSharpLexer csharpLexer = new new CSharpLexer(input);

CommonTokenStream commonTokenStream = new CommonTokenStream(csharpLexer);

List<Token> allTokens = commonTokenStream.getTokens();

but in the result:
The allTokens from C++ does not include token for whitespace symbols, but the allTokens from C# does include token for whitespace characters.

If someone could tell me whether this is my problem or if it's a bug, and also how I can obtain the tokenList with the blank token, that would be great. Thanks.

[kaby76]

It's probably not a bug, certainly not with Antlr. For whatever reason, the token types are handled differently on purpose between the two grammars. There are advantages for either approach.

In the csharp grammar, whitespace is specified to be have a token created with channel HIDDEN (see code). For the cpp grammar, whitespace is marked as skip, which means that no tokens are created for whitespace (see code). It's hard to say why one grammar was written one way and the other grammar written rthe other way, but the requirements may have been different for the two authors.

[q1anz1]

Oh, well. I thought that all grammar will add the blank characters to the hidden tokens. Thank you for your answer, I got it.