Bug: Some Unicode symbols encoded as \uHexHexHexHex in JSON strings are not decoded properly

[lcn2]

Is there an existing issue for this?

• I have searched for existing issues and did not find anything like this

Describe the bug

Certain strings seem to not be decoded properly by the jstrdecode(1) tool, or so it seems.

SUGGESTION: See GH-issuecomment-2350854096 for some updated commentary on this issue.

Some background

Consider this JSON document containing just a JSON string:

"œßåé"

When you enter that JSON document into https://jsonlint.com/ is shows that the JSON is valid.

Moreover, the jparse(1) tool also shows that the JSON document is valid:

$ jparse -J 3 -s '"œßåé"'
JSON tree[3]:	lvl: 0	type: JTYPE_STRING	len{p,c:q}: 8	value:	"\xc5\x93\xc3\x9f\xc3\xa5\xc3\xa9"
in parse_json(): JSON debug[1]: valid JSON

it is GOOD that the JSON parser is happy with this JSON document.

FYI: Here is the hexdump(1) shows about the string (we include the enclosing double quotes and trailing new for consistency with the JSON file):

$ echo '"œßåé"' | hexdump -C
00000000  22 c5 93 c3 9f c3 a5 c3  a9 22 0a                 |"........".|
0000000b

Here is how jstrencode(1) processes the string:

$ jstrencode '"œßåé"'
\"œßåé\"

This is CORRECT as double quotes are back-slashed (i.e., ") is correct because within a JSON string, all double quotes MUST be backslashed.

Now if we use -Q, the enclosing double quotes are ignored:

$ jstrencode -Q '"œßåé"'
œßåé

If we consult https://codebeautify.org/json-encode-online, and we put "œßåé"' on the input side, we see that output side shows the same string.

Evidence of a decoding bug

Now, some JSON tools, such as jsp (see https://github.com/kjozsa/jsp) encodes "œßåé"' as:

$ echo '"œßåé"' | jsp --no-color --indent 4
"\u0153\u00df\u00e5\u00e9"

When we put the "\u0153\u00df\u00e5\u00e9" string into the input side of https://codebeautify.org/json-encode-online, the output side shows "œßåé".

However if we give the "\u0153\u00df\u00e5\u00e9" string to jstrdecode(1), we get something odd:

$ jstrdecode "\u0153\u00df\u00e5\u00e9"
S���

Using hexdump(1) we see that the expected decoded output:

$ echo 'œßåé' | hexdump -C
00000000  c5 93 c3 9f c3 a5 c3 a9  0a                       |.........|
00000009

However this is what jstrdecode(1) produces:

$ jstrdecode "\u0153\u00df\u00e5\u00e9" | hexdump -C
00000000  01 53 df e5 e9 0a                                 |.S....|
00000006

This suggests that the JSON string decoding may be incorrect.

What you expect

We expect that the output of

jstrdecode "\u0153\u00df\u00e5\u00e9"

to be the same as:

echo 'œßåé`

Environment

- OS:SO
- Device:eciveD
- Compiler:relipmoC

bug_report.sh output

n/a

Anything else?

Consider the result of using -v 3:

$ jstrdecode -v 3 "\u0153\u00df\u00e5\u00e9"
debug[1]: enclose in quotes: false
debug[1]: newline output: true
debug[1]: silence warnings: false
debug[1]: processing arg: 0: <\u0153\u00df\u00e5\u00e9>
debug[3]: arg length: 24
debug[3]: decode length: 5
S���

The debug messages look OK, but the output is not:

$ jstrdecode "\u0153\u00df\u00e5\u00e9" | hexdump -C
00000000  01 53 df e5 e9 0a                                 |.S....|
00000006

which is not the same as the expected decoded output:

$ echo 'œßåé' | hexdump -C
00000000  c5 93 c3 9f c3 a5 c3 a9  0a                       |.........|
00000009

So it appears that strings with various \u<hex><hex><hex><hex> might not be decoded correctly.

UPDATE 0C

SUGGESTION: See GH-issuecomment-2350854096 for some updated commentary on this issue.

UPDATE 0D - todo list

It appears that the decoding issue itself is resolved. However a few things have to be done before this can be marked fixed.

• We should make sure that an error or warning is issued when invalid UTF-8 codes are given.
• We need to make sure that the unicode boolean is updated properly.
• We need to fix the test cases in jstr_test.sh that had to be temporarily disabled.

Final todo:

• Clean up the files json_utf8.c and json_utf8.h to include only things we need, making sure to keep credit.

I believe that with those done, as long as things look good, we can mark this as complete.

Side todo:

• Sync this all to mkiocccentry (this can be done before this of course).
• If all is good, update the IOCCC website repo to use jstrdecode instead of jsp.

[lcn2]

How and why we filed 3 bugs reports

We found something that seemed odd regarding JSON string encoding and decoding. It relates to the bin tools from the "Other repo" that we are working on.

While investigating, we discovered the issue with the man page (issue #11). We would have simplex fixed the man page, but while we investigating further we found the issue with the quotes (issue #12).

The encode / decode puzzled us. Stings with wide variety of non-ASCII characters (like "àßç") seemed to be valid within JSON strings without the need to "encode" them as "\u". However other JSON implementations seemed to do just that.

It appears that among the multiple "bogons" of the JSON spec is that there is a "choice": that one can use "é" or one can use "\uc3a9" (because é in UTF-8 is c3 a9). Both are valid ways to refer to the same thing. In fact, one can refer to the ASCII "a" or to it as "\u0061" (ASCII "a" is 0x61). One cannot compare two JSON strings by a simple byte comparison. One MUST decode the JSON string and then compare byte for byte (grown / sigh).

So that is how we started trying to use jstrencode(1) and jstrdecode(1) to play around encoding and decoding. That is how we discovered what seems to be a decode issue (issue #13).

It is a good thing we did discover these issues now. They do block a fix/improvement to how the "other repo" will be able to render web pages that contain non-ASCII characters (such as in an author's name or an award title). Those happen to work because the JSON so-called spec chose to render the symbols. Had someone produced a string using with the equivalent "\u", things would have been broken (at least in terms of how the web pages would have looked).

Anyway we wanted you to know why we were looking into JSON string encoding/decoding, and why we filed these issues.

Now we will return back to the bin tools (we are nearly done, but for these issues and the need for another tool) so then we can return to the submit server, get the screen snapshots and completed one of the TODOs needed for the Great Fork Merge.

[lcn2]

a proposed 2 new options to jstrencode

• In GH-issuecomment-2337677465, we see that there are two ways "represent" a symbol: as the symbol itself and as \u<hex><hex><hex><hex>.

How should the jstrencode(1) and jstrdecode(1) tools deal with these two ways to represent a symbol?

By default, jstrencode(1) would encode symbols as themselves unless they were one of the symbols that must be backslash-escaped (such as the double-quote, the newline, the NUL byte, etc.).

With jstrencode -a, every symbol must be encoded as \u<hex><hex><hex><hex> according to the 2-byte encoded hex value. Here -a stands for "all".

NOTE: With -a, the double-quote, instead of being backslash-escaped as " would he encoded as \u0022 (the double-quote symbol has the value of 0x22). And lower-case a would be encoded as \u0061 (a has the value of 0x61).

With jstrencode -8, any symbol with a value > 127 MUST be encoded as \u<hex><hex><hex><hex> according to the 2-byte encoded hex value. Here -8 stands for "encode if value is >= 2^8".

Of course, the jstrdecode(1) tool will always decode \u<hex><hex><hex><hex> back into the original symbol, so -a and -8 do not apply to that tool.

UPDATE 0

Do we need to encode at 2-byte UTF-8 or ???

On the one hand, ß is UTF-8 c3 9f, so \uc39f.

On the other hand, ß has a byte value of 0xdf, so \u00df. We DO have examples of external applications encoding characters thus way, so at a minimum jstrdecode(1) needs to be able to decode \u00df back to ß.

Which is needed for JSON encoding?

UPDATE 1

According to RFC 4627 Section 3:

3.  Encoding

   JSON text SHALL be encoded in Unicode.  The default encoding is
   UTF-8.

   Since the first two characters of a JSON text will always be ASCII
   characters [RFC0020], it is possible to determine whether an octet
   stream is UTF-8, UTF-16 (BE or LE), or UTF-32 (BE or LE) by looking
   at the pattern of nulls in the first four octets.

           00 00 00 xx  UTF-32BE
           00 xx 00 xx  UTF-16BE
           xx 00 00 00  UTF-32LE
           xx 00 xx 00  UTF-16LE
           xx xx xx xx  UTF-8

Does that provide the vital clue?

[xexyl]

Environment

- OS:SO
- Device:eciveD
- Compiler:relipmoC

:-)

BTW: did you ever notice that the word device is in deceive if you rearrange letters a bit?

[xexyl]

I am waking up more now. The ac might be helping there. Unfortunately I am away from the computer for a little bit.

Your use of Eszett did make me check. It seems that's missing from the UTF-8 map in the mkiocccentry repo.

If I can determine the right \u value I can add it when back but since it was here I might finish the writing of my comment here (that I believe I am almost done writing but I do have a bit more to say when back).

Some of these bug reports can be done easily though with the new options it will of course take more time.

UPDATE 0

Oh! You do have it .. sharp S but you don't have it as ss which it should be. I'll fix that.

UPDATE 1

Eszett fixed in ioccc-src/mkiocccentry@f1cb9e6.

[xexyl]

Okay now back to more serious details.

Caveat

I am very tired so I might be missing something with the thought below. If so I'll either see it later when more alert/awake/clear-headed or maybe what I write will make you think of something.

With that out of the way...

First things first (actually this is the second thing :-) or third if you count the above comment first)

It seems like it might be a good idea to figure out what has to map to what. And since there are so many characters the question I have is if we need a table like struct encode jenc[] found in json_parse.c. But maybe it's more like the table struct utf8_posix_map hmap[] in mkiocccentry's soup/utf8_posix_map.c. Or perhaps it's kind of a combination of the two?

I am not sure how this might go though due to the way the enc table works in particular it being 8-bit values.

But if we need to encode the Eszett (again as per your example) then what should it be encoded to?

We have found external applications that do encode ß as \u00df. So clearly jstrdecode(1) needs to convert \u00df back into ß. So at a minimum this needs to happen.

Will other external applications encode ß as \uc39f, and if so will jstrdecode(1) needs to convert \uc39f back into ß as well?

At a QUICK look at json_parse.c it seems like the reason jparse.c works is due to the function json_conv_string(). The function that calls that is parse_json_string() and it does so like:

str = json_conv_string(string, len, true);

and the boolean indicates:

 *      quote   true ==>  ignore JSON double quotes: both ptr[0] & ptr[len-1]
 *                        must be '"'

Of course it's entirely possible that at this quick glance I am parsing :-) this wrong :-( but I wonder if this might be something to look at. If it is how the parser does it (the parse_json_string() is in jparse.y so it seems likely) then the question is if that's why jparse gets it right. If it does maybe it's something to consider? Do we need (in this case at least?) the json_decode()? I'd say it's not as simple as replacing it given what that function also does but the question is what is different here?

Again this is assuming that I'm not missing something obvious at my quick glance.

UPDATE 0

I was thinking of the reverse step so the above is not going to work even if it was as simple as swapping (with some modifications) it ... which it would not be almost certainly. Even so it's interesting that that function does seem to work okay and the decode one does not.

[xexyl]

Ah .. but of course it's the reverse. The parser does the opposite. So it definitely is not so simple. I wonder if something similar can be done though? I'm not sure. I'm hoping I can read all three bug reports more carefully soon. But I also wonder about the other repo's is_available.sh script we were talking about me writing. I might do that sometime soon. But I do have other things I have to do today too.

[xexyl]

QUESTION - priority wrt the IOCCC

As I have other things I have to do including over at mkiocccentry but also things unrelated to programming, I just wanted to ask what the priority of this issue is versus the IOCCC.

As a programmer, and I'm sure you are very familiar with this, this is an issue that I do not like having, but I also know there is such a thing as prioritising so I am asking this here. I hope to look at the is_available.sh script sometime soon but I have those other things I have to do too so I'm not sure what I'll get done today.

[xexyl]

Another funny thing I noticed is reading from stdin seems wrong. Look:

$ jstrdecode 
foo bar
Warning: json_decode: found non-\-escaped char: 0x0a
Warning: jstrdecode_stream: error while encoding stdin buffer

... when I hit ctrl-d obviously. But look:

$ echo foo bar | jstrdecode 
Warning: json_decode: found non-\-escaped char: 0x0a
Warning: jstrdecode_stream: error while encoding stdin buffer

What am I missing? I don't recall this behaviour at all. In fact in the man page there's an example where this works.

[xexyl]

My thinking is that I should resolve the jstrdecode problems brought up (new option included) and commit and then we can worry about this one.

But I must leave for now possibly for the day.

[xexyl]

My thinking is that I should resolve the jstrdecode problems brought up (new option included) and commit and then we can worry about this one.

Actually I wonder if this should be looked at now .. might require a change in how the above is done. But maybe not. I don't know how to even begin to address it however.

[xexyl]

I'm afraid that that bug has to be fixed first in order to properly test this. Trouble is at this time I'm not sure where to begin with it other than checking that warning message (of course). But what that'll show is another matter.

[xexyl]

The problematic code seems to be in json_parse.c:

            /*
             * disallow characters that should have been escaped
             */
            switch (c) {
            case '\b':  /*fallthrough*/
            case '\t':  /*fallthrough*/
            case '\n':  /*fallthrough*/
            case '\f':  /*fallthrough*/
            case '\r':
                /* error - clear allocated length */
                if (retlen != NULL) {
                    *retlen = 0;
                }
                warn(__func__, "found non-\\-escaped char: 0x%02x", (uint8_t)c);
                return NULL;
                break;

But this is coming from sending EOF.