LibTidy does not support all entities! It should!!

[geoffmcl]

At present libtidy, in entities.c, supports only about some 253, case sensitive, entities... it uses TY_(tmbstrcmp) in the entitiesLookup static function...

Yet W3C pages like these - https://dev.w3.org/html5/html-author/charref - or - https://www.w3.org/TR/xml-entity-names/byalpha.html - and probably others, indicate in excess of 2,000 entities, some like &, &, ... in multiple case forms... not yet exactly counted, and there may be more, yet to be fully explored...

Here is a live web page - https://support.apple.com/en-us/HT202021 - containing the likes of 	, 
, pointed out to me by Karl Dahlke, of the edbrowse project, which uses libtidy to load and view web pages - thanks Karl...

So at present, edbrowse sees warnings about unknown entities, and unless preserve-entities: yes is added to the config, they will be rendered/returned as text &Tab, &NewLine, along with the warnings... not good!

I propose libtidy support all known entities, and seek feedback, and more, other W3C references, on this... thanks...

[geoffmcl]

Have push a new branch issue-643 which contains an entities.c, with some 2031 entities, scraped from the charref page with some quick perl...

Now processing that downloaded web page HT202021 using my new tidy 5.5.76.I643, and adding --drop-empty-elements no, the warnings fell from 104 to just 18, and of course none of the warnings were about unknown entities.

And the test file pushed to my repo, in_643.html, goes from 16 warnings, including 2 spurious missing <li>, to clean...

As noted in the source, I have removed the column alignment to keep the total size down to 80K, rather than the nearly 150K for adding sort of meaningless alignment. This is of course much larger than the original 13K.

I hope this is kept sane... it does break a little from tidy's usual good code, especially table alignment...

And as also noted in the source, I have added some more small optimization steps to the entitiesLookup service which depend upon the entities table being kept in alphabetic order, and think this should be fast enough for now, rather than adding to the lib setup time by creating some type of hash.

Also previously the entities were grouped, but not sure why... did not seem necessary...

Also I do not specifically know the html version that should be used on these new entities.

For the 253 that existed, I used the version they had. But for all the new have used VERS_CHECK, which I define as VERS_ALL. But if some of these entities do really belong to specific html versions, then this should be replaced as appropriate...

As noted above, I have only used those in the charref W3 page, and not cross referenced this with other lists of entities... but it is now certainly sufficient to address this issue... but maybe there are even more...

If others, especially edbrowse, could checkout this issue-643 branch and test, this could be merged to next, maybe in time for the upcoming 5.6 release, hopefully later this month... thanks...

[geoffmcl]

Branch issue-643 now re based to next to add the fixed header install location, see issue #646, PR #647 ...

Testers must do a pull to update their local repo, issue-643 branch, for further testing, and should make sure any installs to include/tidy are deleted... thanks...

[balthisar]

@geoffmcl, nice work. I've added some comments over in the PR.

[geoffmcl]

@balthisar thanks for your comment in the PR #644...

I have now had a chance to run my test sample in_643.html through the W3C validator, and it passes.

So then I added a in_643-1.html, using a legacy 4.01 doctype, and it FAILED...

So I think you are right, maybe VERS_CHECK should be set to only html5, or something, until we know better... but...

I do not think a browser test is a valid test of this...

If I was a browser writer, I think I would simply support all known entities, and not even read, or change anything, according to the doctype... they are not in the business of caring about validation... just render the document as well as possible... why not support all for any doctype...

The wikpedia citation adds a Standard column which seems to clearly set a html version for each, but then does not seem to include the two that started this issue, namely 	 and 
...

And that article cites various other references, like html40, which does indeed suggest many of these entities were allowed in html4 documents, in that case... Then another reference, html32 which again shows a list greatly expanded on the some 252? only that were in tidy...

So I do think my defining VERS_CHECK to equal VERS_ALL is not correct, and suggest at this time it be defined as VERS_FROM40, which I think would get many of them nearly right...

Until such time someone is willing to take the time and big effort and set the actual version for each... and then, where possible validate the choice... not a simple task...

And I have not yet had the chance to run the regression tests on this... but can understand there may be some changes... thanks for starting to look at this...

Of course, Karl, of edbrowse, has reported this issue-643 branch, as is, works for them... but again they just want libtidy to parse the document, and probably do not have much interest in the doctype...

What do you, and others, think about using VERS_FROM40 for now? Thanks...

[geoffmcl]

Further exploration and understandings...

First, I implied the html 3 and 4 versions had a considerably enhanced list of entities, but seems I did not exactly count and compare them, which I have now done, and no!, it seems up to html 4 there were just some 253 known entities, which tidy has always supported...

And the version indicated in the wikpedia page largely agrees with what tidy currently uses for these... still to tie down the minimal differences, and try to decide which is right, tidy or wikipedia...

It was only in html 5 that the list jumped upwards to about 2031 entities, so those not already in tidy should all be marked as VERS_HTML5 only. These came from the charref file...

Then I found this slightly bigger html 5 list, which is available as a json file... and note the change from calling them entities, to Named character references... wonder why, but seems ok...

This adds about an additional 94, and the only reason I can see that these did not make it into the charref list is that most, maybe all?, have more than one codepoint, namely 2 different codepoints!.

So to add these to the tidy supported list, we can only add one or the other value, since our table can not have 2 values, and even if the 2 were added separately, our search and lookup services would only ever find the first value, so maybe no point... but this has not been fully checked, and to decide which is the best value if using only one...

Also, I have found a small number of cases where the various tables do not agree on even a single codepoint... Need to dig deeper here to see if I can discover the reason, and which is correct! Two known are entity lang, code 9001 vs 10216, and rang, code 9002 vs 10217... there are about 10 or so, but that seems a separate issue, after we have the bulk of the support included...

Anyway, at this moment working on adding these 94 or so, and as indicated defining VERS_CHECK to be VERS_HTML5, but also still to explore XML versions...

Also, at the same time as generating the C table structure for tidy, also generated some HTML, and it seems my chrome browser display a glyph for very nearly all - great to see... seems independent of doctype... will get around to pushing this/these somewhere...

So moving this forward slowly, and hopefully carefully... further feedback very welcome... thanks...

[geoffmcl]

Wow, it is really interesting what you discover by outputting a little HTML ;=))

I can now see why the 94, with what I thought had two numeric values, are not in charref! It is not two values, but an entity that sort of combines two values...

Take the first, &NotEqualTilde. It is a single glyph, sort of combining two 'glyphs', value ≂, with ̸, that overlay each other, so like tidy, charref has no current way to represent this as a single value!

That is they are entities or named character references that do not have a single numeric value...

Have to give some thought about how to support this in tidy... feedback welcome... thanks...

[geoffmcl]

Current Tidy

If you want to view the 253 entities currently supported by Tidy, an uploaded html display is here, temptidyents.html.

And have uploaded a number of other temp html pages... see below...

Two value entities

Still pondering how to make tidy support this 94, or maybe 93, entities that consist of two(2) values. Not sure what happened to the 94th? Maybe I miss counted before...

You can view these 93 here, tempmultents.html...

You can see how each is actually a combined glyph, made up of 2 values... sort of overlaying each other, at least in most cases...

And the last column shows that if you add the two values one after another they sort of work as a sort a surrogate pair, and do seem to produce the same single glyph.

That is, for example ≂̸ looks the same as ≂̸, so maybe this gives a clue on how they could be supported in tidy...

As stated still pondering, exploring, this...

Comparison of tables

Have now looked deeper into the 9 differences, plus 1 missed from first, and 2 missed from the second.

A html visual is uploaded here, tempdiffents.html. Some are slight variations, and some appear the same, or equivalent...

This is from a comparison of the 2031 entities from charref page, [reference]((https://dev.w3.org/html5/html-author/charref), and the 2032 entities from the json page, reference...

In that table I show the entity, then each value as a numeric entity. So, for example, the first, ‾ shows a glyph ‾, and ¯ shows ¯, and &#8254 shows ‾, if these appear ok in this github markdown. So in this case the second value 8254 appears more like what the entity shows, so I guess that should be the value... and it seems other choices of the best value can be made...

And others like Ω looks to really have two possible values, 8486 or 937. How to choose between these?

Some conclusions

Perhaps need a second table to support the 93 that are like surrogate pairs... seek ideas, feedback on these... maybe could even be a separate issue... thanks...

It does seem the branch issue-643, taken from the charref page is very close... perhaps 2 or 3 could be adjusted, and 1 added, considering the above difference display, but is already largely fully correct, for all the single value entities...

So, aside from redefining VERS_CHECK to VERS_HTML5, seems PR #644 is good to go, and will merge this shortly... comments welcome... thanks...

[balthisar]

Wow, @geoffmcl, this has been a lot of work.

In cases where they aren't acting in concert with each other, or maybe in all cases, I'd suggest the values from (the W3 reference)[https://dev.w3.org/html5/html-author/charref], at least where they look identical.

As for how they should be internally represented, I'm not certain how Tidy handles this internally. If it encounters a named entity, is it converted to a byte? Or is it just passed through as text? I suppose that might be dependent upon output-encoding and numeric-entities, and maybe ncr, although in the latter case going from NCR to entity should be easy.

In any case, I'm in favor of the merge as-is, deferring the answers until later.

[geoffmcl]

@balthisar seems we sort of cross posted...

After visually examining the value displays carefully, have now chosen one value over another... and made some other changes, after carefully comparing charref and the json, so think I have now answered most of my own questions...

Presently headed out for a while, but will write this up more on my return later... certainly now consider the PR, with 2032 known, single value, entities very complete and accurate...

This of course does not yet address the 94 - yes found the other one I think - which are like surrogate pairs... anyway, more later...

[geoffmcl]

@balthisar as suggested above I have now visually examined all the differences between the two references, charref and json. They are the SAME for some 2021 entities, and as mentioned the differences are enumerated here tempdiffents.html.

Note in the last two the Name and G columns have been correct! That was just a code error in the perl script, tidyentities.pl, used to generate that page... it previously erroniously showed ‌ for both, but their correct names are ⟈ and ⟉ as now shown...

That visual examination led me to the 12 changes made in commit 54f14ff. 9 changes in value, removal of 1, ∽̱ since it is one of the 93 special pairs, and the addition of 2, ⟈ and ⟉, for some reason strangely missing from the charref reference.

For most of the changed values, the choice is simple, except perhaps Å although the glyph for 197 does look slightly better?, and Ω, which does seem to have 2 exactly equivalent values.

These changes were also effected in the perl script, except for the additions, so now the difference between the list generated for the charref reference, tempents1.c.html, and the list generated for the json reference, tempents12.c.html is nothing, except for the 2 missed, namely tempdiffents1.html.

In other words, tidy's entities.c agrees 100% to the json reference, except for the 93 - yes, back to 93 since the extra one I thought I had found, race was already there - see tempmultents1.html, which is unchanged from the previous list...

I too have yet to examine all this, at least for the entities with a single numeric value, in libtidy code terms, that is how Tidy handles this internally...

As you suggest, I think when an alphabetic, or ascii entity is encountered, it is looked up, and if found, only the codepoint is stored in the lexer - I think! Of couse, if it is numeric, either as a decimal, or hex value, then again the codepoint is stored...

Then we have quite a set of entity options, MarkupEntitiesReference. Namely, with their default - ascii-chars no, ncr yes, numeric-entities no, preserve-entities no, quote-ampersand yes, quote-marks no, quote-nbsp yes, and hopefully no others...

So I am still to explore how each of these options effect, either the input and decode, or the output. I certainly feel some of the descriptions leave a lot to be desired...

But the extension of this entities list does effect two regression tests, namely 412 and 437468, that you mentioned...

Was really puzzled by 412 until I dumped it and find the - in the title is E2 80 93. Current libtidy converts that to E2 26 23 38 33 36 34 3B 26 23 38 32 32 30 3B, ie 0xE2€“, which seems meaningless!?!?.

And the new tidy I643 finds the value 8220 matches “ so substitues that... If it searched more it would find 8220 also matches “, but the reverse lookup will always return just the first found...

Still trying to get my head around what is happening here???

The document/doctype is html 5, so utf-8 assumed. E2 80 93 seems a valid utf-8 sequence, at least my chkutf8 app says so...

Why do both current and new tidy decide to output numeric entities, especially after leaving the E2 there??? This is CRAZY!!! This seems a bug, but maybe unrelated to this entities issue...

The test 437468 seems simpler, but maybe not

Inputs of “ and ” were output in new tidy as “ and ”, in a HTML 4.01 document/doctype...

Hmmm, not particularly wrong, but why, why, why??? And why were not “ and ” found? Some questions here...

While I am not particularly sure either of these regression test changes neccessarily call into question the PR #644, there does seem some questions to answer...

What are your further thoughts, or the comments of others... thanks...