Use Lua to Python to handle templates ?

[lasconic]

Following up on : #2504 (comment)

I tried to render the place template from https://en.wiktionary.org/wiki/FYROM with WTP.

{{place|en|@acronym of:former Yugoslav Republic of Macedonia|@former long form of:North Macedonia|country|r/Southeast Europe|*on the [[Balkan Peninsula]]}}

result :

<span class='form-of-definition use-with-mention'>[[Appendix:Glossary#acronym|Acronym]] of <span class='form-of-definition-link'><i class="Latn mention" lang="en">[[:former Yugoslav Republic of Macedonia#English|former Yugoslav Republic of Macedonia]]</i></span></span>, <span class='form-of-definition use-with-mention'>former long form of <span class='form-of-definition-link'><i class="Latn mention" lang="en">[[:North Macedonia#English|North Macedonia]]</i></span></span>: a [[country]] in <span class="Latn" lang="en">[[:Southeast Europe#English|Southeast Europe]]</span> on the [[Balkan Peninsula]][[Category:English acronyms|WORD]][[Category:en:Abbreviations of countries|WORD]][[Category:en:Former long-form names of countries|WORD]][[Category:en:North Macedonia|WORD]][[Category:en:Countries in Europe|WORD]]

We will need to do some work before we process the template and probably after to remove Category for example.

@BoboTiG Do you already have code to read the DB integrated in reader-dict? How do you to deal with the WTP dependency ?

[BoboTiG]

What do you mean by DB?

[lasconic]

WTP uses a database (DB) to store the parsed dump.

[BoboTiG]

Ah, I did not use WTP at the end. I was trying to run lupa from scratch.
WTP seems nice, but it handles only a part of the locales we support (plus, I need to think about multilingual support when doing such a move).

[BoboTiG]

FTR I have this working code to save module content to specifc files: 8c0d425

I skipped templates content since we still need to handle them via python. It could be improved later maybe, but to "start", module is the core we need.

[lasconic]

I have a working prototype to use Lua to render the "place" template in english. It's hacky but here is what I found.

1. Using Lupa alone means we need to reinvent a lot of what Wtp is already doing, so I just went with Wtp core
2. As far as I understand Wtp is language agnostic so it would work for all the languages we support (including multi if necessary)
3. Testing is hard... Since we don't have the dump during the tests we don't have the lua code to render the template. I can't really think of a good way to handle this problem. We could add the current version of the code to the repo for testing purpose, including the dependencies but it seems ugly.
4. We do need templates. For "place", we need at least the place template. If we want to use the same mechanism for other templates, we will need them and all the templates they use.

See master...lasconic:ebook-reader-dict:lua_parse

[BoboTiG]

1. I feel you.
2. That looks promising!
3. That will be a pain to handle, do not bother with that now, I'll see later.
4. Noted.

Do you plan to touch your code more? I'll do more tests, and iterate on your code then.

[lasconic]

No, I don't plan to touch it today. As you see, render_utils.py is not great. Passing wtp objets down to the template handlers is not great either. I look forward to reading your approach :)

More notes:

1. namespace_ids are important when reading templates/modules. With namespace_ids to 0, it doesn't work. For now they are hardcoded.
2. we don't handle template redirection while wtp's dump_parser does. It could come back to bite us later.

[BoboTiG]

I just tested a variation of your code on the entire Danish dictionary (because that's the smallest, but it still contains thousands of modules/templates). Zero errors. Only redirections remain to be handled.

I will continue on my end, do not alter your code since I changed too many lines and recovering the conflict will be a mess.

EDIT: redirections are working now. I move to variants support.

[BoboTiG]

Result for the Danish dictionary:

• errors: 0
• warnings: 0
• redirections: OK
• variants support: OK
• total templates count written in Python: 0
• word count before/after: 12,050 / 12,238

For now, I see only good stuff. Looking at the file generated by --render, all seems excellent.

I'll move to the French dictionary to battle test the code ♨️

I also need to double check those features are still supported:

• chemical/math formulas rendering
• hieroglyphs
• tables

[lasconic]

What does that mean ?

total templates count written in Python: 0

You replaced all the previous template handlers with call to wtp ?

[BoboTiG]

You replaced all the previous template handlers with call to wtp ?

Yes, as a test first. And everything seems to pass. The only templates kept are for variants support. But this is not perfect, we also loose the filtering pre-check for ignored templates, for instance.
All in all, we will be able to do a massive code clean-up on our side when we will be ready for the move.

I am currently hitting errors within the Lua engine on the FR dict: too many calls. I am also concerned about the thread-safety.

Well, this is not yet ready, I'll share more info soon^™️