The structure of a stubs installation

[aivarannamaa]

First, thanks for this effort!

I'm curious about the structure of the files and folders in the installation target directory.

Let's say I do pip3 install micropython-rp2-rpi_pico_w-stubs --no-user --target ./typings

I see that typings/os.pyi is a small file describing a subset of CPython's os module, but there's also stdlib/os, which seems to describe CPython's os.

• Why are there two variants of os-stubs?
• Is stdlib/os meant to be activated via typeshedPaths Pyright setting and os.pyi via stubsPath setting?
• If two copies (in different folders) are required, then have you considered keeping only MicroPython members in stdlib/os stubs? I understand that extra members don't harm diagnostics, but they do obstruct discovering available members during code completion.

[Josverl]

First of all,
Thank you for sparking the initial idea that I found as part of Thonny
That was the first usable piece of information I found on how to overcome the too limited documentation,
and which set me on this path.

on the stub structure - ive learning and hacking workarounds as I built them over the years,
also the typing ecosystem is evolving and changing, so there is a bit of history involved.

model 1

the stubs were still .py files , which makes the type checkers work differently ( as I found out much later)

model 2

each stub had its own copy of `os
but that copy was ignored by the type checkers

model 3

I introduced this with the micropython-stdlib-stubs as before that the typecheckers just ignored the stubs for any of the stdlib librarries,
for instance time.time.sleep_us was nor recognized,
I added a manual modified copy of stdlib/os.pyi to micropython-stdlib-stubs

I added a hack to not need to reverse engineer / rewrite the _typeshed stubs for stdlib :

• I kept the 'os.pyi' in the normal stubs, but start with the from stdlib.os import *
• then I overwrite any additional methods in the stub level

the hack works somewhat - but not for all typecheckers and linters.
its a OK-ish solution for only the simple things - but breaks when i try to make it do more.

model 4

and os and sys appeared to have way to many methods and properties that are not in micropython.
so I created a stdlib update script to

• create a copy of typeshed
• tune and optimize stdlib modules to be close(r) to micropython
• remove a lot of the not supported modules and sub-modules

Model 5

Im currently working on an update to:

• include additional type information for parameters and return that is not included in the documentation in a machine readable manner
• for this I will need to add a micropython typeshed that is currenlty located at : https://github.com/Josverl/micropython-stubs/tree/main/micropython-reference/_mpy_shed
  or latest
• with this , and the stubs bases on this method I should no longer need the 2-levels of os and stdlib.os

This is still a work in progress , and I had hoped to complete it for the 1.24.x release of micropython - but Im still working through some issues in auto generating stubs in this manner.

I hope to have it ready in the next few weeks, but creating code that can merge source files in a clear and consistent way is tens to turn up new surprises quite frequently.

Model 6 - Structure of a stub package ( todo)

Then there is one other thing, apparently I misread PEP 561 ( or it was written to be understood by people that already understood it )
and the names naming of stubs and packages changes during my initial writing.
Ive not yet decided if and when I want to take that on. but perhaps if I see clear advantages of this work.

#760

[aivarannamaa]

It looks like I underestimated the complexity of the problem. I had only focused on Pyright, but now I see that MyPy has its own requirements about typeshed and also uses different import resolution order. I made some experiments but ended up more confused than before: https://discuss.python.org/t/foo-vs-foo-stubs-with-mypypath/77314

I suppose you have already tried it, but out of curiosity -- what would happen if you replaced the content of stdlib/os/init.pyi (the one containing extra methods) with the content of os.pyi?

[Josverl]

No, I dont think I've tried that yet.
I have not found a way to get mypy to use a modified stdlib at all, but it might be worth a shot.

In order to getypy to work at all I need to

del typings/os.pyi 
del typings/sys.pyi

So I do that even in testing

[Josverl]

I've been working on improving the mypy support as it had problems with the MicroPython-stdlib.
See #781 for details.

I plan to publish these in the next week, only for V1.24.1 and newer.

Part of that work was to remove most of the duplicates between modules such as typings/foo and typings/stdlib/foo

I need to check if there is additional duplication I can remove.

HTH