'autosummary: failed to import' when using numpydoc instead of sphinx.ext.napoleon

[mscheltienne]

I've been playing with doc build for a while now, using this sandbox repository with a simple project and code-base. I am running into an issue a bit similar to #80, I'm at a complete loss and I only managed to narrow it down to 'bad things happen when I used numpydoc on macOS and on Windows'.

When building the documentation on my Linux PC, or on the GitHub CI with this workflow, the build works without any issue. Here is the current version deployed.
When building the documentation locally on my Windows or macOS PC, I'm greeted with 52 similar warnings. You can find 2 of them below:

C:\Users\Mathieu\Documents\git\mscheltienne\simple-stimuli\stimuli\audio\noise.py:docstring of stimuli.audio.noise.Noise:36: WARNING: autosummary: failed to import color.
Possible hints:
* ModuleNotFoundError: No module named 'color'
* AttributeError: module 'stimuli.audio' has no attribute 'color'
* ModuleNotFoundError: No module named 'stimuli.audio.color'
* KeyError: 'color'
* AttributeError: module 'stimuli.audio.Noise' has no attribute 'color'
* ModuleNotFoundError: No module named 'stimuli.audio.Noise.color'; 'stimuli.audio.Noise' is not a package
* ValueError: not enough values to unpack (expected 2, got 1)
C:\Users\Mathieu\Documents\git\mscheltienne\simple-stimuli\stimuli\audio\noise.py:docstring of stimuli.audio.noise.Noise:36: WARNING: autosummary: failed to import duration.
Possible hints:
* ModuleNotFoundError: No module named 'stimuli.audio.duration'
* AttributeError: module 'stimuli.audio' has no attribute 'duration'
* ModuleNotFoundError: No module named 'stimuli.audio.Noise.duration'; 'stimuli.audio.Noise' is not a package
* ModuleNotFoundError: No module named 'duration'
* KeyError: 'duration'
* AttributeError: module 'stimuli.audio.Noise' has no attribute 'duration'
* ValueError: not enough values to unpack (expected 2, got 1)

And obviously, the documentation is a bit broken. Based on the pasted warnings, take the Noise class defined in noise.py: Left -> local (windows). Right -> CI (ubuntu). Both build on the same commit,

---

Why is numpydoc involved?

It turns out that if I remove numpydoc from the extensions and load sphinx.ext.napoleon instead, the doc builds without warnings (except the missing x-ref since they are not defined anymore). However, the render is missing the summary tables, so at least it makes sense that warnings about those tables are missing:

And the attributes and method tables that are failing are a feature of numpydoc, right?

In both cases, the generated .rst is the same:

stimuli.audio.Noise
===================

.. currentmodule:: stimuli.audio

.. autoclass:: Noise
   :members:
   :inherited-members:

I tried with different templates, and with the default templates, but nothing changed.

---

I'm at a complete loss as to why the doc build is failing on macOS or Windows, and not on Linux, I'm not even sure it's a bug, or an issue with my configuration. Here is a couple of lines to set-up the project if you want to give it a try:

git clone https://github.com/mscheltienne/simple-stimuli
python3 -m venv test --copies
source test/bin/activate
python3 -m pip install --upgrade pip setuptools wheel
cd simple-stimuli
python3 -m pip install -e ".[doc]"
cd doc
make html

[rossbar]

There's a lot here so it's difficult to say anything concrete.

Given the build warnings you've posted at the top and the fact that your problem is platform specific, my best guess is that there is some sort of environment management/install problem. The warnings make it look like the stimuli package isn't fully installed, at least not correctly, in the environment where you got the warnings. This of course doesn't explain why there's a difference between using napoleon/numpydoc, but perhaps you installed them in different envs? Or ran/re-ran the set of commands with one installed and not the other? In other words if I were trying to debug this I'd start by ensuring that the environment/package installation is 100% correct and behaving exactly the way I expect. The test procedure outlined at the end of your post looks fine to me... at least there's nothing that jumps out to me as potentially problematic, so I really don't know 🤷

Sorry this isn't that helpful "/ - there's only so much one can do when the failing example is so non-specific & doesn't reproduce on all platforms.

[mscheltienne]

Hello @rossbar! Thank you for looking into it. I've played with it more after the initial post, and I figured out a couple of additional points.

1. Installation issue in the environment: that's not it, the github doc-build CI fails in the same way as it does locally if it does not run on Ubuntu. And the doc-build also works locally on a Linux computer. In each case, I install the package in the exact same way.

2. Napoleon vs Numpydoc: The second adds an autosummary-like table for the attributes and methods. See the screenshots above. This element is failing, and since it is absent in Napoleon, the latter works.

3. Following the last point, I've set numpydoc_attributes_as_param_list = sys.platform != "linux" since at least this way it can build on all platforms (but with a different render..)

I've written those bullet points from memory, from the experimentations I did a couple of months ago on this sandbox repository. So maybe some version changes in sphinx, numpydoc, and furo might have fixed the issue by now. I'll try to make the time to test this again!

[mbaudin47]

Is there any link to #204 ?

[mbaudin47]

I found the configuration which makes the bug appear or disappear. Consider the following directory structure.

|
+- packagename
   |
   +- __init__.py
   +- _A.py
   +- B.py
+- doc
   |
   +- conf.py
   +- index.rst
   +- Makefile
   +- user_manual
      +- user_manual.rst

This is the content of the next archive file :
test_sphinx.tar.gz

The help page for A is OK:

But the help page for B is wrong. We can see that the "methods" summary is empty:

The files _A.py and B.py have the same content. The only difference is that the class A is located in the file "_A.py" (this is OK) while the class B is in the file "B.py" (this fails). In summary, the bug is that the "Methods" section is not correctly produced by Sphinx in some cases.

I hope that this may help to find the bug.

[mbaudin47]

@jschueller: From what I can see in the file _AxialStressedBeamReliability.py, you also found this bug, isn'it?

[jschueller]

yes, the error message is a bit different though:

test_sphinx]$ PYTHONPATH=$PWD make html -C doc
make: Entering directory '/tmp/test_sphinx/doc'
set -xe
sphinx-build -b html -d build/doctrees   . build/html
Running Sphinx v8.1.3
loading translations [python]... not available for built-in messages
Converting `source_suffix = ['.rst']` to `source_suffix = {'.rst': 'restructuredtext'}`.
loading pickled environment... done
[autosummary] generating autosummary for: index.rst, user_manual/_generated/packagename.A.rst, user_manual/_generated/packagename.B.rst, user_manual/user_manual.rst
building [mo]: targets for 0 po files that are out of date
writing output... 
building [html]: targets for 0 source files that are out of date
updating environment: 0 added, 1 changed, 0 removed
reading sources... [100%] user_manual/_generated/packagename.B
/tmp/test_sphinx/packagename/B.py:docstring of packagename.B.B:9: WARNING: autosummary: failed to import get_name.
Possible hints:
* AttributeError: module 'packagename.B' has no attribute 'get_name'
* ValueError: not enough values to unpack (expected 2, got 1)
* ModuleNotFoundError: No module named 'packagename.get_name'
* ModuleNotFoundError: No module named 'packagename.B.get_name'; 'packagename.B' is not a package
* ModuleNotFoundError: No module named 'get_name'
* KeyError: 'get_name'
* AttributeError: module 'packagename' has no attribute 'get_name'
/tmp/test_sphinx/packagename/B.py:docstring of packagename.B.B:9: WARNING: autosummary: failed to import get_x.
Possible hints:
* AttributeError: module 'packagename' has no attribute 'get_x'
* ValueError: not enough values to unpack (expected 2, got 1)
* ModuleNotFoundError: No module named 'packagename.get_x'
* KeyError: 'get_x'
* AttributeError: module 'packagename.B' has no attribute 'get_x'
* ModuleNotFoundError: No module named 'get_x'
* ModuleNotFoundError: No module named 'packagename.B.get_x'; 'packagename.B' is not a package
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
copying assets... 
copying static files... 
Writing evaluated template result to /tmp/test_sphinx/doc/build/html/_static/basic.css
Writing evaluated template result to /tmp/test_sphinx/doc/build/html/_static/documentation_options.js
Writing evaluated template result to /tmp/test_sphinx/doc/build/html/_static/language_data.js
Writing evaluated template result to /tmp/test_sphinx/doc/build/html/_static/alabaster.css
copying static files: done
copying extra files... 
copying extra files: done
copying assets: done
writing output... [100%] user_manual/user_manual
generating indices... genindex done
writing additional pages... search done
dumping search index in English (code: en)... done
dumping object inventory... done
build succeeded, 2 warnings.

The HTML pages are in build/html.

Build finished. The HTML pages are in build/html.
make: Leaving directory '/tmp/test_sphinx/doc'

[jschueller]

maybe numpydoc is confused when the package/class/method have the same name ?

[mscheltienne]

maybe numpydoc is confused when the package/class/method have the same name ?

That does ring the bell, I think the class should not have the name of the module in which it is defined.

[mbaudin47]

maybe numpydoc is confused when the package/class/method have the same name ?

That does ring the bell, I think the class should not have the name of the module in which it is defined.

But there is no reason to require such a constraint, is it?