coverage collection fails when using pytest-xdist and pytest 8.4.0 #693
Comments
rjdbcm
changed the title
|
Thanks, was tracking this down on my own side too. I use a combination of |
|
interesting, all of my test setups are similar, will check if taking asyncio out of the equation changes anything |
|
xdist + cov results in a module-not-measured warning. this was also the case in 8.3.0 in pyproject.toml (or whereever you configure pytest-cov) silences the warning and makes the tests pass again. we still get good coverage reports out of it. |
|
After upgrading to pytest 8.4.0, our test suite also started failing, though with a different traceback. @fzimmermann89's solution of Platform: macOS ARM64 (also fails in containerized Linux x86 CI) |
|
So that's why my jobs started failing suddenly! I was just about to report the same… No The reason for failures is that we have @johnthagen you have a different underlying error (a corrupted SQLite file instead of an unmeasured module), so it'd have a different code to suppress (if cc @nedbat @ionelmc need some 👀 on this, to understand the underlying cause of Should the warnings be suppressed around https://github.com/pytest-dev/pytest-cov/blob/9463242/src/pytest_cov/engine.py#L469? |
|
Having #675 flashbacks, I'm curious whether it'd make sense for |
|
I also suddenly started having this problem, and noticed the same set of interactions: pytest 8.4, pytest-cov, pytest-xdist (versions don't seem to matter). Then Adding I do not have Pinning I have no idea why the module not measured warning is being raised -- I had noticed this earlier (and tried and failed to fix it), but I seem to get proper coverage outputs regardless. I assume it has something to do with the interaction between EDIT: Okay, adding the suggested |
|
@billbrod only warnings in this list have codes that can be suppressed through the coverage config: https://coverage.readthedocs.io/en/latest/cmd.html#warnings A workaround on the runtime level would be adding an I'm sure if you were to reveal the warnings with older pytests, they'd show up in the log. P.S. I've also noticed in the past that this warning was being printed out while coverage was actually collected. So the assumption that this could be related to |
|
I'm not certain this is I'm looking into amending the configs to silence this issue as suggested above, but also wonder if this could have been picked up in CI before 8.4 release time if there was a job testing with the pytest dev version? |
|
This seems to apply to all warnings. You would have to adjust what warning to ignore or fix the root cause. |
|
@bsipocz do any of your plugins run subprocesses or threads? There was something in the pytest change log about resurfacing exceptions in threads, I think. My working theory is that it's one of the previously unraisable cases. |
To address [coverage collection fails when using pytest-xdist and pytest 8.4.0](pytest-dev/pytest-cov#693), disable the no-data-collected warning.
|
@webknjaz - not that I know of, but now forcing it to use the older pytest, I see the CoverageWarning of module-not-measures. It makes little sense as it complains about the package in question, but nevertheless I would have expected this warning to be raised as exception rather than hidden in the log of a passing job. |
|
https://pytest-cov.readthedocs.io/en/latest/plugins.html claims that using |
|
FWIW, the explicit ignore of that one warning works, so I emerge from the rabbit hole of figuring out which plugin combination is responsible. |
Yeah, I've been using this same workaround whenever there were problems with |
I quickly verified that copying a few of those private ignores does work locally. |
FWIW this didn't work for me, I still get the warnings with |
|
@billbrod with env vars set along with the CLI flag? |
|
Seems to me like all the breakages have in common is some form of "filterwarnings = error". When warnings were added in pytest-cov (and I believe coverage as well) nobody thought that this "filterwarnings = error" pattern would be so prevalent, or dare I say, so abused. The intention was we add some warning to help the user figure out if their project setup is fine or needs some attention in certain areas (like help the user figure out that the code they want to cover is ran before coverage starts, and that can't be measured). The way I see it "filterwarnings = error" is for temporary or situational usage. You want to upgrade django and want to figure out easy what exactly triggers your warning? You use "filterwarnings = error" cause stacklevel is rarely that helpful unless you know exactly where to look. |
I would very strongly push back on this notion, it's far from being abuse, instead it is best practice. Flipping the warnings to errors in CI is part of the best practices a lot of libraries follow, that would allow us maintainers to pick up deprecation and other changes in behaviour. This follows the notion that warnings have the purpose of informing the users about things they can do about something (otherwise it's just noise), and therefore all warnings should be handled appropriately in the test suite. And once that is done, we can flip the switch and raise the remaining (==new) ones as errors to notice them in CI. (one can use the logger to give verbose advise and notices about stuff the user can improve, and keep the warnings for actual warnings) |
|
I was just about to post the same. I think that pytest itself was one of the places preaching it. To me, it's a fundamental principle even. |
|
@billbrod I meant not one or the other but both. |
One or the other, or both, behavior is the same. To make sure I'm being clear: changing both |
|
On the topic of |
UPD: with a little bit of experimentation, I'm now convinced that the I moved those hacks from the master to the worker class (in my tox's This is something that I've been thinking through since yesterday and so far, it seems like a reliable solution to the bug. |
|
I made a oneliner PR that would disable |
It's not that old. I think it only became really popular circa 2022: https://til.simonwillison.net/pytest/treat-warnings-as-errors
It may be best practice now, but it's still an abuse of the warnings system. If I wanted to raise exceptions all the time from pytest-cov I'd just raise it. You need to understand that This breakage only occurs because an extremely strict But alas, pytest-cov is a convenience tool full of workarounds anyway, so it's not unreasonable to have this pattern supported somehow. What can we do here? The way I see it there are few ways to resolve this issue:
I have created a reproducing test in https://github.com/pytest-dev/pytest-cov/pull/696/files#diff-462f05b103e164981d6272e222b8579cf45f6ba30f8b1d76b2e00909e09ecd91 Which do you prefer, or do you see a different solution? |
@ionelmc so this reproducer seems to trigger the outer layer of the problem I'm having, not the underlying root cause. I'd be perfectly fine not disabling the And with that, the overall experience is that the warning is misleading, causing the end-users to second-guess themselves. I'd argue that as long as it's unreliable in such a setup, it should be disabled by the plugin, not by the users. The I wish So let's agree that practicality beats purity and go for the option 1. |
To explicitly address this, there could be another flag to display this or it could be made conditional on |
They are only surfaced under pytest 8.4, with `pytest-cov` and `pytest-xdist` being both active [[1]]. [1]: pytest-dev/pytest-cov#693
|
So I am trying to figure out if the original issue reported here by @rjdbcm is indeed caused by a warning that should not be issued. I have looked at the tox config and I can't figure out how pytest is ran from that. Also, https://github.com/OZI-Project/OZI doesn't seem to have a tox.ini so I am not sure what to run to reproduce this. |
|
Fwiw, I think the warning I see is also spurious, it warns about the package I'm testing (I suspect the other plugins do an import early and that confuses the cov plugin) |
Uh oh!
There was an error while loading. Please reload this page.
Summary
pytest-xdist and pytest-cov plugins cause an internal error when used together but only on pytest version 8.4.0
Expected vs actual result
Expected pytest-xdist+pytest-cov to succeed.
See the following chart for the actual results:
with the following output:
Reproducer
seems to fail for any test I try given the above conditions
Versions
platform linux -- Python 3.13.3, pytest-8.4.0, pluggy-1.6.0Config
tox config
pytest config
My failing test
The text was updated successfully, but these errors were encountered: