[1.0.4 (and older)] File `tests/conftest.py` is arguably missing from source tarballs?

[hartwork]

Hi!

I just noticed that running the test from the source tarball hashin-1.0.4.tar.gz (but also hashin-1.0.3.tar.gz) a la …

cd "$(mktemp -d)"
wget https://files.pythonhosted.org/packages/04/04/9061632da906161ebbc6ebf326267e5be2ce135552666c555272ee07806f/hashin-1.0.4.tar.gz
tar xf hashin-1.0.4.tar.gz 
cd hashin-1.0.4/
python3.13 -m venv venv
source venv/bin/activate
pip3 install -e .
PYTHONPATH=. pytest -v

…fails with error …

________________________________________________________________________________ ERROR at setup of test_get_latest_version_simple ________________________________________________________________________________
file /tmp/tmp.nul9E8JlHf/hashin-1.0.4/tests/test_cli.py, line 33
  def test_get_latest_version_simple(murlopen):
E       fixture 'murlopen' not found
>       available fixtures: _session_event_loop, anyio_backend, anyio_backend_name, anyio_backend_options, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, event_loop, event_loop_policy, git_repo, make_git_repo, make_repo, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, repo, tests/test_cli.py::<event_loop>, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory, unused_tcp_port, unused_tcp_port_factory, unused_udp_port, unused_udp_port_factory
>       use 'pytest --fixtures [testpath]' for help on them.

/tmp/tmp.nul9E8JlHf/hashin-1.0.4/tests/test_cli.py:33

…due to missing file tests/conftest.py.

Probably the source tarball should either (a) ship all files needed to run the tests or (b) not ship the tests at all (and then distros need to use GitHub-generated Git archives in packaging as a consequence rather than PyPI). @peterbe what do you think?

Best, Sebastian

[peterbe]

Probably the source tarball should either (a) ship all files needed to run the tests or (b) not ship the tests at all

It's been a while since I've worried about this and I appreciate that there are different schools of thought.

My gut tells; Why run tests on distributed packages? Running tests belongs to the development of the package, before it gets packaged and distributed.
Equally, why would you download a package and check that it's correctly PEP8 formatted with ruff check?

Additionally, if packages didn't have the weight of code specifically for testing the code quality, it'd make the packages smaller and use less electricity/disk on the CDNs as files are downloaded and unpacked into Python environments.

In hashin's case the conftest.py is tiny and the only (is it??) thing missing from being able to run pytest on it, from source.

distros need to use GitHub-generated Git archives

I suspect you know something I don't :)
Why do distros need to run unit tests, at all?

Suppose the don't use the tarball and instead download either a .zip or regular git clone, by tag, then A) you better be certain you get the same code as was put into the tarball, B) that's a waste of compute resources too (but much much smaller).

Thanks for bringing this up! I'm eager to learn although I'm a bit disassociated from the project because I don't use it.

[hartwork]

@peterbe just a quick meta reply that I need sleep first and plan to get back to you with an actual reply within the next 24 hours. Stay tuned 😃

[hartwork]

Hi @peterbe ,

so here's the promised actual reply. No AI/LLMs were involved. I hope you have five minutes:

---

First, yes, conftest.py is the only file missing to successfully run pytest tests, I tried.

I cannot speak for all downstreams/distros but I know these to run tests as an integral part of their packaging:

• Gentoo — https://devmanual.gentoo.org/ebuild-writing/functions/src_test/index.html
• Debian — https://manpages.debian.org/testing/debhelper/dh_auto_test.1.en.html
• Arch Linux — https://wiki.archlinux.org/title/Creating_packages#check()
• Homebrew — https://docs.brew.sh/Formula-Cookbook#add-a-test-to-the-formula

Ouf of these, being a Gentoo developer myself I know Gentoo best, and in Gentoo, these tests are run on the end user machines if enabled in the local packager manager.

The key idea in my view is (1) to detect issues with the concrete "integration" — the specific combinations of packages and their versions on that live system — to protect that specific system from breakage, (2) to find things that upstream did not find in their setups with less variety to uncover unknown bugs and (3) to find regressions to previous releases or packages to serve both causes (1) and (2) at the same time.

And there is plenty to find: If you look just for "fails tests" in Gentoo's bug tracker you'll find both users and developers (sometimes with with dedicated infrastructure) to cover up issues like that continuously.

Sometimes it's just running (some or all) of the existing upstream tests…

• docker-compose 1.x (Python, Gentoo)
• LCOV (C, Gentoo)
• servefile (Python, Gentoo)
• gcovr (Python, Gentoo)

…sometimes it's running hand-made downstream test cases…

• dcfldd (C, Gentoo)
• git-delete-merged-branches (Python, Homebrew)

In some sense a distro package needs a justification to not add (or enable) tests at least in Homebrew and also Gentoo(, the same way that bundled dependency are unbundled for security).

For Python packages, that often means the packagers need to use the GitHub release archives because they contain all the test files that the sdist results on PyPI did not. In the case of hashin, only one single file is missing because some Setuptools magic seems to already pull in the tests/test_*.py. I believe we can…

• a) ship the missing file and make things easy for all downstreams
• b) continue shipping as is and ask test-focuses dowstreams to use GitHub-generated Git archives instead
• c) (stop Setuptools from even including tests/test_*.py to not ship any tests)

Approach (a) seemed best and cheap enough to me, so I went for pull request #220 in the meantime.

What do you think?

Best, Sebastian

[peterbe]

Approach (a) has my approval. You can make the merge and the next version update and then I'll make the pypi deploy.
Then, in 1.0.5 you'll be able to run the unit tests off the tarball, right?

[hartwork]

Approach (a) has my approval.

@peterbe cool, thank you! 🙏

You can make the merge and the next version update and then I'll make the pypi deploy. Then, in 1.0.5 you'll be able to run the unit tests off the tarball, right?

Correct, for proof that 1.0.4 plus that one file makes the tests pass:

# cd "$(mktemp -d)"
# wget https://files.pythonhosted.org/packages/04/04/9061632da906161ebbc6ebf326267e5be2ce135552666c555272ee07806f/hashin-1.0.4.tar.gz
# tar xf hashin-1.0.4.tar.gz 
# cd hashin-1.0.4/
# ( cd tests && wget -q https://raw.githubusercontent.com/peterbe/hashin/refs/heads/master/tests/conftest.py )
# python3.13 -m venv venv
# source venv/bin/activate
# pip3 install -e .
# PYTHONPATH=. pytest -v
[..]
==[..]== 69 passed in 2.65s ==[..]==

And for proof that #220 pulls that file into the sdist tarball:

# cd "$(mktemp -d)"
# git clone --branch issue-217-ship-missing-file-tests-conftest-py --depth 1 https://github.com/peterbe/hashin
# cd hashin/
# python3.13 -m venv venv
# source venv/bin/activate
# pip3 install setuptools
# python3 setup.py sdist
# tar tf dist/hashin-1.0.4.tar.gz | grep -F tests/conftest.py
hashin-1.0.4/tests/conftest.py

I'll go ahead and merge…