'portmaster -a' ordering port builds based on insufficient dependency tracking?

[jhgit]

I had not run portmaster -a in nearly a month on FreeBSD 12/amd64. A bunch of ports were queued up including textproc/libxml2, misc/gnome-user-docs and textproc/py-libxml2, in that order.

Those three ports and itstool were already installed:
libxml2-2.9.12
gnome-user-docs-41.1
py38-libxml2-2.9.12
itstool-2.0.7

textproc/libxml2 was successfully updated from 2.9.12 to 2.9.13

Then it came time for misc/gnome-user-docs to be built and the following error occurred:

Traceback (most recent call last):
  File "/usr/local/bin/itstool", line 27, in <module>
    import libxml2
  File "/usr/local/lib/python3.8/site-packages/libxml2.py", line 1, in <module>
    import libxml2mod
ImportError: /usr/local/lib/libxml2.so.2: version LIBXML2_2.4.30 required by /usr/local/lib/python3.8/site-packages/libxml2mod.cpython-38.so not defined
gmake[2]: *** [Makefile:893: as/as.stamp] Error 1

gnome-user-docs correctly has a build dependency on textproc/itstool. itstool, has a run time dependency on textproc/py-libxml2 (which has a run time dependency on textproc/libxml2).

The itstool python script imports the 'libxml2' python module. But the shared object, libxml2mod.cypthon-38.so, installed by textproc/py-libxml2 had not been rebuilt yet (to catch up to the libxml2-2.9.13 update) when the gnome-user-docs build tried to use itstool. py-libxml2 should be rebuilt because of the libxml2.so.2 change (which in this case with the update from libxml2 2.9.12 to 2.9.13, had a breaking change between the shared lib, libxml.so.2, and the shared object module, libxml2mod.cypthon-38.so). The ldd output showing the dependency on libxml2.so.2 is below.

% ldd /usr/local/lib/python3.8/site-packages/libxml2mod.cpython-38.so
/usr/local/lib/python3.8/site-packages/libxml2mod.cpython-38.so:
        libcrypt.so.5 => /lib/libcrypt.so.5 (0x800706000)
        libintl.so.8 => /usr/local/lib/libintl.so.8 (0x800727000)
        libdl.so.1 => /usr/lib/libdl.so.1 (0x800736000)
        libutil.so.9 => /lib/libutil.so.9 (0x80073a000)
        libm.so.5 => /lib/libm.so.5 (0x800752000)
        libxml2.so.2 => /usr/local/lib/libxml2.so.2 (0x800e00000)
        libz.so.6 => /lib/libz.so.6 (0x800788000)
        libc.so.7 => /lib/libc.so.7 (0x80024e000)
        libicudata.so.70 => /usr/local/lib/libicudata.so.70 (0x8007a4000)
        libicui18n.so.70 => /usr/local/lib/libicui18n.so.70 (0x800f9b000)
        libicuuc.so.70 => /usr/local/lib/libicuuc.so.70 (0x8012e8000)
        liblzma.so.5 => /usr/lib/liblzma.so.5 (0x8007a7000)
        libthr.so.3 => /lib/libthr.so.3 (0x8007d3000)
        libc++.so.1 => /usr/lib/libc++.so.1 (0x8014fb000)
        libcxxrt.so.1 => /lib/libcxxrt.so.1 (0x8015c8000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x80160c000)
        libmd.so.6 => /lib/libmd.so.6 (0x801626000)

The question is: Is this a ports bug (where itstool's PORTREVISION should have been bumped due to the py-libxml2 update)? Or is this a portmaster bug where it should have been able to follow the dependency chain and get py-libxml2 (a run time dependency of itstool) built before building gnome-user-docs (which has the build time dependency on itstool)?

The workaround is to build textproc/py-libxml2 alone before going back to build all ports with portmaster -a.

[stesser]

Hi John,

I have tried to reproduce the issue, but did not succeed.

I had already upgraded the libxml2 port, but for a test have performed the following steps:

# pkg delete -f textproc/libxml2
--> removes
	libxml2: 2.9.13

# pkg delete py38-libxml2
--> removes
	itstool: 2.0.7
	py38-libxml2: 2.9.13_2

Then I have re-installed the previous versions of these packages:

# pkg add pkg add /usr/packages/portmaster-backup/libxml2-2.9.12.pkg
# pkg add /usr/packages/portmaster-backup/py38-libxml2-2.9.12.pkg
# pkg add /usr/packages/portmaster-backup/itstool-2.0.6.pkg

I have then re-built the gnome-user-docs port:

# portmaster misc/gnome-user-docs
===>>> The following actions will be taken if you choose to proceed:
	Install misc/gnome-user-docs
	Upgrade itstool-2.0.6 to itstool-2.0.7
	Upgrade py38-libxml2-2.9.12 to py38-libxml2-2.9.13_2
	Upgrade libxml2-2.9.12 to libxml2-2.9.13

===>>> The following actions were performed:
	Upgrade of libxml2-2.9.12 to libxml2-2.9.13
	Upgrade of py38-libxml2-2.9.12 to py38-libxml2-2.9.13_2
	Upgrade of itstool-2.0.6 to itstool-2.0.7
	Installation of misc/gnome-user-docs (gnome-user-docs-41.2)

It seems that the dependency tracking worked in this case, and I have no idea how to better reproduce the issue you observed with reasonable effort.

There is a difference between package building in poudriere (which builds all build and run dependencies in advance and installs them all before proceeding to build the port that has these dependencies) - i.e. it treats build and run dependencies identically. (The only difference is that run depends are registered as package dependencies by poudriere.)

Portmaster does depend on the correct specification of BUILD_DEPENDS and it seems that this has not been done in all ports. But a short check of the itstool, libxml2, py-libxml2, and gnome-user-docs ports does not show any issues:

# make -C /usr/ports/textproc/libxml2 build-depends-list
/usr/ports/ports-mgmt/pkg
/usr/ports/devel/cmake
/usr/ports/devel/ninja
/usr/ports/converters/libiconv
/usr/ports/devel/pkgconf
/usr/ports/devel/icu
/usr/ports/devel/readline

# make -C /usr/ports/textproc/py-libxml2  build-depends-list
/usr/ports/ports-mgmt/pkg
/usr/ports/lang/python38
/usr/ports/devel/cmake
/usr/ports/devel/ninja
/usr/ports/converters/libiconv
/usr/ports/devel/pkgconf
/usr/ports/textproc/libxml2 (*)

# make -C /usr/ports/textproc/itstool build-depends-list
/usr/ports/ports-mgmt/pkg
/usr/ports/textproc/py-libxml2 (*)
/usr/ports/lang/python38

# make -C /usr/ports/misc/gnome-user-docs build-depends-list
/usr/ports/ports-mgmt/pkg
/usr/ports/textproc/itstool (*)
/usr/ports/devel/gettext-tools
/usr/ports/devel/gmake
/usr/ports/textproc/libxml2 (*)

The dependencies marked with (*) should suffice to guarantee the correct build order.

I do not doubt that you got this build issue, but I cannot spot or reproduce the issue and thus cannot suggest a fix.

[jhgit]

I will try to come up with a simple test case that can reproduce the observed problem.

[jhgit]

Instead of deleting gnome-user-docs (as you did above), I left the old version (41.1) in place. I also left itstool-2.0.7 installed. 'portmaster -a' gave:

===>>> The following actions will be taken if you choose to proceed:
        Upgrade gnome-user-docs-41.1 to gnome-user-docs-41.2
        Upgrade libxml2-2.9.12 to libxml2-2.9.13
        Upgrade py38-libxml2-2.9.12 to py38-libxml2-2.9.13
 .
 .
Traceback (most recent call last):
  File "/usr/local/bin/itstool", line 27, in <module>
    import libxml2
  File "/usr/local/lib/python3.8/site-packages/libxml2.py", line 1, in <module>
    import libxml2mod
ImportError: /usr/local/lib/libxml2.so.2: version LIBXML2_2.4.30 required by /usr/local/lib/python3.8/site-packages/libxml2mod.cpython-38.so not defined
 .
 .
===>>> Update for misc/gnome-user-docs failed
===>>> Aborting update

===>>> The following actions were performed:
        Upgrade of libxml2-2.9.12 to libxml2-2.9.13


===>>> You can restart from the point of failure with this command line:
       portmaster <flags> misc/gnome-user-docs lang/go sysutils/packer textproc/py-libxml2@py38

I don't know yet if this detail matters (I don't think it does), but in this test, I started with a ports tree at git hash 13977a9f03290d0580ad09f62d8903571cf7e4c2 (Mar 15). Then I updated gnome-user-docs, libxml2, and py-libxml2 to b6ac38cf27573d6d2037fcfffcdf7cffdd35f9a2 (Mar 28). This is a little different than what you did (build the latest, then revert to old packages, then try to build again).

<later...>

I removed gnome-user-docs and ran portmaster misc/gnome-user-docs (instead of portmaster -a this time) and left libxml2 and py-libxml2 installed at 2.9.12. That still produces the same error:

===>>> The following actions will be taken if you choose to proceed:
        Install misc/gnome-user-docs
        Upgrade libxml2-2.9.12 to libxml2-2.9.13
 .
 .
===>>> make build failed for misc/gnome-user-docs
===>>> Aborting update

===>>> Upgrade of libxml2-2.9.12 to libxml2-2.9.13 complete


===>>> You can restart from the point of failure with this command line:
       portmaster <flags> misc/gnome-user-docs

Notice that portmaster did not even detect that py-libxml2 needs to be updated (not listed in 'The following actions') in this case, unlike portmaster -a. But even though portmaster -a would add py-libxml2 update to its actions, there is still an ordering problem (the py-libxml2 upgrade is not ordered ahead of gnome-user-docs).

Then I tried deleting itstool. gnome-user-docs is still not installed. {py-,}libxml2 are still both at 2.9.12. That succeeded - because the itstool install triggered py-libxml2 to be built earlier (an update from itstool 2.0.6 to 2.0.7 would do the same).

% portmaster misc/gnome-user-docs
 .
 .
===>>> The following actions will be taken if you choose to proceed:
        Install misc/gnome-user-docs
        Install textproc/itstool
        Upgrade py38-libxml2-2.9.12 to py38-libxml2-2.9.13
        Upgrade libxml2-2.9.12 to libxml2-2.9.13
 .
 .
===>>> The following actions were performed:
        Upgrade of libxml2-2.9.12 to libxml2-2.9.13
        Upgrade of py38-libxml2-2.9.12 to py38-libxml2-2.9.13
        Installation of textproc/itstool (itstool-2.0.7)
        Installation of misc/gnome-user-docs (gnome-user-docs-41.2)

So the problem seems to be that if itstool is already up to date, it does not need to be built, and its run time dependency is not checked (because it is already installed and happy). So portmaster does not think it needs to order the py-libxml2 build before gnome-user-docs.

Bumping the itstool PORTREVISION after the py-libxml2 update would also avoid this problem, of course. But should that be necessary? 'portmaster' COULD see that there is a dependency relationship from gnome-user-docs -> itstool -> py-libxml2 and decide to order py-libxml2 before gnome-user-docs even though we are not updating itstool.

[stesser]

Yes, your observations match my knowledge about the inner workings of portmaster (which has not been developed by me, but I have taken over the maintainer role after the original author left the project and FLAVOR support needed to be added to portmaster).

There is a -t (thorough) option that should help with the issue you report.

It checks run time dependencies of dependencies that need not be rebuilt, too, and I think that is exactly what is required in this case. This is a time-consuming operation and therefore not performed in general. You may want to test whether is solves the issue (it should, IMHO).

Adding a direct build dependency to a port affected by such a "deep" version mismatch would solve the issue, but is up to the maintainers of the affected ports. (And many do not bother too much if it successfully builds in poudriere ...)

The portmaster script has evolved over 2 decades and had to be adapted to significant changes in the ports infrastructure. It has severe limitations (e.g. uses child processes for per port state, instead of complex data structures).

I have been using a mostly compatible rewrite of portmaster (in LUA) for more than 2 years (available from the lua branch of stesser/portmaster on Github, but I do not push local commits often enough). It is not ready for general release (I have to re-implement the conflicts handling functions, for example), by I work on it as my spare time permits.

An example run as speed test (with --no-confirm -n) on my development system is:

$ time portmaster.lua --no-confirm -n -a

#-- Scanning 2537 installed packages and dependencies

#-- [1] Upgrade zrepl-0.5.0 to zrepl-0.5.0_1 from sysutils/zrepl

#-- [2] Upgrade vlc-3.0.16_10,4 to vlc-3.0.17.3_1,4 from multimedia/vlc

#-- [3] Upgrade syncthing-1.19.1 to syncthing-1.19.1_1 from net/syncthing

#-- [4] Install yarn-1.22.17_2 from yarn-1.22.17_2.pkg
#   [5] Upgrade anki-2.1.48_4 to anki-2.1.49 from games/anki

#-- [6] Install node14-14.18.1_1 from node14-14.18.1_1.pkg

#-- [7] Install py38-pytest-4.6.11,1 from py38-pytest-4.6.11,1.pkg
#   [8] Install py38-pytest-flake8-1.1.0 from py38-pytest-flake8-1.1.0.pkg

#-- [9] Install ImageMagick6-nox11-6.9.12.44,1 from graphics/ImageMagick6-nox11

#-- [10] Install py38-pytest-black-0.3.12 from devel/py-pytest-black@py38

#-- [11] Install py38-pytest-enabler-1.2.1 from devel/py-pytest-enabler@py38

#-- [12] Install py38-pytest-mypy-0.9.1 from devel/py-pytest-mypy@py38

#-- [13] Install yarn-node14-1.22.17_2 from yarn-node14-1.22.17_2.pkg

#-- [14] Install poudriere-3.3.7_1 from poudriere-3.3.7_1.pkg

#-- [15] Install py38-html2text-2020.1.16 from textproc/py-html2text@py38

#-- [16] Install tshark-3.6.2_1 from net/tshark

#-- [17] Install py38-pytest4-cov-2.9.0 from py38-pytest4-cov-2.9.0.pkg

#-- Statistics of 17 planned actions:
#      13 packages will be installed
#       4 packages will be upgraded

#-- Build results:

#-- Install failures
#   devel/py-pytest4-cov@py38: install conflict: py38-pytest4-cov-2.9.0 conflicts with installed package py38-pytest-cov-2.9.0
#   devel/py-pytest@py38: install conflict: py38-pytest-4.6.11,1 conflicts with installed package py38-pytest4-4.6.11,1
#   graphics/ImageMagick6-nox11: install conflict: ImageMagick6-nox11-6.9.12.44,1 conflicts with installed package ImageMagick7-7.1.0.28
#   net/tshark: install conflict: tshark-3.6.2_1 conflicts with installed package wireshark-3.6.2_1
#   ports-mgmt/poudriere: install conflict: poudriere-3.3.7_1 conflicts with installed package poudriere-devel-3.3.99.20211130
#   textproc/py-html2text@py38: install conflict: py38-html2text-2020.1.16 conflicts with installed package html2text-1.3.2a,1
#   www/node14: install conflict: node14-14.18.1_1 conflicts with installed package node-17.0.1
#   www/yarn: install conflict: yarn-1.22.17_2 conflicts with planned package yarn-node14-1.22.17_2
#   www/yarn-node14: install conflict: yarn-node14-1.22.17_2 conflicts with planned package yarn-1.22.17_2

#-- Finish background fetching and checking for 5 ports:
#   	games/anki
#   	graphics/ImageMagick6-nox11
#   	multimedia/vlc
#   	net/tshark
#   	textproc/py-html2text

real	0m7,422s
user	0m24,292s
sys	0m17,727s

That run included a "thorough" check of dependencies and was performed after updating all ports that had no issues (i.e. with a "warm" buffer cache, not requiring many actual disk accesses). The "fetching" message is due to the scan for required updates being faster then the (parallel) make checksum of all distfiles - without the distfiles checksum verification the time used to check my > 2500 ports for required updates would be lower.

But the missing conflicts handling leads to e.g. dependencies on tshark not being redirected to the installed (and conflicting) wireshark. This can be worked around with the "--jailed" option of this new portmaster tool, which builds the port in a clean jail. But as long as the conflicts checks (which worked with a few flaws and needed to be re-implemented from scratch) is missing, this version is not suitable as a complete replacement of the existing portmaster ...

(You may want to check it ou anyway, it has a number of other functions that are much faster than in the official portmaster version, e.g. pruning the distfiles or packages directories from stale files.)

[jhgit]

Thanks for the reply. And I understand about the history on portmaster. Thanks for helping keep it functional.

I tried -t. It did not find that it needed to update py-libxml2, unfortunately. Same error. This was in the situation with {py38-,}libxml2-2.9.12 and itstool-2.0.7 installed, no gnome-user-docs - and invoking 'portmaster -t misc/gnome-user-docs'.

It looks like -t does not actually call 'make all-depends-list' (as the documentation for -t states). It just adds 'run depends' (as you stated). Since py-libxml2 is not directly in the run depends for gnome-user-docs, it doesn't add it to the build list, so -t is not a workaround (currently) for the stated problem. 'make all-depends-list' is recursive, so it does include py-libxml2. It also includes a LOT more packages because it also includes all TEST_DEPENDS.

I have a patch for make_dep_list to fix -t to be more like the actual all-depends-list, minus test depends which could be a "super thorough" mode - maybe -tt or --test-depends or something. Most people don't typically turn on test depends unless they have a specific need or are doing wider ports tree testing. I'll verify my patch and submit.

I'll also take a look at the lua version. Sounds interesting.