Compilation/link failures when switching between branches

[stinos]

From time to time I have the problem that when switching branches there'll be build failures either because of missing qstr definitions or because of missing module definitions. I mostly use the msvc port so I wanted to fix that one, but it seems the core issue is actually in how makeqstrdefs.py works:

• remove all build output
• do a new build
• makeqstrdefs's 'split' phase outputs all files which contain qstr definitions and module registrations etc in a genhdr/ subdirectory
• makeqstrdefs's 'cat' phase uses those split files to build a concatenated source file
• build completes
• switch to a branch which has for instance less source files and less module registrations
• do a build
• makeqstrdefs's 'split' phase runs as above, outputting less files
• makeqstrdefs's 'cat' phase has a problem now because the previous files are still in the output directory and it'll still use those, thereby resulting in generated source code which typically has more qstr definitions than needed (that doesn't result in errors though) but also more module definitions than needed (linker error because the source for those modules is gone)

All of this is easily solved by doing a full rebuild, but it can be annoying at times. So I'm wondering: do others also find this annoying or is this more a 'once every year type' of problem? I don't think it's too hard to fix makeqstrdefs.py to deal with this, but first wanted to check if there's any interest at all.

[Josverl]

Annoying for sure, and thanks for the time spent to dissect the cause.

For me it's definitely an Aha Erlebnis, to need to go back and
make clean

[robert-hh]

So I'm wondering: do others also find this annoying or is this more a 'once every year type' of problem?

I got used to the habit of clearing the previous build after switching the branch. But there have been several issues or discussion item by people getting caught by this behavior.

[stinos]

I made a POC where makeqstrdefs split records it actual input to a file which makeqstrdefs cat can then comsume. Loses the ability to call makeqstrdefs split several times then one makeqstrdefs cat, but that doesn't seem to be used anywhere. Will check.

Also noticed that makemanifest.py has a similar issue: removing entries from a manifest.py file does nothing, i.e. does not update frozen_content.c accordingly. Will also check that.

[stinos]

Hmm this is a bit of a rabbit hole. The opposite case, source code generation being invoked while it shouldn't be, can also happen. Just doing a clean build, changing one file, then building again keeps on invoking makeqstrdefs because it doesn't update its output file because it sees the result of split is the same as last time. But without the correct timestamp the Makefile treats it as not up-to-date. I.e. currently no matter how many times I run make, it always does

Use make V=1 or set BUILD_VERBOSE in your environment to increase build verbosity.
GEN build-standard/genhdr/moduledefs.collected
GEN build-standard/genhdr/root_pointers.collected
Module registrations not updated
Root pointer registrations not updated

which takes several seconds more than when it doesn't do that.