mypy_primer is very slow

[srittau]

Within the last week, mypy_primer has become very slow. Where previously, it ran in about 10 minutes, it now needs over 45. What could be the cause for that?

[AlexWaygood]

#6451 (comment)

[srittau]

So what can we do to bring it back to run under 10 minutes again? Everything above that, and it just becomes a hindrance more than a help. Can we segment it? Or only run for projects that are potentially affected by the changes?

[hauntsaninja]

Options:

• 0.920 should speed it up a lot (could use a prerelease wheel)
• we could shard it more
• we could try adding some GitHub Actions caching
• we could reduce the number of projects we run against (note mypy_primer has an option that lets you measure project runtimes)

Not sure how you would determine projects potentially affected by changes (note we don't run primer on changes to third party stubs anyway, since modular typeshed)

[srittau]

One idea to determine potentially affected projects: Cache the modules imported by a project (basically do a grep -r ^import | cut -f 2 | cut -d . -f 1 once a day) and compare to the changes files.

Do we have any performance data what the bottlenecks are? What would make sense to cache?

[hauntsaninja]

Given that mypy 0.920 is like 2x faster for mypy_primer like usage, I was surprised to not see the gain.

It looks like I broke the logic that made typeshed use wheels from PyPI, so we have been running pure Python mypy for the last couple weeks :-) Anyway, all fixed, and I expect that should resolve this.