Skip to content

Conversation

@patcon
Copy link
Member

@patcon patcon commented Mar 13, 2025

This PR is about reproducing all the output of the repness key in the polis /api/v3/math/pca2 endpoint. Once this is done, all the existing pieces can be used to generate consensus key, but that will be another PR.

Needs a lot of work and cleanup. Much of functionality is in debug.py right now.

To Dos

  • port all of repness generating functions from Polis
  • add unit test of repness against small conversation from polismath
  • clean up code to make more legible
  • add polismath types for type checking

@patcon
Copy link
Member Author

patcon commented Mar 14, 2025

Very messy first stub for reproducing Polis' per-group representative statements 🎉
https://gist.github.com/patcon/26e47b3b63a4d8e6b1bd9a94bd70d362

Need to extract the debug.py logic into its own performant functions, and maybe start teasing apart utils.py because it's getting large and complex.

We're skipping concern with the clustering here by extracting cluster_labels from Polis API. Since we're not using the harder-to-reproduce clustering data for now, I can confirm our calculated stats match the Polis API math output with high precision.

@patcon patcon force-pushed the add-stats-repness-test branch from ceb9cd0 to 0bc33c0 Compare March 17, 2025 05:31
@patcon patcon marked this pull request as ready for review March 17, 2025 05:35
@patcon
Copy link
Member Author

patcon commented Mar 17, 2025

This is done and with tests. Will restructure gigantic utils.py in another PR :)

@patcon patcon merged commit e491167 into main Mar 17, 2025
4 checks passed
@patcon patcon deleted the add-stats-repness-test branch March 17, 2025 05:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants