Journal-based in-game regression tests

[mrwonko]

Use journal file (or variant thereof) to record test interactions to compare with stored previous results and check for regressions.

Use cases:

• saber attack damage
• player / jump heights
• movement techniques (bunny hopping etc.)
• maybe also just regular unit tests? These would be stored differently, but could be executed the same.
• impact of FPS on movement etc.

Features

• Support parallel execution (with optional limit)
• Ability to view the rendered test output at various speeds
• Command-line version works without renderer as part of CI

Open Questions

• How do we store the desired outcomes? Needs to be forward-compatible for regression tests to make sense.
  • Can we re-use the snapshots shared with clients on connect?
• What exactly do we compare? Depending on tests maybe not always everything, but choice of e.g.
  • bone transformation
  • character transformation
  • health / shields / force / ammo / weapons

[mrwonko]

Could also be used for benchmarks

[mrwonko]

I remember seeing a talk by Jonathan Blow that featured a platformer where the position of the player was visualised over time, with the ability to see changes to e.g. jump height in realtime. Similar to what I envision for these tests.

• Try to find that again
• Rewatch it
• Also check out Blow's other talks about his programming language etc., I think there are some insights to be gained