Sharing data between workers in pytest-xdist

[dimakozin]

Hi! 👋

I'm looking for a reliable way to share data between xdist workers during test execution.
The use case is to have one test (or fixture) produce some data that other tests (running in parallel on other workers) can consume.

I understand that xdist runs each worker as a separate process and doesn't provide built-in support for inter-worker communication.
Still, I wanted to ask:

💬 Questions:

• Are there any recommended or semi-official ways to implement shared state between workers (e.g. via hooks, plugin APIs, or file-based mechanisms)?

• Are there plans or thoughts on enabling coordinated data exchange (even minimal) between workers in future versions of xdist?

• Would a plugin-based shared store (e.g. JSON + filelock, or Redis abstraction) be considered acceptable or encouraged in the ecosystem?

🛠️ What I've tried:

• Using temporary files with file-based locking (works, but clunky)

🔍 What I’m looking for:

• Ideally, a pattern or guidance on how to safely share data between workers within the pytest-xdist context (e.g. via hooks like pytest_configure_node, or any internal APIs).

Any input would be much appreciated! 🙏
Thanks for maintaining such a great tool.

[nicoddemus]

Hi,

Are there any recommended or semi-official ways to implement shared state between workers (e.g. via hooks, plugin APIs, or file-based mechanisms)?

The current suggestion is using temporary files with file-based locking, as described here: https://pytest-xdist.readthedocs.io/en/stable/how-to.html#making-session-scoped-fixtures-execute-only-once.

Would a plugin-based shared store (e.g. JSON + filelock, or Redis abstraction) be considered acceptable or encouraged in the ecosystem?

I think so, if only at least described as an alternative "How-to" in the documentation -- however I don't think we have the bandwidth to implement such a system directly in pytest-xdist.

Hope this helps!