Support using pytest-trio and hypothesis together

[zmitchell]

I'm a big fan of hypothesis for property-based testing. It provides a decorator, @given, that you use to pass in placeholders for randomly generated values. However, when I use it @pytest.mark.trio I get the following error:

Failed: test function `<Function 'test_my_test_func'>` is marked trio but is not async

This happens regardless of the order that you apply the decorators. Is it possible to do something like this right now?

[njsmith]

Oh, interesting bug. We definitely want to support hypothesis, it's awesome. But, the problem is that @given actually wants to take over running the test: it takes your test function and then wraps a loop around it, calling it repeatedly with different inputs. And it doesn't know how to run a trio function. And there's no way for pytest-trio to like, reach inside that loop and teach it. Making this work automatically will require some kind of coordination between them.

As a temporary workaround, I think you should be able to do:

from trio.testing import trio_test

@given(...)
@trio_test
async def test_whatever(...):
    ...

(make sure to put @given on top, and @trio_test underneath). If you do this then async fixture support won't work, but otherwise it should be OK. (Even the clock fixtures should work, I think; it's just fixture functions that need to themselves execute under trio that this can't support.)

From a quick glance it looks like this is an issue that hypothesis upstream has thought about some for asyncio, but probably not for trio, and where maybe they're hoping someone who knows/cares about async will show up to help them figure it out:

• Possibility of support coroutine tests HypothesisWorks/hypothesis#52
• Coroutine tests do not work HypothesisWorks/hypothesis#292
• Feature request: support for twisted trial (unittest) ? HypothesisWorks/hypothesis#703
• Decorator for async functions HypothesisWorks/hypothesis#968
• maybe others

I haven't read all the above, but I suspect this is an issue that's totally solvable but may be a bit tricky because doing it right will require some coordination between pytest-trio/pytest-asyncio/hypothesis/pytest. Unfortunately right now I'm in the middle of adjusting some meds with annoying side-effects and have only slept like 14 hours in the last 4 days so uh... I will not attempt to actually come up with a design for the ideal solution.

[njsmith]

Hmm, Hypothesis's existing execute_example hook isn't quite usable for us, but it might be the basis of a fairly painless solution – like if pytest-trio could somehow detect hypothesis tests that have the @pytest.mark.trio mark, and then somehow tell hypothesis to use pytest-trio's special clever test-running function as the execute_example hook?

source code dive Ah, it looks like when you decorate a test function with @given, then hypothesis adds an .is_hypothesis_test = True attribute, so detecting a hypothesis test is easy.

So now the problem is: the only configuration mechanism that hypothesis provides for execute_example is to define a test class and define that as a method, which is not at all convenient here. We want some other way to configure this that we can do from a pytest plugin. A crude but effective approach would be for hypothesis to support with hypothesis.local_executor(...): ..., that internally sets a thread-local that hypothesis later knows to check for.

[njsmith]

Another tricky bit is that for #9, we'll want to be able to detect async tests, even if they're wrapped by hypothesis.

[njsmith]

Filed HypothesisWorks/hypothesis#1257

[touilleMan]

Not totally related but still interesting I guess: I'm currently using hypothesis's stateful feature along with trio in my own project:
https://github.com/Scille/parsec-cloud/blob/7593869b2be4b22fb524bc36d90b2ceea0b8cf94/tests/hypothesis/conftest.py
https://github.com/Scille/parsec-cloud/blob/7593869b2be4b22fb524bc36d90b2ceea0b8cf94/tests/hypothesis/test_core_offline_rwfile.py

Basically the idea is to have a regular trio test (i.e. an async function with pytest.mark.trio) which define a state machine class inheriting the TrioDriverRuleBasedStateMachine.
When calling the statemachine run_test function, the hypothesis stuff is run inside it own thread and communicates with trio through some queues. On trio's side a trio_runner async function is run each time the test is reseted to do initialization and then loop to listen instruction from the hypothesis thread.

[touilleMan]

For people wondering about this, here is a simple workaround using the execute_example feature of hypothesis (see http://hypothesis.readthedocs.io/en/latest/details.html#custom-function-execution):

import trio
from hypothesis import given, strategies as st

class TestWithHypothesisAndTrio:

    @given(st.binary(max_size=10000))
    async def test_base(self, data):
        left, right = trio.testing.memory_stream_pair()
        await left.send_all(data)
        right_received = await right.receive_some(len(data))
        await right.send_all(right_received)
        left_received = await left.receive_some(len(data))

        assert right_received == left_received == data

    def execute_example(self, f):
        from unittest.mock import Mock
        node = Mock()
        node.function = f
        from pytest_trio.plugin import _trio_test_runner_factory
        return _trio_test_runner_factory(node)()

Note this is far from perfect given fixtures (even synchronous ones) are not supported :'-(

[njsmith]

If you read HypothesisWorks/hypothesis#1257, then the discussion is currently focused on how pytest-trio could automagically set up something like that execute_example without the user having to think about it :-).

We've so far focused on @given tests though; stateful tests are trickier. If you're interested in getting this working though I'm sure that could help push it along :-). (In fact if someone wants to submit them a PR for just the @given part that'd be great, I probably won't have time soon.)

[zmitchell]

I'm not sure using trio_test works properly in this context. I have an async test function that takes two normal fixtures, config_port1 and config_port2, and an @given from hypothesis.

@given(msg_sent=some_strategy)
@trio_test
async def test_can_await_newline_terminated_msg(config_port1, config_port2, msg_sent):
    # Test body ...

This is the error message I get for that test:

==================================================================== test session starts =====================================================================
platform darwin -- Python 3.6.5, pytest-3.5.0, py-1.5.3, pluggy-0.6.0
rootdir: /Users/zmitchell/Projects/mhz-equip, inifile:
plugins: trio-0.4.1, hypothesis-3.55.6
collected 6 items

tests/lockin_test.py F.....                                                                                                                            [100%]

========================================================================== FAILURES ==========================================================================
___________________________________________________________ test_can_await_newline_terminated_msg ____________________________________________________________

kwargs = {}

    @wraps(fn)
>   def wrapper(**kwargs):

../../.local/share/virtualenvs/mhz-equip-IHw2MkON/lib/python3.6/site-packages/trio/testing/_trio_test.py:19:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../.local/share/virtualenvs/mhz-equip-IHw2MkON/lib/python3.6/site-packages/hypothesis/core.py:583: in execute
    result = self.test_runner(data, run)
../../.local/share/virtualenvs/mhz-equip-IHw2MkON/lib/python3.6/site-packages/hypothesis/executors.py:58: in default_new_style_executor
    return function(data)
../../.local/share/virtualenvs/mhz-equip-IHw2MkON/lib/python3.6/site-packages/hypothesis/core.py:575: in run
    return test(*args, **kwargs)
../../.local/share/virtualenvs/mhz-equip-IHw2MkON/lib/python3.6/site-packages/trio/testing/_trio_test.py:19: in test_can_await_newline_terminated_msg
    def wrapper(**kwargs):
../../.local/share/virtualenvs/mhz-equip-IHw2MkON/lib/python3.6/site-packages/hypothesis/core.py:522: in test
    result = self.test(*args, **kwargs)
../../.local/share/virtualenvs/mhz-equip-IHw2MkON/lib/python3.6/site-packages/trio/_core/_result.py:119: in unwrap
    raise self.error
../../.local/share/virtualenvs/mhz-equip-IHw2MkON/lib/python3.6/site-packages/contextvars/__init__.py:66: in run
    return callable(*args, **kwargs)
../../.local/share/virtualenvs/mhz-equip-IHw2MkON/lib/python3.6/site-packages/trio/_core/_run.py:934: in init
    async_fn, args, system_nursery, None
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = Runner(clock=SystemClock(offset=87411.59816923397), instruments=[], io_manager=KqueueIOManager(_kqueue=<select.kqueue ...read.RLock object owner=0 count=0 at 0x1022d0450>), trio_token=None, ki_pending=False, waiting_for_idle=SortedDict({}))
async_fn = functools.partial(<function test_can_await_newline_terminated_msg at 0x10215e2f0>, msg_sent=''), args = ()
nursery = <trio._core._run.Nursery object at 0x102405fd0>, name = None

    def spawn_impl(
        self, async_fn, args, nursery, name, *, ki_protection_enabled=False
    ):

        ######
        # Make sure the nursery is in working order
        ######

        # This sorta feels like it should be a method on nursery, except it
        # has to handle nursery=None for init. And it touches the internals of
        # all kinds of objects.
        if nursery is not None and nursery._closed:
            raise RuntimeError("Nursery is closed to new arrivals")
        if nursery is None:
            assert self.init_task is None

        ######
        # Call the function and get the coroutine object, while giving helpful
        # errors for common mistakes.
        ######

        def _return_value_looks_like_wrong_library(value):
            # Returned by legacy @asyncio.coroutine functions, which includes
            # a surprising proportion of asyncio builtins.
            if inspect.isgenerator(value):
                return True
            # The protocol for detecting an asyncio Future-like object
            if getattr(value, "_asyncio_future_blocking", None) is not None:
                return True
            # asyncio.Future doesn't have _asyncio_future_blocking until
            # 3.5.3. We don't want to import asyncio, but this janky check
            # should work well enough for our purposes. And it also catches
            # tornado Futures and twisted Deferreds. By the time we're calling
            # this function, we already know something has gone wrong, so a
            # heuristic is pretty safe.
            if value.__class__.__name__ in ("Future", "Deferred"):
                return True
            return False

        try:
>           coro = async_fn(*args)
E           TypeError: test_can_await_newline_terminated_msg() missing 2 required positional arguments: 'config_port1' and 'config_port2'

../../.local/share/virtualenvs/mhz-equip-IHw2MkON/lib/python3.6/site-packages/trio/_core/_run.py:756: TypeError

As you can see, the fixtures don't get passed to the test function.

[njsmith]

It looks like something is going wrong with pytest introspecting the test function signature. But that's very odd, because pytest-trio also uses @trio_test internally...

Oh, but make pytest-trio doesn't apply the decorator until after pytest has already introspected the function?

I guess this should be filed as a bug on trio, because that's where @trio_test lives.

[Zac-HD]

HypothesisWorks/hypothesis#1343 will enable a patch such as master...Zac-HD:hypothesis-experiment, which in turn would fix this issue 😄