Support for hypothesis in async methods

[carver]

hypothesis exits early when used on pytest-asyncio tests, complaining that the test returns a coroutine instead of None. This issue talks about how trio added support for hypothesis: HypothesisWorks/hypothesis#968

I can work around this with a decorator:

def run_in_event_loop(async_func):
    @functools.wraps(async_func)
    def wrapped(operations, queue_size, add_size, get_size, event_loop):
        event_loop.run_until_complete(asyncio.ensure_future(
            async_func(operations, queue_size, add_size, get_size, event_loop),
            loop=event_loop,
        ))
    return wrapped

@hypothesis.given(...)
@run_in_event_loop
@pytest.mark.asyncio
async def test_...

But it looks like the recommended approach is to tinker with the hypothesis attribute added to the test, like: https://github.com/python-trio/pytest-trio/pull/44/files#diff-aacce3c1178656a02aecc59c94209433R231

Is this something pytest-asyncio is interested in supporting?

[japrogramer]

what is the status of this, I would like to use hypothesis for my async project so my tests must also be async ... and don't want to use trio.

I was thinking of doing something like this

@given(st.lists(
  st.floats(allow_nan=False, allow_infinity=False), min_size=1))
def test_mean(xs):
    old_loop = asyncio.get_event_loop()
    new_loop = asyncio.new_event_loop()
    try:
        asyncio.set_event_loop(new_loop)
        new_loop.run_until_complete(my_corrutine(xs))
    except Exception as e:
         raise e
    finally:
        asyncio.set_event_loop(old_loop)

[japrogramer]

turned it into a fixture

@pytest.fixture(scope="session")
def event_loop():
    """
    Run coroutines in non async code
    by using a new event_loop
    """
    old_loop = asyncio.get_event_loop()
    new_loop = asyncio.new_event_loop()
    asyncio.set_event_loop(new_loop)
    try:
        yield new_loop
    finally:
        asyncio.set_event_loop(old_loop)

[nicoddemus]

Just a quick remark: the try/finally block is not required for pytest fixtures, the code after the yield will always be executed regardless if the test raised an exception or not (contrary to contextlib.contextmanager).

[japrogramer]

neat, makes sense .. i copy pasted the code into a fixture, thats why i kept it like that

[Zac-HD]

Hi all, maintainer of Hypothesis + Trio + Pytest here - I wrote the linked integration and would be happy to have a go here. Only reason I hadn't already added this was that I don't use asyncio much myself and hoped someone else might take on the task 😄

FYI @japrogramer the integration is basically equivalent to your fixture / @carver's decorator, automatically applied to all tests with pytest.mark.trio that also use Hypothesis. It might be a bit more complicated for asyncio because of explicit event loops, but we'll see I guess!

[Zac-HD]

OK, a first attempt is up as #102. @carver @japrogramer - can you check that this works for you?

[carver]

OK, a first attempt is up as #102. @carver @japrogramer - can you check that this works for you?

Yes, the change does appear to work on the hypothesis tests, when I pip install -e from the Zac-HD/hypothesis checkout (at d4c1b92).

However... my other pytest.mark.asyncio tests start emitting a bunch of warnings like:

tests/trinity/utils/test_task_queue.py::test_queue_size_reset_after_complete
  /home/jcarver/code/py-evm/venv/lib/python3.6/site-packages/_pytest/python.py:195: RuntimeWarning: coroutine 'test_queue_size_reset_after_complete' was never awaited
    testfunction(**testargs)

(they are all green with pytest-asyncio v0.9.0)

I haven't had a chance to dig into why, but maybe it is unrelated to your change.

[japrogramer]

@carver that error means that a coroutine was never awaited but called as if it where.

I merged the pull request into master and this is what i got
@Zac-HD

▶ coverage run -m pytest                                                                                                                                                                                 ∞
=============================================================================================== test session starts ================================================================================================
platform linux -- Python 3.7.1, pytest-4.0.1, py-1.7.0, pluggy-0.8.0
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/home/archangel/VENV/pytest/pytest-asyncio/.hypothesis/examples')
rootdir: /home/archangel/VENV/pytest/pytest-asyncio, inifile: setup.cfg
plugins: asyncio-0.10.0.dev0, hypothesis-3.83.0
collected 38 items

tests/test_dependent_fixtures.py .                                                                                                                                                                           [  2%]
tests/test_event_loop_scope_35.py ...                                                                                                                                                                        [ 10%]
tests/test_hypothesis_integration.py ....                                                                                                                                                                    [ 21%]
tests/test_simple.py ..x......                                                                                                                                                                               [ 44%]
tests/test_simple_35.py ......                                                                                                                                                                               [ 60%]
tests/test_subprocess.py ..                                                                                                                                                                                  [ 65%]
tests/async_fixtures/test_async_fixtures_35.py .                                                                                                                                                             [ 68%]
tests/async_fixtures/test_async_fixtures_scope.py .                                                                                                                                                          [ 71%]
tests/async_fixtures/test_async_gen_fixtures_35.py ..                                                                                                                                                        [ 76%]
tests/async_fixtures/test_async_gen_fixtures_36.py ..                                                                                                                                                        [ 81%]
tests/async_fixtures/test_coroutine_fixtures.py s                                                                                                                                                            [ 84%]
tests/async_fixtures/test_nested_36.py .                                                                                                                                                                     [ 86%]
tests/markers/test_class_marker_35.py .                                                                                                                                                                      [ 89%]
tests/markers/test_module_marker_35.py ..                                                                                                                                                                    [ 94%]
tests/multiloop/test_alternative_loops.py ..                                                                                                                                                                 [100%]
============================================================================================= short test summary info ==============================================================================================
SKIP [1] tests/async_fixtures/test_coroutine_fixtures.py:23: @asyncio.coroutine fixtures are not supported yet
XFAIL tests/test_simple.py::test_asyncio_marker_fail
  need a failure

================================================================================= 36 passed, 1 skipped, 1 xfailed in 1.05 seconds ==================================================================================

[carver]

@carver that error means that a coroutine was never awaited but called as if it where.

Right, but the method that isn't being awaited is the whole test. pytest.mark.asyncio is supposed to await it for you. Also, the tests pass without warning in pytest-asyncio v0.9.0.

[Zac-HD]

It looks like my detection to avoid wrapping Hypothesis tests twice was a bit too loose - try again with 1b66314?

[carver]

Yup, all green!

[japrogramer]

@Zac-HD not for me .. hmm, am i running this correctly?

▶ coverage run -m pytest                                                                                                                                                                               86%
=============================================================================================== test session starts ================================================================================================
platform linux -- Python 3.7.1, pytest-4.0.1, py-1.7.0, pluggy-0.8.0
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/home/archangel/VENV/pytest/pytest-asyncio/.hypothesis/examples')
rootdir: /home/archangel/VENV/pytest/pytest-asyncio, inifile: setup.cfg
plugins: asyncio-0.10.0.dev0, hypothesis-3.83.0
collected 38 items

tests/test_dependent_fixtures.py .                                                                                                                                                                           [  2%]
tests/test_event_loop_scope_35.py ...                                                                                                                                                                        [ 10%]
tests/test_hypothesis_integration.py ....                                                                                                                                                                    [ 21%]
tests/test_simple.py ..x......                                                                                                                                                                               [ 44%]
tests/test_simple_35.py ......                                                                                                                                                                               [ 60%]
tests/test_subprocess.py ..                                                                                                                                                                                  [ 65%]
tests/async_fixtures/test_async_fixtures_35.py .                                                                                                                                                             [ 68%]
tests/async_fixtures/test_async_fixtures_scope.py .                                                                                                                                                          [ 71%]
tests/async_fixtures/test_async_gen_fixtures_35.py ..                                                                                                                                                        [ 76%]
tests/async_fixtures/test_async_gen_fixtures_36.py ..                                                                                                                                                        [ 81%]
tests/async_fixtures/test_coroutine_fixtures.py s                                                                                                                                                            [ 84%]
tests/async_fixtures/test_nested_36.py .                                                                                                                                                                     [ 86%]
tests/markers/test_class_marker_35.py .                                                                                                                                                                      [ 89%]
tests/markers/test_module_marker_35.py ..                                                                                                                                                                    [ 94%]
tests/multiloop/test_alternative_loops.py ..                                                                                                                                                                 [100%]
============================================================================================= short test summary info ==============================================================================================
SKIP [1] tests/async_fixtures/test_coroutine_fixtures.py:23: @asyncio.coroutine fixtures are not supported yet
XFAIL tests/test_simple.py::test_asyncio_marker_fail
  need a failure

================================================================================= 36 passed, 1 skipped, 1 xfailed in 1.58 seconds ==================================================================================

and here is the diff from master

diff --git a/.gitignore b/.gitignore
index a49cb17..0975808 100644
--- a/.gitignore
+++ b/.gitignore
@@ -22,6 +22,7 @@ var/
 *.egg-info/
 .installed.cfg
 *.egg
+.hypothesis/
 
 # PyInstaller
 #  Usually these files are written by a python script from a template
diff --git a/README.rst b/README.rst
index 8c36da8..69ace41 100644
--- a/README.rst
+++ b/README.rst
@@ -178,6 +178,9 @@ Changelog
 
 0.10.0. (UNRELEASED)
 ~~~~~~~~~~~~~~~~~~~~
+- ``pytest-asyncio`` integrates with `Hypothesis <https://hypothesis.readthedocs.io>`_
+  to support ``@given`` on async test functions using ``asyncio``.
+  `#102` <https://github.com/pytest-dev/pytest-asyncio/pull/102>
 
 0.9.0 (2018-07-28)
 ~~~~~~~~~~~~~~~~~~
diff --git a/pytest_asyncio/plugin.py b/pytest_asyncio/plugin.py
index 584efbf..26a37d4 100644
--- a/pytest_asyncio/plugin.py
+++ b/pytest_asyncio/plugin.py
@@ -1,6 +1,7 @@
 """pytest-asyncio implementation."""
 import asyncio
 import contextlib
+import functools
 import inspect
 import socket
 
@@ -139,7 +140,8 @@ def pytest_pyfunc_call(pyfuncitem):
     function call.
     """
     for marker_name, fixture_name in _markers_2_fixtures.items():
-        if marker_name in pyfuncitem.keywords:
+        if marker_name in pyfuncitem.keywords \
+                and not getattr(pyfuncitem.obj, 'is_hypothesis_test', False):
             event_loop = pyfuncitem.funcargs[fixture_name]
 
             funcargs = pyfuncitem.funcargs
@@ -152,11 +154,39 @@ def pytest_pyfunc_call(pyfuncitem):
             return True
 
 
+def wrap_in_sync(func):
+    """Return a sync wrapper around an async function."""
+
+    @functools.wraps(func)
+    def inner(**kwargs):
+        loop = asyncio.get_event_loop_policy().new_event_loop()
+        try:
+            coro = func(**kwargs)
+            if coro is not None:
+                future = asyncio.ensure_future(coro, loop=loop)
+                loop.run_until_complete(future)
+        finally:
+            loop.close()
+
+    return inner
+
+
 def pytest_runtest_setup(item):
     for marker, fixture in _markers_2_fixtures.items():
         if marker in item.keywords and fixture not in item.fixturenames:
             # inject an event loop fixture for all async tests
             item.fixturenames.append(fixture)
+    if item.get_closest_marker("asyncio") is not None:
+        if hasattr(item.obj, 'hypothesis'):
+            # If it's a Hypothesis test, we insert the wrap_in_sync decorator
+            item.obj.hypothesis.inner_test = wrap_in_sync(
+                item.obj.hypothesis.inner_test
+            )
+        elif getattr(item.obj, 'is_hypothesis_test', False):
+            pytest.fail(
+                'test function `%r` is using Hypothesis, but pytest-asyncio '
+                'only works with Hypothesis 3.64.0 or later.' % item
+            )
 
 
 # maps marker to the name of the event loop fixture that will be available
diff --git a/setup.py b/setup.py
index 2c26c42..97ef460 100644
--- a/setup.py
+++ b/setup.py
@@ -43,7 +43,11 @@ setup(
     install_requires=["pytest >= 3.0.6"],
     extras_require={
         ':python_version == "3.5"': "async_generator >= 1.3",
-        "testing": ["coverage", "async_generator >= 1.3"],
+        "testing": [
+            "coverage",
+            "async_generator >= 1.3",
+            "hypothesis >= 3.64",
+        ],
     },
     entry_points={"pytest11": ["asyncio = pytest_asyncio.plugin"]},
 )
diff --git a/tests/test_hypothesis_integration.py b/tests/test_hypothesis_integration.py
new file mode 100644
index 0000000..562f477
--- /dev/null
+++ b/tests/test_hypothesis_integration.py
@@ -0,0 +1,27 @@
+"""Tests for the Hypothesis integration, which wraps async functions in a
+sync shim for Hypothesis.
+"""
+
+import pytest
+
+from hypothesis import given, strategies as st
+
+
+@given(st.integers())
+@pytest.mark.asyncio
+async def test_mark_inner(n):
+    assert isinstance(n, int)
+
+
+@pytest.mark.asyncio
+@given(st.integers())
+async def test_mark_outer(n):
+    assert isinstance(n, int)
+
+
+@pytest.mark.parametrize("y", [1, 2])
+@given(x=st.none())
+@pytest.mark.asyncio
+async def test_mark_and_parametrize(x, y):
+    assert x is None
+    assert y in (1, 2)

pip freeze

async-generator==1.10
atomicwrites==1.2.1
attrs==18.2.0
coverage==4.5.2
hypothesis==3.83.0
more-itertools==4.3.0
pluggy==0.8.0
py==1.7.0
pytest==4.0.1
-e git+https://github.com/pytest-dev/pytest-asyncio.git@923fe9a95392148fb73985ac0e5a47b4e4d18098#egg=pytest_asyncio
six==1.12.0