Persistent failure when submitting many API requests concurrently

[tbarsballe]

When submitting many (1000+) quick-search requests concurrently using asyncio.gather, they consistently fail with a planet.exceptions.TooManyRequests: max rate reached: retry-in 200ms error.

I have also occasionally seen a planet.exceptions.ServerError: Internal Server Error instead, but only when using a shared session among all requests; using a unique session for each request seems to only produce the TooManyRequests error.

This failure is consistent, and happens every time I attempt this.

Stacktrace of the TooManyRequests error is:

unhandled exception during asyncio.run() shutdown
task: <Task finished name='Task-111' coro=<do_searches.<locals>.do_search_geom() done, defined at /Users/torben.barsballe/repos/Planet/order-util/order_aois.py:108> exception=CloseError(SSLError(1, '[SSL: APPLICATION_DATA_AFTER_CLOSE_NOTIFY] application data after close notify (_ssl.c:2756)'))>
Traceback (most recent call last):
  File "/Users/torben.barsballe/repos/Planet/order-util/order_aois.py", line 113, in do_search_geom
    itms = await do_search(start, end, geom)
  File "/Users/torben.barsballe/repos/Planet/order-util/order_aois.py", line 97, in do_search
    items.append(item)
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/httpcore/_async/connection.py", line 183, in aclose
    await self.connection.aclose()
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/httpcore/_async/http11.py", line 202, in aclose
    await self.socket.aclose()
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/httpcore/_backends/asyncio.py", line 198, in aclose
    await self.stream_writer.wait_closed()  # type: ignore
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/httpcore/_exceptions.py", line 12, in map_exceptions
    raise to_exc(exc) from None
httpcore.CloseError: [SSL: APPLICATION_DATA_AFTER_CLOSE_NOTIFY] application data after close notify (_ssl.c:2756)
unhandled exception during asyncio.run() shutdown
task: <Task finished name='Task-79' coro=<do_searches.<locals>.do_search_geom() done, defined at /Users/torben.barsballe/repos/Planet/order-util/order_aois.py:108> exception=CloseError(SSLError(1, '[SSL: APPLICATION_DATA_AFTER_CLOSE_NOTIFY] application data after close notify (_ssl.c:2756)'))>
Traceback (most recent call last):
  File "/Users/torben.barsballe/repos/Planet/order-util/order_aois.py", line 113, in do_search_geom
    itms = await do_search(start, end, geom)
  File "/Users/torben.barsballe/repos/Planet/order-util/order_aois.py", line 97, in do_search
    items.append(item)
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/httpcore/_async/connection.py", line 183, in aclose
    await self.connection.aclose()
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/httpcore/_async/http11.py", line 202, in aclose
    await self.socket.aclose()
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/httpcore/_backends/asyncio.py", line 198, in aclose
    await self.stream_writer.wait_closed()  # type: ignore
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/httpcore/_exceptions.py", line 12, in map_exceptions
    raise to_exc(exc) from None
httpcore.CloseError: [SSL: APPLICATION_DATA_AFTER_CLOSE_NOTIFY] application data after close notify (_ssl.c:2756)
Traceback (most recent call last):
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/planet/models.py", line 317, in __anext__
    item = self._items.pop(0)
IndexError: pop from empty list
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/Users/torben.barsballe/repos/Planet/order-util/order_aois.py", line 118, in do_searches
    results = await asyncio.gather(*tasks)
  File "/Users/torben.barsballe/repos/Planet/order-util/order_aois.py", line 113, in do_search_geom
    itms = await do_search(start, end, geom)
  File "/Users/torben.barsballe/repos/Planet/order-util/order_aois.py", line 95, in do_search
    async for item in await search_results:
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/planet/models.py", line 321, in __anext__
    page = await self._pages.__anext__()
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/planet/models.py", line 333, in _get_pages
    resp = await self._do_request(self.request)
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/planet/clients/data.py", line 86, in _do_request
    return await self._session.request(request)
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/planet/http.py", line 230, in request
    return await self._retry(self._request, request, stream=stream)
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/planet/http.py", line 178, in _retry
    raise e
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/planet/http.py", line 174, in _retry
    resp = await func(*a, **kw)
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/planet/http.py", line 234, in _request
    http_resp = await self._client.send(request.http_request,
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/httpx/_client.py", line 1422, in send
    await hook(response)
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/planet/http.py", line 135, in araise_for_status
    return self._raise_for_status(*args, **kwargs)
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/planet/http.py", line 77, in _raise_for_status
    raise exception(msg)
planet.exceptions.TooManyRequests: max rate reached: retry-in 200ms

Stacktrace of the ServerError is:

unhandled exception during asyncio.run() shutdown
task: <Task finished name='Task-938' coro=<do_searches.<locals>.do_search_geom() done, defined at /Users/torben.barsballe/repos/Planet/order-util/order_aois.py:109> exception=RemoteProtocolError("can't handle event type ConnectionClosed when role=SERVER and state=SEND_RESPONSE")>
Traceback (most recent call last):
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/httpx/_exceptions.py", line 326, in map_exceptions
    yield
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/httpx/_client.py", line 1502, in _send_single_request
    (status_code, headers, stream, ext,) = await transport.arequest(
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 218, in arequest
    response = await connection.arequest(
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/httpcore/_async/connection.py", line 106, in arequest
    return await self.connection.arequest(method, url, headers, stream, ext)
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/httpcore/_async/http11.py", line 72, in arequest
    ) = await self._receive_response(timeout)
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/httpcore/_async/http11.py", line 133, in _receive_response
    event = await self._receive_event(timeout)
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/httpcore/_async/http11.py", line 169, in _receive_event
    event = self.h11_state.next_event()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/httpcore/_exceptions.py", line 12, in map_exceptions
    raise to_exc(exc) from None
httpcore.RemoteProtocolError: can't handle event type ConnectionClosed when role=SERVER and state=SEND_RESPONSE
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/Users/torben.barsballe/repos/Planet/order-util/order_aois.py", line 114, in do_search_geom
    itms = await do_search(sess, start, end, geom)
  File "/Users/torben.barsballe/repos/Planet/order-util/order_aois.py", line 96, in do_search
    async for item in await search_results:
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/planet/models.py", line 321, in __anext__
    page = await self._pages.__anext__()
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/planet/http.py", line 174, in _retry
    resp = await func(*a, **kw)
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/planet/http.py", line 234, in _request
    http_resp = await self._client.send(request.http_request,
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/httpx/_client.py", line 1406, in send
    response = await self._send_handling_auth(
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/httpx/_client.py", line 1444, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/httpx/_client.py", line 1476, in _send_handling_redirects
    response = await self._send_single_request(request, timeout)
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/httpx/_client.py", line 1502, in _send_single_request
    (status_code, headers, stream, ext,) = await transport.arequest(
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/httpx/_exceptions.py", line 343, in map_exceptions
    raise mapped_exc(message, **kwargs) from exc  # type: ignore
httpx.RemoteProtocolError: can't handle event type ConnectionClosed when role=SERVER and state=SEND_RESPONSE
Traceback (most recent call last):
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/planet/models.py", line 317, in __anext__
    item = self._items.pop(0)
IndexError: pop from empty list
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/Users/torben.barsballe/repos/Planet/order-util/order_aois.py", line 120, in do_searches
    results = await asyncio.gather(*tasks)
  File "/Users/torben.barsballe/repos/Planet/order-util/order_aois.py", line 114, in do_search_geom
    itms = await do_search(sess, start, end, geom)
  File "/Users/torben.barsballe/repos/Planet/order-util/order_aois.py", line 96, in do_search
    async for item in await search_results:
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/planet/models.py", line 321, in __anext__
    page = await self._pages.__anext__()
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/planet/models.py", line 333, in _get_pages
    resp = await self._do_request(self.request)
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/planet/clients/data.py", line 86, in _do_request
    return await self._session.request(request)
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/planet/http.py", line 230, in request
    return await self._retry(self._request, request, stream=stream)
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/planet/http.py", line 174, in _retry
    resp = await func(*a, **kw)
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/planet/http.py", line 234, in _request
    http_resp = await self._client.send(request.http_request,
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/httpx/_client.py", line 1422, in send
    await hook(response)
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/planet/http.py", line 135, in araise_for_status
    return self._raise_for_status(*args, **kwargs)
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/planet/http.py", line 77, in _raise_for_status
    raise exception(msg)
planet.exceptions.ServerError: Internal Server Error
python-BaseException

[jreiberkyle]

First thought: we definitely want to use one Session for all requests. Within a Session we can throttle to avoid TooManyRequests. If one Session is used for each function call, we can't help. @tbarsballe was it your first instinct to use multiple Session instances? Any ideas how we can direct the user to avoid this approach in favor of using a single session?

[jreiberkyle]

Findings from using one Session with 100 or 1000 concurrent calls, adjusting the httpx settings read timeout and max_connections:

• A httpx.PoolTimout is raised when 1) max_connections is 50 and number of concurrent calls is 100 or 1000 and 2) max_connections is 100 (the httpx default), read timeout is high, and number of concurrent calls is 1000. Conclusion: limit the number of concurrent calls to the same number as max_connections to avoid httpx.PoolTimeout
• The test completed successfully when read timeout was set to 60s, max_connections was set to 100, num_concurrent was 100, and limit was 1000, aka a high read_timeout and num_concurrent = max_connections. The number of TooManyRequests handled was 243. Total there were 400 successful requests. So the ratio of TooManyRequests to successful response is ~ 1:2, pretty high. Conclusion: throttle requests so we don't trigger so many TooManyRequests errors.

tests/functional/test_concurrent_calls.py

import asyncio
import logging

import pytest

import planet
from planet import data_filter

LOGGER = logging.getLogger(__name__)


@pytest.mark.asyncio
@pytest.mark.parametrize('limit, num_concurrent', [(1000, 100), (100, 1000)])
@pytest.mark.parametrize('read_timeout', [10, 60])
@pytest.mark.parametrize('max_connections', [50, 100])
async def test_no_limiting_throttling(limit,
                                      num_concurrent,
                                      read_timeout,
                                      max_connections,
                                      monkeypatch):
    LOGGER.warning(
        'order: max_connections, read_timeout, limit, num_concurrent')

    async def _search(cl):
        sfilter = data_filter.permission_filter()
        items = await cl.quick_search(['PSScene'], sfilter, limit=limit)
        return list([i async for i in items])

    monkeypatch.setattr(planet.http, 'READ_TIMEOUT', read_timeout)
    monkeypatch.setattr(planet.http, 'MAX_CONNECTIONS', max_connections)
    monkeypatch.setattr(planet.http, 'RATE_LIMIT', 0)  # disable throttling
    monkeypatch.setattr(planet.http, 'MAX_ACTIVE', 0)  # disable limiting

    async with planet.Session() as sess:
        cl = planet.DataClient(sess)

        tasks = [asyncio.create_task(func(cl)) for _ in range(num_concurrent)]
        try:
            await asyncio.gather(*tasks, return_exceptions=False)
        except BaseException as e:
            LOGGER.warning(f'{type(e)} raised. message: {e}')
            raise
        finally:
            for t in tasks:
                t.cancel()
            LOGGER.warning(sess.handled_exceptions)

command line

pytest tests/functional/test_concurrent_calls.py --tb=line --durations=0 --log-cli-level=warning | tee no-limit-throttling.log

no-limit-throttling.log

tests/functional/test_concurrent_calls.py::test_no_limiting_throttling[50-10-1000-100] 
-------------------------------- live log call ---------------------------------
2022-07-11 10:47:35 WARNING order: max_connections, read_timeout, limit, num_concurrent
2022-07-11 10:47:51 WARNING <class 'httpx.PoolTimeout'> raised. message: 
2022-07-11 10:47:51 WARNING Counter({'TooManyRequests': 88, 'Successful': 12})
FAILED                                                                   [ 12%]
tests/functional/test_concurrent_calls.py::test_no_limiting_throttling[50-10-100-1000] 
-------------------------------- live log call ---------------------------------
2022-07-11 10:47:51 WARNING order: max_connections, read_timeout, limit, num_concurrent
2022-07-11 10:48:01 WARNING <class 'httpx.PoolTimeout'> raised. message: 
2022-07-11 10:48:01 WARNING Counter({'TooManyRequests': 43, 'Successful': 7})
FAILED                                                                   [ 25%]
tests/functional/test_concurrent_calls.py::test_no_limiting_throttling[50-60-1000-100] 
-------------------------------- live log call ---------------------------------
2022-07-11 10:48:01 WARNING order: max_connections, read_timeout, limit, num_concurrent
2022-07-11 10:48:11 WARNING <class 'httpx.PoolTimeout'> raised. message: 
2022-07-11 10:48:11 WARNING Counter({'TooManyRequests': 44, 'Successful': 6})
FAILED                                                                   [ 37%]
tests/functional/test_concurrent_calls.py::test_no_limiting_throttling[50-60-100-1000] 
-------------------------------- live log call ---------------------------------
2022-07-11 10:48:11 WARNING order: max_connections, read_timeout, limit, num_concurrent
2022-07-11 10:48:22 WARNING <class 'httpx.PoolTimeout'> raised. message: 
2022-07-11 10:48:22 WARNING Counter({'TooManyRequests': 44, 'Successful': 6})
FAILED                                                                   [ 50%]
tests/functional/test_concurrent_calls.py::test_no_limiting_throttling[100-10-1000-100] 
-------------------------------- live log call ---------------------------------
2022-07-11 10:48:22 WARNING order: max_connections, read_timeout, limit, num_concurrent
2022-07-11 10:48:32 WARNING <class 'httpx.ReadTimeout'> raised. message: 
2022-07-11 10:48:32 WARNING Counter({'TooManyRequests': 195, 'Successful': 16})
FAILED                                                                   [ 62%]
tests/functional/test_concurrent_calls.py::test_no_limiting_throttling[100-10-100-1000] 
-------------------------------- live log call ---------------------------------
2022-07-11 10:48:32 WARNING order: max_connections, read_timeout, limit, num_concurrent
2022-07-11 10:48:43 WARNING <class 'httpx.ReadTimeout'> raised. message: 
2022-07-11 10:48:43 WARNING Counter({'TooManyRequests': 85, 'Successful': 11})
FAILED                                                                   [ 75%]
tests/functional/test_concurrent_calls.py::test_no_limiting_throttling[100-60-1000-100] 
-------------------------------- live log call ---------------------------------
2022-07-11 10:48:43 WARNING order: max_connections, read_timeout, limit, num_concurrent
2022-07-11 10:50:06 WARNING Counter({'Successful': 400, 'TooManyRequests': 243})
PASSED                                                                   [ 87%]
tests/functional/test_concurrent_calls.py::test_no_limiting_throttling[100-60-100-1000] 
-------------------------------- live log call ---------------------------------
2022-07-11 10:50:07 WARNING order: max_connections, read_timeout, limit, num_concurrent
2022-07-11 10:50:17 WARNING <class 'httpx.PoolTimeout'> raised. message: 
2022-07-11 10:50:17 WARNING Counter({'TooManyRequests': 88, 'Successful': 12})
FAILED                                                                   [100%]


=================================== FAILURES ===================================
/Users/jennifer.kyle/.pyenv/versions/3.7.9/envs/planet-client-python-3.7.9/lib/python3.7/site-packages/httpx/_exceptions.py:343: httpx.PoolTimeout
/Users/jennifer.kyle/.pyenv/versions/3.7.9/envs/planet-client-python-3.7.9/lib/python3.7/site-packages/httpx/_exceptions.py:343: httpx.PoolTimeout
/Users/jennifer.kyle/.pyenv/versions/3.7.9/envs/planet-client-python-3.7.9/lib/python3.7/site-packages/httpx/_exceptions.py:343: httpx.PoolTimeout
/Users/jennifer.kyle/.pyenv/versions/3.7.9/envs/planet-client-python-3.7.9/lib/python3.7/site-packages/httpx/_exceptions.py:343: httpx.PoolTimeout
/Users/jennifer.kyle/.pyenv/versions/3.7.9/envs/planet-client-python-3.7.9/lib/python3.7/site-packages/httpx/_exceptions.py:343: httpx.ReadTimeout
/Users/jennifer.kyle/.pyenv/versions/3.7.9/envs/planet-client-python-3.7.9/lib/python3.7/site-packages/httpx/_exceptions.py:343: httpx.ReadTimeout
/Users/jennifer.kyle/.pyenv/versions/3.7.9/envs/planet-client-python-3.7.9/lib/python3.7/site-packages/httpx/_exceptions.py:343: httpx.PoolTimeout
=============================== warnings summary ===============================
../../.pyenv/versions/3.7.9/envs/planet-client-python-3.7.9/lib/python3.7/site-packages/pytest_asyncio/plugin.py:191
  /Users/jennifer.kyle/.pyenv/versions/3.7.9/envs/planet-client-python-3.7.9/lib/python3.7/site-packages/pytest_asyncio/plugin.py:191: DeprecationWarning: The 'asyncio_mode' default value will change to 'strict' in future, please explicitly use 'asyncio_mode=strict' or 'asyncio_mode=auto' in pytest configuration file.
    config.issue_config_time_warning(LEGACY_MODE, stacklevel=2)

-- Docs: https://docs.pytest.org/en/stable/warnings.html
============================== slowest durations ===============================
84.32s call     functional/test_concurrent_calls.py::test_no_limiting_throttling[100-60-1000-100]
15.93s call     functional/test_concurrent_calls.py::test_no_limiting_throttling[50-10-1000-100]
10.51s call     functional/test_concurrent_calls.py::test_no_limiting_throttling[100-10-1000-100]
10.36s call     functional/test_concurrent_calls.py::test_no_limiting_throttling[100-10-100-1000]
10.35s call     functional/test_concurrent_calls.py::test_no_limiting_throttling[50-60-100-1000]
10.33s call     functional/test_concurrent_calls.py::test_no_limiting_throttling[100-60-100-1000]
10.32s call     functional/test_concurrent_calls.py::test_no_limiting_throttling[50-10-100-1000]
10.10s call     functional/test_concurrent_calls.py::test_no_limiting_throttling[50-60-1000-100]

(16 durations < 0.005s hidden.  Use -vv to show these durations.)
=========================== short test summary info ============================
FAILED tests/functional/test_concurrent_calls.py::test_no_limiting_throttling[50-10-1000-100]
FAILED tests/functional/test_concurrent_calls.py::test_no_limiting_throttling[50-10-100-1000]
FAILED tests/functional/test_concurrent_calls.py::test_no_limiting_throttling[50-60-1000-100]
FAILED tests/functional/test_concurrent_calls.py::test_no_limiting_throttling[50-60-100-1000]
FAILED tests/functional/test_concurrent_calls.py::test_no_limiting_throttling[100-10-1000-100]
FAILED tests/functional/test_concurrent_calls.py::test_no_limiting_throttling[100-10-100-1000]
FAILED tests/functional/test_concurrent_calls.py::test_no_limiting_throttling[100-60-100-1000]
============== 7 failed, 1 passed, 1 warning in 162.27s (0:02:42) ==============

[jreiberkyle]

Findings from using one Session with 500 concurrent calls, adjusting the httpx settings read timeout and the throttling rate and limit on active (aka, sending, waiting, or receiving response from server) communications with the server:

• Almost all tests failed with a httpx.ReadTimeout error with read timeout set to 10s. The only test that succeeded had a limit of 50 active communications with the server and a throttle rate of 5s. Conclusion: minimum read_timeout should be 30s.
• For tests with read_timeout of 30s (all successful):
  • The number of TooManyRequests errors handled does not have much of a pattern. Generally, the number is ~200 for throttle rates 4.5/5, ~425 for throttle rates 9.8/10, but there is an anomaly where only 145 were handled for throttle rate 9.8, max active communications 50. Conclusion: lower throttle rates will lead to fewer TooManyRequests errors but will not completely remove them.
  • There are two sets of runtimes, the shortest being with throttle rate of 9.8/10. For each of these sets, the shortest runtime is with active communications to server limited to 50. For 50 active connections, the runtime of 9.8/10 (~112s) was 40s shorter than the runtime of 4.5/5 (~153s). Conclusion: Limit active communications to 50 and throttle rate to 10s.
  • One httpx.RemoteProtocolError was caught and handled in the test. This needs to be added to the exceptions that result in a retry instead of failure.

tests/functional/test_concurrent_calls.py

import asyncio
import logging

import pytest

import planet
from planet import data_filter

LOGGER = logging.getLogger(__name__)


@pytest.mark.asyncio
@pytest.mark.parametrize('execution_number', range(1))
@pytest.mark.parametrize('limit, num_concurrent', [(10, 500)])
@pytest.mark.parametrize('read_timeout', [10, 30])
@pytest.mark.parametrize('max_connections', [100])
@pytest.mark.parametrize('max_active', [50, 100])
@pytest.mark.parametrize('rate_limit', [10, 9.8, 5, 4.5])
async def test_limiting_throttling(execution_number,
                                   limit,
                                   num_concurrent,
                                   read_timeout,
                                   max_connections,
                                   rate_limit,
                                   max_active,
                                   monkeypatch):
    LOGGER.warning(
        'order: rate_limit, max_active, max_connections, read_timeout, limit, num_concurrent, execution_number'
    )

    async def _search(cl):
        sfilter = data_filter.permission_filter()
        items = await cl.quick_search(['PSScene'], sfilter, limit=limit)
        return list([i async for i in items])

    await _runit(_search,
                 planet.DataClient,
                 monkeypatch,
                 num_concurrent=num_concurrent,
                 read_timeout=read_timeout,
                 max_connections=max_connections,
                 rate_limit=rate_limit,
                 max_active=max_active)


async def _runit(func,
                 client,
                 monkeypatch,
                 num_concurrent,
                 read_timeout,
                 max_connections,
                 rate_limit,
                 max_active):
    monkeypatch.setattr(planet.http, 'READ_TIMEOUT', read_timeout)
    monkeypatch.setattr(planet.http, 'MAX_CONNECTIONS', max_connections)
    monkeypatch.setattr(planet.http, 'RATE_LIMIT', rate_limit)
    monkeypatch.setattr(planet.http, 'MAX_ACTIVE', max_active)

    async with planet.Session() as sess:
        cl = client(sess)

        tasks = [asyncio.create_task(func(cl)) for _ in range(num_concurrent)]
        try:
            await asyncio.gather(*tasks, return_exceptions=False)
        except BaseException as e:
            LOGGER.warning(f'{type(e)} raised. message: {e}')
            raise
        finally:
            for t in tasks:
                t.cancel()
            LOGGER.warning(sess.handled_exceptions)

command line

pytest tests/functional/test_concurrent_calls.py --tb=line --durations=0 --log-cli-level=warning -k test_limiting_throttling | tee limiting-throttling.log

limiting-throttling.log

============================= test session starts ==============================
platform darwin -- Python 3.7.9, pytest-6.2.1, py-1.10.0, pluggy-0.13.1
rootdir: /Users/jennifer.kyle/pl/planet-client-python/tests, configfile: pytest.ini
plugins: requests-mock-1.8.0, anyio-3.4.0, cov-2.10.1, asyncio-0.18.0
asyncio: mode=legacy
collected 24 items / 8 deselected / 16 selected

tests/functional/test_concurrent_calls.py::test_limiting_throttling[10-50-100-10-10-500-0] 
-------------------------------- live log call ---------------------------------
2022-07-11 10:54:09 WARNING order: rate_limit, max_active, max_connections, read_timeout, limit, num_concurrent, execution_number
2022-07-11 10:54:36 WARNING <class 'httpx.ReadTimeout'> raised. message: 
2022-07-11 10:54:36 WARNING Counter({'TooManyRequests': 141, 'Successful': 95})
FAILED                                                                   [  6%]
tests/functional/test_concurrent_calls.py::test_limiting_throttling[10-50-100-30-10-500-0] 
-------------------------------- live log call ---------------------------------
2022-07-11 10:54:36 WARNING order: rate_limit, max_active, max_connections, read_timeout, limit, num_concurrent, execution_number
2022-07-11 10:56:33 WARNING Counter({'Successful': 500, 'TooManyRequests': 381})
PASSED                                                                   [ 12%]
tests/functional/test_concurrent_calls.py::test_limiting_throttling[10-100-100-10-10-500-0] 
-------------------------------- live log call ---------------------------------
2022-07-11 10:56:33 WARNING order: rate_limit, max_active, max_connections, read_timeout, limit, num_concurrent, execution_number
2022-07-11 10:56:51 WARNING <class 'httpx.ReadTimeout'> raised. message: 
2022-07-11 10:56:51 WARNING Counter({'TooManyRequests': 91, 'Successful': 61})
FAILED                                                                   [ 18%]
tests/functional/test_concurrent_calls.py::test_limiting_throttling[10-100-100-30-10-500-0] 
-------------------------------- live log call ---------------------------------
2022-07-11 10:56:51 WARNING order: rate_limit, max_active, max_connections, read_timeout, limit, num_concurrent, execution_number
2022-07-11 10:59:01 WARNING Counter({'Successful': 500, 'TooManyRequests': 468})
PASSED                                                                   [ 25%]
tests/functional/test_concurrent_calls.py::test_limiting_throttling[9.8-50-100-10-10-500-0] 
-------------------------------- live log call ---------------------------------
2022-07-11 10:59:01 WARNING order: rate_limit, max_active, max_connections, read_timeout, limit, num_concurrent, execution_number
2022-07-11 10:59:25 WARNING <class 'httpx.ReadTimeout'> raised. message: 
2022-07-11 10:59:25 WARNING Counter({'TooManyRequests': 116, 'Successful': 87})
FAILED                                                                   [ 31%]
tests/functional/test_concurrent_calls.py::test_limiting_throttling[9.8-50-100-30-10-500-0] 
-------------------------------- live log call ---------------------------------
2022-07-11 10:59:25 WARNING order: rate_limit, max_active, max_connections, read_timeout, limit, num_concurrent, execution_number
2022-07-11 11:01:17 WARNING Counter({'Successful': 500, 'TooManyRequests': 145})
PASSED                                                                   [ 37%]
tests/functional/test_concurrent_calls.py::test_limiting_throttling[9.8-100-100-10-10-500-0] 
-------------------------------- live log call ---------------------------------
2022-07-11 11:01:17 WARNING order: rate_limit, max_active, max_connections, read_timeout, limit, num_concurrent, execution_number
2022-07-11 11:01:51 WARNING <class 'httpx.ReadTimeout'> raised. message: 
2022-07-11 11:01:51 WARNING Counter({'TooManyRequests': 178, 'Successful': 131})
FAILED                                                                   [ 43%]
tests/functional/test_concurrent_calls.py::test_limiting_throttling[9.8-100-100-30-10-500-0] 
-------------------------------- live log call ---------------------------------
2022-07-11 11:01:51 WARNING order: rate_limit, max_active, max_connections, read_timeout, limit, num_concurrent, execution_number
2022-07-11 11:04:14 WARNING Counter({'Successful': 500, 'TooManyRequests': 478})
PASSED                                                                   [ 50%]
tests/functional/test_concurrent_calls.py::test_limiting_throttling[5-50-100-10-10-500-0] 
-------------------------------- live log call ---------------------------------
2022-07-11 11:04:14 WARNING order: rate_limit, max_active, max_connections, read_timeout, limit, num_concurrent, execution_number
2022-07-11 11:06:52 WARNING Counter({'Successful': 500, 'TooManyRequests': 198})
PASSED                                                                   [ 56%]
tests/functional/test_concurrent_calls.py::test_limiting_throttling[5-50-100-30-10-500-0] 
-------------------------------- live log call ---------------------------------
2022-07-11 11:06:52 WARNING order: rate_limit, max_active, max_connections, read_timeout, limit, num_concurrent, execution_number
2022-07-11 11:09:25 WARNING Counter({'Successful': 500, 'TooManyRequests': 178, 'RemoteProtocolError': 1})
PASSED                                                                   [ 62%]
tests/functional/test_concurrent_calls.py::test_limiting_throttling[5-100-100-10-10-500-0] 
-------------------------------- live log call ---------------------------------
2022-07-11 11:09:25 WARNING order: rate_limit, max_active, max_connections, read_timeout, limit, num_concurrent, execution_number
2022-07-11 11:10:02 WARNING <class 'httpx.ReadTimeout'> raised. message: 
2022-07-11 11:10:02 WARNING Counter({'Successful': 116, 'TooManyRequests': 49})
FAILED                                                                   [ 68%]
tests/functional/test_concurrent_calls.py::test_limiting_throttling[5-100-100-30-10-500-0] 
-------------------------------- live log call ---------------------------------
2022-07-11 11:10:02 WARNING order: rate_limit, max_active, max_connections, read_timeout, limit, num_concurrent, execution_number
2022-07-11 11:12:39 WARNING Counter({'Successful': 500, 'TooManyRequests': 202})
PASSED                                                                   [ 75%]
tests/functional/test_concurrent_calls.py::test_limiting_throttling[4.5-50-100-10-10-500-0] 
-------------------------------- live log call ---------------------------------
2022-07-11 11:12:39 WARNING order: rate_limit, max_active, max_connections, read_timeout, limit, num_concurrent, execution_number
2022-07-11 11:14:09 WARNING <class 'httpx.ReadTimeout'> raised. message: 
2022-07-11 11:14:09 WARNING Counter({'Successful': 292, 'TooManyRequests': 134})
FAILED                                                                   [ 81%]
tests/functional/test_concurrent_calls.py::test_limiting_throttling[4.5-50-100-30-10-500-0] 
-------------------------------- live log call ---------------------------------
2022-07-11 11:14:09 WARNING order: rate_limit, max_active, max_connections, read_timeout, limit, num_concurrent, execution_number
2022-07-11 11:16:42 WARNING Counter({'Successful': 500, 'TooManyRequests': 181})
PASSED                                                                   [ 87%]
tests/functional/test_concurrent_calls.py::test_limiting_throttling[4.5-100-100-10-10-500-0] 
-------------------------------- live log call ---------------------------------
2022-07-11 11:16:42 WARNING order: rate_limit, max_active, max_connections, read_timeout, limit, num_concurrent, execution_number
2022-07-11 11:17:01 WARNING <class 'httpx.ReadTimeout'> raised. message: 
2022-07-11 11:17:01 WARNING Counter({'Successful': 44, 'TooManyRequests': 23})
FAILED                                                                   [ 93%]
tests/functional/test_concurrent_calls.py::test_limiting_throttling[4.5-100-100-30-10-500-0] 
-------------------------------- live log call ---------------------------------
2022-07-11 11:17:01 WARNING order: rate_limit, max_active, max_connections, read_timeout, limit, num_concurrent, execution_number
2022-07-11 11:19:36 WARNING Counter({'Successful': 500, 'TooManyRequests': 203, 'RemoteProtocolError': 1})
PASSED                                                                   [100%]

=================================== FAILURES ===================================
/Users/jennifer.kyle/.pyenv/versions/3.7.9/envs/planet-client-python-3.7.9/lib/python3.7/site-packages/httpx/_exceptions.py:343: httpx.ReadTimeout
/Users/jennifer.kyle/.pyenv/versions/3.7.9/envs/planet-client-python-3.7.9/lib/python3.7/site-packages/httpx/_exceptions.py:343: httpx.ReadTimeout
/Users/jennifer.kyle/.pyenv/versions/3.7.9/envs/planet-client-python-3.7.9/lib/python3.7/site-packages/httpx/_exceptions.py:343: httpx.ReadTimeout
/Users/jennifer.kyle/.pyenv/versions/3.7.9/envs/planet-client-python-3.7.9/lib/python3.7/site-packages/httpx/_exceptions.py:343: httpx.ReadTimeout
/Users/jennifer.kyle/.pyenv/versions/3.7.9/envs/planet-client-python-3.7.9/lib/python3.7/site-packages/httpx/_exceptions.py:343: httpx.ReadTimeout
/Users/jennifer.kyle/.pyenv/versions/3.7.9/envs/planet-client-python-3.7.9/lib/python3.7/site-packages/httpx/_exceptions.py:343: httpx.ReadTimeout
/Users/jennifer.kyle/.pyenv/versions/3.7.9/envs/planet-client-python-3.7.9/lib/python3.7/site-packages/httpx/_exceptions.py:343: httpx.ReadTimeout
=============================== warnings summary ===============================
../../.pyenv/versions/3.7.9/envs/planet-client-python-3.7.9/lib/python3.7/site-packages/pytest_asyncio/plugin.py:191
  /Users/jennifer.kyle/.pyenv/versions/3.7.9/envs/planet-client-python-3.7.9/lib/python3.7/site-packages/pytest_asyncio/plugin.py:191: DeprecationWarning: The 'asyncio_mode' default value will change to 'strict' in future, please explicitly use 'asyncio_mode=strict' or 'asyncio_mode=auto' in pytest configuration file.
    config.issue_config_time_warning(LEGACY_MODE, stacklevel=2)

-- Docs: https://docs.pytest.org/en/stable/warnings.html
============================== slowest durations ===============================
157.61s call     functional/test_concurrent_calls.py::test_limiting_throttling[5-50-100-10-10-500-0]
156.18s call     functional/test_concurrent_calls.py::test_limiting_throttling[5-100-100-30-10-500-0]
155.04s call     functional/test_concurrent_calls.py::test_limiting_throttling[4.5-100-100-30-10-500-0]
153.64s call     functional/test_concurrent_calls.py::test_limiting_throttling[4.5-50-100-30-10-500-0]
153.54s call     functional/test_concurrent_calls.py::test_limiting_throttling[5-50-100-30-10-500-0]
143.27s call     functional/test_concurrent_calls.py::test_limiting_throttling[9.8-100-100-30-10-500-0]
129.79s call     functional/test_concurrent_calls.py::test_limiting_throttling[10-100-100-30-10-500-0]
116.90s call     functional/test_concurrent_calls.py::test_limiting_throttling[10-50-100-30-10-500-0]
111.38s call     functional/test_concurrent_calls.py::test_limiting_throttling[9.8-50-100-30-10-500-0]
90.12s call     functional/test_concurrent_calls.py::test_limiting_throttling[4.5-50-100-10-10-500-0]
37.09s call     functional/test_concurrent_calls.py::test_limiting_throttling[5-100-100-10-10-500-0]
34.09s call     functional/test_concurrent_calls.py::test_limiting_throttling[9.8-100-100-10-10-500-0]
27.06s call     functional/test_concurrent_calls.py::test_limiting_throttling[10-50-100-10-10-500-0]
24.25s call     functional/test_concurrent_calls.py::test_limiting_throttling[9.8-50-100-10-10-500-0]
18.38s call     functional/test_concurrent_calls.py::test_limiting_throttling[4.5-100-100-10-10-500-0]
18.08s call     functional/test_concurrent_calls.py::test_limiting_throttling[10-100-100-10-10-500-0]

(32 durations < 0.005s hidden.  Use -vv to show these durations.)
=========================== short test summary info ============================
FAILED tests/functional/test_concurrent_calls.py::test_limiting_throttling[10-50-100-10-10-500-0]
FAILED tests/functional/test_concurrent_calls.py::test_limiting_throttling[10-100-100-10-10-500-0]
FAILED tests/functional/test_concurrent_calls.py::test_limiting_throttling[9.8-50-100-10-10-500-0]
FAILED tests/functional/test_concurrent_calls.py::test_limiting_throttling[9.8-100-100-10-10-500-0]
FAILED tests/functional/test_concurrent_calls.py::test_limiting_throttling[5-100-100-10-10-500-0]
FAILED tests/functional/test_concurrent_calls.py::test_limiting_throttling[4.5-50-100-10-10-500-0]
FAILED tests/functional/test_concurrent_calls.py::test_limiting_throttling[4.5-100-100-10-10-500-0]
====== 7 failed, 9 passed, 8 deselected, 1 warning in 1526.52s (0:25:26) =======

[jreiberkyle]

Ok, so I realized something about throttling: the API (specifically, data, search endpoint) returns a TooManyRequests error if just sending two requests, but with less than 200ms delay. This is different than my initial understanding, which was that it would return the TooManyRequests if one submitted more than 5 requests in a second. That changed how I needed to do throttling. Now throttling based on cadence, that is, enforcing a delay. see new _Limiter class. This change has significantly decreased the TooManyRequest errors that hand to be handled.

I also changed the test code to wait until all concurrent calls are returned and report all errors that occurred, instead of just bubbling up the first error. This gives us more information about failure paths. See test.

Findings:

• All of the tests with limiter set to max rate of 10/s, max workers of 100 failed. Errors were: ReadTimeout, ConnectError, and planet.APIError (I am guessing bad gateway based on previous tests when I was monitoring the debug logs). Two of the 5/s tests were affected by my computer going to sleep, which resulted in failures out of the control of the SDK. All of the other tests passed.
• The shortest runtime in both the limit=1000, num_concurrent=100 and limit=100, num_concurrent=1000 scenarios was with the max rate of 10/s, max workers of 50. for limit/num_concurrent 100/1000 10s/50workers was approx 200s, 5s/100workers was approx 235s and 5s/50workers was approx 315s. for lim/num_concurrent 1000/100 10s/50workers was approx 87s, 5s/100workers and 5s/50workers was approx 100s
• For num_concurrent=1000 scenarios, at 10s rate limit/50 workers, ~140 TooManyRequests were handled. This dropped to ~20 for 5s rate limit (with both 50 and 100 workers).

Conclusions:

• Add ReadTimeout, ConnectError, and bad gateway to errors to retry
• Best setting before adding those errors to retry is 10s/50workers

command line

pytest tests/functional/test_concurrent_calls.py --tb=line --durations=0 --log-cli-level=warning \
    -k test_max_active_limit_concurrent

rate_limit_max_active-100-1000.log

============================= test session starts ==============================
platform darwin -- Python 3.7.9, pytest-6.2.1, py-1.10.0, pluggy-0.13.1
rootdir: /Users/jennifer.kyle/pl/planet-client-python/tests, configfile: pytest.ini
plugins: asyncio-0.16.0, requests-mock-1.8.0, anyio-3.4.0, cov-2.10.1
collected 72 items / 48 deselected / 24 selected

tests/functional/test_concurrent_calls.py::test_max_active_limit_concurrent[100-1000-10-50-0] 
-------------------------------- live log call ---------------------------------
2022-07-12 13:05:34 WARNING order: limit, num_concurrent, rate_limit, max_active, execution_number
2022-07-12 13:08:55 WARNING Session outcomes: Counter({'Successful': 1000, <class 'planet.exceptions.TooManyRequests'>: 138, <class 'httpx.RemoteProtocolError'>: 1})
PASSED                                                                   [  4%]
tests/functional/test_concurrent_calls.py::test_max_active_limit_concurrent[100-1000-10-50-1] 
-------------------------------- live log call ---------------------------------
2022-07-12 13:08:56 WARNING order: limit, num_concurrent, rate_limit, max_active, execution_number
2022-07-12 13:12:13 WARNING Session outcomes: Counter({'Successful': 1000, <class 'planet.exceptions.TooManyRequests'>: 135, <class 'httpx.RemoteProtocolError'>: 1})
PASSED                                                                   [  8%]
tests/functional/test_concurrent_calls.py::test_max_active_limit_concurrent[100-1000-10-50-2] 
-------------------------------- live log call ---------------------------------
2022-07-12 13:12:14 WARNING order: limit, num_concurrent, rate_limit, max_active, execution_number
2022-07-12 13:15:47 WARNING Session outcomes: Counter({'Successful': 1000, <class 'planet.exceptions.TooManyRequests'>: 140})
PASSED                                                                   [ 12%]
tests/functional/test_concurrent_calls.py::test_max_active_limit_concurrent[100-1000-10-100-0] 
-------------------------------- live log call ---------------------------------
2022-07-12 13:15:47 WARNING order: limit, num_concurrent, rate_limit, max_active, execution_number
2022-07-12 13:19:02 WARNING Session outcomes: Counter({'Successful': 774, <class 'planet.exceptions.TooManyRequests'>: 136})
2022-07-12 13:19:02 WARNING Task exceptions: Counter({<class 'httpx.ReadTimeout'>: 157, <class 'httpx.ConnectError'>: 69})
FAILED                                                                   [ 16%]
tests/functional/test_concurrent_calls.py::test_max_active_limit_concurrent[100-1000-10-100-1] 
-------------------------------- live log call ---------------------------------
2022-07-12 13:19:02 WARNING order: limit, num_concurrent, rate_limit, max_active, execution_number
2022-07-12 13:22:16 WARNING Session outcomes: Counter({'Successful': 880, <class 'planet.exceptions.TooManyRequests'>: 167})
2022-07-12 13:22:16 WARNING Task exceptions: Counter({<class 'httpx.ReadTimeout'>: 115, <class 'planet.exceptions.APIError'>: 5})
FAILED                                                                   [ 20%]
tests/functional/test_concurrent_calls.py::test_max_active_limit_concurrent[100-1000-10-100-2] 
-------------------------------- live log call ---------------------------------
2022-07-12 13:22:16 WARNING order: limit, num_concurrent, rate_limit, max_active, execution_number
2022-07-12 13:25:41 WARNING Session outcomes: Counter({'Successful': 980, <class 'planet.exceptions.TooManyRequests'>: 184})
2022-07-12 13:25:41 WARNING Task exceptions: Counter({<class 'httpx.ReadTimeout'>: 20})
FAILED                                                                   [ 25%]
tests/functional/test_concurrent_calls.py::test_max_active_limit_concurrent[100-1000-5-50-0] 
-------------------------------- live log call ---------------------------------
2022-07-12 13:25:41 WARNING order: limit, num_concurrent, rate_limit, max_active, execution_number
2022-07-12 13:30:43 WARNING Session outcomes: Counter({'Successful': 1000, <class 'planet.exceptions.TooManyRequests'>: 18})
PASSED                                                                   [ 29%]
tests/functional/test_concurrent_calls.py::test_max_active_limit_concurrent[100-1000-5-50-1] 
-------------------------------- live log call ---------------------------------
2022-07-12 13:30:44 WARNING order: limit, num_concurrent, rate_limit, max_active, execution_number
2022-07-12 17:58:14 WARNING Session outcomes: Counter({'Successful': 885, <class 'planet.exceptions.TooManyRequests'>: 23, <class 'httpx.ReadError'>: 16})
2022-07-12 17:58:14 WARNING Task exceptions: Counter({<class 'httpx.ReadTimeout'>: 92, <class 'httpcore.CloseError'>: 18, <class 'httpx.CloseError'>: 3, <class 'httpx.ConnectError'>: 2})
FAILED                                                                   [ 33%]
tests/functional/test_concurrent_calls.py::test_max_active_limit_concurrent[100-1000-5-50-2] 
-------------------------------- live log call ---------------------------------
2022-07-12 17:58:14 WARNING order: limit, num_concurrent, rate_limit, max_active, execution_number
2022-07-12 18:03:28 WARNING Session outcomes: Counter({'Successful': 1000, <class 'planet.exceptions.TooManyRequests'>: 22, <class 'httpx.RemoteProtocolError'>: 3})
PASSED                                                                   [ 37%]
tests/functional/test_concurrent_calls.py::test_max_active_limit_concurrent[100-1000-5-100-0] 
-------------------------------- live log call ---------------------------------
2022-07-12 18:03:29 WARNING order: limit, num_concurrent, rate_limit, max_active, execution_number
2022-07-12 18:07:39 WARNING Session outcomes: Counter({'Successful': 999, <class 'planet.exceptions.TooManyRequests'>: 16})
2022-07-12 18:07:39 WARNING Task exceptions: Counter({<class 'httpx.WriteError'>: 1})
FAILED                                                                   [ 41%]
tests/functional/test_concurrent_calls.py::test_max_active_limit_concurrent[100-1000-5-100-1] 
-------------------------------- live log call ---------------------------------
2022-07-12 18:07:39 WARNING order: limit, num_concurrent, rate_limit, max_active, execution_number
2022-07-12 18:11:32 WARNING Session outcomes: Counter({'Successful': 1000, <class 'planet.exceptions.TooManyRequests'>: 18, <class 'httpx.RemoteProtocolError'>: 1})
PASSED                                                                   [ 45%]
tests/functional/test_concurrent_calls.py::test_max_active_limit_concurrent[100-1000-5-100-2] 
-------------------------------- live log call ---------------------------------
2022-07-12 18:11:33 WARNING order: limit, num_concurrent, rate_limit, max_active, execution_number
2022-07-12 18:15:29 WARNING Session outcomes: Counter({'Successful': 1000, <class 'planet.exceptions.TooManyRequests'>: 21})
PASSED                                                                   [ 50%]
tests/functional/test_concurrent_calls.py::test_max_active_limit_concurrent[1000-100-10-50-0] 
-------------------------------- live log call ---------------------------------
2022-07-12 18:15:30 WARNING order: limit, num_concurrent, rate_limit, max_active, execution_number
2022-07-12 18:16:59 WARNING Session outcomes: Counter({'Successful': 400, <class 'planet.exceptions.TooManyRequests'>: 51})
PASSED                                                                   [ 54%]
tests/functional/test_concurrent_calls.py::test_max_active_limit_concurrent[1000-100-10-50-1] 
-------------------------------- live log call ---------------------------------
2022-07-12 18:17:00 WARNING order: limit, num_concurrent, rate_limit, max_active, execution_number
2022-07-12 18:18:24 WARNING Session outcomes: Counter({'Successful': 400, <class 'planet.exceptions.TooManyRequests'>: 43})
PASSED                                                                   [ 58%]
tests/functional/test_concurrent_calls.py::test_max_active_limit_concurrent[1000-100-10-50-2] 
-------------------------------- live log call ---------------------------------
2022-07-12 18:18:24 WARNING order: limit, num_concurrent, rate_limit, max_active, execution_number
2022-07-12 18:19:48 WARNING Session outcomes: Counter({'Successful': 400, <class 'planet.exceptions.TooManyRequests'>: 35})
PASSED                                                                   [ 62%]
tests/functional/test_concurrent_calls.py::test_max_active_limit_concurrent[1000-100-10-100-0] 
-------------------------------- live log call ---------------------------------
2022-07-12 18:19:49 WARNING order: limit, num_concurrent, rate_limit, max_active, execution_number
2022-07-12 18:21:14 WARNING Session outcomes: Counter({'Successful': 388, <class 'planet.exceptions.TooManyRequests'>: 46})
2022-07-12 18:21:14 WARNING Task exceptions: Counter({<class 'httpx.ReadTimeout'>: 4, <class 'httpx.ConnectTimeout'>: 1})
FAILED                                                                   [ 66%]
tests/functional/test_concurrent_calls.py::test_max_active_limit_concurrent[1000-100-10-100-1] 
-------------------------------- live log call ---------------------------------
2022-07-12 18:21:14 WARNING order: limit, num_concurrent, rate_limit, max_active, execution_number
2022-07-12 18:22:36 WARNING Session outcomes: Counter({'Successful': 396, <class 'planet.exceptions.TooManyRequests'>: 54})
2022-07-12 18:22:36 WARNING Task exceptions: Counter({<class 'httpx.ConnectError'>: 1})
FAILED                                                                   [ 70%]
tests/functional/test_concurrent_calls.py::test_max_active_limit_concurrent[1000-100-10-100-2] 
-------------------------------- live log call ---------------------------------
2022-07-12 18:22:36 WARNING order: limit, num_concurrent, rate_limit, max_active, execution_number
2022-07-12 18:23:50 WARNING Session outcomes: Counter({'Successful': 302, <class 'planet.exceptions.TooManyRequests'>: 41})
2022-07-12 18:23:50 WARNING Task exceptions: Counter({<class 'httpx.ReadTimeout'>: 35})
FAILED                                                                   [ 75%]
tests/functional/test_concurrent_calls.py::test_max_active_limit_concurrent[1000-100-5-50-0] 
-------------------------------- live log call ---------------------------------
2022-07-12 18:23:50 WARNING order: limit, num_concurrent, rate_limit, max_active, execution_number
2022-07-12 18:25:31 WARNING Session outcomes: Counter({'Successful': 400, <class 'planet.exceptions.TooManyRequests'>: 5, <class 'httpx.RemoteProtocolError'>: 1})
PASSED                                                                   [ 79%]
tests/functional/test_concurrent_calls.py::test_max_active_limit_concurrent[1000-100-5-50-1] 
-------------------------------- live log call ---------------------------------
2022-07-12 18:25:32 WARNING order: limit, num_concurrent, rate_limit, max_active, execution_number
2022-07-12 18:27:14 WARNING Session outcomes: Counter({'Successful': 400, <class 'planet.exceptions.TooManyRequests'>: 5})
PASSED                                                                   [ 83%]
tests/functional/test_concurrent_calls.py::test_max_active_limit_concurrent[1000-100-5-50-2] 
-------------------------------- live log call ---------------------------------
2022-07-12 18:27:15 WARNING order: limit, num_concurrent, rate_limit, max_active, execution_number
2022-07-12 18:28:52 WARNING Session outcomes: Counter({'Successful': 400, <class 'planet.exceptions.TooManyRequests'>: 6, <class 'httpx.RemoteProtocolError'>: 1})
PASSED                                                                   [ 87%]
tests/functional/test_concurrent_calls.py::test_max_active_limit_concurrent[1000-100-5-100-0] 
-------------------------------- live log call ---------------------------------
2022-07-12 18:28:53 WARNING order: limit, num_concurrent, rate_limit, max_active, execution_number
2022-07-12 18:30:31 WARNING Session outcomes: Counter({'Successful': 400, <class 'planet.exceptions.TooManyRequests'>: 4})
PASSED                                                                   [ 91%]
tests/functional/test_concurrent_calls.py::test_max_active_limit_concurrent[1000-100-5-100-1] 
-------------------------------- live log call ---------------------------------
2022-07-12 18:30:31 WARNING order: limit, num_concurrent, rate_limit, max_active, execution_number
2022-07-12 18:32:16 WARNING Session outcomes: Counter({'Successful': 400, <class 'planet.exceptions.TooManyRequests'>: 6})
PASSED                                                                   [ 95%]
tests/functional/test_concurrent_calls.py::test_max_active_limit_concurrent[1000-100-5-100-2] 
-------------------------------- live log call ---------------------------------
2022-07-12 18:32:17 WARNING order: limit, num_concurrent, rate_limit, max_active, execution_number
2022-07-12 18:33:57 WARNING Session outcomes: Counter({'Successful': 400, <class 'planet.exceptions.TooManyRequests'>: 3})
PASSED                                                                   [100%]

=================================== FAILURES ===================================
/Users/jennifer.kyle/.pyenv/versions/3.7.9/envs/planet-client-python-3.7.9/lib/python3.7/site-packages/httpx/_exceptions.py:343: httpx.ConnectError: [Errno 8] nodename nor servname provided, or not known
/Users/jennifer.kyle/.pyenv/versions/3.7.9/envs/planet-client-python-3.7.9/lib/python3.7/site-packages/httpx/_exceptions.py:343: httpx.ReadTimeout
/Users/jennifer.kyle/.pyenv/versions/3.7.9/envs/planet-client-python-3.7.9/lib/python3.7/site-packages/httpx/_exceptions.py:343: httpx.ReadTimeout
/Users/jennifer.kyle/.pyenv/versions/3.7.9/envs/planet-client-python-3.7.9/lib/python3.7/site-packages/httpx/_exceptions.py:343: httpx.ReadTimeout
/Users/jennifer.kyle/.pyenv/versions/3.7.9/envs/planet-client-python-3.7.9/lib/python3.7/site-packages/httpx/_exceptions.py:343: httpx.WriteError: Connection lost
/Users/jennifer.kyle/.pyenv/versions/3.7.9/envs/planet-client-python-3.7.9/lib/python3.7/site-packages/httpx/_exceptions.py:343: httpx.ReadTimeout
/Users/jennifer.kyle/.pyenv/versions/3.7.9/envs/planet-client-python-3.7.9/lib/python3.7/site-packages/httpx/_exceptions.py:343: httpx.ConnectError: [Errno 60] Connect call failed ('35.190.57.55', 443)
/Users/jennifer.kyle/.pyenv/versions/3.7.9/envs/planet-client-python-3.7.9/lib/python3.7/site-packages/httpx/_exceptions.py:343: httpx.ReadTimeout
============================== slowest durations ===============================
335.85s call     functional/test_concurrent_calls.py::test_max_active_limit_concurrent[100-1000-5-50-1]
315.12s call     functional/test_concurrent_calls.py::test_max_active_limit_concurrent[100-1000-5-50-2]
303.19s call     functional/test_concurrent_calls.py::test_max_active_limit_concurrent[100-1000-5-50-0]
249.89s call     functional/test_concurrent_calls.py::test_max_active_limit_concurrent[100-1000-5-100-0]
237.35s call     functional/test_concurrent_calls.py::test_max_active_limit_concurrent[100-1000-5-100-2]
233.77s call     functional/test_concurrent_calls.py::test_max_active_limit_concurrent[100-1000-5-100-1]
213.53s call     functional/test_concurrent_calls.py::test_max_active_limit_concurrent[100-1000-10-50-2]
205.14s call     functional/test_concurrent_calls.py::test_max_active_limit_concurrent[100-1000-10-100-2]
202.15s call     functional/test_concurrent_calls.py::test_max_active_limit_concurrent[100-1000-10-50-0]
197.76s call     functional/test_concurrent_calls.py::test_max_active_limit_concurrent[100-1000-10-50-1]
195.06s call     functional/test_concurrent_calls.py::test_max_active_limit_concurrent[100-1000-10-100-0]
193.42s call     functional/test_concurrent_calls.py::test_max_active_limit_concurrent[100-1000-10-100-1]
106.01s call     functional/test_concurrent_calls.py::test_max_active_limit_concurrent[1000-100-5-100-1]
102.66s call     functional/test_concurrent_calls.py::test_max_active_limit_concurrent[1000-100-5-50-1]
101.85s call     functional/test_concurrent_calls.py::test_max_active_limit_concurrent[1000-100-5-50-0]
100.69s call     functional/test_concurrent_calls.py::test_max_active_limit_concurrent[1000-100-5-100-2]
98.37s call     functional/test_concurrent_calls.py::test_max_active_limit_concurrent[1000-100-5-100-0]
98.23s call     functional/test_concurrent_calls.py::test_max_active_limit_concurrent[1000-100-5-50-2]
89.63s call     functional/test_concurrent_calls.py::test_max_active_limit_concurrent[1000-100-10-50-0]
85.29s call     functional/test_concurrent_calls.py::test_max_active_limit_concurrent[1000-100-10-100-0]
84.80s call     functional/test_concurrent_calls.py::test_max_active_limit_concurrent[1000-100-10-50-1]
84.25s call     functional/test_concurrent_calls.py::test_max_active_limit_concurrent[1000-100-10-50-2]
81.81s call     functional/test_concurrent_calls.py::test_max_active_limit_concurrent[1000-100-10-100-1]
74.16s call     functional/test_concurrent_calls.py::test_max_active_limit_concurrent[1000-100-10-100-2]
0.01s teardown functional/test_concurrent_calls.py::test_max_active_limit_concurrent[100-1000-5-50-1]
0.01s setup    functional/test_concurrent_calls.py::test_max_active_limit_concurrent[100-1000-5-50-2]

(46 durations < 0.005s hidden.  Use -vv to show these durations.)
=========================== short test summary info ============================
FAILED tests/functional/test_concurrent_calls.py::test_max_active_limit_concurrent[100-1000-10-100-0]
FAILED tests/functional/test_concurrent_calls.py::test_max_active_limit_concurrent[100-1000-10-100-1]
FAILED tests/functional/test_concurrent_calls.py::test_max_active_limit_concurrent[100-1000-10-100-2]
FAILED tests/functional/test_concurrent_calls.py::test_max_active_limit_concurrent[100-1000-5-50-1]
FAILED tests/functional/test_concurrent_calls.py::test_max_active_limit_concurrent[100-1000-5-100-0]
FAILED tests/functional/test_concurrent_calls.py::test_max_active_limit_concurrent[1000-100-10-100-0]
FAILED tests/functional/test_concurrent_calls.py::test_max_active_limit_concurrent[1000-100-10-100-1]
FAILED tests/functional/test_concurrent_calls.py::test_max_active_limit_concurrent[1000-100-10-100-2]
========== 8 failed, 16 passed, 48 deselected in 19704.23s (5:28:24) ===========

[tbarsballe]

I tried both approaches - single session and multiple. If I recall, I wound up trying multiple sessions first, just because that's what was easiest based on my code architecture, then switched to single when that didn't work.

With regards to incentivizing users towards single session, I think we just need to make sure it is clearly documented, and the rate limiting failures will push them towards using the proper approach. I don't think there's much we can do from the API design side to push towards multiple sessions, unless you can detect use of multiple sessions and log a warning if this is detected?

[jreiberkyle]

Thanks for the insight! I feel fine with just documenting and having one Session be successful while multiple are likely not.

[jreiberkyle]

Increasing the types of errors that are retried turns out to be pretty successful. In this test, the following errors were retried:

RETRY_EXCEPTIONS = [
    httpx.ConnectError,
    httpx.ReadError,
    httpx.ReadTimeout,
    httpx.RemoteProtocolError,
    exceptions.BadGateway,
    exceptions.TooManyRequests
]

In the test, 1000 concurrent calls were made with a limit of 100. 18 of 20 tests passed. Runtime of passing tests was 200-230s. The two that failed did so with a single error: httpcore.CloseError. This should be added to the retry list.

command line

pytest tests/functional/test_concurrent_calls.py --tb=line --durations=0 --log-cli-level=warning \
    -k test_reliability

reliability.log

============================= test session starts ==============================
platform darwin -- Python 3.7.9, pytest-6.2.1, py-1.10.0, pluggy-0.13.1
rootdir: /Users/jennifer.kyle/pl/planet-client-python/tests, configfile: pytest.ini
plugins: asyncio-0.16.0, requests-mock-1.8.0, anyio-3.4.0, cov-2.10.1
collected 101 items / 81 deselected / 20 selected

tests/functional/test_concurrent_calls.py::test_reliability[100-1000-10-50-0] 
-------------------------------- live log call ---------------------------------
2022-07-13 08:33:06 WARNING order: limit, num_concurrent, rate_limit, max_active, execution_number
2022-07-13 08:36:32 WARNING Session outcomes: Counter({'Successful': 1000, <class 'planet.exceptions.TooManyRequests'>: 156, <class 'httpx.RemoteProtocolError'>: 1})
PASSED                                                                   [  5%]
tests/functional/test_concurrent_calls.py::test_reliability[100-1000-10-50-1] 
-------------------------------- live log call ---------------------------------
2022-07-13 08:36:33 WARNING order: limit, num_concurrent, rate_limit, max_active, execution_number
2022-07-13 08:39:57 WARNING Session outcomes: Counter({'Successful': 1000, <class 'planet.exceptions.TooManyRequests'>: 129})
PASSED                                                                   [ 10%]
tests/functional/test_concurrent_calls.py::test_reliability[100-1000-10-50-2] 
-------------------------------- live log call ---------------------------------
2022-07-13 08:39:58 WARNING order: limit, num_concurrent, rate_limit, max_active, execution_number
2022-07-13 08:43:28 WARNING Session outcomes: Counter({'Successful': 1000, <class 'planet.exceptions.TooManyRequests'>: 129})
PASSED                                                                   [ 15%]
tests/functional/test_concurrent_calls.py::test_reliability[100-1000-10-50-3] 
-------------------------------- live log call ---------------------------------
2022-07-13 08:43:29 WARNING order: limit, num_concurrent, rate_limit, max_active, execution_number
2022-07-13 08:46:53 WARNING Session outcomes: Counter({'Successful': 1000, <class 'planet.exceptions.TooManyRequests'>: 143})
PASSED                                                                   [ 20%]
tests/functional/test_concurrent_calls.py::test_reliability[100-1000-10-50-4] 
-------------------------------- live log call ---------------------------------
2022-07-13 08:46:54 WARNING order: limit, num_concurrent, rate_limit, max_active, execution_number
2022-07-13 08:50:33 WARNING Session outcomes: Counter({'Successful': 1000, <class 'planet.exceptions.TooManyRequests'>: 144})
PASSED                                                                   [ 25%]
tests/functional/test_concurrent_calls.py::test_reliability[100-1000-10-50-5] 
-------------------------------- live log call ---------------------------------
2022-07-13 08:50:33 WARNING order: limit, num_concurrent, rate_limit, max_active, execution_number
2022-07-13 08:54:00 WARNING Session outcomes: Counter({'Successful': 1000, <class 'planet.exceptions.TooManyRequests'>: 121, <class 'httpx.RemoteProtocolError'>: 1})
PASSED                                                                   [ 30%]
tests/functional/test_concurrent_calls.py::test_reliability[100-1000-10-50-6] 
-------------------------------- live log call ---------------------------------
2022-07-13 08:54:01 WARNING order: limit, num_concurrent, rate_limit, max_active, execution_number
2022-07-13 08:57:21 WARNING Session outcomes: Counter({'Successful': 1000, <class 'planet.exceptions.TooManyRequests'>: 135})
PASSED                                                                   [ 35%]
tests/functional/test_concurrent_calls.py::test_reliability[100-1000-10-50-7] 
-------------------------------- live log call ---------------------------------
2022-07-13 08:57:22 WARNING order: limit, num_concurrent, rate_limit, max_active, execution_number
2022-07-13 09:00:44 WARNING Session outcomes: Counter({'Successful': 1000, <class 'planet.exceptions.TooManyRequests'>: 146})
PASSED                                                                   [ 40%]
tests/functional/test_concurrent_calls.py::test_reliability[100-1000-10-50-8] 
-------------------------------- live log call ---------------------------------
2022-07-13 09:00:45 WARNING order: limit, num_concurrent, rate_limit, max_active, execution_number
2022-07-13 09:04:13 WARNING Session outcomes: Counter({'Successful': 1000, <class 'planet.exceptions.TooManyRequests'>: 133})
PASSED                                                                   [ 45%]
tests/functional/test_concurrent_calls.py::test_reliability[100-1000-10-50-9] 
-------------------------------- live log call ---------------------------------
2022-07-13 09:04:14 WARNING order: limit, num_concurrent, rate_limit, max_active, execution_number
2022-07-13 09:08:15 WARNING Session outcomes: Counter({'Successful': 1000, <class 'planet.exceptions.TooManyRequests'>: 146, <class 'httpx.ReadTimeout'>: 1})
PASSED                                                                   [ 50%]
tests/functional/test_concurrent_calls.py::test_reliability[100-1000-10-50-10] 
-------------------------------- live log call ---------------------------------
2022-07-13 09:08:15 WARNING order: limit, num_concurrent, rate_limit, max_active, execution_number
2022-07-13 09:12:13 WARNING Session outcomes: Counter({'Successful': 1000, <class 'planet.exceptions.TooManyRequests'>: 124, <class 'httpx.ReadTimeout'>: 1})
PASSED                                                                   [ 55%]
tests/functional/test_concurrent_calls.py::test_reliability[100-1000-10-50-11] 
-------------------------------- live log call ---------------------------------
2022-07-13 09:12:14 WARNING order: limit, num_concurrent, rate_limit, max_active, execution_number
2022-07-13 09:15:45 WARNING Session outcomes: Counter({'Successful': 1000, <class 'planet.exceptions.TooManyRequests'>: 147, <class 'httpx.RemoteProtocolError'>: 1})
PASSED                                                                   [ 60%]
tests/functional/test_concurrent_calls.py::test_reliability[100-1000-10-50-12] 
-------------------------------- live log call ---------------------------------
2022-07-13 09:15:46 WARNING order: limit, num_concurrent, rate_limit, max_active, execution_number
2022-07-13 09:19:13 WARNING Session outcomes: Counter({'Successful': 1000, <class 'planet.exceptions.TooManyRequests'>: 150})
PASSED                                                                   [ 65%]
tests/functional/test_concurrent_calls.py::test_reliability[100-1000-10-50-13] 
-------------------------------- live log call ---------------------------------
2022-07-13 09:19:14 WARNING order: limit, num_concurrent, rate_limit, max_active, execution_number
2022-07-13 09:22:35 WARNING Session outcomes: Counter({'Successful': 999, <class 'planet.exceptions.TooManyRequests'>: 150, <class 'httpx.RemoteProtocolError'>: 2})
2022-07-13 09:22:35 WARNING Task exceptions: Counter({<class 'httpcore.CloseError'>: 1})
FAILED                                                                   [ 70%]
tests/functional/test_concurrent_calls.py::test_reliability[100-1000-10-50-14] 
-------------------------------- live log call ---------------------------------
2022-07-13 09:22:35 WARNING order: limit, num_concurrent, rate_limit, max_active, execution_number
2022-07-13 09:26:01 WARNING Session outcomes: Counter({'Successful': 1000, <class 'planet.exceptions.TooManyRequests'>: 136})
PASSED                                                                   [ 75%]
tests/functional/test_concurrent_calls.py::test_reliability[100-1000-10-50-15] 
-------------------------------- live log call ---------------------------------
2022-07-13 09:26:02 WARNING order: limit, num_concurrent, rate_limit, max_active, execution_number
2022-07-13 09:29:40 WARNING Session outcomes: Counter({'Successful': 1000, <class 'planet.exceptions.TooManyRequests'>: 146, <class 'httpx.RemoteProtocolError'>: 1})
PASSED                                                                   [ 80%]
tests/functional/test_concurrent_calls.py::test_reliability[100-1000-10-50-16] 
-------------------------------- live log call ---------------------------------
2022-07-13 09:29:40 WARNING order: limit, num_concurrent, rate_limit, max_active, execution_number
2022-07-13 09:33:05 WARNING Session outcomes: Counter({'Successful': 1000, <class 'planet.exceptions.TooManyRequests'>: 149, <class 'httpx.ReadTimeout'>: 1, <class 'planet.exceptions.BadGateway'>: 1})
PASSED                                                                   [ 85%]
tests/functional/test_concurrent_calls.py::test_reliability[100-1000-10-50-17] 
-------------------------------- live log call ---------------------------------
2022-07-13 09:33:06 WARNING order: limit, num_concurrent, rate_limit, max_active, execution_number
2022-07-13 09:36:50 WARNING Session outcomes: Counter({'Successful': 1000, <class 'planet.exceptions.TooManyRequests'>: 131})
PASSED                                                                   [ 90%]
tests/functional/test_concurrent_calls.py::test_reliability[100-1000-10-50-18] 
-------------------------------- live log call ---------------------------------
2022-07-13 09:36:51 WARNING order: limit, num_concurrent, rate_limit, max_active, execution_number
2022-07-13 09:40:28 WARNING Session outcomes: Counter({'Successful': 1000, <class 'planet.exceptions.TooManyRequests'>: 129})
PASSED                                                                   [ 95%]
tests/functional/test_concurrent_calls.py::test_reliability[100-1000-10-50-19] 
-------------------------------- live log call ---------------------------------
2022-07-13 09:40:29 WARNING order: limit, num_concurrent, rate_limit, max_active, execution_number
2022-07-13 09:44:14 WARNING Session outcomes: Counter({'Successful': 999, <class 'planet.exceptions.TooManyRequests'>: 122})
2022-07-13 09:44:14 WARNING Task exceptions: Counter({<class 'httpcore.CloseError'>: 1})
FAILED                                                                   [100%]

=================================== FAILURES ===================================
/Users/jennifer.kyle/.pyenv/versions/3.7.9/envs/planet-client-python-3.7.9/lib/python3.7/site-packages/httpcore/_exceptions.py:12: httpcore.CloseError: [Errno 54] Connection reset by peer
/Users/jennifer.kyle/.pyenv/versions/3.7.9/envs/planet-client-python-3.7.9/lib/python3.7/site-packages/httpcore/_exceptions.py:12: httpcore.CloseError: [Errno 54] Connection reset by peer
============================== slowest durations ===============================
227.37s call     functional/test_concurrent_calls.py::test_reliability[100-1000-10-50-10]
226.80s call     functional/test_concurrent_calls.py::test_reliability[100-1000-10-50-9]
225.05s call     functional/test_concurrent_calls.py::test_reliability[100-1000-10-50-17]
224.58s call     functional/test_concurrent_calls.py::test_reliability[100-1000-10-50-19]
219.55s call     functional/test_concurrent_calls.py::test_reliability[100-1000-10-50-4]
218.66s call     functional/test_concurrent_calls.py::test_reliability[100-1000-10-50-15]
217.95s call     functional/test_concurrent_calls.py::test_reliability[100-1000-10-50-18]
212.11s call     functional/test_concurrent_calls.py::test_reliability[100-1000-10-50-11]
211.28s call     functional/test_concurrent_calls.py::test_reliability[100-1000-10-50-2]
209.23s call     functional/test_concurrent_calls.py::test_reliability[100-1000-10-50-8]
207.80s call     functional/test_concurrent_calls.py::test_reliability[100-1000-10-50-12]
207.69s call     functional/test_concurrent_calls.py::test_reliability[100-1000-10-50-5]
206.88s call     functional/test_concurrent_calls.py::test_reliability[100-1000-10-50-14]
206.70s call     functional/test_concurrent_calls.py::test_reliability[100-1000-10-50-0]
205.49s call     functional/test_concurrent_calls.py::test_reliability[100-1000-10-50-16]
204.74s call     functional/test_concurrent_calls.py::test_reliability[100-1000-10-50-1]
204.50s call     functional/test_concurrent_calls.py::test_reliability[100-1000-10-50-3]
203.19s call     functional/test_concurrent_calls.py::test_reliability[100-1000-10-50-7]
200.98s call     functional/test_concurrent_calls.py::test_reliability[100-1000-10-50-13]
200.69s call     functional/test_concurrent_calls.py::test_reliability[100-1000-10-50-6]

(40 durations < 0.005s hidden.  Use -vv to show these durations.)
=========================== short test summary info ============================
FAILED tests/functional/test_concurrent_calls.py::test_reliability[100-1000-10-50-13]
FAILED tests/functional/test_concurrent_calls.py::test_reliability[100-1000-10-50-19]
=========== 2 failed, 18 passed, 81 deselected in 4267.24s (1:11:07) ===========

[jreiberkyle]

Re-ran this 20 times with httpcore.ConnectError added to the retry list. One of 1000 calls on one of the tests failed with httpx.ConnectTimeout. I am not sure it makes sense to catch and retry that error. So, I think we have our final set of retry errors and limit configurations. We can communicate to the user that they may want to return exceptions in the asyncio.gather function and retry those (very few) that fail. (@tbarsballe interested in your thoughts here)

Is limiting required with the new exceptions being caught? This test comparing non-limited (max rate and max active both set to zero) to max rate of 10/s and 100 or 50 max active shows that the non-limited test fails all three tries, while the limited tests all succeed. (and, we also find that the 10/s, 50 max active is faster than 10/s, 100 max active). So, I think we have our final setting for the limiter. I will go ahead and clean up the code and prepare a PR.

results.log

============================= test session starts ==============================
platform darwin -- Python 3.7.9, pytest-6.2.1, py-1.10.0, pluggy-0.13.1
rootdir: /Users/jennifer.kyle/pl/planet-client-python/tests, configfile: pytest.ini
plugins: asyncio-0.16.0, requests-mock-1.8.0, anyio-3.4.0, cov-2.10.1
collected 81 items / 72 deselected / 9 selected

tests/functional/test_concurrent_calls.py::test_speed_with_exceptions[100-1000-10-50-0] 
-------------------------------- live log call ---------------------------------
2022-07-12 21:47:22 WARNING order: limit, num_concurrent, rate_limit, max_active, execution_number
2022-07-12 21:50:43 WARNING Session outcomes: Counter({'Successful': 1000, <class 'planet.exceptions.TooManyRequests'>: 223, <class 'httpx.RemoteProtocolError'>: 2})
PASSED                                                                   [ 11%]
tests/functional/test_concurrent_calls.py::test_speed_with_exceptions[100-1000-10-50-1] 
-------------------------------- live log call ---------------------------------
2022-07-12 21:50:43 WARNING order: limit, num_concurrent, rate_limit, max_active, execution_number
2022-07-12 21:53:51 WARNING Session outcomes: Counter({'Successful': 1000, <class 'planet.exceptions.TooManyRequests'>: 165, <class 'httpx.RemoteProtocolError'>: 2})
PASSED                                                                   [ 22%]
tests/functional/test_concurrent_calls.py::test_speed_with_exceptions[100-1000-10-50-2] 
-------------------------------- live log call ---------------------------------
2022-07-12 21:53:52 WARNING order: limit, num_concurrent, rate_limit, max_active, execution_number
2022-07-12 21:57:00 WARNING Session outcomes: Counter({'Successful': 1000, <class 'planet.exceptions.TooManyRequests'>: 205})
PASSED                                                                   [ 33%]
tests/functional/test_concurrent_calls.py::test_speed_with_exceptions[100-1000-10-100-0] 
-------------------------------- live log call ---------------------------------
2022-07-12 21:57:01 WARNING order: limit, num_concurrent, rate_limit, max_active, execution_number
2022-07-12 22:00:24 WARNING Session outcomes: Counter({'Successful': 1000, <class 'planet.exceptions.TooManyRequests'>: 230, <class 'httpx.ReadTimeout'>: 34})
PASSED                                                                   [ 44%]
tests/functional/test_concurrent_calls.py::test_speed_with_exceptions[100-1000-10-100-1] 
-------------------------------- live log call ---------------------------------
2022-07-12 22:00:25 WARNING order: limit, num_concurrent, rate_limit, max_active, execution_number
2022-07-12 22:03:45 WARNING Session outcomes: Counter({'Successful': 1000, <class 'planet.exceptions.TooManyRequests'>: 237, <class 'httpx.ReadTimeout'>: 63})
PASSED                                                                   [ 55%]
tests/functional/test_concurrent_calls.py::test_speed_with_exceptions[100-1000-10-100-2] 
-------------------------------- live log call ---------------------------------
2022-07-12 22:03:45 WARNING order: limit, num_concurrent, rate_limit, max_active, execution_number
2022-07-12 22:08:04 WARNING Session outcomes: Counter({'Successful': 1000, <class 'planet.exceptions.TooManyRequests'>: 267, <class 'httpx.ReadTimeout'>: 209})
PASSED                                                                   [ 66%]
tests/functional/test_concurrent_calls.py::test_speed_with_exceptions[100-1000-0-0-0] 
-------------------------------- live log call ---------------------------------
2022-07-12 22:08:05 WARNING order: limit, num_concurrent, rate_limit, max_active, execution_number
2022-07-12 22:12:14 WARNING Session outcomes: Counter({<class 'planet.exceptions.TooManyRequests'>: 3910, 'Successful': 474})
2022-07-12 22:12:14 WARNING Task exceptions: Counter({<class 'planet.exceptions.TooManyRequests'>: 520, <class 'httpx.PoolTimeout'>: 6})
FAILED                                                                   [ 77%]
tests/functional/test_concurrent_calls.py::test_speed_with_exceptions[100-1000-0-0-1] 
-------------------------------- live log call ---------------------------------
2022-07-12 22:12:14 WARNING order: limit, num_concurrent, rate_limit, max_active, execution_number
2022-07-12 22:16:09 WARNING Session outcomes: Counter({<class 'planet.exceptions.TooManyRequests'>: 3821, 'Successful': 511})
2022-07-12 22:16:09 WARNING Task exceptions: Counter({<class 'planet.exceptions.TooManyRequests'>: 483, <class 'httpx.PoolTimeout'>: 6})
FAILED                                                                   [ 88%]
tests/functional/test_concurrent_calls.py::test_speed_with_exceptions[100-1000-0-0-2] 
-------------------------------- live log call ---------------------------------
2022-07-12 22:16:09 WARNING order: limit, num_concurrent, rate_limit, max_active, execution_number
2022-07-12 22:20:28 WARNING Session outcomes: Counter({<class 'planet.exceptions.TooManyRequests'>: 3722, 'Successful': 532, <class 'httpx.RemoteProtocolError'>: 1})
2022-07-12 22:20:28 WARNING Task exceptions: Counter({<class 'planet.exceptions.TooManyRequests'>: 464, <class 'httpx.PoolTimeout'>: 4})
FAILED                                                                   [100%]

=================================== FAILURES ===================================
/Users/jennifer.kyle/pl/planet-client-python/planet/http.py:93: planet.exceptions.TooManyRequests: max rate reached: retry-in 200ms
/Users/jennifer.kyle/pl/planet-client-python/planet/http.py:93: planet.exceptions.TooManyRequests: max rate reached: retry-in 200ms
/Users/jennifer.kyle/pl/planet-client-python/planet/http.py:93: planet.exceptions.TooManyRequests: max rate reached: retry-in 200ms
============================== slowest durations ===============================
259.57s call     functional/test_concurrent_calls.py::test_speed_with_exceptions[100-1000-0-0-2]
259.53s call     functional/test_concurrent_calls.py::test_speed_with_exceptions[100-1000-10-100-2]
249.02s call     functional/test_concurrent_calls.py::test_speed_with_exceptions[100-1000-0-0-0]
234.74s call     functional/test_concurrent_calls.py::test_speed_with_exceptions[100-1000-0-0-1]
204.21s call     functional/test_concurrent_calls.py::test_speed_with_exceptions[100-1000-10-100-0]
201.74s call     functional/test_concurrent_calls.py::test_speed_with_exceptions[100-1000-10-50-0]
200.41s call     functional/test_concurrent_calls.py::test_speed_with_exceptions[100-1000-10-100-1]
188.69s call     functional/test_concurrent_calls.py::test_speed_with_exceptions[100-1000-10-50-2]
188.57s call     functional/test_concurrent_calls.py::test_speed_with_exceptions[100-1000-10-50-1]

(18 durations < 0.005s hidden.  Use -vv to show these durations.)
=========================== short test summary info ============================
FAILED tests/functional/test_concurrent_calls.py::test_speed_with_exceptions[100-1000-0-0-0]
FAILED tests/functional/test_concurrent_calls.py::test_speed_with_exceptions[100-1000-0-0-1]
FAILED tests/functional/test_concurrent_calls.py::test_speed_with_exceptions[100-1000-0-0-2]
=========== 3 failed, 6 passed, 72 deselected in 1986.74s (0:33:06) ============

[jreiberkyle]

A note on httpcore.CloseError: It is likely being thrown because the server did not sign off nicely and a fix to httpx actually does not raise this error in this case (since the data was indeed exchanged). So, an update of the httpx version (#276) should clean this bug up. But, for now, we are catching it and retrying.

[tbarsballe]

Re-ran this 20 times with httpcore.ConnectError added to the retry list. One of 1000 calls on one of the tests failed with httpx.ConnectTimeout. I am not sure it makes sense to catch and retry that error. So, I think we have our final set of retry errors and limit configurations. We can communicate to the user that they may want to return exceptions in the asyncio.gather function and retry those (very few) that fail. (@tbarsballe interested in your thoughts here)

In past rate limiting implementations, I've retried on both requests.exceptions.ReadTimeout and google.api_core.exceptions.GatewayTimeout (not applicable here, but at least another data point towards retrying on timeouts being safe) without issue. So, I'm a little on the fence - including httpx.ConnectTimeout in the set of things we retry on might be fine. There are other instances it could be thrown, but those are all stuff like no internet or firewalls, where retrying a few extra times probably won't do any damage, as long as there is a hard limit to max number of retries.

Regardless of which way we you there, a section on the docs on catching exceptions with asyncio.gather would be a good idea, even if only to catch legitimate exceptions like order failures and such.