test_ssl: test_get_server_certificate() and test_msg_callback_deadlock_bpo43577() fail randomly on the macOS CI

[erlend-aasland]

BPO	44229
Nosy	@terryjreedy, @ronaldoussoren, @vstinner, @tiran, @pablogsal, @miss-islington, @erlend-aasland
PRs	bpo-44229: Intest_ssl, ignore spurious EPROTOTYPE on macOS #26893 [3.9] bpo-44229: Ignore spurious EPROTOTYPE on macOS in test_ssl (GH-26893) #26894 [3.8] bpo-44229: Ignore spurious EPROTOTYPE on macOS in test_ssl (GH-26893) #26895 [3.10] bpo-44229: Ignore spurious EPROTOTYPE on macOS in test_ssl (GH-26893) #26896
Files	5_Display build info.txt

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-06-27.14:17:20.638>
created_at = <Date 2021-05-25.07:24:56.023>
labels = ['3.8', 'tests', '3.9', '3.10', '3.11']
title = 'test_ssl: test_get_server_certificate() and test_msg_callback_deadlock_bpo43577() fail randomly on the macOS CI'
updated_at = <Date 2021-06-28.08:11:11.070>
user = 'https://github.com/erlend-aasland'

bugs.python.org fields:

activity = <Date 2021-06-28.08:11:11.070>
actor = 'erlendaasland'
assignee = 'none'
closed = True
closed_date = <Date 2021-06-27.14:17:20.638>
closer = 'erlendaasland'
components = ['Tests']
creation = <Date 2021-05-25.07:24:56.023>
creator = 'erlendaasland'
dependencies = []
files = ['50063']
hgrepos = []
issue_num = 44229
keywords = ['patch']
message_count = 40.0
messages = ['394299', '394321', '394339', '394341', '394342', '394344', '394347', '394348', '394349', '394355', '394357', '394367', '394418', '394420', '394739', '394741', '394751', '394752', '394798', '394805', '396149', '396425', '396431', '396436', '396438', '396441', '396464', '396466', '396471', '396474', '396479', '396481', '396482', '396485', '396487', '396490', '396580', '396613', '396615', '396616']
nosy_count = 7.0
nosy_names = ['terry.reedy', 'ronaldoussoren', 'vstinner', 'christian.heimes', 'pablogsal', 'miss-islington', 'erlendaasland']
pr_nums = ['26893', '26894', '26895', '26896']
priority = 'high'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue44229'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10', 'Python 3.11']

[erlend-aasland]

See https://github.com/python/cpython/pull/26104/checks?check_run_id=2662511684

======================================================================
ERROR: test_get_server_certificate (test.test_ssl.SimpleBackgroundTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/Users/runner/work/cpython/cpython/Lib/test/test_ssl.py", line 2132, in test_get_server_certificate
    _test_get_server_certificate(self, *self.server_addr, cert=SIGNING_CA)
  File "/Users/runner/work/cpython/cpython/Lib/test/test_ssl.py", line 2329, in _test_get_server_certificate
    pem = ssl.get_server_certificate((host, port), ca_certs=cert)
  File "/Users/runner/work/cpython/cpython/Lib/ssl.py", line 1520, in get_server_certificate
    with create_connection(addr, timeout=timeout) as sock:
  File "/Users/runner/work/cpython/cpython/Lib/socket.py", line 844, in create_connection
    raise err
  File "/Users/runner/work/cpython/cpython/Lib/socket.py", line 832, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused

[tiran]

The problem could be related to bpo-43921. I neither have a macOS nor a Windows machine to reproduce and debug the issue. Since I'm cannot reproduce the problem on Linux, I'm unable to debug and fix it.

[pablogsal]

I neither have a macOS nor a Windows machine to reproduce and debug the issue.

Can you maybe use a VM for Windows?

[erlend-aasland]

I’ll see if I can provoke it on my Mac.

What’s the preferred solution?

1. Catch the connection failure and skip the test?
2. Retry connection?
3. Find the root cause and make sure it never ever happens :)

Maybe 2. is acceptable :)

[pablogsal]

Is possible that these are flaky tests but I am afraid this may be something more important so o would prefer to go with 3 ;)

[erlend-aasland]

I feared that ;)

[erlend-aasland]

Getting there. The error message is similar:

$ ./python.exe -m test test_ssl -m test_get_server_certificate -u all -F
...
0:02:11 load avg: 2.10 [328] test_ssl
test test_ssl failed -- Traceback (most recent call last):
  File "/Users/erlendaasland/src/cpython-ssl/Lib/test/test_ssl.py", line 2132, in test_get_server_certificate
    _test_get_server_certificate(self, *self.server_addr, cert=SIGNING_CA)
  File "/Users/erlendaasland/src/cpython-ssl/Lib/test/test_ssl.py", line 2329, in _test_get_server_certificate
    pem = ssl.get_server_certificate((host, port), ca_certs=cert)
  File "/Users/erlendaasland/src/cpython-ssl/Lib/ssl.py", line 1521, in get_server_certificate
    with context.wrap_socket(sock, server_hostname=host) as sslsock:
  File "/Users/erlendaasland/src/cpython-ssl/Lib/ssl.py", line 518, in wrap_socket
    return self.sslsocket_class._create(
  File "/Users/erlendaasland/src/cpython-ssl/Lib/ssl.py", line 1070, in _create
    self.do_handshake()
  File "/Users/erlendaasland/src/cpython-ssl/Lib/ssl.py", line 1339, in do_handshake
    self._sslobj.do_handshake()
ConnectionResetError: [Errno 54] Connection reset by peer

test_ssl failed

== Tests result: FAILURE ==

[erlend-aasland]

Got it:

0:02:29 load avg: 2.81 [389] test_ssl
test test_ssl failed -- Traceback (most recent call last):
  File "/Users/erlendaasland/src/cpython-ssl/Lib/test/test_ssl.py", line 2132, in test_get_server_certificate
    _test_get_server_certificate(self, *self.server_addr, cert=SIGNING_CA)
  File "/Users/erlendaasland/src/cpython-ssl/Lib/test/test_ssl.py", line 2329, in _test_get_server_certificate
    pem = ssl.get_server_certificate((host, port), ca_certs=cert)
  File "/Users/erlendaasland/src/cpython-ssl/Lib/ssl.py", line 1520, in get_server_certificate
    with create_connection(addr, timeout=timeout) as sock:
  File "/Users/erlendaasland/src/cpython-ssl/Lib/socket.py", line 844, in create_connection
    raise err
  File "/Users/erlendaasland/src/cpython-ssl/Lib/socket.py", line 832, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused

test_ssl failed

== Tests result: FAILURE ==

[erlend-aasland]

$ git diff
diff --git a/Lib/ssl.py b/Lib/ssl.py
index 2b131de043..9c281d8028 100644
--- a/Lib/ssl.py
+++ b/Lib/ssl.py
@@ -257,8 +257,9 @@ class _TLSMessageType:
 if sys.platform == "win32":
     from _ssl import enum_certificates, enum_crls
 
+from test.support import SHORT_TIMEOUT as _GLOBAL_DEFAULT_TIMEOUT
 from socket import socket, SOCK_STREAM, create_connection
-from socket import SOL_SOCKET, SO_TYPE, _GLOBAL_DEFAULT_TIMEOUT
+from socket import SOL_SOCKET, SO_TYPE
 import socket as _socket
 import base64        # for DER-to-PEM translation
 import errno

$ % ./python.exe -m test test_ssl -m test_get_server_certificate -u all -F
...
0:03:27 load avg: 2.24 [535] test_ssl
test test_ssl failed -- Traceback (most recent call last):
  File "/Users/erlendaasland/src/cpython-ssl/Lib/test/test_ssl.py", line 2132, in test_get_server_certificate
    _test_get_server_certificate(self, *self.server_addr, cert=SIGNING_CA)
  File "/Users/erlendaasland/src/cpython-ssl/Lib/test/test_ssl.py", line 2329, in _test_get_server_certificate
    pem = ssl.get_server_certificate((host, port), ca_certs=cert)
  File "/Users/erlendaasland/src/cpython-ssl/Lib/ssl.py", line 1521, in get_server_certificate
    with create_connection(addr, timeout=timeout) as sock:
  File "/Users/erlendaasland/src/cpython-ssl/Lib/socket.py", line 844, in create_connection
    raise err
  File "/Users/erlendaasland/src/cpython-ssl/Lib/socket.py", line 832, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused

test_ssl failed

== Tests result: FAILURE ==