Our TLS tests currently depend on our own fork of nassl (the internetnl branch). This fork is based on a 4 year old version of nassl, built on a fork of an older openssl version. The build process is fragile, and this is not a sustainable setup. We have already talked/mailed about this - this issue is meant to summarize plans and thoughts about improving this.

The custom nassl fork adds support for TLS 1.3, ChaCha20 and Poly1305. These have since been added to nassl itself, so we likely do not need our own fork anymore. However, there are also API differences, which may be APIs we added to our own fork, and/or changes in nassl in the last years.

In addition to nassl dependencies, our TLS checks are a few thousand lines of our own code. From what I’ve seen so far, a lot of this code also has a high complexity. Together with the vague dependencies and their build process, this makes TLS one of the more complex and fragile parts of the project.

Built on top of nassl is sslyze, an SSL/TLS scanning tool by the same author. I played around with it a bit, and I am impressed by how thorough it is, and the interface is really neat. Potentially, we could replace a lot of our code with calls to sslyze instead. It is currently not feature complete enough for us, but perhaps we could build some features and contribute them back. It does make judgements about the results, but can report enough raw data that we can still apply our own rules and scoring. This is an example JSON output to show the kind of checks and detail.

@gthess contributed this list of probable requirements:

• Are all the current TLS checks that we support covered by the library?
• Is it possible to pass IP addresses for testing instead of domain
  names?
• Can you run individual checks against a server or you call a
  "check_server" function that runs the whole thing? If the latter is
  the case, can checks be conditionally turned on/off for a domain? For
  example to limit connections/testing time when testing a mail server
  for checks that are not relevant for mail servers e.g., OCSP-stapling.
• Can we influence how connections are made to a server during testing
  (consecutive/parallel). Especially for email we want the connections
  to be done in series.
• In what degree can we influence the checks themselves? For example if
  we only care for non good ciphers we don't want the test to waste
  connections (and possibly hitting connection limits) on testing
  ciphers that we don't want to take any action about. Another example
  is to stop the cipher test on the first offending cipher; no need to
  waste connections if we know the verdict already.
• Can it handle cipher order preference testing?
• Some tests are HTTP based (HTTPS available/HTTPS redirect/HTTP
  compression/HSTS/HTTP header tests). For these we use the clients that
  nassl provides for initiating the TLS connection. We would still need
  to use these for connecting via HTTPS (we can't rely on python's
  requests for example in case the TLS connection is considered
  insecure/old for the system's cryptolibraries)
  This bullet point could be up for discussion to fail those HTTP tests if a
  TLS connection cannot be made by a modern system.

Personally I do not think the last bullet should be a hard requirement. If your TLS setup is so exceptionally broken that an ordinary Python client can't connect anymore, I feel fine with reporting that we skipped some tests - because whatever those tests would report, it isn't remotely the main issue that site is having.

Also important to consider some configuration cases like #753.

Currently sslyze does not support cipher order, but this is planned, so that is one clear missing feature.

I see a few potential paths:

• Upgrade our nassl dependency to the latest nassl version. Unclear how difficult this is, it might just be changing/adding a few API calls. If we need API calls that are missing from nassl now, try to integrate them in a way that we can contribute them upstream. We should really try to minimize maintaining custom forks when possible.
• Identify in more detail which sslyze features are missing to have a better idea of how feasible it is to use sslyze as our TLS test backend, or what we additional features we would need to ask for or add ourselves. Cipher order is a definite one, there may be more. We could also see if we can fund nassl to add these features. This has a lot of potential, because it means we have much less custom code, and contribute to the wider TLS testing ecosystem.
• Another TLS testing library. I'm not aware of any others with this much functionality though.

Some mix may also be a good choice. It's probably worth exploring all options in some more detail.

(Note that our nassl fork does have a "how to keep the fork up to date" section in the readme, but this only updates the main branch, not our custom branch).