Unable to read fleet.ssl options: Invalid TLS validation configuration - endpoint security integration

[narph]

Ex configuration:

elastic-agent.yml:

....
fleet:
  hosts:
    - 'https://fleet.test.com:8220'

fleet.yml

ssl:
certificate_authorities:
- |
-----BEGIN CERTIFICATE-----
...

Generated elastic-endpoint.yaml

....
fleet:
  access_api_key: ....
  agent:
    id: ....
    logging:
      level: info
    monitoring:
      http:
        enabled: false
        host: ""
        port: ...
  enabled: true
  host:
    id: ....
  hosts:
  - https://fleet.test.com:8220
  protocol: http
  reporting:
    check_frequency_sec: 30
    threshold: 10000
  ssl:
    certificate_authorities:
    - .....
    renegotiation: never
    verification_mode: ""
....

Following errors are seen in the endpoint logs:

{"@timestamp":"2022-02-21T20:49:48.6605404Z","agent":{"id":"...","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"error","origin":{"file":{"line":285,"name":"Config.cpp"}}},"message":"Config.cpp:285 Unable to read fleet.ssl options: Invalid TLS validation configuration","process":{"pid":21240,"thread":{"id":7176}}}
{"@timestamp":"2022-02-21T21:09:48.7862266Z","agent":{"id":"...","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":160,"name":"TlsConfig.cpp"}}},"message":"TlsConfig.cpp:160 Unrecognized fleet.ssl.verification_mode: ","process":{"pid":21240,"thread":{"id":7176}}}

This is due to the validation in endpoint where empty string verification mode is not recognized https://github.com/elastic/endpoint-dev/blob/4e2c19673e9654dd83e26d32ec8d045f81c931e9/Plugins/Policy/Lib/TlsConfig.cpp#L180-L208

It looks like empty string might be an accepted verification mode option https://github.com/elastic/beats/blob/main/libbeat%2Fcommon%2Ftransport%2Ftlscommon%2Ftypes.go#L133 and it is considered full.

[ph]

I believe the best way to fix this issue is to make it explicit which is Full, one way to fix it would be to change how the VerificationMode type is serialized into YML or JSON, when the value is empty we should have full instead of ``

[jlind23]

@belimawr any learnings around verification mode to share here? I remember you faced some issues with beats one or two releases ago.

[belimawr]

Yes, it was pretty much the same situation.

Ideally Elastic-Agent would not set an empty string on ssl.verification_mode, however even though Endpoint is logging an error, it's also setting the verification mode to full, hence it's not a security issue.

@ph I agree we should explicitly set it. Either set it to what we (Elastic-Agent) considers the default or not serialise it into the YAMAL file, thus allowing Endpoint to use it's defaults.

[belimawr]

Just as a sanity check, @narph have you experienced any issues in the TLS behaviour? Or just the error/warning logs?

[narph]

@belimawr , we have not made any changes yet so I have not experienced any issues in the TLS behavior, only the validation errors in the endpoint logs.
Since empty string is considered full verification mode then I suggested using full as default instead of `` so we were interested in the effects this change will have.

[ph]

This seems to be a low effort to me, is this just a matter of making MarshalText return full when string is empty?

https://github.com/elastic/elastic-agent-libs/blob/6825ccb76ed5b79371f7598a8c96dc2bd863fa86/transport/tlscommon/types.go#L151-L156

[belimawr]

This seems to be a low effort to me, is this just a matter of making MarshalText return full when string is empty?

https://github.com/elastic/elastic-agent-libs/blob/6825ccb76ed5b79371f7598a8c96dc2bd863fa86/transport/tlscommon/types.go#L151-L156

I believe it's more about ensuring a default is set rather than changing the marshal method (I'm not sure if it's already enforced somewhere).

I took a quick look at the file yo linked @ph, and it seems there might be a change of Elastic-Agent marshaling the VerifyFull as an empty string 😱 .

There is a map define here:
https://github.com/elastic/elastic-agent-libs/blob/6825ccb76ed5b79371f7598a8c96dc2bd863fa86/transport/tlscommon/types.go#L135-L141

That gets "reversed" here:
https://github.com/elastic/elastic-agent-libs/blob/6825ccb76ed5b79371f7598a8c96dc2bd863fa86/transport/tlscommon/types.go#L88-L92

There is no guarantee that the "name" to VerifyFull will be full instead of an empty string. Maybe those cases where the verification mode is an empty string is due to the way tlsVerificationModesInverse is built rather than it not being set.