Using custom certificate for NiFi and automated restarts

[dmasice]

Hi,

In order to avoid as much downtime as possible, I'm adding a custom certificate for NiFi, to be used instead of the auto-generated one, with a higher expiration time. I managed to add it to the keystore/truststore (expanding on #29) and configure NiFi to use it. So far, so good.

The problem now is the restarter.stackable.tech/expires-at annotation added by the secret-operator to the pods. Is there a way to disable this annotation? I could remove it manually, but if the pods are restarted for whatever reason, it will get added again. For now I could have a cronjob to remove the annotation via kubectl annotate. Is it possible to do this from within a pod?

Thanks.

Best regards.

[nightkr]

Hi.

The intended way to use a custom certificate is to create a custom SecretClass that uses the k8sSearch backend.

Nifi-Operator specifically doesn't currently support setting the secretclass (stackabletech/nifi-operator#499), but in the meantime it can be done using pod overrides:

apiVersion: nifi.stackable.tech/v1alpha1
kind: NifiCluster
metadata: ...
spec:
  ...
  nodes:
    ...
    podOverrides:
      spec:
        volumes:
          - name: keystore
            ephemeral:
              volumeClaimTemplate:
                metadata:
                  annotations:
                    secrets.stackable.tech/class: foo

This should prevent the restart annotation from being created, and should not require any custom init containers or such hacks. In fact, writing into the secret volume by hand is unsupported, and may break in the future.

That said, I would strongly suggest that the correct way to avoid downtime is to add redundant replicas, rather than to compromise the system to try to avoid the single-point-of-failure going down.

[dmasice]

The intended way to use a custom certificate is to create a custom SecretClass that uses the k8sSearch backend.

I was just reading about this, but have not tried it yet.

Nifi-Operator specifically doesn't currently support setting the secretclass (stackabletech/nifi-operator#499), but in the meantime it can be done using pod overrides:

I'll try that. Thanks :-)

This should prevent the restart annotation from being created, and should not require any custom init containers or such hacks. In fact, writing into the secret volume by hand is unsupported, and may break in the future.

Yes, I know. My current solution was more a proof of concept (and a hack) to see if there was something else that needs to be changed.

That said, I would strongly suggest that the correct way to avoid downtime is to add redundant replicas, rather than to compromise the system to try to avoid the single-point-of-failure going down.

The problem is that if all the replicas are created at the same time, all the pods will have roughly the same expiration/restart time. That means that all the replicas could be down at the same time (and NiFi takes a while to be ready after starting the pod). That said, there's this bit from the Nifi Administration Guide:

Once the nifi.security.autoreload.enabled property is set to true, any valid changes to the configured keystore and truststore will cause NiFi’s SSL context factory to be reloaded, allowing clients to pick up the changes. This is intended to allow expired certificates to be updated in the keystore and new trusted certificates to be added in the truststore, all without having to restart the NiFi server.

If there is a way to generate the new certificates and update the keystore/truststore without having to restart the pod, enabling that option would be enough, at least for NiFi, when using auto-generated certificates.

Anyway, having our own certificate instead of the generated one should allow us to avoid all of this.

Thanks again.

[nightkr]

The problem is that if all the replicas are created at the same time, all the pods will have roughly the same expiration/restart time. That means that all the replicas could be down at the same time (and NiFi takes a while to be ready after starting the pod).

Yeah, this is a concern. We've been planning to add jittering to reduce the likelihood of this (stackabletech/secret-operator#159). Another thing you can do is define a PodDisruptionBudget, which defines cluster health constraints that must be met before commons-operator is allowed to kill the replicas.

If there is a way to generate the new certificates and update the keystore/truststore without having to restart the pod, enabling that option would be enough, at least for NiFi, when using auto-generated certificates.

Yeah, I'd like to see this happen where possible, but it's not currently supported by our secret-operator (stackabletech/secret-operator#11).