Support deploying Kafka with KRaft instead of ZooKeeper for consensus building

[soenkeliebau]

In later versions Kafka supports running without ZooKeeper and provides its own KRaft consensus algorithm.

We should look into enabling the operator to deploy Kafka in this mode.

To discuss:

• New CRD role for controllers
  • optional field
• New CRD version
  • no
  • make zookeeper config map name optional
  • sanity checks for kraft & zk relevant fields
• Provision log dirs with meta.properties
• New set of integration tests

Migration guide: https://docs.confluent.io/platform/current/installation/migrate-zk-kraft.html

• test manually with 3.9.0 and decide later if automation is feasible
• automated migration is out of scope
• Users need to migrate to KRaft (with 3.9.0) before upgrading to 4.0.0.

[sbernauer]

I just wanted to mention that according to https://www.confluent.io/blog/zookeeper-to-kraft-with-confluent-kubernetes/:

With Kafka 4.0 and Confluent Platform 8.0, Kafka will complete its journey to KRaft by removing the deprecated ZK mode,

So we need to tackle this at some point of time

[NickLarsenNZ]

I assume we want a CRD change to remove the Zookeeper ConfigMap reference.

There will be a CRD versioning discussion during the on-site about how to version CRDs (eg: how to choose from v1alpha2 vs v1beta1 vs v2alpha1).