Kafka 4.x

[razvan]

Which new version of Apache Kafka should we support?

Kafka 4 has been released.

Additional information

Most important changes are the new coordinator roles and zookeeperless clusters.

Changes required

Are there any upstream changes that we need to support?
e.g. new features, changed features, deprecated features etc.

Implementation checklist

• Update the Docker image
• Update documentation to include supported version(s)
• Update and test getting started guide with updated version(s)
• Update operator to support the new version (if needed)
• Update integration tests to test use the new versions (in addition or replacing old versions
• Update examples to use new versions

Refinement

• Out of Scope

• Kafka Connect and Stream ( NEXT )

• Kafka Clients

• Steps

• General

  • [OPTIONAL] Add metrics plugin to directly retrieve client metics from the broker #871
  • Support deploying Kafka with KRaft instead of ZooKeeper for consensus building #690
  • Update logging for Kafka 4.x #872
  • Kafka Demos: Update to Kafka 4.x demos#232

• 🟢 Kafka 4 in Docker-Images ( Seems to be done already )

  • Update Java-Base image to new Java requirements
    • Kafka Clients and Kafka Streams require Java 11.
    • Kafka Brokers, Connect, and Tools, now require Java 17.
  • Adopt versions.py
  • Add necessary patches

• Kafka 4 in Operator

  • Add Controller role for Kafka ( KRaft quorum ) #873
  • [OPTIONAL] The Next Generation of the Consumer Rebalance Protocol should be configureable #874
  • Defaults for Kafka 4 may have changed #875
  • Update integration tests

• Documentation: Update for kafka 4.x #876

• NEXT

  • Implement Kafka connect
  • Evaluate Kafka Streams

[lfrancke]

Maxi and Malte to look at this - @razvan to dig up previous notes.
Requires refinement first to decide the scope of what/how we want to support Kafka 4.

[razvan]

Previous notes: #690