kafka_internal topics

[HARIHARAN1103]

in our setup using 3 node redpanda cluster and using kafka-scylla sink connector to write redpanda message to scylla.

We noticed two topics from Prometheus log which are not created from our side.

1. group
2. controller

vectorized_cluster_partition_high_watermark{instance="redpanda1:9644", job="redpanda", namespace="kafka", partition="5", shard="2", topic="streamdata"}	61723318
vectorized_cluster_partition_high_watermark{instance="redpanda1:9644", job="redpanda", namespace="kafka", partition="8", shard="3", topic="streamdata"}	62150954
vectorized_cluster_partition_high_watermark{instance="redpanda1:9644", job="redpanda", namespace="kafka_internal", partition="0", shard="6", topic="group"}	57644927
vectorized_cluster_partition_high_watermark{instance="redpanda1:9644", job="redpanda", namespace="redpanda", partition="0", shard="0", topic="controller"}	1204
vectorized_cluster_partition_high_watermark{instance="redpanda2:9644", job="redpanda", namespace="kafka", partition="0", shard="1", topic="streamdata"}	62929419
vectorized_cluster_partition_high_watermark{instance="redpanda2:9644", job="redpanda", namespace="kafka", partition="12", shard="5", topic="streamdata"}	62343768

� The question is what does this two topic group & controller describes, since using rpk topic list and rpk topic describe its say no topic created in that name.

redpanda1:~ $ rpk topic list
  Name        Partitions  Replicas  
  _schemas    1           1         
  streamdata  16          1         

redpanda1:~ $ rpk topic describe group
Error: topic 'group' not found

Whenever we trying loadtest the counts are increasing. After clearing topics we created also the counts are not reseting.

My question are:

1. What does group & controller topics describes
2. We face data loss on writing to scylla, whether these increase count causing any issue.
3. Clearing these topics are possible.

[emaxerrno]

Hi @HARIHARAN1103 they are internal to Redpanda.

One manages the consumer groups.
One stores global state, like ip of computers and such.

They should have no impact on correctness and completeness.

What we found with the Scylla connector is most ppl use timestamp as key, sometimes 2 keys get produced in the same millisecond and it looks like dropped data. So I would look at the keys on the Kafka batches

we’ve seen this before and there hasn’t been a bug in either system

[HARIHARAN1103]

Hi @HARIHARAN1103 they are internal to Redpanda.

One manages the consumer groups.
One stores global state, like ip of computers and such.

They should have no impact on correctness and completeness.

What we found with the Scylla connector is most ppl use timestamp as key, sometimes 2 keys get produced in the same millisecond and it looks like dropped data. So I would look at the keys on the Kafka batches

we’ve seen this before and there hasn’t been a bug in either system

Is that possible to read or clear those data of internal-topics.

[emaxerrno]

No. That is internal state of redpanda. You can start a brand new cluster or delete the folders but users cannot interact with those topics.

[HARIHARAN1103]

In default what is the dir_name or location of this topics. I can able to locate user defined topics folder in ./mnt/vectorized/redpanda/data/kafka/ but not this.

[emaxerrno]

You want to delete data/*