Not able to configure geo-replication with SSL

[ragaur-tibco]

I was trying to setup the geo replication server and created clusters but after completing the geo-replication server with SSL
facing below error

> java.util.concurrent.CompletionException: java.lang.RuntimeException: java.lang.IllegalArgumentException: Param serviceUrl must not be blank.
> 	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315) ~[?:?]
> 	at java.util.concurrent.CompletableFuture.uniApplyNow(CompletableFuture.java:687) ~[?:?]
> 	at java.util.concurrent.CompletableFuture.uniApplyStage(CompletableFuture.java:662) ~[?:?]
> 	at java.util.concurrent.CompletableFuture.thenApply(CompletableFuture.java:2168) ~[?:?]
> 	at org.apache.pulsar.broker.service.persistent.PersistentTopic.lambda$addReplicationCluster$72(PersistentTopic.java:1878) ~[org.apache.pulsar-pulsar-broker-3.2.2.jar:3.2.2]
> 	at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1187) ~[?:?]
> 	at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2309) ~[?:?]
> 	at org.apache.pulsar.broker.service.persistent.PersistentTopic.addReplicationCluster(PersistentTopic.java:1871) ~[org.apache.pulsar-pulsar-broker-3.2.2.jar:3.2.2]
> 	at org.apache.pulsar.broker.service.persistent.PersistentTopic.initialize(PersistentTopic.java:352) ~[org.apache.pulsar-pulsar-broker-3.2.2.jar:3.2.2]
> 	at org.apache.pulsar.broker.service.BrokerService$2.openLedgerComplete(BrokerService.java:1655) ~[org.apache.pulsar-pulsar-broker-3.2.2.jar:3.2.2]
> 	at org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl.lambda$asyncOpen$7(ManagedLedgerFactoryImpl.java:427) ~[org.apache.pulsar-managed-ledger-3.2.2.jar:3.2.2]
> 	at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:718) ~[?:?]
> 	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
> 	at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147) ~[?:?]
> 	at org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl$2.initializeComplete(ManagedLedgerFactoryImpl.java:394) ~[org.apache.pulsar-managed-ledger-3.2.2.jar:3.2.2]
> 	at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl$3$1.operationComplete(ManagedLedgerImpl.java:593) ~[org.apache.pulsar-managed-ledger-3.2.2.jar:3.2.2]
> 	at org.apache.bookkeeper.mledger.impl.ManagedCursorImpl$2.operationComplete(ManagedCursorImpl.java:510) ~[org.apache.pulsar-managed-ledger-3.2.2.jar:3.2.2]
> 	at org.apache.bookkeeper.mledger.impl.ManagedCursorImpl$2.operationComplete(ManagedCursorImpl.java:468) ~[org.apache.pulsar-managed-ledger-3.2.2.jar:3.2.2]
> 	at org.apache.bookkeeper.mledger.impl.MetaStoreImpl.lambda$asyncGetCursorInfo$13(MetaStoreImpl.java:226) ~[org.apache.pulsar-managed-ledger-3.2.2.jar:3.2.2]
> 	at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:718) ~[?:?]
> 	at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482) ~[?:?]
> 	at org.apache.bookkeeper.common.util.OrderedExecutor$TimedRunnable.run(OrderedExecutor.java:201) ~[org.apache.bookkeeper-bookkeeper-common-4.16.4.jar:4.16.4]
> 	at org.apache.bookkeeper.common.util.SingleThreadSafeScheduledExecutorService$SafeRunnable.run(SingleThreadSafeScheduledExecutorService.java:46) ~[org.apache.bookkeeper-bookkeeper-common-4.16.4.jar:4.16.4]
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[?:?]
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
> 	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.105.Final.jar:4.1.105.Final]
> 	at java.lang.Thread.run(Thread.java:840) ~[?:?]
> Caused by: java.lang.RuntimeException: java.lang.IllegalArgumentException: Param serviceUrl must not be blank.
> 	at org.apache.pulsar.broker.service.BrokerService.lambda$getReplicationClient$51(BrokerService.java:1316) ~[org.apache.pulsar-pulsar-broker-3.2.2.jar:3.2.2]
> 	at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.put(ConcurrentOpenHashMap.java:438) ~[org.apache.pulsar-pulsar-common-3.2.2.jar:3.2.2]
> 	at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.computeIfAbsent(ConcurrentOpenHashMap.java:257) ~[org.apache.pulsar-pulsar-common-3.2.2.jar:3.2.2]
> 	at org.apache.pulsar.broker.service.BrokerService.getReplicationClient(BrokerService.java:1240) ~[org.apache.pulsar-pulsar-broker-3.2.2.jar:3.2.2]
> 	at org.apache.pulsar.broker.service.persistent.PersistentTopic.lambda$addReplicationCluster$71(PersistentTopic.java:1879) ~[org.apache.pulsar-pulsar-broker-3.2.2.jar:3.2.2]
> 	at java.util.concurrent.CompletableFuture.uniApplyNow(CompletableFuture.java:684) ~[?:?]
> 	... 28 more
> Caused by: java.lang.IllegalArgumentException: Param serviceUrl must not be blank.
> 	at com.google.common.base.Preconditions.checkArgument(Preconditions.java:143) ~[com.google.guava-guava-32.1.2-jre.jar:?]
> 	at org.apache.pulsar.client.impl.ClientBuilderImpl.serviceUrl(ClientBuilderImpl.java:86) ~[org.apache.pulsar-pulsar-client-original-3.2.2.jar:3.2.2]
> 	at org.apache.pulsar.broker.service.BrokerService.configTlsSettings(BrokerService.java:1330) ~[org.apache.pulsar-pulsar-broker-3.2.2.jar:3.2.2]
> 	at org.apache.pulsar.broker.service.BrokerService.lambda$getReplicationClient$51(BrokerService.java:1285) ~[org.apache.pulsar-pulsar-broker-3.2.2.jar:3.2.2]
> 	at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.put(ConcurrentOpenHashMap.java:438) ~[org.apache.pulsar-pulsar-common-3.2.2.jar:3.2.2]
> 	at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.computeIfAbsent(ConcurrentOpenHashMap.java:257) ~[org.apache.pulsar-pulsar-common-3.2.2.jar:3.2.2]
> 	at org.apache.pulsar.broker.service.BrokerService.getReplicationClient(BrokerService.java:1240) ~[org.apache.pulsar-pulsar-broker-3.2.2.jar:3.2.2]
> 	at org.apache.pulsar.broker.service.persistent.PersistentTopic.lambda$addReplicationCluster$71(PersistentTopic.java:1879) ~[org.apache.pulsar-pulsar-broker-3.2.2.jar:3.2.2]
> 	at java.util.concurrent.CompletableFuture.uniApplyNow(CompletableFuture.java:684) ~[?:?]
> 	... 28 more

NOTE: Able to successfully start the main server but not able to start the connected server because of above mention issue

@lhotari
@visortelle

[visortelle]

Hi @ragaur-tibco

From my understanding, you have two running clusters cluster-a and cluster-b.

For cluster-a you ran:

bin/pulsar-admin clusters create \
    --broker-url-secure pulsar+ssl://clustes-b:6651 \
    --url-secure https://cluster-b:8081 \
    cluster-b

And for cluster-b you ran:

bin/pulsar-admin clusters create \
    --broker-url-secure pulsar+ssl://clustes-a:6651 \
    --url-secure https://cluster-a:8081 \
    cluster-a

Able to successfully start the main server but not able to start the connected server because of above mention issue

Now you're not able to consume or produce messages from the topic.

Am I understood it correctly?

Did you specify the serviceUrl for the client with the pulsar+ssl:// schema?

pulsar/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ClientBuilderImpl.java

Line 86 in 5d9ccd4

public ClientBuilder serviceUrl(String serviceUrl) {

[visortelle]

@ragaur-tibco interesting. Does TLS encryption work as expected for a single Pulsar standalone cluster when you configure the cluster and client in the same way? Without doing any steps related to geo-replication.

[ragaur-tibco]

yes @visortelle I tried with the same configuration for SSL in both the servers and was able to make the server up

[ragaur-tibco]

@visortelle after configuring for geo-replication facing attached issue

[ragaur-tibco]

@visortelle any update or the cause of the this issue, I am using the same configuration for SSL but after creating cluster facing this issue in cluster B but on cluster A server was UP

[visortelle]

@ragaur-tibco hi 👋
It would help a lot if you could provide some simple way to reproduce exactly what you're doing.
Maybe in the form of a docker-compose.yml file and a shell script with all the certs generation and cluster registration steps.
Or Helm deployment that I could install to Minikube or k3s + scripts.

[lucianogreiner]

@ragaur-tibco Were you able to resolve this somehow? I am facing the same issue.

[lucianogreiner]

I got it working guys. Here's a guide:
https://gist.github.com/lucianogreiner/a4cfb7302477b74a5168b26353770217