Change storage class for listener pvc

[rajarshp]

Hi Team,

Is there a way to change the listener class PVC's storage class ? We are seeing one issue when we have cluster upgrade/patching, these PVCs are bound to a particular node. But post these activities those nodes are not available so the namenode is not coming up. We have to delete these PVCs and then re-deploy HDFS (which we want to avid)

We are using AWS Rosa and we have changed the listener class to cluster-internal. But it seems these listener PVCs are taking local storage which is not recommended in the distributed platform.

oc get sc | grep stackable
listeners.stackable.tech                   listeners.stackable.tech                                   Delete          WaitForFirstConsumer   false                  204d
secrets.stackable.tech                     secrets.stackable.tech                                     Delete          Immediate              false                  204d

[nightkr]

Hey, sorry about the slow answer.

Because of some Kubernetes limitations, you'll need to delete the StatefulSet and the old listener PVCs for the ListenerClass change to take effect. The product operator should then recreate the StatefulSet with the new settings. This will incur some downtime while the product is being redeployed, but existing saved data shouldn't be affected as long as you leave the data PVCs alone.

(You'd still use the same StorageClass in either case, but it changes its pod binding behaviour depending on the ListenerClass in question.)

[rajarshp]

Thanks @nightkr for the details.
So can we add this feature which will give more flexibility f we can change the storage class for this listener pvcs ?

[nightkr]

They aren't "storage classes" in the traditional sense, trying to swap them out with a regular CSI provider is just going to cause more problems.

[rajarshp]

thanks @nightkr for the explanation.
I have one more doubt on this. We were testing this with Cluster-Internal listener class and I can see the listener pvc has an annotation

volume.kubernetes.io/selected-node: ip-10-X-X-X3.YYY.compute.internal

so we have intentionally deleted this node, but we saw the annotation didnt change. so how it is going to impact the HDFS or it will update with new node details eventually ? Though we saw the PVC is fine and all the HDFS nodes came up.

[nightkr]

nodeAffinity is used to direct pod scheduling, that annotation should not.

[rajarshp]

as I am using cluster-internal, so I couldn't find any nodeAffinity, but I can see this annotation. so wanted to know what is significance of this selected-node?

[nightkr]

It's used as a hint by Kubernetes for where the initial provisioner should run (which, in our case, then looks at the ListenerClass to decide whether to actually bind to that node).

[rajarshp]

thanks for the details.
what is the SLA to bring all the services up for HDFS ? Yesterday we were testing in AWS Rosa and it took almost 20 min to bring all the services up, which seems really high. is it possible to make it faster ?