Skip to content

Commit 079575e

Browse files
lfranckemaltesander
lfrancke
and
maltesander
authored
Change the Listener documentation (#326)
* Change the Listener documentation * Update docs/modules/listener-operator/pages/index.adoc Co-authored-by: Malte Sander <[email protected]> * Update docs/modules/listener-operator/pages/listenerclass.adoc Co-authored-by: Malte Sander <[email protected]> * Update docs/modules/listener-operator/pages/listenerclass.adoc Co-authored-by: Malte Sander <[email protected]> * Update docs/modules/listener-operator/pages/listenerclass.adoc Co-authored-by: Malte Sander <[email protected]> * Update docs/modules/listener-operator/pages/listenerclass.adoc Co-authored-by: Malte Sander <[email protected]> * Update docs/modules/listener-operator/pages/usage.adoc Co-authored-by: Malte Sander <[email protected]> * Update docs/modules/listener-operator/pages/usage.adoc Co-authored-by: Malte Sander <[email protected]> * Update docs/modules/listener-operator/pages/usage.adoc Co-authored-by: Malte Sander <[email protected]> * Update docs/modules/listener-operator/pages/usage.adoc Co-authored-by: Malte Sander <[email protected]> * Update docs/modules/listener-operator/pages/usage.adoc Co-authored-by: Malte Sander <[email protected]> --------- Co-authored-by: Malte Sander <[email protected]>
1 parent 8761eeb commit 079575e

File tree

5 files changed

+291
-97
lines changed

5 files changed

+291
-97
lines changed

‎Cargo.lock

Lines changed: 2 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Cargo.nix

Lines changed: 2 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎docs/modules/listener-operator/pages/index.adoc

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,4 +7,10 @@
77
* {github}[GitHub {external-link-icon}^]
88
* {crd}[CRD documentation {external-link-icon}^]
99

10-
This is an operator for Kubernetes that provisions network listeners according to the cluster policy, and injects connection parameters into Pods.
10+
The Listener Operator solves a common Kubernetes challenge for Stackable data applications: making services accessible across different cloud providers and on-premise environments without rewriting networking configurations.
11+
12+
Instead of manually creating Services with different types (LoadBalancer, NodePort, ClusterIP) that work differently on each platform, Stackable operators automatically handle this when you specify a `listenerClass` (like "external-stable") in your cluster configuration.
13+
The Listener Operator automatically creates the appropriate Kubernetes Service resources for your environment and injects the actual connection details (IP addresses, remapped ports) into the application Pods via a special CSI volume.
14+
15+
This means the same Stackable cluster definition works identically whether you are running on GKE with LoadBalancers, on-premise with NodePorts, or any other Kubernetes setup.
16+
All you need to adapt are the xref:listenerclass.adoc[ListenerClasses] to match your infrastructure, enabling truly portable data platform configurations across any environment.

‎docs/modules/listener-operator/pages/listenerclass.adoc

Lines changed: 198 additions & 90 deletions
Original file line numberDiff line numberDiff line change
@@ -1,158 +1,266 @@
11
= ListenerClass
22
:description: The ListenerClass defines listener types and exposure rules for Kubernetes Pods, supporting various service types like ClusterIP, NodePort, and LoadBalancer.
33

4-
A ListenerClass defines a category of listeners.
5-
For example, this could be "VPC-internal service", "internet-accessible service", or "K8s-internal service".
6-
The ListenerClass then defines how this intent is realized in a given cluster.
4+
A ListenerClass defines a category of listeners and how to expose them in your specific Kubernetes environment.
5+
Think of it as a policy that says "when an application asks for 'external-stable' networking, here's how we provide it in this cluster".
76

8-
For example, a Google Kubernetes Engine (GKE) cluster might want to expose all internet-facing services using a managed load balancer, since GKE nodes are
9-
relatively short-lived and don't have stable addresses:
7+
== Common Examples
8+
9+
=== Cloud Environment (GKE, EKS, AKS)
10+
11+
In managed cloud environments, you typically want to use LoadBalancers since nodes are short-lived:
1012

1113
[source,yaml]
1214
----
1315
include::example$listenerclass-public-gke.yaml[]
1416
----
1517

16-
On the other hand, an on-premise cluster might not have dedicated load balancer infrastructure at all, but instead use "pet" Nodes which may be expected to live for years.
17-
This might lead administrators of such systems to prefer exposing node ports directly instead:
18+
=== On-Premise Environment
19+
20+
In on-premise clusters with stable, long-lived nodes, a NodePort Service is often preferred.
21+
Sometimes these clusters lack the necessary LoadBalancer infrastructure:
1822

1923
[source,yaml]
2024
----
2125
include::example$listenerclass-public-onprem.yaml[]
2226
----
2327

24-
Finally, it can be desirable to add additional annotations to a Service.
25-
For example, a user might want to only expose some services inside a given cloud vendor VPC.
28+
=== Internal-Only Services / Additional Service Annotations
29+
30+
Sometimes it is required to add additional annotations to a Service.
2631
How exactly this is accomplished depends on the cloud provider in question, but for GKE this requires the annotation `networking.gke.io/load-balancer-type`:
2732

2833
[source,yaml]
2934
----
3035
include::example$listenerclass-internal-gke.yaml[]
3136
----
3237

33-
[#servicetype]
34-
== Service types
38+
== Default ListenerClasses
39+
40+
The Stackable Data Platform expects these three ListenerClasses to exist:
3541

36-
The service type is defined by `ListenerClass.spec.serviceType`.
37-
The following service types are currently supported by the Stackable Listener Operator:
42+
`cluster-internal`:: Used for internal cluster communication (e.g., ZooKeeper nodes talking to each other)
43+
`external-unstable`:: Used for external access where clients discover addresses dynamically and no stable address is required (e.g., individual Kafka brokers)
44+
`external-stable`:: Used for external access where clients need predictable addresses (e.g., Kafka bootstrap servers, Web UIs)
3845

39-
[#servicetype-clusterip]
40-
=== `ClusterIP`
4146

42-
The Listener can be accessed from inside the Kubernetes cluster.
43-
The Listener addresses will direct clients to the cluster-internal address.
47+
[#presets]
48+
== Presets
49+
50+
To help users get started, the Stackable Listener Operator ships different ListenerClass _presets_ for different environments.
51+
These are configured using the `preset` Helm value, with `stable-nodes` being the default.
52+
53+
=== Installation Commands
54+
55+
*For cloud environments:*
56+
[source,bash]
57+
----
58+
helm install listener-operator oci://oci.stackable.tech/sdp-charts/listener-operator --set preset=ephemeral-nodes
59+
----
60+
61+
*For clusters with stable nodes:*
62+
[source,bash]
63+
----
64+
helm install listener-operator oci://oci.stackable.tech/sdp-charts/listener-operator --set preset=stable-nodes
65+
----
66+
67+
*To define your own ListenerClasses:*
68+
[source,bash]
69+
----
70+
helm install listener-operator oci://oci.stackable.tech/sdp-charts/listener-operator --set preset=none
71+
----
72+
73+
[#preset-details]
74+
=== What Each Preset Creates
75+
76+
Both `stable-nodes` and `ephemeral-nodes` create the same three ListenerClasses that Stackable operators expect, but with different service types:
77+
78+
|===
79+
|ListenerClass Name |`stable-nodes` |`ephemeral-nodes`
80+
81+
|`cluster-internal`
82+
|ClusterIP
83+
|ClusterIP
84+
85+
|`external-unstable`
86+
|NodePort
87+
|NodePort
88+
89+
|`external-stable`
90+
|NodePort
91+
|LoadBalancer
92+
|===
93+
94+
==== Why the Difference?
95+
96+
* **stable-nodes**: Uses NodePort for external access and pins pods to specific nodes for address stability.
97+
+
98+
[CAUTION]
99+
====
100+
This creates a dependency on specific nodes. If a pinned node becomes unavailable, the pod cannot start on other nodes until you either restore the node or manually delete the PVC to allow rescheduling.
101+
====
102+
+
103+
.To recover from node failures:
104+
1. `kubectl delete pvc <listener-pvc-name>` - Allows the pod to reschedule (address may change)
105+
2. Or restore/replace the failed node with the same identity
106+
+
107+
This does _not_ require any particular networking setup, but is best suited for environments with reliable, long-lived nodes.
108+
109+
* **ephemeral-nodes**: Uses LoadBalancer for stable external access, allowing pods to move freely between nodes but requires LoadBalancer infrastructure
110+
111+
Managed cloud environments should generally already provide an integrated LoadBalancer controller.
112+
For on-premise environments, an external implementation such as https://docs.tigera.io/calico/latest/networking/configuring/advertise-service-ips[Calico] or https://metallb.org/[MetalLB] can be used.
113+
114+
NOTE: K3s' built-in https://docs.k3s.io/networking#service-load-balancer[ServiceLB] (Klipper) is _not_ recommended, because it doesn't allow multiple Services to bind the same Port.
115+
If you use ServiceLB, use the `stable-nodes` preset instead.
116+
117+
== Creating Custom ListenerClasses
118+
119+
If these presets are inadequate, you can create custom ListenerClasses.
120+
The key is understanding your environment's requirements.
121+
122+
=== Choosing the Right Service Type
123+
124+
[#servicetype-clusterip]
125+
==== ClusterIP
126+
* **Use for**: Internal cluster communication only
127+
* **Access**: Only from within the Kubernetes cluster
128+
* **Address**: Cluster-internal IP address
44129

45130
[#servicetype-nodeport]
46-
=== `NodePort`
131+
==== NodePort
132+
* **Use for**: External access (from outside the Kubernetes cluster) in environments with stable nodes
133+
* **Access**: From outside the cluster via `<NodeIP>:<NodePort>`
134+
* **Behavior**: Pins pods to specific nodes for address stability
47135

48-
The Listener can be accessed from outside the Kubernetes cluster.
49-
This may include the internet, if the Nodes have public IP addresses.
50-
The Listener address will direct clients to connect to a randomly assigned port on the Nodes running the Pods.
136+
[WARNING]
137+
====
138+
NodePort services may expose your applications to the internet if your Kubernetes nodes have public IP addresses.
139+
Ensure you understand your cluster's network topology and have appropriate firewall rules in place.
140+
====
51141

52-
Additionally, Pods bound to `NodePort` listeners will be xref:volume.adoc#pinning[pinned] to a specific Node.
53-
If this is undesirable, consider using xref:#servicetype-loadbalancer[] instead.
142+
[CAUTION]
143+
====
144+
When using NodePort with pinned pods, service addresses depend on specific nodes. If a pinned node becomes unavailable, the service may become unreachable until the pod can be rescheduled to a new node, potentially changing the service address.
145+
====
54146

55-
[#servicetype-loadbalancer]
56-
=== `LoadBalancer`
147+
Pods bound to `NodePort` listeners will be xref:volume.adoc#pinning[pinned] to a specific Node for address stability.
148+
If this behavior is undesirable, consider using xref:#servicetype-loadbalancer[] instead.
57149

58-
The Listener can be accessed from outside the Kubernetes cluster.
59-
This may include the internet, depending on the configuration of the Kubernetes cloud controller manager.
60-
A dedicated address will be allocated for the Listener.
61150

62-
Compared to xref:#servicetype-nodeport[], this service type allows Pods to be moved freely between Nodes.
63-
However, it requires https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer[a cloud controller manager that supports load balancers].
64-
Additionally, many cloud providers charge for load-balanced traffic.
151+
[#servicetype-loadbalancer]
152+
==== LoadBalancer
153+
* **Use for**: External access in environments without stable nodes or other reasons for a LoadBalancer
154+
* **Access**: From outside the cluster via dedicated load balancer
155+
* **Behavior**: Allows pods to move freely between nodes
156+
* **Requirements**: Kubernetes cluster must have a LoadBalancer controller
157+
* **Cost**: Cloud providers typically charge for load balancer usage
158+
159+
=== Advanced Configuration
65160

66161
[#servicetype-loadbalancer-class]
67-
==== Custom load-balancer classes
162+
==== Custom Load Balancer Classes
68163

69164
Kubernetes supports using multiple different load balancer types in the same cluster by configuring a unique https://kubernetes.io/docs/concepts/services-networking/service/#load-balancer-class[load-balancer class] for each provider.
70165

71166
The Stackable Listener Operator supports using custom classes setting the `ListenerClass.spec.loadBalancerClass` field.
72167

73168
NOTE: `loadBalancerClass` is _only_ respected when using the xref:#servicetype-loadbalancer[] service type. Otherwise, the field will be ignored.
74169

170+
[source,yaml]
171+
----
172+
apiVersion: listeners.stackable.tech/v1alpha1
173+
kind: ListenerClass
174+
metadata:
175+
name: my-custom-lb
176+
spec:
177+
serviceType: LoadBalancer
178+
loadBalancerClass: "example.com/my-loadbalancer"
179+
----
180+
75181
[#servicetype-loadbalancer-nodeportallocation]
76-
==== Load-balancer NodePort allocation
182+
==== Disabling NodePort Allocation
77183

78-
Normally, Kubernetes https://kubernetes.io/docs/concepts/services-networking/service/#load-balancer-nodeport-allocation[also enables] xref:#servicetype-nodeport[] access for any Services that use the xref:#servicetype-loadbalancer[] type.
184+
By default, LoadBalancer services https://kubernetes.io/docs/concepts/services-networking/service/#load-balancer-nodeport-allocation[also create NodePorts].
79185

80-
If your LoadBalancer controller does not require this then it can be disabled using the `ListenerClass.spec.loadBalancerAllocateNodePorts` field.
186+
This can be disabled using the `ListenerClass.spec.loadBalancerAllocateNodePorts` field.
81187

82188
NOTE: `loadBalancerAllocateNodePorts` is _only_ respected when using the xref:#servicetype-loadbalancer[] service type. Otherwise, the field will be ignored.
83189

84-
[#addresstype]
85-
== Address types
86-
87-
The Stackable Listener Operator supports both IP addresses and DNS hostnames. The preferred address type for a given ListenerClass can be configured using the `ListenerClass.spec.preferredAddressType` field. If no `preferredAddressType` is specified then it defaults to xref:#addresstype-hostname-conservative[].
88-
89-
NOTE: If the preferred address type is not supported for a given environment then another type will be used.
90-
91-
[#addresstype-ip]
92-
=== IP
93-
94-
The IP address of a resource. The addresses will be less predictable (especially for xref:#servicetype-clusterip[] services),
95-
but does not require any special client configuration (beyond what the xref:#servicetype[] requires).
190+
[source,yaml]
191+
----
192+
apiVersion: listeners.stackable.tech/v1alpha1
193+
kind: ListenerClass
194+
metadata:
195+
name: lb-no-nodeports
196+
spec:
197+
serviceType: LoadBalancer
198+
loadBalancerAllocateNodePorts: false
199+
----
96200

97-
[#addresstype-hostname]
98-
=== Hostname
201+
[#addresstype]
202+
=== Address Types
99203

100-
The DNS hostname of a resource. Clients must be able to resolve these addresses in order to connect, which may require special DNS configuration.
204+
Control whether clients receive IP addresses or hostnames:
101205

102-
[#addresstype-hostname-conservative]
103-
=== HostnameConservative
206+
`IP`:: Returns IP addresses (more compatible, less predictable especially for ClusterIP services)
207+
`Hostname`:: Returns DNS hostnames (requires proper DNS setup)
208+
`HostnameConservative`:: _(default)_ Uses hostnames for LoadBalancer/ClusterIP, IPs for NodePort
104209

105-
A pseudo-addresstype that is equivalent to xref:#addresstype-ip[] for xref:#servicetype-nodeport[] services, and xref:#addresstype-hostname[] for all others.
106210
This means that we default to hostnames where "safe", but don't assume that nodes are resolvable by external clients.
107211

108-
== Default ListenerClasses
109-
110-
The Stackable Data Platform assumes the existence of a few predefined ListenerClasses, and will use them by default as appropriate:
111-
112-
`cluster-internal`:: Used for listeners that are only accessible internally from the cluster. For example: communication between ZooKeeper nodes.
113-
`external-unstable`:: Used for listeners that are accessible from outside the cluster, but which do not require a stable address. For example: individual Kafka brokers.
114-
`external-stable`:: Used for listeners that are accessible from outside the cluster, and do require a stable address. For example: Kafka bootstrap.
212+
NOTE: If the preferred address type is not supported for a given environment then another type will be used.
115213

116-
[#presets]
117-
=== Presets
214+
[source,yaml]
215+
----
216+
apiVersion: listeners.stackable.tech/v1alpha1
217+
kind: ListenerClass
218+
metadata:
219+
name: hostname-preferred
220+
spec:
221+
serviceType: LoadBalancer
222+
preferredAddressType: Hostname
223+
----
118224

119-
To help users get started, the Stackable Listener Operator ships different ListenerClass _presets_ for different environments.
120-
These are configured using the `preset` Helm value.
225+
=== Adding Service Annotations
121226

122-
[#preset-stable-nodes]
123-
==== `stable-nodes`
227+
Many cloud providers require specific annotations for advanced features:
124228

125-
The `stable-nodes` preset installs ListenerClasses appropriate for Kubernetes clusters that use long-lived "pet nodes".
126-
This does _not_ require any particular networking setup, but makes pods that require
127-
stable addresses "sticky" to the Kubernetes Node that they were scheduled to.
128-
In addition, downstream operators may generate configurations that refer to particular nodes by name.
229+
[source,yaml]
230+
----
231+
apiVersion: listeners.stackable.tech/v1alpha1
232+
kind: ListenerClass
233+
metadata:
234+
name: aws-internal-nlb
235+
spec:
236+
serviceType: LoadBalancer
237+
serviceAnnotations:
238+
service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
239+
service.beta.kubernetes.io/aws-load-balancer-internal: "true"
240+
----
129241

130-
The following ListenerClasses are installed:
242+
== Frequently Asked Questions
131243

132-
`cluster-internal`:: xref:#servicetype-clusterip[]
133-
`external-unstable`:: xref:#servicetype-nodeport[]
134-
`external-stable`:: xref:#servicetype-nodeport[]
244+
=== Why aren't ListenerClasses namespace-scoped?
135245

136-
[#preset-ephemeral-nodes]
137-
==== `ephemeral-nodes`
246+
ListenerClasses are intentionally cluster-scoped to encourage separation of concerns between platform administrators (who understand infrastructure) and application developers (who choose policies).
247+
While this limits flexibility for application-specific customizations, it promotes networking standardization across the cluster.
138248

139-
The `ephemeral-nodes` preset installs ListenerClasses appropriate for Kubernetes clusters that use short-lived "cattle nodes".
140-
This makes them appropriate for managed cloud environments, but requires that
141-
a LoadBalancer controller is present in the cluster.
249+
If you need more granular control, consider creating additional ListenerClasses or using the `none` preset for full customization.
142250

143-
Managed cloud environments should generally already provide an integrated LoadBalancer controller.
144-
For on-premise environments, an external implementation such as https://docs.tigera.io/calico/latest/networking/configuring/advertise-service-ips[Calico] or https://metallb.org/[MetalLB] can be used.
251+
=== My pods won't start after a node failure - what do I do?
145252

146-
NOTE: K3s' built-in https://docs.k3s.io/networking#service-load-balancer[ServiceLB] (Klipper) is _not_ recommended, because it doesn't allow multiple Services to bind the same Port.
147-
If you use ServiceLB, use the xref:#preset-stable-nodes[] preset instead.
253+
If you're using the `stable-nodes` preset (or custom NodePort ListenerClasses), pods may get stuck when their pinned node becomes unavailable.
148254

149-
The following ListenerClasses are installed:
255+
*Quick fix:*
150256

151-
`cluster-internal`:: xref:#servicetype-clusterip[]
152-
`external-unstable`:: xref:#servicetype-nodeport[]
153-
`external-stable`:: xref:#servicetype-loadbalancer[]
257+
[source,bash]
258+
----
259+
# Find the stuck PVC
260+
kubectl get pvc | grep listener-
154261
155-
[#preset-none]
156-
==== `none`
262+
# Delete it to allow rescheduling (address may change)
263+
kubectl delete pvc <listener-pvc-name>
264+
----
157265

158-
The `none` (pseudo-)preset installs no ListenerClasses, leaving the administrator to define them for themself.
266+
For more details on why this happens and prevention strategies, see the xref:#preset-details[preset details section].

0 commit comments

Comments
 (0)