Skip to content

Commit b723367

Browse files
hiboyanghiboyang
authored
Support exposing extra TCP ports in Spark Driver via K8s Ingress (#1998)
* Add driverIngressOptions in SparkApplication CRD Signed-off-by: Bo (AIML) Yang <[email protected]> * Update chart version to 1.3.0 Signed-off-by: Bo (AIML) Yang <[email protected]> * Update helm chart README Signed-off-by: Bo (AIML) Yang <[email protected]> * Fix make detect-crds-drift Signed-off-by: Bo (AIML) Yang <[email protected]> * Update api-docs.md Signed-off-by: Bo (AIML) Yang <[email protected]> --------- Signed-off-by: Bo (AIML) Yang <[email protected]>
1 parent 153537e commit b723367

14 files changed

+1535
-260
lines changed

‎charts/spark-operator-chart/Chart.yaml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
apiVersion: v2
22
name: spark-operator
33
description: A Helm chart for Spark on Kubernetes operator
4-
version: 1.2.15
5-
appVersion: v1beta2-1.4.6-3.5.0
4+
version: 1.3.0
5+
appVersion: v1beta2-1.4.2-3.5.0
66
keywords:
77
- spark
88
home: https://github.com/kubeflow/spark-operator

‎charts/spark-operator-chart/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# spark-operator
22

3-
![Version: 1.2.15](https://img.shields.io/badge/Version-1.2.15-informational?style=flat-square) ![AppVersion: v1beta2-1.4.6-3.5.0](https://img.shields.io/badge/AppVersion-v1beta2--1.4.6--3.5.0-informational?style=flat-square)
3+
![Version: 1.3.0](https://img.shields.io/badge/Version-1.3.0-informational?style=flat-square) ![AppVersion: v1beta2-1.4.2-3.5.0](https://img.shields.io/badge/AppVersion-v1beta2--1.4.2--3.5.0-informational?style=flat-square)
44

55
A Helm chart for Spark on Kubernetes operator
66

‎charts/spark-operator-chart/crds/sparkoperator.k8s.io_scheduledsparkapplications.yaml

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3779,6 +3779,43 @@ spec:
37793779
serviceType:
37803780
type: string
37813781
type: object
3782+
driverIngressOptions:
3783+
items:
3784+
properties:
3785+
serviceAnnotations:
3786+
additionalProperties:
3787+
type: string
3788+
type: object
3789+
serviceLabels:
3790+
additionalProperties:
3791+
type: string
3792+
type: object
3793+
ingressURLFormat:
3794+
type: string
3795+
ingressAnnotations:
3796+
additionalProperties:
3797+
type: string
3798+
type: object
3799+
ingressTLS:
3800+
items:
3801+
properties:
3802+
hosts:
3803+
items:
3804+
type: string
3805+
type: array
3806+
secretName:
3807+
type: string
3808+
type: object
3809+
type: array
3810+
servicePort:
3811+
format: int32
3812+
type: integer
3813+
servicePortName:
3814+
type: string
3815+
serviceType:
3816+
type: string
3817+
type: object
3818+
type: array
37823819
sparkVersion:
37833820
type: string
37843821
timeToLiveSeconds:

‎charts/spark-operator-chart/crds/sparkoperator.k8s.io_sparkapplications.yaml

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3767,6 +3767,43 @@ spec:
37673767
serviceType:
37683768
type: string
37693769
type: object
3770+
driverIngressOptions:
3771+
items:
3772+
properties:
3773+
serviceAnnotations:
3774+
additionalProperties:
3775+
type: string
3776+
type: object
3777+
serviceLabels:
3778+
additionalProperties:
3779+
type: string
3780+
type: object
3781+
ingressURLFormat:
3782+
type: string
3783+
ingressAnnotations:
3784+
additionalProperties:
3785+
type: string
3786+
type: object
3787+
ingressTLS:
3788+
items:
3789+
properties:
3790+
hosts:
3791+
items:
3792+
type: string
3793+
type: array
3794+
secretName:
3795+
type: string
3796+
type: object
3797+
type: array
3798+
servicePort:
3799+
format: int32
3800+
type: integer
3801+
servicePortName:
3802+
type: string
3803+
serviceType:
3804+
type: string
3805+
type: object
3806+
type: array
37703807
sparkVersion:
37713808
type: string
37723809
timeToLiveSeconds:

‎docs/api-docs.md

Lines changed: 144 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -601,6 +601,20 @@ SparkUIConfiguration
601601
</tr>
602602
<tr>
603603
<td>
604+
<code>driverIngressOptions</code><br/>
605+
<em>
606+
<a href="#sparkoperator.k8s.io/v1beta2.DriverIngressConfiguration">
607+
[]DriverIngressConfiguration
608+
</a>
609+
</em>
610+
</td>
611+
<td>
612+
<em>(Optional)</em>
613+
<p>DriverIngressOptions allows configuring the Service and the Ingress to expose ports inside Spark Driver</p>
614+
</td>
615+
</tr>
616+
<tr>
617+
<td>
604618
<code>dynamicAllocation</code><br/>
605619
<em>
606620
<a href="#sparkoperator.k8s.io/v1beta2.DynamicAllocation">
@@ -991,6 +1005,122 @@ string
9911005
</tr>
9921006
</tbody>
9931007
</table>
1008+
<h3 id="sparkoperator.k8s.io/v1beta2.DriverIngressConfiguration">DriverIngressConfiguration
1009+
</h3>
1010+
<p>
1011+
(<em>Appears on:</em><a href="#sparkoperator.k8s.io/v1beta2.SparkApplicationSpec">SparkApplicationSpec</a>)
1012+
</p>
1013+
<div>
1014+
<p>DriverIngressConfiguration is for driver ingress specific configuration parameters.</p>
1015+
</div>
1016+
<table>
1017+
<thead>
1018+
<tr>
1019+
<th>Field</th>
1020+
<th>Description</th>
1021+
</tr>
1022+
</thead>
1023+
<tbody>
1024+
<tr>
1025+
<td>
1026+
<code>servicePort</code><br/>
1027+
<em>
1028+
int32
1029+
</em>
1030+
</td>
1031+
<td>
1032+
<p>ServicePort allows configuring the port at service level that might be different from the targetPort.</p>
1033+
</td>
1034+
</tr>
1035+
<tr>
1036+
<td>
1037+
<code>servicePortName</code><br/>
1038+
<em>
1039+
string
1040+
</em>
1041+
</td>
1042+
<td>
1043+
<p>ServicePortName allows configuring the name of the service port.
1044+
This may be useful for sidecar proxies like Envoy injected by Istio which require specific ports names to treat traffic as proper HTTP.</p>
1045+
</td>
1046+
</tr>
1047+
<tr>
1048+
<td>
1049+
<code>serviceType</code><br/>
1050+
<em>
1051+
<a href="https://v1-18.docs.kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#servicetype-v1-core">
1052+
Kubernetes core/v1.ServiceType
1053+
</a>
1054+
</em>
1055+
</td>
1056+
<td>
1057+
<em>(Optional)</em>
1058+
<p>ServiceType allows configuring the type of the service. Defaults to ClusterIP.</p>
1059+
</td>
1060+
</tr>
1061+
<tr>
1062+
<td>
1063+
<code>serviceAnnotations</code><br/>
1064+
<em>
1065+
map[string]string
1066+
</em>
1067+
</td>
1068+
<td>
1069+
<em>(Optional)</em>
1070+
<p>ServiceAnnotations is a map of key,value pairs of annotations that might be added to the service object.</p>
1071+
</td>
1072+
</tr>
1073+
<tr>
1074+
<td>
1075+
<code>serviceLabels</code><br/>
1076+
<em>
1077+
map[string]string
1078+
</em>
1079+
</td>
1080+
<td>
1081+
<em>(Optional)</em>
1082+
<p>ServiceLables is a map of key,value pairs of labels that might be added to the service object.</p>
1083+
</td>
1084+
</tr>
1085+
<tr>
1086+
<td>
1087+
<code>ingressURLFormat</code><br/>
1088+
<em>
1089+
string
1090+
</em>
1091+
</td>
1092+
<td>
1093+
<p>IngressURLFormat is the URL for the ingress.</p>
1094+
</td>
1095+
</tr>
1096+
<tr>
1097+
<td>
1098+
<code>ingressAnnotations</code><br/>
1099+
<em>
1100+
map[string]string
1101+
</em>
1102+
</td>
1103+
<td>
1104+
<em>(Optional)</em>
1105+
<p>IngressAnnotations is a map of key,value pairs of annotations that might be added to the ingress object. i.e. specify nginx as ingress.class</p>
1106+
</td>
1107+
</tr>
1108+
<tr>
1109+
<td>
1110+
<code>ingressTLS</code><br/>
1111+
<em>
1112+
<a href="https://v1-18.docs.kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#ingresstls-v1-networking">
1113+
[]Kubernetes networking/v1.IngressTLS
1114+
</a>
1115+
</em>
1116+
</td>
1117+
<td>
1118+
<em>(Optional)</em>
1119+
<p>TlsHosts is useful If we need to declare SSL certificates to the ingress object</p>
1120+
</td>
1121+
</tr>
1122+
</tbody>
1123+
</table>
9941124
<h3 id="sparkoperator.k8s.io/v1beta2.DriverSpec">DriverSpec
9951125
</h3>
9961126
<p>
@@ -2493,6 +2623,20 @@ SparkUIConfiguration
24932623
</tr>
24942624
<tr>
24952625
<td>
2626+
<code>driverIngressOptions</code><br/>
2627+
<em>
2628+
<a href="#sparkoperator.k8s.io/v1beta2.DriverIngressConfiguration">
2629+
[]DriverIngressConfiguration
2630+
</a>
2631+
</em>
2632+
</td>
2633+
<td>
2634+
<em>(Optional)</em>
2635+
<p>DriverIngressOptions allows configuring the Service and the Ingress to expose ports inside Spark Driver</p>
2636+
</td>
2637+
</tr>
2638+
<tr>
2639+
<td>
24962640
<code>dynamicAllocation</code><br/>
24972641
<em>
24982642
<a href="#sparkoperator.k8s.io/v1beta2.DynamicAllocation">

‎manifest/crds/sparkoperator.k8s.io_scheduledsparkapplications.yaml

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3779,6 +3779,43 @@ spec:
37793779
serviceType:
37803780
type: string
37813781
type: object
3782+
driverIngressOptions:
3783+
items:
3784+
properties:
3785+
serviceAnnotations:
3786+
additionalProperties:
3787+
type: string
3788+
type: object
3789+
serviceLabels:
3790+
additionalProperties:
3791+
type: string
3792+
type: object
3793+
ingressURLFormat:
3794+
type: string
3795+
ingressAnnotations:
3796+
additionalProperties:
3797+
type: string
3798+
type: object
3799+
ingressTLS:
3800+
items:
3801+
properties:
3802+
hosts:
3803+
items:
3804+
type: string
3805+
type: array
3806+
secretName:
3807+
type: string
3808+
type: object
3809+
type: array
3810+
servicePort:
3811+
format: int32
3812+
type: integer
3813+
servicePortName:
3814+
type: string
3815+
serviceType:
3816+
type: string
3817+
type: object
3818+
type: array
37823819
sparkVersion:
37833820
type: string
37843821
timeToLiveSeconds:

‎manifest/crds/sparkoperator.k8s.io_sparkapplications.yaml

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3767,6 +3767,43 @@ spec:
37673767
serviceType:
37683768
type: string
37693769
type: object
3770+
driverIngressOptions:
3771+
items:
3772+
properties:
3773+
serviceAnnotations:
3774+
additionalProperties:
3775+
type: string
3776+
type: object
3777+
serviceLabels:
3778+
additionalProperties:
3779+
type: string
3780+
type: object
3781+
ingressURLFormat:
3782+
type: string
3783+
ingressAnnotations:
3784+
additionalProperties:
3785+
type: string
3786+
type: object
3787+
ingressTLS:
3788+
items:
3789+
properties:
3790+
hosts:
3791+
items:
3792+
type: string
3793+
type: array
3794+
secretName:
3795+
type: string
3796+
type: object
3797+
type: array
3798+
servicePort:
3799+
format: int32
3800+
type: integer
3801+
servicePortName:
3802+
type: string
3803+
serviceType:
3804+
type: string
3805+
type: object
3806+
type: array
37703807
sparkVersion:
37713808
type: string
37723809
timeToLiveSeconds:

0 commit comments

Comments
 (0)