Skip to content

Commit d0daf2f

Browse files
ChenYi015ChenYi015
authored
Support pod template for Spark 3.x applications (#2141)
* Update API definition to support pod template Signed-off-by: Yi Chen <[email protected]> * Mark pod template field as schemaless Signed-off-by: Yi Chen <[email protected]> * Add kubebuilder marker to preserve unknown fields Signed-off-by: Yi Chen <[email protected]> * Add example for using pod template Signed-off-by: Yi Chen <[email protected]> * Support pod template Signed-off-by: Yi Chen <[email protected]> --------- Signed-off-by: Yi Chen <[email protected]>
1 parent fd2e125 commit d0daf2f

17 files changed

+566
-29
lines changed

‎Makefile

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -109,7 +109,7 @@ print-%: ; @echo $*=$($*)
109109

110110
.PHONY: manifests
111111
manifests: controller-gen ## Generate CustomResourceDefinition, RBAC and WebhookConfiguration manifests.
112-
$(CONTROLLER_GEN) crd rbac:roleName=spark-operator-controller webhook paths="./..." output:crd:artifacts:config=config/crd/bases
112+
$(CONTROLLER_GEN) crd:generateEmbeddedObjectMeta=true rbac:roleName=spark-operator-controller webhook paths="./..." output:crd:artifacts:config=config/crd/bases
113113

114114
.PHONY: generate
115115
generate: controller-gen ## Generate code containing DeepCopy, DeepCopyInto, and DeepCopyObject method implementations.

‎api/v1beta2/sparkapplication_types.go

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -409,6 +409,14 @@ type Dependencies struct {
409409
// SparkPodSpec defines common things that can be customized for a Spark driver or executor pod.
410410
// TODO: investigate if we should use v1.PodSpec and limit what can be set instead.
411411
type SparkPodSpec struct {
412+
// Template is a pod template that can be used to define the driver or executor pod configurations that Spark configurations do not support.
413+
// Spark version >= 3.0.0 is required.
414+
// Ref: https://spark.apache.org/docs/latest/running-on-kubernetes.html#pod-template.
415+
// +optional
416+
// +kubebuilder:validation:Schemaless
417+
// +kubebuilder:validation:Type:=object
418+
// +kubebuilder:pruning:PreserveUnknownFields
419+
Template *corev1.PodTemplateSpec `json:"template,omitempty"`
412420
// Cores maps to `spark.driver.cores` or `spark.executor.cores` for the driver and executors, respectively.
413421
// +optional
414422
// +kubebuilder:validation:Minimum=1

‎api/v1beta2/zz_generated.deepcopy.go

Lines changed: 5 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎charts/spark-operator-chart/crds/sparkoperator.k8s.io_scheduledsparkapplications.yaml

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4755,6 +4755,13 @@ spec:
47554755
- name
47564756
type: object
47574757
type: array
4758+
template:
4759+
description: |-
4760+
Template is a pod template that can be used to define the driver or executor pod configurations that Spark configurations do not support.
4761+
Spark version >= 3.0.0 is required.
4762+
Ref: https://spark.apache.org/docs/latest/running-on-kubernetes.html#pod-template.
4763+
type: object
4764+
x-kubernetes-preserve-unknown-fields: true
47584765
terminationGracePeriodSeconds:
47594766
description: Termination grace period seconds for the pod
47604767
format: int64
@@ -9512,6 +9519,13 @@ spec:
95129519
- name
95139520
type: object
95149521
type: array
9522+
template:
9523+
description: |-
9524+
Template is a pod template that can be used to define the driver or executor pod configurations that Spark configurations do not support.
9525+
Spark version >= 3.0.0 is required.
9526+
Ref: https://spark.apache.org/docs/latest/running-on-kubernetes.html#pod-template.
9527+
type: object
9528+
x-kubernetes-preserve-unknown-fields: true
95159529
terminationGracePeriodSeconds:
95169530
description: Termination grace period seconds for the pod
95179531
format: int64
@@ -10351,6 +10365,23 @@ spec:
1035110365
May contain labels and annotations that will be copied into the PVC
1035210366
when creating it. No other fields are allowed and will be rejected during
1035310367
validation.
10368+
properties:
10369+
annotations:
10370+
additionalProperties:
10371+
type: string
10372+
type: object
10373+
finalizers:
10374+
items:
10375+
type: string
10376+
type: array
10377+
labels:
10378+
additionalProperties:
10379+
type: string
10380+
type: object
10381+
name:
10382+
type: string
10383+
namespace:
10384+
type: string
1035410385
type: object
1035510386
spec:
1035610387
description: |-

‎charts/spark-operator-chart/crds/sparkoperator.k8s.io_sparkapplications.yaml

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4694,6 +4694,13 @@ spec:
46944694
- name
46954695
type: object
46964696
type: array
4697+
template:
4698+
description: |-
4699+
Template is a pod template that can be used to define the driver or executor pod configurations that Spark configurations do not support.
4700+
Spark version >= 3.0.0 is required.
4701+
Ref: https://spark.apache.org/docs/latest/running-on-kubernetes.html#pod-template.
4702+
type: object
4703+
x-kubernetes-preserve-unknown-fields: true
46974704
terminationGracePeriodSeconds:
46984705
description: Termination grace period seconds for the pod
46994706
format: int64
@@ -9421,6 +9428,13 @@ spec:
94219428
- name
94229429
type: object
94239430
type: array
9431+
template:
9432+
description: |-
9433+
Template is a pod template that can be used to define the driver or executor pod configurations that Spark configurations do not support.
9434+
Spark version >= 3.0.0 is required.
9435+
Ref: https://spark.apache.org/docs/latest/running-on-kubernetes.html#pod-template.
9436+
type: object
9437+
x-kubernetes-preserve-unknown-fields: true
94249438
terminationGracePeriodSeconds:
94259439
description: Termination grace period seconds for the pod
94269440
format: int64
@@ -10257,6 +10271,23 @@ spec:
1025710271
May contain labels and annotations that will be copied into the PVC
1025810272
when creating it. No other fields are allowed and will be rejected during
1025910273
validation.
10274+
properties:
10275+
annotations:
10276+
additionalProperties:
10277+
type: string
10278+
type: object
10279+
finalizers:
10280+
items:
10281+
type: string
10282+
type: array
10283+
labels:
10284+
additionalProperties:
10285+
type: string
10286+
type: object
10287+
name:
10288+
type: string
10289+
namespace:
10290+
type: string
1026010291
type: object
1026110292
spec:
1026210293
description: |-

‎config/crd/bases/sparkoperator.k8s.io_scheduledsparkapplications.yaml

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4755,6 +4755,13 @@ spec:
47554755
- name
47564756
type: object
47574757
type: array
4758+
template:
4759+
description: |-
4760+
Template is a pod template that can be used to define the driver or executor pod configurations that Spark configurations do not support.
4761+
Spark version >= 3.0.0 is required.
4762+
Ref: https://spark.apache.org/docs/latest/running-on-kubernetes.html#pod-template.
4763+
type: object
4764+
x-kubernetes-preserve-unknown-fields: true
47584765
terminationGracePeriodSeconds:
47594766
description: Termination grace period seconds for the pod
47604767
format: int64
@@ -9512,6 +9519,13 @@ spec:
95129519
- name
95139520
type: object
95149521
type: array
9522+
template:
9523+
description: |-
9524+
Template is a pod template that can be used to define the driver or executor pod configurations that Spark configurations do not support.
9525+
Spark version >= 3.0.0 is required.
9526+
Ref: https://spark.apache.org/docs/latest/running-on-kubernetes.html#pod-template.
9527+
type: object
9528+
x-kubernetes-preserve-unknown-fields: true
95159529
terminationGracePeriodSeconds:
95169530
description: Termination grace period seconds for the pod
95179531
format: int64
@@ -10351,6 +10365,23 @@ spec:
1035110365
May contain labels and annotations that will be copied into the PVC
1035210366
when creating it. No other fields are allowed and will be rejected during
1035310367
validation.
10368+
properties:
10369+
annotations:
10370+
additionalProperties:
10371+
type: string
10372+
type: object
10373+
finalizers:
10374+
items:
10375+
type: string
10376+
type: array
10377+
labels:
10378+
additionalProperties:
10379+
type: string
10380+
type: object
10381+
name:
10382+
type: string
10383+
namespace:
10384+
type: string
1035410385
type: object
1035510386
spec:
1035610387
description: |-

‎config/crd/bases/sparkoperator.k8s.io_sparkapplications.yaml

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4694,6 +4694,13 @@ spec:
46944694
- name
46954695
type: object
46964696
type: array
4697+
template:
4698+
description: |-
4699+
Template is a pod template that can be used to define the driver or executor pod configurations that Spark configurations do not support.
4700+
Spark version >= 3.0.0 is required.
4701+
Ref: https://spark.apache.org/docs/latest/running-on-kubernetes.html#pod-template.
4702+
type: object
4703+
x-kubernetes-preserve-unknown-fields: true
46974704
terminationGracePeriodSeconds:
46984705
description: Termination grace period seconds for the pod
46994706
format: int64
@@ -9421,6 +9428,13 @@ spec:
94219428
- name
94229429
type: object
94239430
type: array
9431+
template:
9432+
description: |-
9433+
Template is a pod template that can be used to define the driver or executor pod configurations that Spark configurations do not support.
9434+
Spark version >= 3.0.0 is required.
9435+
Ref: https://spark.apache.org/docs/latest/running-on-kubernetes.html#pod-template.
9436+
type: object
9437+
x-kubernetes-preserve-unknown-fields: true
94249438
terminationGracePeriodSeconds:
94259439
description: Termination grace period seconds for the pod
94269440
format: int64
@@ -10257,6 +10271,23 @@ spec:
1025710271
May contain labels and annotations that will be copied into the PVC
1025810272
when creating it. No other fields are allowed and will be rejected during
1025910273
validation.
10274+
properties:
10275+
annotations:
10276+
additionalProperties:
10277+
type: string
10278+
type: object
10279+
finalizers:
10280+
items:
10281+
type: string
10282+
type: array
10283+
labels:
10284+
additionalProperties:
10285+
type: string
10286+
type: object
10287+
name:
10288+
type: string
10289+
namespace:
10290+
type: string
1026010291
type: object
1026110292
spec:
1026210293
description: |-

‎docs/api-docs.md

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2828,6 +2828,22 @@ TODO: investigate if we should use v1.PodSpec and limit what can be set instead.
28282828
<tbody>
28292829
<tr>
28302830
<td>
2831+
<code>template</code><br/>
2832+
<em>
2833+
<a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#podtemplatespec-v1-core">
2834+
Kubernetes core/v1.PodTemplateSpec
2835+
</a>
2836+
</em>
2837+
</td>
2838+
<td>
2839+
<em>(Optional)</em>
2840+
<p>Template is a pod template that can be used to define the driver or executor pod configurations that Spark configurations do not support.
2841+
Spark version &gt;= 3.0.0 is required.
2842+
Ref: <a href="https://spark.apache.org/docs/latest/running-on-kubernetes.html#pod-template">https://spark.apache.org/docs/latest/running-on-kubernetes.html#pod-template</a>.</p>
2843+
</td>
2844+
</tr>
2845+
<tr>
2846+
<td>
28312847
<code>cores</code><br/>
28322848
<em>
28332849
int32

0 commit comments

Comments
 (0)