kustomize re-orders config causing fail when crd or admission webhook used

[donbowman]

If we look @ cert-manager we see that it has in its output:

• crd
• objects
• webhooks
• objects using webooks + crd

thus the order must be preserved.

So if I apply kustomize across their input and add

---
apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
  namespace: kube-system
  name: selfsigning-issuer
spec:
  selfSigned: {}

to the 'end', i get a failure

Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "clusterissuers.admission.certmanager.k8s.io": the server could not find the requested resource

the attached output file shows. The selfsigning-issuer ClusterIssuer is emitted in the middle, before its webhook is defined

cert-manager-to-be-applied.yaml.gz

[donbowman]

possible solutions?

• kubectl gets a 'sort' or 'order' concept
• kubectl gets a 'begin/commit' phase like sql
• kustomize maintains order
• kubectl understands crd + webhooks, orders crd first, then webhook creation first
• kustomize understands order and emits crd first then webhook
• kubectl gets a 'filter' argument, i can run 3 times with kubectl apply --filter=crd ; kubectl apply --filter=webhook first

others?

[donbowman]

In the short term I have a workaround:
https://github.com/Agilicus/yaml_filter

kustomize build . | yaml_filter.py -i CustomResourceDefinition,ValidatingWebhookConfiguration |kubectl apply -f -
kustomize build . | yaml_filter.py -o CustomResourceDefinition,ValidatingWebhookConfiguration |kubectl apply -f -

[pwittrock]

kustomize already sorts things like namespaces for this reason. CRDs and Webhooks should also be sorted IMHO.

[pwittrock]

I think we just need to add MutatingWebhooks here:

kustomize/pkg/gvk/gvk.go

Line 73 in 92fc368

"Namespace",

[donbowman]

This issue is not resolved by that commit.
the commit doesn't even add the test case above.

[monopole]

Re-opened.

kubernetes/enhancements#865 proposed as a general solution to order.

Open to other ideas.

@donbowman You mention that the PR that attempted to close this issue did not include the data you provided, and it did not.

But the test case you mention was a large tarball of resources - which can be a lot to digest.

The readme suggests a good approach for reporting undesired output, but i realize that it's buried in a bunch of commentary.

basically, if you convert your data to a unit test in the pkg/target package, e.g. this config map generator test, then we all get multiple wins.

1. The test unambiguously records the bad behavior currently exhibited by the code. The test must pass (so we can merge it), but it should be commented to show why the actual output is undesired, and what would be desired instead (e.g. resource Foo must be declared in the output before resource Bar). These tests are formulated to be very easy to build and read.

2. When a purported fix comes along, it must modify this test, encoding the desired behavior, or the PR cannot be submitted. So proper modification of the test serves as proof that the bug is closeable.

3. The test remains as part of the regression suite forever in either form - either the bad behavior stands as a proven thing, or the fixed behavior stands to protect itself.

please consider converting your data to such a test. It's not that i don't understand the problem, it's that i want help with regression coverage.

[anguslees]

Fwiw, I suggest replacing #822 with something that orders Webhook declarations last.

In a single invocation (and without more human-driven direction), we have to assume that we aren't able to perform the webhook until after Deployments/etc have started, and we also have to assume that other resources in the same invocation don't require the (mutating) webhook to act on them (that would be a bootstrapping cycle).

(Edit: or better yet for kustomize: preserve original document order)

[mgoltzsche]

I changed the order so that ValidatingWebhookConfiguration now comes last.
This makes a cert-manager.yaml installation work with kustomize as well.

However unnecessary problems caused by ordering may still appear in theory when using objects with a CustomResourceDefinition's kind where one has a hard dependency to the other.
This could be avoided when the original input source order would be respected as kubectl apply -f does it. For this reason I agree with @anguslees that the original order should be preserved.

The question is: why was a kustomize-specific order introduced in the first place?

Edit: Also I didn't touch the MutatingWebhookConfiguration order since I had no example to test at hand but I think it must come last as well.