Operator crashes for very long certificate lifetimes

[soenkeliebau]

A user recently noticed that commons op kept crashing on startup in their environment.

The root cause turned out to be certificates with very long lifetimes (expiry in 2069). When the commons operator sees these, it tries to schedule a restart for those pods in 2069, which panics, because it is "too far" in the future.

An upstream issue in kube-rs has been opened: kube-rs/kube#1772

We need to decide, whether it is realistic to get this into 25.7 via a released kube-rs version, or if we should implement a simple workaround for now that limits scheduling tasks to 6 months or something similar.
We can then remove this, once kube-rs releases a version with a fix.

[soenkeliebau]

We decided to wait for someone to fix this upstream. We won't do anything about it for now.

[dervoeti]

My plan would be:
Clamp the rescheduling delay to 6 months in commons-operator and add a comment referring to the upstream issue.

If it works easily, try to fix it in kube-rs, so the issue doesn't just sit there upstream and we run into the same problem again somewhere else in the future.
Once it's merged and released, remove the workaround in commons-op and add a comment stating that kube-rs will clamp the delay if it's too long.

[lfrancke]

Just so I understand: Clamping the rescheduling delay does not stop user from having longer certificate lifetimes, right?

[dervoeti]

Just so I understand: Claming the rescheduling delay does not stop user from having longer certificate lifetimes, right?

No, commons-operator would just reschedule more often than strictly neccessary. For a lifetime of 15 months it would look like:

• wait 6 months
• wait 6 months
• wait 3 months

Instead of waiting 15 months once.

I will test this scenario specifically (with shorter test lifetimes of course).

[lfrancke]

Great, thanks. The plan sounds good to me. Go ahead with step 1 and feel free to spend a day on trying to fix it in kube-rs but if it takes longer than that let me know.

[nightkr]

IMO: The "solution" (if we don't count on upstream changes in Tokio, anyway) would be doing the same clamp in kube-rs, and it's the same 5 minute fix to do it there.

The only benefit I see to doing it in commons-op would be a band-aid while waiting for the kube-rs review/release.