[BUG] Flyte Admin Creates New Workflow CRDs When Attempting To Create Duplicate Executions

[Sovietaced]

Flyte & Flytekit version

Flyte v1.15.X

Describe the bug

We had some alerts fire that the P99 acceptance latency of workflows was greater than 30s. We use these alerts to understand if something is wrong with Flyte Propeller. The workflow latency was extremely high and didn't seem to match up with any recent executions.

After doing some investigation I found a workflow that had completed several hours ago that was somehow re-evaluated by Flyte Propeller. After doing some additional investigation it appears that Flyte Scheduler was the trigger for the workflow as it was on a scheduled launch plan.
It appears that the first workflow execution failed, which was tracked in the propeller's terminated tracking store. A couple hours later the second workflow execution was triggered. This created a new CRD but did not create a new DB model since the previous execution already exists. The second workflow CRD was ignored because it was still tracked as terminated by Flyte Propeller until the statically configured LRU cache evicted the entry and the second workflow CRD was finally processed, thus creating a very large acceptance latency.

Expected behavior

I would expect that duplicate executions would not create a duplicate workflow CRD

Additional context to reproduce

No response

Screenshots

No response

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[runllm]

FlyteAdmin can create duplicate workflow CRDs for executions if a scheduled launch plan triggers a new execution with the same identifier as a previously failed one. In this scenario, a new CRD is created, but no new DB model is written since the execution already exists, causing FlytePropeller to ignore the CRD until its terminated state is evicted from the LRU cache. This results in high acceptance latency when the CRD is eventually processed. This behavior is not explicitly documented as a supported or intended feature, and the data sources do not provide a built-in mechanism to prevent duplicate CRD creation for the same execution identifier. The issue is discussed in community reports, but no official fix or configuration is described in the available documentation or codebase references.

If you need to avoid this, you may need to implement additional checks or cleanup logic outside of Flyte's default behavior, as the platform currently does not guarantee deduplication of workflow CRDs for duplicate executions triggered by the scheduler. Would you like more detail on the internal workflow execution and CRD handling process in Flyte?

Sources:

• [BUG] Race condition leads to workflows stuck in UNDEFINED/UNKNOWN phase #6556
• RFC: Add workflow execution concurrency #5659

---

This is an AI-generated response and your feedback is appreciated! Please leave a 👍 if this is helpful and 👎 if it is not.

Mention @RunLLM to ask follow-up questions. See runllm.com for more!

[Sovietaced]

I have an implementation of this and will file a PR soon.