Alert if a task misses deadline

[yuqian90]

Description

We often need to be notified if a task is not finished by certain deadline. This may sound very similar to the existing SLA concept, but unfortunately, the SLA implementation makes it not useful for such cases for a few reasons:

1. sla_miss_callback only fires after the task finishes. That means if the task is never finished in the first place due to it being blocked, or is still running, sla_miss_callback is not fired.
2. SLA is defined as a timedelta relative to the execution_date. But we may have deadlines specified in various timezone that is difficult to define as a simple timedelta relative to the execution_date. Some DAGs are triggered externally, meaning they don't have a fixed schedule or fixed start time, making dag.following_schedule() and thus SLA not to work.
3. sla_miss_callback is an attribute of the DAG. But users often need task level notification if deadlines are missed.

Given all these shortcomings of SLA, I'm proposing to create a new task level concept called deadline and its corresponding deadline_miss_callback, where deadline is defined as a jinja-template str that can be converted to a pendulum.DateTime object, and deadline_miss_callback is a callable to be called if the task is not finished by the given deadline.

The alternative is to revamp SLA to become a timezone-aware datetime object rather than a timedelta, and making sure sla_miss_callback is called at the deadline rather than after the task finishes. These two changes may make the SLA concept very different from what it currently is.

Use case/motivation

For example, given the simple DAG shown below, we need to know at 20210904 05:00 America/New_York if the generate_model task of the 20210903 DagRun is not yet finished. So if either download_file or generate_model takes too long, causing generate_model not to be finished by this deadline, the users should be notified.

download_file >> generate_model

Related issues

I see a few attempts to revamp/improve SLA. There's some overlap, but none of them does exact what's needed here.

#12008
#16389
#8545

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[eladkal]

I find no value with the current SLA feature. I don't know many people who are using it due to it's limitations.
I would say that a better approch is to deprecate the current SLA feature and replace it with a new one. I'm willing to pick up some tasks on that project if we decide to move forawrd with it.

[potiuk]

Yeah. I think we should have a more comprehensive/better/complete/usable SLA implementation replacing the current one.

[malthe]

Seems like this could use the new triggerer service. Essentially, it is like branching out and having a suspended task with a trigger that activates at the deadline.

[yuqian90]

Seems like this could use the new triggerer service. Essentially, it is like branching out and having a suspended task with a trigger that activates at the deadline.

Hi @malthe, could you elaborate a bit more? What's "triggerer service" and how can it be used to warn about the deadline?

[malthe]

@yuqian90 what I'm referring to are the new deferrable operators.

There's a framework in there which allows us to set up future actions such as reacting to a "missed deadline". It might need a little reworking in order to implement SLAs but I think it's pretty close since you could also just branch out and use the new DateTimeSensorAsync.

Note that this framework is available only from Airflow 2.2 onwards.

[uranusjr]

Yeah, this feels like a good use for async sensors indeed. I think it's even possible to do it with the old sensors (with obvious resource issues, but possible if you have practically unlimited worker resource).

[alexbegg]

Maybe I am not understanding the ticket, but I don't find this to be true:

1. sla_miss_callback only fires after the task finishes. That means if the task is never finished in the first place due to it being blocked, or is still running, sla_miss_callback is not fired.

if I have a DAG that should be hourly and for example I set a 1-hour SLA, I will get an SLA miss email if a single DAG run is still running past 1 hour. If a DAG finished and it failed it will use on_failure_callback, and if a DAG finished and it succeeded it will use on_success_callback. So that covers both cases of a DAG being complete (which is only possible if no tasks are running or blocked) so obviously the sla_miss_callback is if the DAG is missing its SLA, even if it is running.

I do agree that there should be better control for task-specific deadlines, but also this can be accomplished partially today by putting the part of the DAG that needs a deadline should be by itself in a separate DAG with an SLA in place, and then the remainder of a DAG will be in a 2nd task and use ExternalTaskSensor.

[tanvn]

I tried to use ExternalTaskSensor to check other tasks deadline
but because the try_number is not increased when the sensor mode is reschedule
so it will run forever until the conditions are met and marked as SUCCESS (never be marked as FAILED and call the on_failure_callback function)
The related issue is here: #18080

[alexbegg]

I tried to use ExternalTaskSensor to check other tasks deadline

but because the try_number is not increased when the sensor mode is reschedule

so it will run forever until the conditions are met and marked as SUCCESS (never be marked as FAILED and call the on_failure_callback function)

The related issue is here: #18080

I believe the intention of using ExternalTaskSensor with a deadline planned is to set a timeout and have no retries so if it doesn't complete by the timeout it fails, not just run forever.

[alexbegg]

Also as of Airflow 2.2.0 I think sensors will no longer retry if the sensor times out: #12058

[tanvn]

I believe the intention of using ExternalTaskSensor with a deadline planned is to set a timeout and have no retries so if it doesn't complete by the timeout it fails, not just run forever.

Thank you for the information.
(I am quite new to Airflow so please excuse my lack of knowledge 🙇 )
Actually I tried adding a timeout to my ExternalTaskSensor before but its behavior seemed not change. Maybe I should retry with a shorter one.
And by the way, with soft_fail=True, my on_failure_callback will not be called, I believe.
(on_retry_callback won't be called either with mode : "reschedule".)
Therefore, I have to create a new sensor based on the ExternalTaskSensor (the logic is quite the same but there is no inheritance relationship between them) to check deadlines of other tasks in the same DAG, to send a notification when the deadline missed while make it possible to let Airflow mark the DAG as SUCCESS when it tasks finished after deadline.

[potiuk]

I think this is a wider effort - improving SLAs. We know current SLAs are not really useful and there should be an effort to redefine SLA behaviour basically from scratch.

However it requires devlist discussion and AIP added here as this is important, big fetature to add to Airflow. https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals

Converting it to discussion, but would be great if somoene starts an effort around creating "New SLA AIP".