Add the ability to backfill a DAG based on past Asset Events

[hnomichith]

Hello!

Airflow version

Airflow 3.1.4

Context

I'm currently working with DAG scheduling on Asset Events. I saw that if you create a DAG which triggers on an Asset Event, there's no usable way to backfill it based on past Asset Events. For example, if those Asset Events were created when the DAG was not created yet, or if the DAG was paused.

I saw two work-around, both with their caveats:

1. Backfill the upstream DAG to re-trigger the events. It's a single operation, however it implies re-running the upstream DAG when it may not be useful.
2. Manually re-create the asset events via "Assets" -> "select Asset" -> "Create Event" -> "Manual". However:
   • Here you create the events one by one, which is not scalable.
   • If you want to "re-publish" an Asset Event, you need to copy its extra info, then paste them in the new Asset Event.
   • If your Asset has several downstream DAGs, you'll run them all, while you may only want to backfill one.

I identified one more usable solution creating a script to automate the second solution. It makes it scalable, but still have the caveat of running all downstream DAGs.

Questions

I'm opening the discussion to ask some questions:

• Is it a very specific need, or is there some other people who share it?
• Am I mis-using Airflow in a way?

Potential solution

Instinctively, the solution I see to address my need is:

1. Create a POST /api/v2/dags/:dag_id/assets/queuedEvents endpoint
2. Modify GET /api/v2/assets/events to add a name filter (and everything that could be used to schedule a DAG based on an Asset)
3. In Airflow UI, from an Asset-scheduled DAG, when clicking on "Trigger" -> "Backfill", call the endpoint modified in step 2 to show the number of runs to be triggered ; and on validation call the endpoint modified in step 1.

What do you think?

[potiuk]

This seems a very specific need and I think you should just use a Python script - and use Python Client to trigger such assets in the way you want via the API.

The thing about backfills is that their whole "management" is based on data intervals - and it's easy to specify "between this and that interval" for example. I do not see an easy UX where you would like to "declaratively" define what assets to regenerate - it's way easier and way more flexible to write a simple Python script to generate the set of API calls that you want to "backfill".