Airflow 3.1.3 - DagProccessor CPU throttling

[IIPOCTAK]

Apache Airflow version

3.1.3

If "Other Airflow 2/3 version" selected, which one?

No response

What happened?

We are currently using Airflow 3.1.0. After upgrading to the latest versions:

• 3.1.0 -> 3.1.1
• 3.1.0 -> 3.1.2
• 3.1.0 -> 3.1.3

We observed significant CPU throttling in the DagProcessor.

As shown in the graphs below, CPU throttling has increased dramatically:

CPU usage has also increased, but not nearly as much as throttling:

These spikes on the graph correspond to the activity of the DagFileProcessorManager

Our Airflow resources run in a Kubernetes cluster. We did not change anything except the package versions.
requrements.txt:

-apache-airflow[fab,ldap]==3.1.0
-apache-airflow-providers-fab==2.4.4
+apache-airflow[fab,ldap]==3.1.3
+apache-airflow-providers-fab==3.0.1

values.yaml:

-airflowVersion: 3.1.0
+airflowVersion: 3.1.3

-defaultAirflowTag: 3.1.0-python3.10
+defaultAirflowTag: 3.1.3-python3.10

I were trying to change these parameters after upgrade:

AIRFLOW__DAG_PROCESSOR__FILE_PARSING_SORT_MODE: "random_seeded_by_host"
AIRFLOW__DAG_PROCESSOR__PARSING_PROCESSES: "1"
AIRFLOW__DAG_PROCESSOR__REFRESH_INTERVAL: "300"

And increase CPU limits/requests (x2). Nothing helped.

What you think should happen instead?

No response

How to reproduce

Upgrade airflow version, for example, from 3.1.0 -> 3.1.3 version and check CPU usage and throttling.

Operating System

Debian GNU/Linux 12 (bookworm)

Versions of Apache Airflow Providers

# pip3 list | grep providers

apache-airflow-providers-amazon           9.12.0
apache-airflow-providers-celery           3.12.3
apache-airflow-providers-cncf-kubernetes  10.8.1
apache-airflow-providers-common-compat    1.7.4
apache-airflow-providers-common-io        1.6.3
apache-airflow-providers-common-messaging 2.0.0
apache-airflow-providers-common-sql       1.28.1
apache-airflow-providers-docker           4.4.3
apache-airflow-providers-elasticsearch    6.3.3
apache-airflow-providers-fab              2.4.4
apache-airflow-providers-ftp              3.13.2
apache-airflow-providers-git              0.0.8
apache-airflow-providers-google           18.0.0
apache-airflow-providers-grpc             3.8.2
apache-airflow-providers-hashicorp        4.3.2
apache-airflow-providers-http             5.3.4
apache-airflow-providers-microsoft-azure  12.7.1
apache-airflow-providers-mysql            6.3.4
apache-airflow-providers-odbc             4.10.2
apache-airflow-providers-openlineage      2.7.1
apache-airflow-providers-postgres         6.3.0
apache-airflow-providers-redis            4.3.1
apache-airflow-providers-sendgrid         4.1.3
apache-airflow-providers-sftp             5.4.0
apache-airflow-providers-slack            9.3.0
apache-airflow-providers-smtp             2.2.1
apache-airflow-providers-snowflake        6.5.4
apache-airflow-providers-ssh              4.1.4
apache-airflow-providers-standard         1.8.0

Deployment

Official Apache Airflow Helm Chart

Deployment details

helm

Anything else?

Constantly after upgrade

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[potiuk]

Interesting discussion. But i am not sure what "problem" you are desribing. You describe change in behaviour, for sure, but not necesarily problem,

Yes, if you limit CPU for DagProcess and it will want to use more of it, it will get throttled. But maybe simply it means that DagFileprocessor is just more efficient in parallelims in 3.1.3 and simply your throtling does not let you utilise it's new capabilities now - and for example be more efficient in parsing things in parallel.

There are various configuration option you can set to control parallelism (see configuration of dag Processor) and if your goal is to limit how many CPUS it uses, maybe you should use those options, I don't see how just "presence" of throttling without explaining your reasoning of how you deliberately configured it for throttling, what your decionse were when you tested were - so this post does not classify as "problem" (without having an actual problem statement) - rather than merely an "observation".

If you would like to discuss your assumptions, calculations and results of tests showing that this throtting is something that does not come from simply the fact that you set specific throttling levels - feel free to discuss it further here, but you need to specify what your problem statement is other than ("things changed") - because this is the only reasonable thing that you can see from your graphs.

Yes. They changed .. . So ?

[nassereddinebelghith]

Dears , I am upgrading airflow from 2.9.3 to 3.1.3. In my previous version I was using keycloak and the authentication and authorization were working fine. I used a custom security class in webserver_config.py. As I read in the upgrading manual that airflow is using Fab so I changed the import for the FabAurhManager and the OAUTH and I kept the same approach based on role mapping retrieved from keycloak. After changing parameters like core auth manager to fab , proxy fix to true and even cookie parameters. I am facing a CSRF session token is mismatching. I am really blocked on this upgrade and I am wondering which version of configuration you are using. By the way if anyone can help please I am running airflow on k8s. I tried a lot of parameters without any success the UI is not working but when I curl the api v2 monitor health I got a healthy status of scheduler , dag processor...etc the connection to the db is healthy and I did a airflow db migrate already.