log worker identity

[gudata]

Description

we should know where the task is processed. log worker type and identity - ecs.docker and the id cluster/task id

Use case/motivation

when someone have a lot of instance ids or ecs services as workers it would want to know on which of them the task has been executed.

now this is unclear. are the logs coming from the scheler, worker ot from where

Real production could have different kind of workers.

Related issues

example: I am getting timeout on compiling the dags. where should I check for the error? the scheduler or the worker node?

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[jroachgolf84]

Couple of things here:

• When processing DAGs, logs will go to the Scheduler/DAG Process (if you're running a DAG Processor).
• Synchronous Tasks execute on Workers. These logs are output into the Airflow UI (and can also be persisted elsewhere).
• Asynchronous Tasks will typically execute on the Triggerer. This is where you'd find logs for a deferrable Operator/AssetWatcher. I believe these are also passed through to the Airflow UI, but I may be wrong about that.

Do these answers address your requirements/questions?

[potiuk]

Yep. Details of Airflow architecture are described in https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/overview.html -> there you can understand how Airflow works. You usually also see information about logs in your logging solution that gathers the information about the deployment you - have - either the deployment you manage (in which case you should make sure you add appropriate logging information you miss - following https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/advanced-logging-configuration.html or your managed Airflow solution should have (in which case you should raise an issue to whover manges your Airflow)

Airflow is deployment agnostic - it can be deployed in many ways https://airflow.apache.org/docs/apache-airflow/stable/installation/index.html -> so Airflow has no notion about ECS, Docker ID, Cluster - because it might be deployed without either of those.

It's the role of Deployment Manager (i.e. you if you self-manage airflow or whoever provides you managed airflow) to configure airflow in the way that your monitoring and management of running airflow is good in the context of your deployment. We clearly explain it in the installation pages I quoted above - where responsibility of Deployment Manager is clearly explained. Just follow it.