Need a way to verify code changes of non-DAG files

[alexbegg]

For anyone that uses a "git sync" method for deploying their DAGs:

Without having to go into your git sync container's logs, it looks like currently the only way to confirm the new DAG changes are deployed is to look for the changes in the "Code" page of your DAG (or if there is changes that would cause a grid change you can verify the change on that page too).

However, what if you are importing a function from a separate file within the same DAG folder and you made changes to that function in that other python file? Currently there is no way to verify those changes are deployed without either simply waiting a few minutes and hoping it has synced or checking the git sync container's logs. Neither are a user-friendly way to verify the code change got deployed.

I am thinking we need a way in Airflow UI to view the code of non-DAG files. There is obviously complexity to this ask so I'll try and break it out like so:

Let's say a DAG uses code from different methods of importing modules for use in DAGs:

• A "custom_package" module into your PYTHONPATH like so: PYTHONPATH=/home/arch/projects/custom_package.
• A "my_company" folder inside of your "dags" folder.
• A "third_party_package" package installed via pip install.

(following the methods explained in "Modules Management" documentation page: https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/modules_management.html)

and you imported them like so:

import custom_package
import third_party_package
from my_company.common_package.common_module import SomeClass
from my_company.common_package.subpackage.subpackaged_util_module import AnotherClass

Maybe the "Code" page can be expanded to let you view the code of these modules. I am thinking some possible ways:

Option 1: List of imported modules to switch to on the "Code" page:

Show a list of files based on what was imported in the DAG, so you can click on the name and view the code of that file. For the above examples it will be these files to view:

1. This DAG
2. custom_package
3. third_party_package
4. common_module (my_company.common_package.common_module)
5. subpackaged_util_module (my_company.common_package.subpackage.subpackaged_util_module)

Also, if it is code from an installed and imported Python package:

• If it is an Airflow Provider, we already have a "Providers" page to verify those version changes.
• If it is not an Airflow Provider, then maybe we can provide a way to view the versions of the packages used by that DAG. Possibly an equivalent of pip freeze but filtered to just the modules imported in the top level of that DAG file.

Option 2: Links to the modules in the code view:

It can also be that the module names in the import lines of the code view be turned into hyperlinks which will take you to viewing the code of that module.

Option 3: Symbols panel like in GitHub:

It would be great if there was a way to implement something like GitHub's symbols panel when viewing a file in GitHub. You click on a symbol in the code view and then it brings up a list of the definitions and you click on the definition and it takes you to that file's code view. This might be a bit tricky unless someone built a software that does this and we can utilize that.

I am starting this discussion to see the thoughts of others on this.