clearml-agent 2.x ignoring CLI args

[OliverColeman]

Tried upgrading to version 2.0.3 and found that command line arguments do not seem to be passed when tasks have been launched from a CLI script. Downgraded to the latest 1.x version and it all works perfectly again.

The args:

The script:

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Train a model, tracking results in ClearML.")
    parser.add_argument("config", type=argparse.FileType("r"), help="path to a configuration file, can be YAML or JSON")
    parser.add_argument("--overrides", nargs="*", help='eg --overrides a.1=123 a.b="hello"')
    args = parser.parse_args()
    ...

Console output:

[info     ] Command: `src/train_clearml.py` module=__main__
usage: train_clearml.py [-h] [--overrides [OVERRIDES ...]] config
train_clearml.py: error: the following arguments are required: config
Process failed, exit code 2

[QazyBi]

This seems related to #244 and it's blocking my work

[H4dr1en]

I am also affected by this, last version working is 1.9.3

[jkhenning]

Hi @QazyBi, @H4dr1en, which SDK version are you using when running this?

[H4dr1en]

I used sdk version 2.0.2 and agent version 2.0.3 and I had this issue
Then I switch both agent to 1.9.3 and sdk to 1.18.0 and it worked

[jkhenning]

I assume using SDK version 1.18 and Agent 2.03 did not work? Also, can you perhaps share the task execution log?

[H4dr1en]

I didn't test with sdk 1.18 and agent 2.0.3 to be honest, I wasn't expecting them to be compatible, so I reverted to latest 1.* on both and it worked. I can try later this week

[jkhenning]

Please do - I suspect this might also be related to the SDK, and I want to make sure 🙂

[QazyBi]

I am using clearml SDK version 2.0.2 and clearml-agent version 2.0.3.

When I use pip as the package manager, my script receives the command-line arguments as expected. However, when I switch to using uv as the package manager, the command-line arguments are not passed to my script.

Here is a minimal example to reproduce the issue:

main.py:

import argparse
from clearml import Task


def main(config):
    print(f"Main function called with config: {config}")


if __name__ == "__main__":
    task = Task.init(project_name="test", task_name="test")
    parser = argparse.ArgumentParser(description="Train a model, tracking results in ClearML.")
    parser.add_argument("config", type=argparse.FileType("r"), help="path to a configuration file, can be YAML or JSON")
    args = parser.parse_args()
    main(args.config.read())

I run the script from the command line like this:
python main.py simple.yaml

On the ClearML dashboard, I clone and enqueue the task with two different configurations:

With pip as the package manager: The script receives the command-line argument, but fails due to the file not being present (which is the expected behavior in this stage). ClearML Log

With uv as the package manager: The script does not receive the command-line argument and fails. ClearML Log UV

If you need additional information please let me know

[H4dr1en]

@jkhenning I could not reproduce the error error because of #251

[QazyBi]

@jkhenning, are there any updates regarding this issue?

[ijmiller2]

Looks like I might be encountering this issue too. Trying to enqueue a job and changing epochs (which is specified via CLI) from 3 to 100, but the agent is only running the job for the original 3 epochs (via GCP autoscaler).

I am using the current version of the Web UI and docker mode. Agent version 2.0.4, SDK version 2.0.2.

[wildfluss]

looks like here https://github.com/clearml/clearml-agent/blob/master/clearml_agent/commands/worker.py#L3158 it ignores whatever args were passed altogether as in

Running task id [c6ad656b121543e2b33a2a229a164e9c]:
[.]$ /root/.clearml/venvs-builds/3.12_uv/bin/uv run --python /root/.clearml/venvs-builds/3.12/task_repository/pocs.git/.venv/bin/python python -u models/huggingface/my_model_huggingface/dummy_script.py
Summary - installed python packages:

of this dummy_script.py

#!/usr/bin/env python

import argparse


def train_huggingface_cli():
    parser = argparse.ArgumentParser(description="train huggingface model")
    parser.add_argument("--ads-jsonl", type=str, default="data/raw/ads.jsonl")
    parser.add_argument("--data-processed", type=str, default="data/processed")

    parser.add_argument("--do-train-test-split", type=bool, default=False)

    parser.add_argument("--dataset", type=str, default="")

    args = parser.parse_args()

    print(
        "args.dataset is", args.dataset
    )  # FIXME its empty , https://github.com/clearml/clearml/issues/1466


if __name__ == "__main__":
    train_huggingface_cli()

running it like this against clearml-agent on k8s using latest versions

clearml-task --project huggingface --name dummy_script --script ./models/huggingface/my_model_huggingface/dummy_script.py --branch main --queue default --args dataset=cfee2161443241408159e82d0b6288b0

whereas in UI for the PARENT task "the command line task" which has project "huggingface" (because clearml-task creates 2 tasks actually ) i see args like this

and then in the ANOTHER task "the git clone'ed task" which has project "pocs" in name of the repo (the actual task? which is created from within docker image which clearml-task schedules on the cluster) its like this

[lifefoundry-scott]

Found the same issue on my side. Below is a temporary workaround, but note you'll also need to patch your version of clearml-agent with the pull request attached to the issue mentioned above, which has been merged but not released.

### Temporary workaround to override session parameters due to clearml bug
if not Task.running_locally():
    with contextlib.suppress(BaseException):
        _session_task_id = os.environ["CLEARML_TASK_ID"]
        _session_configuration = Task.get_task(task_id=_session_task_id).get_parameters()
        # var1 = _session_configuration.get("Args/var1")
        # var2 = _session_configuration.get("Args/var2")

Y'all really need to work on the stability of your codebase. Every time I do any development I come across so many bugs that I end up spending more time trying to work around them than on my actual project. I don't have time to report every single bug and even if I did, it seems nothing gets done unless you fix it yourself and submit a PR. Docs are often outdated, API client commands don't work, plenty of SDK functions (e.g. PipelineController clone and create) don't work, agent execution is a buggy mess. I had to spend so much time raising a fuss that some internal bug is causing a memory leak in the metrics database and after over a month it has yet to be resolved, although I was told 2 weeks ago that devops is looking into it. Running ruff and mypy checks on this repo alone shows thousands of errors each. No testing, deprecated project configuration, etc. You can understand how all of this adds up to a very frustrating user experience. I could recommend taking inspiration from something like Pydantic Logfire, which shares many of the similar backend methodologies but just works perfectly out of the box every single time.

I hope this message makes its way to someone who can steer this product in a better direction and that this comes across as constructive feedback rather than being rude. I like the vision and when it works it is great to use, but sunk cost fallacy wears thin at some point.