Task.init() creates new task when running remote

[svolsky]

Test code:

parser = argparse.ArgumentParser(description="Test task")
parser.add_argument('--days_count', type=int, default=365, required=False)
args = parser.parse_args()
task = Task.init(project_name='test-project', task_name=f'test task {datetime.datetime.now()}')

print(f'Days count: {args.days_count}')

Steps to reproduce:

Run the script locally with the argument --days_count 10.

Wait for the task to complete, then clone it and enqueue it.

Expected behavior:

The cloned task should run with the argument value 10.

Actual behavior:

The cloned task shows 10 as the parameter in the UI.

When executed, it creates another task — but this new one has no parameters set.

The printed value is 365 (the default).

This extra task runs with the default value 365 and does not spawn another task.

It seems that the presence of argparse in the script might be interfering with the expected behavior of parameter propagation.

begins with clearml-agent 2.0.1

[svolsky]

It seems that the Task.running_locally() method always returns True, regardless of where the task is actually running.

[and-za]

I have the same issue. It also happens with the task.execute_remotely(queue_name = 'default'). When I have my experiment defined in Jupyter notebook, then suddenly two tasks are created. One, as expected when the task is locally executed. Second is created by the agent which pulls it from the default queue, as a result of it executing Task.init().

In my case it stops there, because I am using Jupyter notebook and the converted python script is not cloned to the third task, so it fails. But I would assume, if it would not fail, this behavior would continue.

From what I see in documentation and what I have also seen in code, both Task.init() and task.execute_remotely() should not create new tasks if it is running in the ClearML agent.

Here is simple script, which can be used to reproduce this issue:

from clearml import Task, Logger

task = Task.init(project_name="Experiments", task_name="Test")
task.execute_remotely(queue_name = 'default')

task.get_logger().report_text('blablabla') # I guess not necessary, but let it do something

task.close()

Based on my little investigation it looks like the CLEARML_TASK_ID env variable, which is set here https://github.com/clearml/clearml-agent/blob/master/clearml_agent/commands/worker.py#L3230 is not correctly propagated to the created sub-process, which is (varies by configuration) on this line https://github.com/clearml/clearml-agent/blob/master/clearml_agent/commands/worker.py#L3230

To verify this I tried following task code (even though I am not sure about it):

from clearml import Task, Logger
import os

print(os.environ.get('CLEARML_TASK_ID'))
print(os.environ.get('TRAINS_TASK_ID'))
task = Task.init(project_name="Experiments", task_name="Test")
task.execute_remotely(queue_name = 'default')

task.get_logger().report_text('blablabla') # I guess not necessary, but let it do something

task.close()

which produced following output

None
None
ClearML Task: created new task id=some_id

My setup

• K8S deployment using ClearML helm charts version 7.14.5
• Agent deployed with following values.yaml

  agentk8sglue:
  apiServerUrlReference: "api_url"
  fileServerUrlReference: "files_url"
  webServerUrlReference: "webserver_url"
  defaultContainerImage: "quay.io/jupyter/tensorflow-notebook:cuda-tensorflow-2.19.0"
  extraEnvs:
    - name: CLEARML_AGENT_UPDATE_VERSION
      value: "==1.9.2"
  
  # Base pod template is used by the agent pod to spawn new worker pods when
  # some task is enqueued to the queue.
  basePodTemplate:
    env:
      - name: CLEARML_AGENT_SKIP_CONTAINER_APT
        value: 'true'
  
      - name: CLEARML_AGENT_GIT_USER
        valueFrom:
          secretKeyRef:
            name: clearml-git-creds
            key: git-user
      - name: CLEARML_AGENT_GIT_PASS
        valueFrom:
          secretKeyRef:
            name: clearml-git-creds
            key: git-pass
      - name: CLEARML_AGENT_GIT_HOST
        value: git_url
  clearml:
    agentk8sglueKey: key
    agentk8sglueSecret: secret
    clearmlConfig: |-
      agent {
        package_manager {
          type: "uv"
        }
      }

  Please note, that even though the agent version is fixed to 1.9.2, it is true only for the "glue agent". The spawned worker agent uses latest release version.

[and-za]

I noticed, that if I use pip as agent.package_manager.type, then it works fine. So I assume it has something to do with how the command for execution of the task with uv is constructed.

[KyrillosL]

Hi,
I have the same problem with the agent 2.0.1 and the client 2.0.2.
On my side, the problem occurs when these two conditions are met:

• Using Git for the repo
• Using UV as a package manager

Pip + Git works, and UV without Git works as well.