Clearml agent suddenly stopped working

[Octoslav]

Description

All my agents suddenly stopped executing tasks, throwing the following error:

clearml_agent: ERROR: expected SCALAR, SEQUENCE-START, MAPPING-START, or ALIAS

Environment:

• WebApp: 2.1.0-664
• Server: 2.1.0-664
• API: 2.32
• ClearML Agent: 1.9.3
• Setup: Self-hosted server

Possible Cause

It appears that agents running in Docker mode install the latest version of the ClearML agent by default, ignoring the system-installed ClearML agent version (1.9.3). So it is seems like a bug of the v2.0.0 version

Temporary Workaround:

Run agents with the --force-current-version flag to enforce the system-installed agent version.

[BrunoCoric]

Same thing is happening to me, but I am using autoscaler, not sure how to pass --force-current-version.

[ainoam]

@BrunoCoric @Octoslav Can you please include the complete logs?

[BrunoCoric]

Currently first time I run an experiment with autoscaler, it runs correctly.
But if that experiment is not running anymore and autoscaler reuses the instance and the worker already running than it fails with this error:
clearml_agent: ERROR: expected SCALAR, SEQUENCE-START, MAPPING-START, or ALIAS

[BrunoCoric]

Here is the complete log.
Also before the update few days ago, everything work great.
One error that had also started is:
clearml_agent: ERROR: Could not install task requirements!
This was solved by having minimal libraries installed locally, but this was not a problem few days before.

Log - 0f058fdde78048059546850738393e6a.txt

[jkhenning]

@BrunoCoric is there any chance you can get more logs when running the agent with --debug?

[BrunoCoric]

How do I run it with debug if I am using GCP Autoscaler that I am not self-hosting?

[and-za]

I have a very similar exception, when the agent tries to install dependencies with pip, even though I am not sure if it is related to this bug report:

clearml_agent: ERROR: Could not install task requirements!
expected SCALAR, SEQUENCE-START, MAPPING-START, or ALIAS

I was able to drill it down to one particular import

from matplotlib import pyplot as plt

When I remove that import, it works. Bellow is full example task

# === ClearML imports
from clearml import Logger
from clearml import Task

from matplotlib import pyplot as plt # This import causes exception

task = Task.init(project_name="WP23 experiments", task_name="Test repo4")
task.execute_remotely(queue_name = 'default')
task.close()

Here is full stack trace from the agent with enabled debugging

INFO:clearml_agent.commands.worker:Found task requirements section, trying to install
INFO:clearml_agent.commands.worker:Traceback (most recent call last):
  File "/opt/conda/lib/python3.12/site-packages/clearml_agent/commands/worker.py", line 3836, in install_requirements_for_package_api
    package_api.load_requirements(cached_requirements)
  File "/opt/conda/lib/python3.12/site-packages/clearml_agent/helper/package/pip_api/venv.py", line 41, in load_requirements
    requirements["pip"] = self.requirements_manager.replace(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/clearml_agent/helper/package/requirements.py", line 693, in replace
    dump_yaml(sorted([str(p) for p in skipped_packages])),
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/clearml_agent/helper/base.py", line 326, in dump_yaml
    return dump_func(obj, **base_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/clearml_agent/_vendor/pyyaml/__init__.py", line 47, in dump
    yaml_instance.dump(data, stream)
  File "/opt/conda/lib/python3.12/site-packages/clearml_agent/_vendor/ruamel/yaml/main.py", line 593, in dump
    return self.dump_all([data], stream, transform=transform)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/clearml_agent/_vendor/ruamel/yaml/main.py", line 603, in dump_all
    self._context_manager.dump(data)
  File "/opt/conda/lib/python3.12/site-packages/clearml_agent/_vendor/ruamel/yaml/main.py", line 940, in dump
    self._yaml.representer.represent(data)
  File "/opt/conda/lib/python3.12/site-packages/clearml_agent/_vendor/ruamel/yaml/representer.py", line 82, in represent
    self.serializer.serialize(node)
  File "_ruamel_yaml.pyx", line 1351, in work._ruamel_yaml.CEmitter.serialize
ruamel.yaml.emitter.EmitterError: expected SCALAR, SEQUENCE-START, MAPPING-START, or ALIAS

Traceback (most recent call last):
  File "/opt/conda/lib/python3.12/site-packages/clearml_agent/commands/worker.py", line 3836, in install_requirements_for_package_api
    package_api.load_requirements(cached_requirements)
  File "/opt/conda/lib/python3.12/site-packages/clearml_agent/helper/package/pip_api/venv.py", line 41, in load_requirements
    requirements["pip"] = self.requirements_manager.replace(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/clearml_agent/helper/package/requirements.py", line 693, in replace
    dump_yaml(sorted([str(p) for p in skipped_packages])),
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/clearml_agent/helper/base.py", line 326, in dump_yaml
    return dump_func(obj, **base_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/clearml_agent/_vendor/pyyaml/__init__.py", line 47, in dump
    yaml_instance.dump(data, stream)
  File "/opt/conda/lib/python3.12/site-packages/clearml_agent/_vendor/ruamel/yaml/main.py", line 593, in dump
    return self.dump_all([data], stream, transform=transform)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/clearml_agent/_vendor/ruamel/yaml/main.py", line 603, in dump_all
    self._context_manager.dump(data)
  File "/opt/conda/lib/python3.12/site-packages/clearml_agent/_vendor/ruamel/yaml/main.py", line 940, in dump
    self._yaml.representer.represent(data)
  File "/opt/conda/lib/python3.12/site-packages/clearml_agent/_vendor/ruamel/yaml/representer.py", line 82, in represent
    self.serializer.serialize(node)
  File "_ruamel_yaml.pyx", line 1351, in work._ruamel_yaml.CEmitter.serialize
ruamel.yaml.emitter.EmitterError: expected SCALAR, SEQUENCE-START, MAPPING-START, or ALIAS

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/opt/conda/lib/python3.12/site-packages/clearml_agent/__main__.py", line 87, in <module>
    sys.exit(main())
             ^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/clearml_agent/__main__.py", line 83, in main
    return run_command(parser, args, command_name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/clearml_agent/__main__.py", line 46, in run_command
    return func(**args_dict)
           ^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/clearml_agent/commands/base.py", line 63, in newfunc
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/clearml_agent/commands/worker.py", line 3118, in execute
    self.install_requirements(
  File "/opt/conda/lib/python3.12/site-packages/clearml_agent/commands/worker.py", line 3745, in install_requirements
    return self.install_requirements_for_package_api(execution, repo_info, requirements_manager,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/clearml_agent/commands/worker.py", line 3840, in install_requirements_for_package_api
    raise ValueError("Could not install task requirements!\n{}".format(e))
ValueError: Could not install task requirements!
expected SCALAR, SEQUENCE-START, MAPPING-START, or ALIAS

Some environment informations:

Agent version: 2.0.1
Deployment: K8S with ClearML helm charts
Agent values.yaml:

agentk8sglue:
apiServerUrlReference: "api_url"
fileServerUrlReference: "files_url"
webServerUrlReference: "webserver_url"
defaultContainerImage: "quay.io/jupyter/tensorflow-notebook:cuda-tensorflow-2.19.0"
extraEnvs:
  # This affects only the "glue" agent, the worker agents will still use latest version
  - name: CLEARML_AGENT_UPDATE_VERSION
    value: "==1.9.2"

# Base pod template is used by the agent pod to spawn new worker pods when
# some task is enqueued to the queue.
basePodTemplate:
  env:
    - name: CLEARML_AGENT_SKIP_CONTAINER_APT
      value: 'true'

    - name: CLEARML_AGENT_GIT_USER
      valueFrom:
        secretKeyRef:
          name: clearml-git-creds
          key: git-user
    - name: CLEARML_AGENT_GIT_PASS
      valueFrom:
        secretKeyRef:
          name: clearml-git-creds
          key: git-pass
    - name: CLEARML_AGENT_GIT_HOST
      value: git_url
clearml:
  agentk8sglueKey: key
  agentk8sglueSecret: secret
  clearmlConfig: |-
    agent {
      package_manager {
        type: "pip"
      }
      debug: true
    }

Btw. a little bit off topic, but the debug configuration parameter is not documented. Is that intentional?

[jkhenning]

@and-za this has been resolved with ClearML Agent v2.0.2, please confirm the agent is now working as expected.

[and-za]

@jkhenning I confirm, that the issue I had is gone (fixed). However I am not sure, as I mentioned in my previous comment, if that was the same issue, as was originally reported by @Octoslav. So I guess @Octoslav should confirm it as well.