ClearML k8s agent suddenly in crashloopbackoff #247
Description
Hi there!
We're running ClearML agents on k8s via the helm chart. Not too long ago they started crashlooping with the following exception:
File "/usr/local/lib/python3.6/dist-packages/clearml_agent/interface/base.py", line 12, in <module>
from clearml_agent.session import Session
File "/usr/local/lib/python3.6/dist-packages/clearml_agent/session.py", line 23, in <module>
from clearml_agent.helper.docker_args import DockerArgsSanitizer, sanitize_urls
File "/usr/local/lib/python3.6/dist-packages/clearml_agent/helper/docker_args.py", line 279, in <module>
class CustomTemplate(Template):
File "/usr/lib/python3.6/string.py", line 74, in init
cls.pattern = _re.compile(pattern, cls.flags | _re.VERBOSE)
File "/usr/lib/python3.6/re.py", line 233, in compile
return _compile(pattern, flags)
File "/usr/lib/python3.6/re.py", line 301, in _compile
p = sre_compile.compile(pattern, flags)
File "/usr/lib/python3.6/sre_compile.py", line 562, in compile
p = sre_parse.parse(p, flags)
File "/usr/lib/python3.6/sre_parse.py", line 855, in parse
p = _parse_sub(source, pattern, flags & SRE_FLAG_VERBOSE, 0)
File "/usr/lib/python3.6/sre_parse.py", line 416, in _parse_sub
not nested and not items))
File "/usr/lib/python3.6/sre_parse.py", line 765, in _parse
p = _parse_sub(source, state, sub_verbose, nested + 1)
File "/usr/lib/python3.6/sre_parse.py", line 416, in _parse_sub
not nested and not items))
File "/usr/lib/python3.6/sre_parse.py", line 765, in _parse
p = _parse_sub(source, state, sub_verbose, nested + 1)
File "/usr/lib/python3.6/sre_parse.py", line 416, in _parse_sub
not nested and not items))
File "/usr/lib/python3.6/sre_parse.py", line 734, in _parse
flags = _parse_flags(source, state, char)
File "/usr/lib/python3.6/sre_parse.py", line 803, in _parse_flags
raise source.error("bad inline flags: cannot turn on global flag", 1)
sre_constants.error: bad inline flags: cannot turn on global flag at position 92 (line 4, column 20)
We solved it by setting the CLEARML_AGENT_NO_UPDATE=true env var. But wanted to raise awareness to this issue that the last update probably broke something!
Cheers