V8 CI is not working

[MylesBorins]

Looks like an error with glient fetching v8

Fetching V8 from chromium.googlesource.com
Error: Command 'download_from_google_storage --no_resume --platform=linux* --no_auth --bucket chromium-clang-format -s v8/buildtools/linux64/clang-format.sha1' returned non-zero exit status 1 in /home/iojs/build/workspace/node-test-commit-v8-linux/nodes/ppcle-ubuntu1404/v8test/v8test/deps
Running: gclient root
Running: gclient config --spec 'solutions = [
  {
    "url": "https://chromium.googlesource.com/v8/v8.git",
    "managed": False,
    "name": "v8",
    "deps_file": "DEPS",
    "custom_deps": {},
  },
]
'
Running: gclient sync --with_branch_heads
Traceback (most recent call last):
  File "/home/iojs/build/workspace/node-test-commit-v8-linux/nodes/ppcle-ubuntu1404/v8test/v8test/depot_tools/fetch.py", line 299, in <module>
    sys.exit(main())
  File "/home/iojs/build/workspace/node-test-commit-v8-linux/nodes/ppcle-ubuntu1404/v8test/v8test/depot_tools/fetch.py", line 294, in main
    return run(options, spec, root)
  File "/home/iojs/build/workspace/node-test-commit-v8-linux/nodes/ppcle-ubuntu1404/v8test/v8test/depot_tools/fetch.py", line 288, in run
    return checkout.init()
  File "/home/iojs/build/workspace/node-test-commit-v8-linux/nodes/ppcle-ubuntu1404/v8test/v8test/depot_tools/fetch.py", line 131, in init
    self.run_gclient(*sync_cmd)
  File "/home/iojs/build/workspace/node-test-commit-v8-linux/nodes/ppcle-ubuntu1404/v8test/v8test/depot_tools/fetch.py", line 78, in run_gclient
    return self.run(cmd_prefix + cmd, **kwargs)
  File "/home/iojs/build/workspace/node-test-commit-v8-linux/nodes/ppcle-ubuntu1404/v8test/v8test/depot_tools/fetch.py", line 68, in run
    return subprocess.check_output(cmd, **kwargs)
  File "/usr/lib/python2.7/subprocess.py", line 573, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '('gclient', 'sync', '--with_branch_heads')' returned non-zero exit status 2
V8 fetch failed

/cc @mhdawson @nodejs/v8

[MylesBorins]

Worth mentioning this is happening for any + all attempts to run CI right now

[gibfahn]

To be clear, this is happening for all runs of node-test-commit-v8-linux, not node-test-commit right?

@joransiu @jBarz do I recall you guys talking about something like this on Slack?

[jbajwa]

This started failing after https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/656565.
@john-yan is working with chromium's infra team on this https://bugs.chromium.org/p/chromium/issues/detail?id=764087 . In the meantime could add the following to the jenkins job

cd <path>/depot_tools
git revert -n 509776e
cd -
export DEPOT_TOOLS_UPDATE=0

[mhdawson]

Have put the work around into the job and kicked off a run to see if it resolves the issue.

[gibfahn]

Assuming it's https://ci.nodejs.org/job/node-test-commit-v8-linux/912/ then it looks like it worked.

We should leave this open to track the proper fix though, sounds like John is making progress.

[mhdawson]

Looks like work around did the trick, lets leave this open until the proper fix is in place.

[mhdawson]

@jBarz @joransiu this looks different than this issue https://bugs.chromium.org/p/chromium/issues/detail?id=764087&can=2&start=0&num=100&q=&colspec=ID%20Pri%20M%20Stars%20ReleaseBlock%20Component%20Status%20Owner%20Summary%20OS%20Modified&groupby=&sort=

But it did remind me to ask where we are on this one. At some point I should remove the work around.

[john-yan]

It should be fixed now. All we need to do is adding an extra env variable.
export VPYTHON_BYPASS="manually managed python not supported by chrome operations"

[MylesBorins]

Looks like the temporary patch is now failing 😭

I'm hacking on the CI job right now to make a permanent fix

[MylesBorins]

Was not able to fix it. I attempted to revert all of the hacks we had currently put in and added export VPYTHON_BYPASS="manually managed python not supported by chrome operations"

Same error reported in OP

[refack]

The error I'm seeing is: env issue

Using https://ci.nodejs.org/job/node-test-commit-v8-linux-fedora24/nodes=fedora24,v8test=v8test/398/console as reference.
Manual on test-rackspace-fedora24-x64-1

[refack]

Ok found some stuff:

1. The fedora machines (https://ci.nodejs.org/job/node-test-commit-v8-linux-fedora24/) are failing because they don't have bzip2 (and their yum repos have been archived to http://archives.fedoraproject.org/pub/archive/fedora/linux), but I see this is only run by timer...
2. ppc getting:

   iojs@test-osuosl-ubuntu14-ppc64-be-3:~/build/workspace/node-test-commit-v8-linux/nodes/ppcbe-
   ubuntu1404/v8test/v8test/depot_tools$ ./update_depot_tools
   UNKNOWN Machine architecture: ppc64
   

   which lead me to https://bugs.chromium.org/p/chromium/issues/detail?id=764087.
   I think it's a depot_tools regression of that issue.

---

The best I got is to revive the test on the fedora24 - https://ci.nodejs.org/job/node-test-commit-v8-linux-fedora24/399/nodes=fedora24,v8test=v8test/

[MylesBorins]

@refack were you testing with VPYTHON_BYPASS="manually managed python not supported by chrome operations"

[refack]

@refack were you testing with VPYTHON_BYPASS="manually managed python not supported by chrome operations"

In multiple variations with no success (with " without, in the same line, with export, before make after make, with just calling gclient sync --with_branch_heads).
Also it wasn't there yesterday when the last green job run https://ci.nodejs.org/view/All/job/node-test-commit-v8-linux/953/nodes=ppcbe-ubuntu1404,v8test=v8test/consoleFull

But now I see that we used to roll back depot_tools, and it broke sometime in the last 24h:

+ git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
Cloning into 'depot_tools'...
+ cd depot_tools
+ git revert -n 509776e
error: could not revert 509776e... [gsutil] run through "vpython" (2)
hint: after resolving the conflicts, mark the corrected paths