roachtest: jepsen/multi-register/majority-ring-start-kill-2 failed

[cockroach-teamcity]

roachtest.jepsen/multi-register/majority-ring-start-kill-2 failed with artifacts on master @ c7a7e423506c88f655c08d6ba6acd6e907e1dae5:

test artifacts and logs in: /artifacts/jepsen/multi-register/majority-ring-start-kill-2/run_1
(jepsen.go:434).runJepsen: output in run_153616.408878639_n6_bash-e-c-cd-mntdata1: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.1.0-21cbebe-standalone.jar test \
    --tarball file://${PWD}/cockroach.tgz \
    --username ${USER} \
    --ssh-private-key ~/.ssh/id_rsa \
    --os ubuntu \
    --time-limit 300 \
    --concurrency 30 \
    --recovery-time 25 \
    --test-count 1 \
    -n 10.142.1.26 -n 10.142.1.39 -n 10.142.1.32 -n 10.142.1.35 -n 10.142.1.24 \
    --test multi-register --nemesis majority-ring --nemesis2 start-kill-2 > invoke.log 2>&1" returned: COMMAND_PROBLEM: exit status 1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-28622

[cockroach-teamcity]

roachtest.jepsen/multi-register/majority-ring-start-kill-2 failed with artifacts on master @ 6e38f676273d08b3a111213d6c9f7945629228d7:

test artifacts and logs in: /artifacts/jepsen/multi-register/majority-ring-start-kill-2/run_1
(jepsen.go:434).runJepsen: output in run_154817.792271259_n6_bash-e-c-cd-mntdata1: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.1.0-21cbebe-standalone.jar test \
    --tarball file://${PWD}/cockroach.tgz \
    --username ${USER} \
    --ssh-private-key ~/.ssh/id_rsa \
    --os ubuntu \
    --time-limit 300 \
    --concurrency 30 \
    --recovery-time 25 \
    --test-count 1 \
    -n 10.142.0.178 -n 10.142.0.175 -n 10.142.0.171 -n 10.142.0.173 -n 10.142.0.169 \
    --test multi-register --nemesis majority-ring --nemesis2 start-kill-2 > invoke.log 2>&1" returned: COMMAND_PROBLEM: exit status 1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.jepsen/multi-register/majority-ring-start-kill-2 failed with artifacts on master @ 7fd4c21157221eae9e7d5892d89d2b5a671aba3e:

(jepsen.go:434).runJepsen: output in run_142234.467898826_n6_bash-e-c-cd-mntdata1: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.1.0-21cbebe-standalone.jar test \
    --tarball file://${PWD}/cockroach.tgz \
    --username ${USER} \
    --ssh-private-key ~/.ssh/id_rsa \
    --os ubuntu \
    --time-limit 300 \
    --concurrency 30 \
    --recovery-time 25 \
    --test-count 1 \
    -n 10.142.1.27 -n 10.142.1.76 -n 10.142.1.23 -n 10.142.1.28 -n 10.142.1.75 \
    --test multi-register --nemesis majority-ring --nemesis2 start-kill-2 > invoke.log 2>&1" returned: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/jepsen/multi-register/majority-ring-start-kill-2/run_1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.jepsen/multi-register/majority-ring-start-kill-2 failed with artifacts on master @ 4a614f89cea81bf94674d6072c3bbf30502244d4:

(jepsen.go:434).runJepsen: output in run_150133.506380153_n6_bash-e-c-cd-mntdata1: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.1.0-21cbebe-standalone.jar test \
    --tarball file://${PWD}/cockroach.tgz \
    --username ${USER} \
    --ssh-private-key ~/.ssh/id_rsa \
    --os ubuntu \
    --time-limit 300 \
    --concurrency 30 \
    --recovery-time 25 \
    --test-count 1 \
    -n 10.142.0.76 -n 10.142.0.12 -n 10.142.0.30 -n 10.142.0.48 -n 10.142.0.80 \
    --test multi-register --nemesis majority-ring --nemesis2 start-kill-2 > invoke.log 2>&1" returned: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/jepsen/multi-register/majority-ring-start-kill-2/run_1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.jepsen/multi-register/majority-ring-start-kill-2 failed with artifacts on master @ 0207c613fa7c8f3ab66c4518ee1e52dabb863426:

(jepsen.go:434).runJepsen: output in run_151204.652222335_n6_bash-e-c-cd-mntdata1: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.1.0-21cbebe-standalone.jar test \
    --tarball file://${PWD}/cockroach.tgz \
    --username ${USER} \
    --ssh-private-key ~/.ssh/id_rsa \
    --os ubuntu \
    --time-limit 300 \
    --concurrency 30 \
    --recovery-time 25 \
    --test-count 1 \
    -n 10.142.0.151 -n 10.142.0.143 -n 10.142.0.113 -n 10.142.0.159 -n 10.142.0.241 \
    --test multi-register --nemesis majority-ring --nemesis2 start-kill-2 > invoke.log 2>&1" returned: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/jepsen/multi-register/majority-ring-start-kill-2/run_1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

[renatolabs]

Some of the artifacts for the earlier issues are already gone; however, there are some recent enough failures (starting from #104602 (comment) at the time of writing) where we still have artifacts.

In all cases, the reason for failure is the same: at some point in the test, Jepsen is no longer able to connect to one of the nodes:

WARN [2023-07-10 15:13:44,600] jepsen worker 15 - jepsen.cockroach.multiregister Error reopening [:cockroach "10.142.0.151"]
java.lang.RuntimeException: Connection to 10.142.0.151 timed out

The error above seems to be coming from:

       :open (fn open []
               (util/timeout max-timeout
                             (throw (RuntimeException.
                                      (str "Connection to " node " timed out")))
                             (util/retry 0.1
                               (let [spec (db-conn-spec node)
                                     conn (j/get-connection spec)
                                     spec' (j/add-connection spec conn)]
                                 (assert spec')
                                 spec'))))

https://github.com/cockroachdb/jepsen/blob/tc-nightly/cockroachdb/src/jepsen/cockroach/client.clj#L85-L94

We retry to connect every 100ms and eventually time-out after 30s.

Nothing jumps at me in the logs as to why this is happening; the way logs are stored in Jepsen tests is less than ideal as everything is in the same cockroach.stderr file, and the logs of multiple runs/restarts are all aggregated in that one file.

Could @cockroachdb/kv take a look at this and decipher what's going on, if possible?

[andrewbaptist]

I'll take more of a look at this tomorrow. The test is confusing because it is stopping and restarting nodes a lot during the run, and during some period, it fails to connect when it thinks it should be able to. The logs are really noisy with all the starts/stops/down nodes so it is hard to tell exactly what is happening.