roachtest: transfer-leases/drain-other-node failed

[cockroach-teamcity]

roachtest.transfer-leases/drain-other-node failed with artifacts on release-22.2 @ cfc200070bbf9fc4b6bc6c6556b5fd56f48db381:

test artifacts and logs in: /artifacts/transfer-leases/drain-other-node/run_1
(cluster.go:1946).Stop: cluster.StopE: parallel execution failure: context deadline exceeded

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-28449

Epic CRDB-27234

[irfansharif]

cockroach/pkg/cmd/roachtest/tests/quit.go

Line 94 in c4d85c9

q.runWithTimeout(ctx, func(ctx context.Context) { method(ctx, q.t, q.c, nodeID) })

Drain is given 1m to finish, but it takes just a bit longer.

node is draining... remaining: 449
I230604 07:34:07.378499 1 cli/rpc_node_shutdown.go:186  [-] 2  drain details: liveness record: 1, range lease iterations: 435, descriptor leases: 13
node is draining... remaining: 3
I230604 07:34:07.601703 1 cli/rpc_node_shutdown.go:186  [-] 3  drain details: range lease iterations: 3
node is draining... remaining: 2
I230604 07:34:07.822336 1 cli/rpc_node_shutdown.go:186  [-] 4  drain details: range lease iterations: 2
node is draining... remaining: 2
...

node is draining... remaining: 2
I230604 07:34:16.229077 1 cli/rpc_node_shutdown.go:186  [-] 42  drain details: range lease iterations: 2
node is draining... remaining: 2
I230604 07:34:16.450075 1 cli/rpc_node_shutdown.go:186  [-] 43  drain details: range lease iterations: 2
node is draining... remaining: 555
I230604 07:34:16.676265 1 cli/rpc_node_shutdown.go:186  [-] 44  drain details: range lease iterations: 555
node is draining... remaining: 2
I230604 07:34:16.896783 1 cli/rpc_node_shutdown.go:186  [-] 45  drain details: range lease iterations: 2
node is draining... remaining: 2
I230604 07:34:17.118054 1 cli/rpc_node_shutdown.go:186  [-] 46  drain details: range lease iterations: 2

...

I230604 07:35:01.597720 1 cli/rpc_node_shutdown.go:186  [-] 247  drain details: range lease iterations: 2
node is draining... remaining: 2
I230604 07:35:01.819723 1 cli/rpc_node_shutdown.go:186  [-] 248  drain details: range lease iterations: 2
node is draining... remaining: 2
I230604 07:35:02.041823 1 cli/rpc_node_shutdown.go:186  [-] 249  drain details: range lease iterations: 2
node is draining... remaining: 555
I230604 07:35:02.274241 1 cli/rpc_node_shutdown.go:186  [-] 250  drain details: range lease iterations: 555
node is draining... remaining: 2
I230604 07:35:02.495407 1 cli/rpc_node_shutdown.go:186  [-] 251  drain details: range lease iterations: 2
node is draining... remaining: 2
I230604 07:35:02.715840 1 cli/rpc_node_shutdown.go:186  [-] 252  drain details: range lease iterations: 2
node is draining... remaining: 2
I230604 07:35:02.938259 1 cli/rpc_node_shutdown.go:186  [-] 253  drain details: range lease iterations: 2
node is draining... remaining: 2
I230604 07:35:03.159378 1 cli/rpc_node_shutdown.go:186  [-] 254  drain details: range lease iterations: 2
node is draining... remaining: 2
I230604 07:35:03.380404 1 cli/rpc_node_shutdown.go:186  [-] 255  drain details: range lease iterations: 2
node is draining... remaining: 2
I230604 07:35:03.601990 1 cli/rpc_node_shutdown.go:186  [-] 256  drain details: range lease iterations: 2
node is draining... remaining: 1
I230604 07:35:03.815661 1 cli/rpc_node_shutdown.go:186  [-] 257  drain details: range lease iterations: 1
node is draining... remaining: 0 (complete)
I230604 07:35:04.025723 68 2@rpc/context.go:1393  [-] 259  closing [{n0: localhost:26257 (system)}]
I230604 07:35:04.025699 16 2@rpc/context.go:1393  [-] 258  closing [{n0: localhost:26257 (system)}]

BTW, what's with the occasional jumps to "node is draining... remaining: 555"?

[irfansharif]

In #104185 too we saw drain get stuck-ish. Wonder if it's related.

[kvoli]

Looks like it got stuck on a single range for a while. Like @irfansharif mentioned it just missed the 1 minute timeout. Successfully draining (without shutdown). The SIGHUP also succeeds but looks like the context is cancelled just after the SIGKILL.

07:34:05 quit.go:93: stopping node 1
...
I230604 07:34:07.590445 63966 13@kv/kvserver/allocator/allocatorimpl/allocator.go:1736 ⋮ [drain,n1,s1,r27/1:‹/Table/2{6-7}›] 6429  no lease transfer target found for r27
... # 511 more attempts
I230604 07:35:03.810899 442218 13@kv/kvserver/replicate_queue.go:1885 ⋮ [drain,n1,s1,r27/1:‹/Table/2{6-7}›] 6941  transferring lease to s3
...
I230604 07:35:04.025188 443390 1@server/drain.go:285 ⋮ [n1] 6809  drain remaining: 0
I230604 07:35:04.025223 443390 1@server/drain.go:167 ⋮ [n1] 6810  drain request completed without server shutdown
...
07:35:04 cluster.go:677: test status: stopping nodes :1
teamcity-10381227-1685856001-25-n3cpu4: stopping
07:35:04 cluster.go:677: test status: stopping nodes :1
teamcity-10381227-1685856001-25-n3cpu4: stopping and waiting
0: context deadline exceeded: 

These log lines are odd. They are probably due the context being called with Done:

cockroach/pkg/kv/kvserver/store.go

Line 1533 in 7754c4b

case <-transferCtx.Done():

I230604 07:35:02.264744 432231 kv/kvserver/store.go:1542 ⋮ [drain,n1,s1,r533/1:‹/Table/104/1/2{0-1}›] 6166  lease transfer aborted due to exceeded timeout
... # 550 more of these
I230604 07:35:02.267873 432429 kv/kvserver/store.go:1542 ⋮ [drain,n1,s1,r3/1:‹/System/{NodeLive…-tsd}›] 6296  lease transfer aborted due to exceeded timeout
...
I230604 07:35:02.273560 432718 kv/kvserver/store.go:1542 ⋮ [drain,n1,s1,r32/1:‹/{NamespaceTab…-Table/32}›] 6718  lease transfer aborted due to exceeded timeout

So the test isn't that far off passing (less than a second), but the stuck lease for r27.

The lease isn't being transferred because the allocator doesn't find a valid lease target. A lease transfer candidate for the prior and following range are found during the first pass:

I230604 07:34:07.246962 61514 13@kv/kvserver/replicate_queue.go:1885 ⋮ [drain,n1,s1,r408/1:‹/Table/104/1/14{5-6}›] 6242  transferring lease to s3
I230604 07:34:07.247129 61541 13@kv/kvserver/allocator/allocatorimpl/allocator.go:1736 ⋮ [drain,n1,s1,r27/1:‹/Table/2{6-7}›] 6243  no lease transfer target found for r27
I230604 07:34:07.247215 61542 13@kv/kvserver/replicate_queue.go:1885 ⋮ [drain,n1,s1,r192/1:‹/Table/104/1/36{1-2}›] 6244  transferring lease to s3

This indicates that it isn't a store/node level condition we are filtering on, but range level. This also occurs for r44 briefly but then succeeds:

I230604 07:34:07.368285 62607 13@kv/kvserver/allocator/allocatorimpl/allocator.go:1736 ⋮ [drain,n1,s1,r44/1:‹/Table/4{3-4}›] 6426  no lease transfer target found for r44
I230604 07:34:07.589451 63726 13@kv/kvserver/replicate_queue.go:1885 ⋮ [drain,n1,s1,r44/1:‹/Table/4{3-4}›] 6428  transferring lease to s2

What range level state would prevent a replica being a valid target? We know it is failing here:

cockroach/pkg/kv/kvserver/allocator/allocatorimpl/allocator.go

Lines 1735 to 1738 in 7754c4b

	if len(existing) == 0 || (len(existing) == 1 && existing[0].StoreID == leaseRepl.StoreID()) {
	log.KvDistribution.VEventf(ctx, 2, "no lease transfer target found for r%d", leaseRepl.GetRangeID())
	return roachpb.ReplicaDescriptor{}
	}

If we exclude any node/store related checks, the following could invalidate a candidate:

1. Range descriptor doesn't contain any voters¹ (excl current LH).

cockroach/pkg/kv/kvserver/allocator/allocatorimpl/allocator.go

Line 1549 in 7754c4b

if err := roachpb.CheckCanReceiveLease(existing[i], replDescs, false /* wasLastLeaseholder */); err != nil {

2. Range descriptor only contains voters (excl current LH) which are in need of a snapshot ².

cockroach/pkg/kv/kvserver/allocator/allocatorimpl/allocator.go

Lines 1605 to 1606 in 7754c4b

	candidates = append(validSnapshotCandidates, excludeReplicasInNeedOfSnapshots(
	ctx, status, leaseRepl.GetFirstIndex(), candidates)...)

The range log shows the last activity for r27 was upreplicating, making n3 a voter from a learner about 80s before the failing drain run.

Range log entry

      "event": {
        "timestamp": "2023-06-04T07:32:33.422399Z",
        "range_id": 27,
        "store_id": 1,
        "event_type": 1,
        "info": {
          "UpdatedDesc": {
            "range_id": 27,
            "start_key": "og==",
            "end_key": "ow==",
            "internal_replicas": [
              {
                "node_id": 1,
                "store_id": 1,
                "replica_id": 1,
                "type": 0
              },
              {
                "node_id": 2,
                "store_id": 2,
                "replica_id": 2,
                "type": 0
              },
              {
                "node_id": 3,
                "store_id": 3,
                "replica_id": 3,
                "type": 0
              }
            ],
            "next_replica_id": 4,
            "generation": 4,
            "sticky_bit": {}
          },
          "AddReplica": {
            "node_id": 3,
            "store_id": 3,
            "replica_id": 3,
            "type": 0
          },
          "Reason": "range under-replicated",
          "Details": "{\"Target\":\"s3, diversity:0.25, converges:0, balance:1, rangeCount:40\"}"
        }
      },
      "pretty_info": {
        "updated_desc": "r27:/Table/2{6-7} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]",
        "added_replica": "(n3,s3):3",
        "reason": "range under-replicated",
        "details": "{\"Target\":\"s3, diversity:0.25, converges:0, balance:1, rangeCount:40\"}"
      }

So I somewhat doubt there aren't any voters, it even shows them in the descriptor being logged on failure throughout. It seems plausible that perhaps we are hitting the 2nd condition where we incorrectly think both of the other replicas may need a snapshot.

I filed #105066 to give an answer. I can't figure out how we ended up in this weird state, nor how it seemingly resolves itself after. The range status from both n1 (to-be drained) and n2 is available about 10s before the drain begins but doesn't show any issues:

"problems": {
  "unavailable": false,
  "leaderNotLeaseHolder": false,
  "noRaftLeader": false,
  "underreplicated": false,
  "overreplicated": false,
  "noLease": false,
  "quiescentEqualsTicking": false,
  "raftLogTooLarge": false,
  "circuitBreakerError": false,
  "pausedFollowers": false
}

I'm going to leave this test open but remove the release blocker. The behavior is unexpected but it does eventually succeed. With added obs into why there were no valid candidates, we could dig deeper - I'm suspecting a bug and not just a flake.

Footnotes

1. Relevant code. The replica needs to either be an (incoming) voter or outgoing voter and previous leaseholder. ↩

2. Relevant code. ↩

[cockroach-teamcity]

roachtest.transfer-leases/drain-other-node failed with artifacts on release-22.2 @ 2124c6b9adc61030bedc4a6a67c025566f989201:

test artifacts and logs in: /artifacts/transfer-leases/drain-other-node/run_1
(quit.go:429).func4: context deadline exceeded

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!