grpcutil: improve gRPC logging

[tbg]

Extracted from #99191, where I ran a bunch of experiments to make sure the logging around connection failures is clean and helpful.

I did manual testing on the changes in this PR. It's tricky to test vmodule-depth-thingies programmatically and I don't want to spend time building new infra here.

• cli: don't lower gRPC severity threshold
• grpcutil: adopt DepthLoggerV2
• sidetransport: suppress logging when peer down

Release note: None
Epic: CRDB-21710

[cockroach-teamcity]

This change is 

[knz]

Reviewed 4 of 4 files at r1, 1 of 1 files at r2, 2 of 2 files at r3, all commit messages.
Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @aliher1911 and @tbg)

---

pkg/util/grpcutil/grpc_log.go line 120 at r1 (raw file):

	// has a multi-line JSON object in it (but not in any structured way),
	// so there's really no way to work with it. Best we can do is throw
	// away the many newlines.

Q: why throw away the newlines? how do they hurt us?

---

pkg/util/grpcutil/grpc_log.go line 123 at r1 (raw file):

	sl := make([]interface{}, 0, len(args))
	for _, arg := range args {
		sl = append(sl, strings.Replace(fmt.Sprint(arg), "\n", "", -1))

nit: replace sequences of multiple newlines with a single space, to avoid two separate words from being concatenated together.

[tbg]

TFTR, appreciate it!

bors r=knz

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @aliher1911 and @knz)

---

pkg/util/grpcutil/grpc_log.go line 120 at r1 (raw file):

Previously, knz (Raphael 'kena' Poss) wrote…

Q: why throw away the newlines? how do they hurt us?

I find these log messages pretty irritating to work with since they are difficult to grep out (merge-logs makes it better, but that's not always at hand). The details printed on these lines also barely add any detail but consume a lot of vertical real estate.

[craig]

Build failed:

• Bazel Essential CI (Cockroach)

[tbg]

bors r=knz

bors failure is

=== RUN   TestStatusEngineStatsJson
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/115b56b7b7bd2dfb6f4ea1f5f6513edb/logTestStatusEngineStatsJson1920103392
    test_log_scope.go:79: use -show-logs to present logs inline
    status_test.go:280: using admin URL https://127.0.0.1:34309/
    status_test.go:287: condition failed to evaluate within 45s: status: 503 Service Unavailable, content-type: application/json, body: {
          "error": "connection error: desc = \"transport: error while dialing: gRPC connection unexpectedly re-dialed: connection interrupted (did the remote node shut down or are there networking issues?)\"",
          "code": 14,
          "message": "connection error: desc = \"transport: error while dialing: gRPC connection unexpectedly re-dialed: connection interrupted (did the remote node shut down or are there networking issues?)\"",
          "details": [
          ]
        }, error: <nil>
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/115b56b7b7bd2dfb6f4ea1f5f6513edb/logTestStatusEngineStatsJson1920103392
--- FAIL: TestStatusEngineStatsJson (64.59s)

I was able to reproduce this on the merge-base 5af7d05 via

./dev test --stress ./pkg/server/ --filter TestStatusEngineStatsJson:

--- FAIL: TestStatusEngineStatsJson (68.44s)
    test_log_scope.go:161: test logs captured to: /Users/tobias/go/src/github.com/cockroachdb/cockroach/tmp/_tmp/eeba368469fff078dbc179626aa9b86d/logTestStatusEngineStatsJson2601962448
    test_log_scope.go:79: use -show-logs to present logs inline
    status_test.go:280: using admin URL https://127.0.0.1:51397
    status_test.go:287: condition failed to evaluate within 45s: status: 503 Service Unavailable, content-type: application/json, body: {
          "error": "connection error: desc = \"transport: error while dialing: gRPC connection unexpectedly re-dialed: connection interrupted (did the remote node shut down or are there networking issues?)\"",
          "code": 14,
          "message": "connection error: desc = \"transport: error while dialing: gRPC connection unexpectedly re-dialed: connection interrupted (did the remote node shut down or are there networking issues?)\"",
          "details": [
          ]
        }, error: <nil>
    panic.go:522: -- test log scope end --
FAIL
I230519 16:09:42.012033 1 (gostd) testmain.go:528  [-] 1  Test //pkg/server:server_test exited with error code 1


ERROR: exit status 1

58 runs completed, 1 failures, over 2m39s

So it isn't introduced in this PR. Still, it's topical to what I'm working on (#99191) so I'll double check what's going on here. This strikes me as completely unexpected in two ways: first, the connection should "just work", second, it shouldn't be unhealthy for >45s, and third, we're not supposed to see "connection unexpectedly re-dialed" pretty much ever, or so I thought.

Touches #99261
Touches #95491

[craig]

Build succeeded:

• Bazel Essential CI (Cockroach)