Sink Healthcheck Bug (clickhouse)

[satellite-no]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

Im not sure if this is more wide spread of just an issue for the Clickhouse sink but healthcheck when enabled fails to validate when using the SECRET feature backend. Vector appears to not be resolving the value indicated during the healthcheck and instead sending literally the value SECRET[vector_clickhouse.username].

Log from Clickhouse:

2025.02.12 10:41:19.837455 [ 62406 ] {} <Error> DynamicQueryHandler: Code: 516. DB::Exception: SECRET[vector_clickhouse.username]: Authentication failed: password is incorrect, or there is no user with such name. (AUTHENTICATION_FAILED), Stack trace (when copying this message, always include the lines below):

Configuration

type: clickhouse
inputs:
  - metric_to_clickhouse
  - event_to_clickhouse
database: "{{ .ch_database }}"
table: "{{ .ch_table }}"
endpoint: https://clickhouse_url:9443
skip_unknown_fields: true
date_time_best_effort: true
compression: zstd
auth:
  strategy: basic
  user: "SECRET[vector_clickhouse.username]"
  password: "SECRET[vector_clickhouse.password]"
format: json_each_row
tls:
  verify_certificate: false
healthcheck:
  enabled: true

Version

vector 0.44.0 (x86_64-unknown-linux-gnu 3cdc7c3 2025-01-13 21:26:04.735691656)

Debug Output

2025-02-12T15:41:19.825972Z DEBUG http: vector::internal_events::http_client: Sending HTTP request. uri=https://clickhouse_url:9443/?query=SELECT%201 method=GET version=HTTP/1.1 headers={"authorization": Sensitive, "user-agent": "Vector/0.44.0 (x86_64-unknown-linux-gnu 3cdc7c3 2025-01-13 21:26:04.735691656)", "accept-encoding": "identity"} body=[empty]
2025-02-12T15:41:19.826318Z DEBUG hyper::client::connect::dns: resolving host="clickhouse_url"
2025-02-12T15:41:19.830373Z DEBUG http: hyper::client::connect::http: connecting to 10.1.2.2:9443
2025-02-12T15:41:19.831885Z DEBUG http: hyper::client::connect::http: connected to 10.1.2.2:9443
2025-02-12T15:41:19.835878Z DEBUG hyper::proto::h1::io: flushed 309 bytes
2025-02-12T15:41:19.838077Z DEBUG hyper::proto::h1::io: parsed 8 headers
2025-02-12T15:41:19.838095Z DEBUG hyper::proto::h1::conn: incoming body is chunked encoding
2025-02-12T15:41:19.838122Z DEBUG hyper::proto::h1::decode: incoming chunked header: 0xD0 (208 bytes)
2025-02-12T15:41:19.838173Z DEBUG http: vector::internal_events::http_client: HTTP response. status=403 Forbidden version=HTTP/1.1 headers={"date": "Wed, 12 Feb 2025 15:41:19 GMT", "connection": "Keep-Alive", "content-type": "text/plain; charset=UTF-8", "x-clickhouse-server-display-name": "ip-10-174-70-37.ec2.internal", "transfer-encoding": "chunked", "x-clickhouse-exception-code": "516", "keep-alive": "timeout=10, max=9999", "x-clickhouse-summary": "{\"read_rows\":\"0\",\"read_bytes\":\"0\",\"written_rows\":\"0\",\"written_bytes\":\"0\",\"total_rows_to_read\":\"0\",\"result_rows\":\"0\",\"result_bytes\":\"0\",\"elapsed_ns\":\"37001\"}"} body=[unknown]
2025-02-12T15:41:19.838251Z ERROR vector::topology::builder: msg="Healthcheck failed." error=Unexpected status: 403 Forbidden component_kind="sink" component_type="clickhouse" component_id=clickhouse
2025-02-12T15:41:19.838293Z DEBUG hyper::proto::h1::conn: incoming body completed
x Health check for "clickhouse" failed: Unexpected status: 403 Forbidden

Example Data

No response

Additional Context

No response

References

No response

[aa1ex]

Hello! I encountered a similar issue with Elasticsearch.
Here is an example of my configuration:

data_dir: .

secret:
  secret_1:
    type: exec
    command: ["curl", "https://gist.githubusercontent.com/aa1ex/8b7b2f97e26d1ef0000c38d9474e95cd/raw/gistfile1.txt"]

sources:
  generate_syslog:
    type:   "demo_logs"
    format: "syslog"
    count:  1

sinks:
  sink-test:
    inputs:
      - generate_syslog
    type: "elasticsearch"
    api_version: auto
    endpoints:
      - https://webhook.site/6d119cd2-401e-4b7d-97e3-813067b42dd4
    mode: bulk
    tls:
      verify_certificate: false
    bulk:
      action: create
      index: "demo-index-%Y-%m-%d"
    auth:
      user: "SECRET[secret_1.ELASTIC_USER]"
      password: "SECRET[secret_1.ELASTIC_PASSWORD]"
      strategy: basic

When running validation, the secrets are not resolved. You can verify this by running

vector validate elastic.yaml

and inspecting the request parameters in a webhook.site

However, when running Vector with this configuration, everything works fine:

vector -c elastic.yaml

This issue is reproducible in the following versions:
• Vector/0.44.0 (aarch64-apple-darwin 3cdc7c3 2025-01-13 21:26:04.735691656)
• Vector/0.45.0 (aarch64-apple-darwin 063cabb 2025-02-24 14:52:02.810034614)

[zvlb]

Hi all.
This issue is blocking the implementation of Secret handling logic in Vector Operator. We’re eagerly awaiting a resolution!

Disabling validations is not an option for us, as it could theoretically completely break the configuration processing logic in Vector Operator.

[nitzan-tz]

I have the same issue with Elasticsearch
I also verified it with tcpdump and the secret is not resolved

[pront]

Hello, I can confirm that secrets are not resolved in the context of config validation.

• Validate entrypoint:

  vector/src/validate.rs

  Line 100 in a51cd8d

  pub async fn validate(opts: &Opts, color: bool) -> ExitCode {

• Secret resolution and config loading: https://github.com/vectordotdev/vector/blob/master/src/config/loading/mod.rs#L221

This should be trivial to change to call load_builder_from_paths_with_secrets instead of load_builder_from_paths but it will need some testing and I believe there are some subtle side-effects. Some users will not want the validation step to rely on HTTP requests for example. We could workaround it by adding a new option to the validate CLI to load secrets.

[MikoLone]

I guess it's an easy workaround to suppress healthchecks by using the

[healthchecks]
enabled = false

flag but man it makes me sad to have to disable the healthchecks.