cannot convert the confmap.Conf: environment variable "1"

[netikras]

Hi,

I'm trying to deploy otel using terraform (see below). However, I'm facing 2 problems:

1. right after deployment (successful):

kubectl -n monitoring logs opentelemetry-kube-stack-daemon-collector-nzj9f
2024-07-11T16:25:18.091Z        warn    [email protected]/expand.go:102  Variable substitution using $VAR will be deprecated in favor of ${VAR} and ${env:VAR}, please update $OTEL_K8S_NODE_NAME        {"variable": "OTEL_K8S_NODE_NAME"}
2024-07-11T16:25:18.091Z        warn    [email protected]/expand.go:102  Variable substitution using $VAR will be deprecated in favor of ${VAR} and ${env:VAR}, please update $1 {"variable": "1"}
Error: failed to resolve config: cannot resolve the configuration: cannot convert the confmap.Conf: environment variable "1" has invalid name: must match regex ^[a-zA-Z_][a-zA-Z0-9_]*$
2024/07/11 16:25:18 collector server run finished with error: failed to resolve config: cannot resolve the configuration: cannot convert the confmap.Conf: environment variable "1" has invalid name: must match regex ^[a-zA-Z_][a-zA-Z0-9_]*$

I can't find where a variable with a name of "1" is specified...

2. I can't cleanly uninstall the helmchart. Pods, services, configmaps, secrets, CRDs -- they all remain there. I managed to begin removing CRDs after kubectl patch crd/opentelemetrycollectors.opentelemetry.io -p '{"metadata":{"finalizers":[]}}' --type=merge. After this patch I can then remove it all, manually.
   For some reason, helm fails to uninstall it completely, even if it says that the release has been uninstalled.

3. P.S. I'm planning to inject otel into applications running applications in various namespaces. I understand that annotating respective namespaces should be enough? Will it be able to instrument across multiple namespaces?

resource "helm_release" "otel_auto_instrumentation_stack" {
  repository       = "https://open-telemetry.github.io/opentelemetry-helm-charts"
  chart            = "opentelemetry-kube-stack"
  version          = "0.0.8"
  name             = "opentelemetry-kube-stack"
  namespace        = "monitoring"
  create_namespace = true

  values = [
    <<-YAML
instrumentation:
  enabled: true
  exporter:
    # This is the default collector's service
    # Upon creation of a tracing collector, edit this endpoint.
    endpoint: http://collector-collector:4317

opentelemetry-operator:
  enabled: true

  admissionWebhooks:
    certManager:
      enabled: false
  crds:
      create: false

defaultCRConfig:
  enabled: true
  presets:
    hostMetrics:
      enabled: true
    logsCollection:
      enabled: false
    kubernetesAttributes:
      enabled: true
    kubeletMetrics:
      enabled: true
    kubernetesEvents:
      enabled: true
    clusterMetrics:
      enabled: true
  config:
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
    processors:
      memory_limiter:
        check_interval: 1s
        limit_percentage: 75
        spike_limit_percentage: 15
      batch:
        send_batch_size: 10000
        timeout: 10s

    exporters:
      debug: {}
      otlp/tempo:
        endpoint: tempo.monitoring.svc:4317
        tls:
          insecure: true
        sending_queue:
          num_consumers: 4
          queue_size: 100
        retry_on_failure:
          enabled: true

    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [memory_limiter, batch]
          exporters: [debug, otlp/tempo]
  observability:
    metrics:
      enableMetrics: true
admissionWebhooks:
  autoGenerateCert:
    enabled: true
YAML
  ]
}

[TylerHelmuth]

@netikras the issue is happening because of a bug in 0.104.0: open-telemetry/opentelemetry-collector#10560.

For the opentelemetry-kube-stack chart specifically, the issue is fixed in #1255. But also the chart is still under development and using it right now isn't recommended.

[TylerHelmuth]

I am going to close this since we have a fix and since the chart is still under developement. Once we have our first minor release (v0.1.0) if the issue persists please ping me