Save name came through ENV vars to let Logstash decide using value from either keystore or ENV.

[mashhurs]

Release notes

What does this PR do?

This PR fixes the historical bug: when running Logstash in a docker container with ENV variables, env2yaml tool saves ENV ${KEY}=${VALUE} by resolving its actual value. This will be an issue when keystore has a same ${KEY} where the expected behavior is to be keystore precedence.

Why is it important/What is the impact to the user?

If users are using same ${KEY} in both keystore and docker ENV, they may see the behavior where keystore ${KEY}=VALUE will be used.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files (and/or docker env variables)
• [ ] I have added tests that prove my fix is effective or that my feature works

Author's Checklist

• [ ]

How to test this PR locally

• pull this PR
• run rake artifact:docker -> creates an docker image (see the logs below)
• run the container with docker run --rm --name={preferredName} -e API_ENABLED=false -e {dockerImageHashBuiltByRake}
• add API_ENABLED same key to the keystore with value of true, bin/logstash-keystore add API_ENABLED
• rerun the docker container
• check the Logstash logs that API server will be executed because Logstash uses keystore API_ENABLED=true
• got to container and check the config/logstash.yml file. Before the fix, logstash.yml would contain api.enabled: false lines. After the fix it is, api.enabled: ${API_ENABLED}

Related issues

• Closes Inconsistent order of resolving ${VAR} value when running Logstash with docker #15766

Use cases

Screenshots

Logs

Using system java: /usr/bin/java
....
[docker] Building docker image
../vendor/jruby/bin/jruby -S erb -T "-" created_date="2024-03-25T12:32:27-07:00" elastic_version="8.14.0-SNAPSHOT" arch="arm64" version_tag="8.14.0-SNAPSHOT" image_flavor="oss" local_artifacts="true" templates/Dockerfile.erb > "/Users/mashhur/Dev/elastic/logstash/build/Dockerfile-oss" ;   ../vendor/jruby/bin/jruby -S erb -T "-" created_date="2024-03-25T12:32:27-07:00" elastic_version="8.14.0-SNAPSHOT" arch="arm64" version_tag="8.14.0-SNAPSHOT" image_flavor="full" local_artifacts="true" templates/Dockerfile.erb > "/Users/mashhur/Dev/elastic/logstash/build/Dockerfile-full" ;   ../vendor/jruby/bin/jruby -S erb -T "-" created_date="2024-03-25T12:32:27-07:00" elastic_version="8.14.0-SNAPSHOT" arch="arm64" version_tag="8.14.0-SNAPSHOT" image_flavor="ubi8" local_artifacts="true" templates/Dockerfile.erb > "/Users/mashhur/Dev/elastic/logstash/build/Dockerfile-ubi8" ; 
docker run --rm \
	  -v "/Users/mashhur/Dev/elastic/logstash/docker/data/logstash/env2yaml:/usr/src/env2yaml" \
		-e GOARCH=arm64 -e GOOS=linux \
		-w /usr/src/env2yaml golang:1 go build -o /usr/src/env2yaml/env2yaml-arm64
Unable to find image 'golang:1' locally
1: Pulling from library/golang
...
Digest: sha256:0b55ab82ac2a54a6f8f85ec8b943b9e470c39e32c109b766bbc1b801f3fa8d3b
Status: Downloaded newer image for golang:1
go: downloading gopkg.in/yaml.v2 v2.4.0
docker run --rm \
	  -v "/Users/mashhur/Dev/elastic/logstash/docker/data/logstash/env2yaml:/usr/src/env2yaml" \
		-e GOARCH=amd64 -e GOOS=linux \
		-w /usr/src/env2yaml golang:1 go build -o /usr/src/env2yaml/env2yaml-amd64
go: downloading gopkg.in/yaml.v2 v2.4.0
docker run --rm -d --name=logstash-docker-artifact-server \
	           -p 8000:8000 --expose=8000 -v /Users/mashhur/Dev/elastic/logstash/build:/mnt \
	           python:3 bash -c 'cd /mnt && python3 -m http.server'
Unable to find image 'python:3' locally
3: Pulling from library/python
...
Digest: sha256:336461f63f4eb1100e178d5acbfea3d1a5b2a53dea88aa0f9b8482d4d02e981c
Status: Downloaded newer image for python:3
c0788dc1c4055d4822d64dfb189f8a82c5d81aefed259b9ddc4013e94195d183
timeout 120 bash -c 'until curl -s localhost:8000 > /dev/null; do sleep 1; done'
docker build --network=host -t docker.elastic.co/logstash/logstash-full:8.14.0-SNAPSHOT -f /Users/mashhur/Dev/elastic/logstash/build/Dockerfile-full data/logstash || \
	  (docker kill logstash-docker-artifact-server; false); \
	docker tag docker.elastic.co/logstash/logstash-full:8.14.0-SNAPSHOT docker.elastic.co/logstash/logstash:8.14.0-SNAPSHOT;
[+] Building 2.5s (19/19) FINISHED                                                                                                                                                       
 => [internal] load build definition from Dockerfile-full                                                                                                                           0.1s
 => => transferring dockerfile: 3.26kB                                                                                                                                              0.0s
 => [internal] load .dockerignore                                                                                                                                                   0.1s
 => => transferring context: 2B                                                                                                                                                     0.0s
 => [internal] load metadata for docker.io/library/ubuntu:20.04                                                                                                                     1.8s
 => [ 1/14] FROM docker.io/library/ubuntu:20.04@sha256:80ef4a44043dec4490506e6cc4289eeda2d106a70148b74b5ae91ee670e9c35d                                                             0.0s
 => [internal] load build context                                                                                                                                                   0.0s
 => => transferring context: 3.16MB                                                                                                                                                 0.0s
 => CACHED [ 2/14] RUN for iter in {1..10}; do export DEBIAN_FRONTEND=noninteractive && apt-get update -y && apt-get upgrade -y && apt-get install -y procps findutils tar gzip &&  0.0s
 => CACHED [ 3/14] RUN groupadd --gid 1000 logstash &&     adduser --uid 1000 --gid 1000     --home /usr/share/logstash --no-create-home     logstash                               0.0s
 => CACHED [ 4/14] RUN  curl -Lo - http://localhost:8000/logstash-8.14.0-SNAPSHOT-linux-$(arch).tar.gz |     tar zxf - -C /usr/share &&     mv /usr/share/logstash-8.14.0-SNAPSHOT  0.0s
...
 => [14/14] RUN chmod 0755 /usr/local/bin/docker-entrypoint                                                                                                                         0.1s
 => exporting to image                                                                                                                                                              0.0s
 => => exporting layers                                                                                                                                                             0.0s
 => => writing image sha256:e89b6d0d04e262f5fa3b51378ca23982756a6b585af411b917dfd2010122806a                                                                                        0.0s
 => => naming to docker.elastic.co/logstash/logstash-full:8.14.0-SNAPSHOT                                                                                                           0.0s
docker kill logstash-docker-artifact-server
logstash-docker-artifact-server

[mashhurs]

I do think we need to also include documentation about order of the resolving the ${VAR} value:

• if the same KEY is defined in both keystore and environment vars, the keystore wins

• if Logstash cannot resolve ${VAR} from either keystore or environment, it throws an error

  ERROR: Failed to load settings file from "path.settings". Aborting... path.setting=/usr/share/logstash/config, 
  exception=LogStash::ConfigurationError, message=>Cannot evaluate `${VAR_NAME}`. Replacement variable 
  `VAR_NAME` is not defined in a Logstash secret store or as an Environment entry and there is no default value given.
  [FATAL] 2024-03-25 23:00:01.350 [main] Logstash - Logstash stopped processing because of an error: (SystemExit) exit
  org.jruby.exceptions.SystemExit: (SystemExit) exit
   at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:912) ~[jruby.jar:?]
   at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:871) ~[jruby.jar:?]
   at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:90) ~[?:?]
  

[kaisecheng]

This is clean and smart solution :) I believe this is acceptable to overwrite logstash.yml from key: value to key: ${KEY}. The original env2yaml do rewrite with actual value whenever the name matched. I see this solution as a bug fix.

[elastic-sonarqube]

Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 0780c9f

History

• 💚 Build #967 succeeded f98171b

cc @mashhurs

[kaisecheng]

LGTM

[jsvd]

The scope for which this PR solves the problem is constrained to when the logstash.yml has settings with references to strings matching environment variable names used by env2yaml. Otherwise this PR will still override the existing logstash.yml entry, for example:

❯ cat logstash.yml 
api.auth.basic.password: ${my_basic_pwd}
api.auth.basic.username: meh
api.auth.type: basic
http.host: 0.0.0.0

❯ docker run -e API_AUTH_BASIC_PASSWORD=1 -p 9600:9600 -v .:/usr/share/logstash/config -it docker.elastic.co/logstash/logstash-full:8.14.0-SNAPSHOT -e "input { generator { count => 1 } }"
2024/03/27 13:37:43 Setting 'api.auth.basic.password' from environment.
[...]

❯ cat logstash.yml 
api.auth.basic.password: ${API_AUTH_BASIC_PASSWORD}
api.auth.basic.username: meh
api.auth.type: basic
http.host: 0.0.0.0

That said, for the env2yaml references the change LGTM, I've tested manually and it works.

[mashhurs]

@logstashmachine backport 8.13