Pull in latests commits from 8.x and update based on new patterns (#17385)

* Fix empty node stats pipelines (#17185) (#17197)

Fixed an issue where the `/_node/stats` API displayed empty pipeline metrics
when X-Pack monitoring was enabled

(cherry picked from commit 8678581)

Co-authored-by: kaisecheng <[email protected]>

* Update z_rubycheck.rake to no longer inject Xmx1g (#17211)

This allows the environment variable JRUBY_OPTS to be used for setting properties like Xmx
original pr: #16420

(cherry picked from commit f562f37)

Co-authored-by: kaisecheng <[email protected]>

* Improve warning for insufficient file resources for PQ max_bytes (#16656) (#17222)

This commit refactors the `PersistedQueueConfigValidator` class to provide a
more detailed, accurate and actionable warning when pipeline's PQ configs are at
risk of running out of disk space. See
#14839 for design considerations. The
highlights of the changes include accurately determining the free resources on a
filesystem disk and then providing a breakdown of the usage for each of the
paths configured for a queue.

(cherry picked from commit 0621544)

Co-authored-by: Cas Donoghue <[email protected]>

* gradle task migrate to the new artifacts-api (#17232) (#17236)

This commit migrates gradle task to the new artifacts-api

- remove dependency on staging artifacts
- all builds use snapshot artifacts
- resolve version from current branch, major.x, previous minor,
   with priority given in that order.

Co-authored-by: Andrea Selva <[email protected]>
(cherry picked from commit 0a74568)

Co-authored-by: kaisecheng <[email protected]>

* tests: ls2ls delay checking until events have been processed (#17167) (#17252)

* tests: ls2ls delay checking until events have been processed

* Make sure upstream sends expected number of events before checking the expectation with downstream. Remove unnecessary or duplicated logics from the spec.

* Add exception handling in `wait_for_rest_api` to make wait for LS REST API retriable.

---------

Co-authored-by: Mashhur <[email protected]>
Co-authored-by: Mashhur <[email protected]>
(cherry picked from commit 73ffa24)

Co-authored-by: Ry Biesemeyer <[email protected]>

* Additional cleanify changes to ls2ls integ tests (#17246) (#17255)

* Additional cleanify changes to ls2ls integ tests: replace heartbeat-input with reload option, set queue drain to get consistent result.

(cherry picked from commit 1e06eea)

Co-authored-by: Mashhur <[email protected]>

* [8.x] Reimplement LogStash::Numeric setting in Java (backport #17127) (#17273)

This is an automatic backport of pull request #17127 done by [Mergify](https://mergify.com).

----

* Reimplement LogStash::Numeric setting in Java (#17127)

Reimplements `LogStash::Setting::Numeric` Ruby setting class into the `org.logstash.settings.NumericSetting` and exposes it through `java_import` as `LogStash::Setting::NumericSetting`.
Updates the rspec tests:
- verifies `java.lang.IllegalArgumentException` instead of `ArgumentError` is thrown because the kind of exception thrown by Java code, during verification.

(cherry picked from commit 07a3c8e)

* Fixed reference of SettingNumeric class (on main modules were removed)

---------

Co-authored-by: Andrea Selva <[email protected]>

* [CI] Health report integration tests use the new artifacts-api (#17274) (#17277)

migrate to the new artifacts-api

(cherry picked from commit feb2b92)

Co-authored-by: kaisecheng <[email protected]>

* Backport 17203 and 17267 8.x (#17270)

* Pluginmanager clean after mutate (#17203)

* pluginmanager: always clean after mutate

* pluginmanager: don't skip updating plugins installed with --version

* pr feedback

(cherry picked from commit 8c96913)

* Pluginmanager install preserve (#17267)

* tests: integration tests for pluginmanager install --preserve

* fix regression where pluginmanager's install --preserve flag didn't

* [Backport 8.x] benchmark script (#17283)

This commit cherry-picked the missing becnhmark script PRs
The deprecated artifacts-api is removed

[CI] benchmark uses the new artifacts-api (#17224)
[CI] benchmark readme (#16783)
Introduce a new flag to explicitly permit legacy monitoring (#16586) (Only take the benchmark script)
[ci] fix wrong queue type in benchmark marathon (#16465)
[CI] fix benchmark marathon (#16447)
[CI] benchmark dashboard and pipeline for testing against multiple versions (#16421)

* Fix pqcheck and pqrepair on Windows (#17210) (#17259)

A recent change to pqheck, attempted to address an issue where the
pqcheck would not on Windows mahcines when located in a folder containing
a space, such as "C:\program files\elastic\logstash". While this fixed an
issue with spaces in folders, it introduced a new issue related to Java options,
and the pqcheck was still unable to run on Windows.

This PR attempts to address the issue, by removing the quotes around the Java options,
which caused the option parsing to fail, and instead removes the explicit setting of
the classpath - the use of `set CLASSPATH=` in the `:concat` function is sufficient
to set the classpath, and should also fix the spaces issue

Fixes: #17209
(cherry picked from commit ba5f215)

Co-authored-by: Rob Bavey <[email protected]>

* Shareable function for partitioning integration tests (#17223) (#17303)

For the fedramp high work https://github.com/elastic/logstash/pull/17038/files a
use case for multiple scripts consuming the partitioning functionality emerged.
As we look to more advanced partitioning we want to ensure that the
functionality will be consumable from multiple scripts.

See #17219 (comment)

(cherry picked from commit d916972)

Co-authored-by: Cas Donoghue <[email protected]>

* [8.x] Surface failures from nested rake/shell tasks (backport #17310) (#17317)

* Surface failures from nested rake/shell tasks (#17310)

Previously when rake would shell out the output would be lost. This
made debugging CI logs difficult. This commit updates the stack with
improved message surfacing on error.

(cherry picked from commit 0d931a5)

# Conflicts:
#	rubyUtils.gradle

* Extend ruby linting tasks to handle file inputs (#16660)

This commit extends the gradle and rake tasks to pass through a list of files
for rubocop to lint. This allows more specificity and fine grained control for
linting when the consumer of the tasks only wishes to lint a select few files.

* Ensure shellwords library is loaded

Without this depending on task load order `Shellwords` may not be available.

---------

Co-authored-by: Cas Donoghue <[email protected]>

* Forward Port of Release notes for `8.16.5` and `8.17.3` (#17187), (#17188) (#17266) (#17321)

* Forward Port of Release notes for 8.17.3 (#17187)

* Update release notes for 8.17.3

---------

Co-authored-by: logstashmachine <[email protected]>
Co-authored-by: Rob Bavey <[email protected]>

* Forward Port of Release notes for 8.16.5 (#17188)

* Update release notes for 8.16.5

---------

Co-authored-by: logstashmachine <[email protected]>
Co-authored-by: Rob Bavey <[email protected]>

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: logstashmachine <[email protected]>
(cherry picked from commit 63e8fd1)

Co-authored-by: Rob Bavey <[email protected]>

* Add Deprecation tag to arcsight module (#17331)

* [8.x] Upgrade elasticsearch-ruby client. (backport #17161) (#17306)

* Upgrade elasticsearch-ruby client. (#17161)

* Fix Faraday removed basic auth option and apply the ES client module name change.

(cherry picked from commit e748488)

* Apply the required changes in elasticsearch_client.rb after upgrading the elasticsearch-ruby client to 8.x

* Swallow the exception and make non-connectable client when ES client raises connection refuses exception.

---------

Co-authored-by: Mashhur <[email protected]>
Co-authored-by: Mashhur <[email protected]>

* Removed unused configHash computation that can be replaced by PipelineConfig.configHash() (#17336) (#17345)

Removed unused configHash computation happening in AbstractPipeline and used only in tests replaced by PipelineConfig.configHash() invocation

(cherry picked from commit 787fd2c)

Co-authored-by: Andrea Selva <[email protected]>

* Use org.logstash.common.Util to hashing by default to SHA256 (#17346) (#17352)

Removes the usage fo Apache Commons Codec MessgeDigest to use internal Util class with embodies hashing methods.

(cherry picked from commit 9c0e50f)

Co-authored-by: Andrea Selva <[email protected]>

* Added test to verify the int overflow happen (#17353) (#17354)

Use long instead of int type to keep the length of the first token.

The size limit validation requires to sum two integers, one with the length of the accumulated chars till now plus the next fragment head part. If any of the two sizes is close to the max integer it generates an overflow and could successfully fail the test https://github.com/elastic/logstash/blob/9c0e50faacc4700da3dc84a3ba729b84bff860a8/logstash-core/src/main/java/org/logstash/common/BufferedTokenizerExt.java#L123.

To fall in this case it's required that sizeLimit is bigger then 2^32 bytes (2GB) and data fragments without any line delimiter is pushed to the tokenizer with a total size close to 2^32 bytes.

(cherry picked from commit afde43f)

Co-authored-by: Andrea Selva <[email protected]>

* [8.x] add ci shared qualified-version script (backport #17311) (#17348)

* add ci shared qualified-version script (#17311)

* ci: add shareable script for generating qualified version

* ci: use shared script to generate qualified version

(cherry picked from commit 10b5a84)

# Conflicts:
#	.buildkite/scripts/dra/build_docker.sh

* resolve merge conflict

---------

Co-authored-by: Rye Biesemeyer <[email protected]>

* tests: make integration split quantity configurable (#17219) (#17367)

* tests: make integration split quantity configurable

Refactors shared splitter bash function to take a list of files on stdin
and split into a configurable number of partitions, emitting only those from
the currently-selected partition to stdout.

Also refactors the only caller in the integration_tests launcher script to
accept an optional partition_count parameter (defaulting to `2` for backward-
compatibility), to provide the list of specs to the function's stdin, and to
output relevant information about the quantity of partition splits and which
was selected.

* ci: run integration tests in 3 parts

(cherry picked from commit 3e0f488)

Co-authored-by: Rye Biesemeyer <[email protected]>

* Update buildkite with new patterns from 8.x

This commit updates the buildkite definitions to be compatible with the
upstream 8.x branch. Specificially:
 - Split integration tests for fips into 3 runners.
 - Use the new shared bash helper for computing QUALIFIED_VERSION

It also continues standardization of using a "fedrampHighMode" for indicating
the tests should be running in the context of our custom image for the SRE team.

* Bug fix: Actually use shared integration_tests.sh file

After refactoring to use the same script, I forgot to actually use it
in the buildkite definition...

---------

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: kaisecheng <[email protected]>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Ry Biesemeyer <[email protected]>
Co-authored-by: Mashhur <[email protected]>
Co-authored-by: Andrea Selva <[email protected]>
Co-authored-by: Rob Bavey <[email protected]>
Co-authored-by: Mashhur <[email protected]>

Author: 8 people

.buildkite/aarch64_pipeline.yml

@@ -35,48 +35,71 @@ steps:
           automatic:
             - limit: 3
 
-      - label: ":lab_coat: Integration Tests / part 1"
-        key: "integration-tests-part-1"
+      - label: ":lab_coat: Integration Tests / part 1-of-3"
+        key: "integration-tests-part-1-of-3"
         command: |
           set -euo pipefail
 
           source .buildkite/scripts/common/vm-agent.sh
-          ci/integration_tests.sh split 0
+          ci/integration_tests.sh split 0 3
         retry:
           automatic:
             - limit: 3
 
-      - label: ":lab_coat: Integration Tests / part 2"
-        key: "integration-tests-part-2"
+      - label: ":lab_coat: Integration Tests / part 2-of-3"
+        key: "integration-tests-part-2-of-3"
         command: |
           set -euo pipefail
 
           source .buildkite/scripts/common/vm-agent.sh
-          ci/integration_tests.sh split 1
+          ci/integration_tests.sh split 1 3
         retry:
           automatic:
             - limit: 3
 
-      - label: ":lab_coat: IT Persistent Queues / part 1"
-        key: "integration-tests-qa-part-1"
+      - label: ":lab_coat: Integration Tests / part 3-of-3"
+        key: "integration-tests-part-3-of-3"
+        command: |
+          set -euo pipefail
+
+          source .buildkite/scripts/common/vm-agent.sh
+          ci/integration_tests.sh split 2 3
+        retry:
+          automatic:
+            - limit: 3
+
+      - label: ":lab_coat: IT Persistent Queues / part 1-of-3"
+        key: "integration-tests-qa-part-1-of-3"
+        command: |
+          set -euo pipefail
+
+          source .buildkite/scripts/common/vm-agent.sh
+          export FEATURE_FLAG=persistent_queues
+          ci/integration_tests.sh split 0 3
+        retry:
+          automatic:
+            - limit: 3
+
+      - label: ":lab_coat: IT Persistent Queues / part 2-of-3"
+        key: "integration-tests-qa-part-2-of-3"
         command: |
           set -euo pipefail
 
           source .buildkite/scripts/common/vm-agent.sh
           export FEATURE_FLAG=persistent_queues
-          ci/integration_tests.sh split 0
+          ci/integration_tests.sh split 1 3
         retry:
           automatic:
             - limit: 3
 
-      - label: ":lab_coat: IT Persistent Queues / part 2"
-        key: "integration-tests-qa-part-2"
+      - label: ":lab_coat: IT Persistent Queues / part 3-of-3"
+        key: "integration-tests-qa-part-3-of-3"
         command: |
           set -euo pipefail
 
           source .buildkite/scripts/common/vm-agent.sh
           export FEATURE_FLAG=persistent_queues
-          ci/integration_tests.sh split 1
+          ci/integration_tests.sh split 2 3
         retry:
           automatic:
             - limit: 3

.buildkite/benchmark_marathon_pipeline.yml

@@ -0,0 +1,11 @@
+agents:
+  provider: gcp
+  imageProject: elastic-images-prod
+  image: family/platform-ingest-logstash-ubuntu-2204
+  machineType: "n2-standard-16"
+  diskSizeGb: 100
+  diskType: pd-ssd
+
+steps:
+  - label: "Benchmark Marathon"
+    command: .buildkite/scripts/benchmark/marathon.sh

.buildkite/pull_request_pipeline.yml

@@ -137,14 +137,13 @@ steps:
     command: |
       set -euo pipefail
       source .buildkite/scripts/common/vm-agent.sh
-      # TODO: Use https://github.com/elastic/logstash/pull/17311 to compute QUALIFIED_VERSION once merged
-      QUALIFIED_VERSION="8.19.0-SNAPSHOT"
+      QUALIFIED_VERSION="$(.buildkite/scripts/common/qualified-version.sh)"
       ./gradlew --stacktrace artifactDockerObservabilitySRE -PfedrampHighMode=true
       docker run docker.elastic.co/logstash/logstash-observability-sre:$${QUALIFIED_VERSION} \
         logstash -e 'input { generator { count => 3 } } output { stdout { codec => rubydebug } }'
 
-  - label: ":lab_coat: Integration Tests - FIPS mode / part 1"
-    key: "integration-tests-fips-part-1"
+  - label: ":lab_coat: Integration Tests - FIPS mode / part 1-of-3"
+    key: "integration-tests-fips-part-1-of-3"
     agents:
       provider: gcp
       imageProject: elastic-images-prod
@@ -158,10 +157,10 @@ steps:
       set -euo pipefail
 
       docker build -t test-runner-image -f x-pack/distributions/internal/observabilitySRE/docker/Dockerfile .
-      docker run test-runner-image ci/run-fips-integration-tests.sh 0
+      docker run -e FEDRAMP_HIGH_MODE=true test-runner-image ci/integration_tests.sh split 0 3
 
-  - label: ":lab_coat: Integration Tests - FIPS mode / part 2"
-    key: "integration-tests-fips-part-2"
+  - label: ":lab_coat: Integration Tests - FIPS mode / part 2-of-3"
+    key: "integration-tests-fips-part-2-of-3"
     agents:
       provider: gcp
       imageProject: elastic-images-prod
@@ -175,9 +174,26 @@ steps:
       set -euo pipefail
 
       docker build -t test-runner-image -f x-pack/distributions/internal/observabilitySRE/docker/Dockerfile .
-      docker run test-runner-image ci/run-fips-integration-tests.sh 1
+      docker run -e FEDRAMP_HIGH_MODE=true test-runner-image ci/integration_tests.sh split 1 3
 
-  - label: ":lab_coat: Integration Tests / part 1"
+  - label: ":lab_coat: Integration Tests - FIPS mode / part 3-of-3"
+    key: "integration-tests-fips-part-3-of-3"
+    agents:
+      provider: gcp
+      imageProject: elastic-images-prod
+      image: family/platform-ingest-logstash-ubuntu-2204
+      machineType: "n2-standard-4"
+      diskSizeGb: 64
+    retry:
+      automatic:
+        - limit: 3
+    command: |
+      set -euo pipefail
+
+      docker build -t test-runner-image -f x-pack/distributions/internal/observabilitySRE/docker/Dockerfile .
+      docker run -e FEDRAMP_HIGH_MODE=true test-runner-image ci/integration_tests.sh split  2 3
+
+  - label: ":lab_coat: Integration Tests / part 1-of-3"
     key: "integration-tests-part-1"
     agents:
       image: "docker.elastic.co/ci-agent-images/platform-ingest/buildkite-agent-logstash-ci-no-root"
@@ -193,10 +209,10 @@ steps:
       set -euo pipefail
 
       source .buildkite/scripts/common/container-agent.sh
-      ci/integration_tests.sh split 0
+      ci/integration_tests.sh split 0 3
 
-  - label: ":lab_coat: Integration Tests / part 2"
-    key: "integration-tests-part-2"
+  - label: ":lab_coat: Integration Tests / part 2-of-3"
+    key: "integration-tests-part-2-of-3"
     agents:
       image: "docker.elastic.co/ci-agent-images/platform-ingest/buildkite-agent-logstash-ci-no-root"
       cpu: "8"
@@ -211,10 +227,47 @@ steps:
       set -euo pipefail
 
       source .buildkite/scripts/common/container-agent.sh
-      ci/integration_tests.sh split 1
+      ci/integration_tests.sh split 1 3
+
+  - label: ":lab_coat: Integration Tests / part 3-of-3"
+    key: "integration-tests-part-3-of-3"
+    agents:
+      image: "docker.elastic.co/ci-agent-images/platform-ingest/buildkite-agent-logstash-ci-no-root"
+      cpu: "8"
+      memory: "16Gi"
+      ephemeralStorage: "100Gi"
+      # Run as a non-root user
+      imageUID: "1002"
+    retry:
+      automatic:
+        - limit: 3
+    command: |
+      set -euo pipefail
+
+      source .buildkite/scripts/common/container-agent.sh
+      ci/integration_tests.sh split 2 3
+
+  - label: ":lab_coat: IT Persistent Queues / part 1-of-3"
+    key: "integration-tests-qa-part-1-of-3"
+    agents:
+      image: "docker.elastic.co/ci-agent-images/platform-ingest/buildkite-agent-logstash-ci-no-root"
+      cpu: "8"
+      memory: "16Gi"
+      ephemeralStorage: "100Gi"
+      # Run as non root (logstash) user. UID is hardcoded in image.
+      imageUID: "1002"
+    retry:
+      automatic:
+        - limit: 3
+    command: |
+      set -euo pipefail
+
+      source .buildkite/scripts/common/container-agent.sh
+      export FEATURE_FLAG=persistent_queues
+      ci/integration_tests.sh split 0 3
 
-  - label: ":lab_coat: IT Persistent Queues / part 1"
-    key: "integration-tests-qa-part-1"
+  - label: ":lab_coat: IT Persistent Queues / part 2-of-3"
+    key: "integration-tests-qa-part-2-of-3"
     agents:
       image: "docker.elastic.co/ci-agent-images/platform-ingest/buildkite-agent-logstash-ci-no-root"
       cpu: "8"
@@ -230,10 +283,10 @@ steps:
 
       source .buildkite/scripts/common/container-agent.sh
       export FEATURE_FLAG=persistent_queues
-      ci/integration_tests.sh split 0
+      ci/integration_tests.sh split 1 3
 
-  - label: ":lab_coat: IT Persistent Queues / part 2"
-    key: "integration-tests-qa-part-2"
+  - label: ":lab_coat: IT Persistent Queues / part 3-of-3"
+    key: "integration-tests-qa-part-3-of-3"
     agents:
       image: "docker.elastic.co/ci-agent-images/platform-ingest/buildkite-agent-logstash-ci-no-root"
       cpu: "8"
@@ -249,7 +302,7 @@ steps:
 
       source .buildkite/scripts/common/container-agent.sh
       export FEATURE_FLAG=persistent_queues
-      ci/integration_tests.sh split 1
+      ci/integration_tests.sh split 2 3
 
   - label: ":lab_coat: x-pack unit tests"
     key: "x-pack-unit-tests"
@@ -302,7 +355,7 @@ steps:
       set -euo pipefail
 
       docker build -t test-runner-image -f x-pack/distributions/internal/observabilitySRE/docker/Dockerfile .
-      docker run -e FIPS_MODE=true test-runner-image x-pack/ci/unit_tests.sh
+      docker run -e FEDRAMP_HIGH_MODE=true test-runner-image x-pack/ci/unit_tests.sh
 
   - label: ":lab_coat: x-pack integration - FIPS mode"
     key: "integration-tests-x-pack-fips"
@@ -319,7 +372,7 @@ steps:
       set -euo pipefail
 
       docker build -t test-runner-image -f x-pack/distributions/internal/observabilitySRE/docker/Dockerfile .
-      docker run -e FIPS_MODE=true test-runner-image x-pack/ci/integration_tests.sh
+      docker run -e FEDRAMP_HIGH_MODE=true test-runner-image x-pack/ci/integration_tests.sh
 
   - wait: ~
     continue_on_failure: true

.buildkite/scripts/benchmark/README.md

@@ -0,0 +1,22 @@
+## Steps to set up GCP instance to run benchmark script
+- Create an instance "n2-standard-16" with Ubuntu image
+- Install docker
+  - `sudo snap install docker`
+  - `sudo usermod -a -G docker $USER`
+- Install jq
+- Install vault
+  - `sudo snap install vault`
+  - `vault login --method github`
+  - `vault kv get -format json secret/ci/elastic-logstash/benchmark`
+- Setup Elasticsearch index mapping and alias with `setup/*`
+- Import Kibana dashboard with `save-objects/*`
+- Run the benchmark script
+  - Send data to your own Elasticsearch. Customise `VAULT_PATH="secret/ci/elastic-logstash/your/path"`
+  - Run the script `main.sh` 
+    - or run in background `nohup bash -x main.sh > log.log 2>&1 &`
+
+## Notes
+- Benchmarks should only be compared using the same hardware setup.
+- Please do not send the test metrics to the benchmark cluster. You can set `VAULT_PATH` to send data and metrics to your own server.
+- Run `all.sh` as calibration which gives you a baseline of performance in different versions.
+- [#16586](https://github.com/elastic/logstash/pull/16586) allows legacy monitoring using the configuration `xpack.monitoring.allow_legacy_collection: true`, which is not recognized in version 8. To run benchmarks in version 8, use the script of the corresponding branch (e.g. `8.16`) instead of `main` in buildkite.

.buildkite/scripts/benchmark/config/logstash.yml

@@ -3,6 +3,7 @@ pipeline.workers: ${WORKER}
 pipeline.batch.size: ${BATCH_SIZE}
 queue.type: ${QTYPE}
 
+xpack.monitoring.allow_legacy_collection: true
 xpack.monitoring.enabled: true
 xpack.monitoring.elasticsearch.username: ${MONITOR_ES_USER}
 xpack.monitoring.elasticsearch.password: ${MONITOR_ES_PW}

.buildkite/scripts/benchmark/config/uuid

@@ -0,0 +1 @@
+f74f1a28-25e9-494f-ba41-ca9f13d4446d