Merge feature branch for observability SRE image creation into 8.x #17541
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…n tests with fips mode (elastic#17029) * Run ruby unit tests under FIPS mode This commit shows a proposed pattern for running automated tests for logstash in FIPS mode. It uses a new identifier in gradle for conditionally setting properties to configure fips mode. The tests are run in a container representative of the base image the final artifacts will be built from. * Move everything from qa/fips -> x-pack This commit moves test setup/config under x-pack dir. * Extend test pipelines for fips mode to java unit tests and integration * Add git to container for gradle * move fips-mode gradle hooks to x-pack * Skip license check for now --------- Co-authored-by: Ry Biesemeyer <[email protected]>
* Split fips integration tests into two steps The integration tests suite takes about 40 minutes. This is far too slow for reasonable feedback on a PR. This commit follows the pattern for the non-fips integration tests whereby the tests are split into two sections that can run in parallel across two steps. This should halve the feedback time. The logic for getting a list of specs files to run has been extracted to a shared shell script for use here and in the integration tests shell script. * Use shared function for splitting integration tests The logic for getting a list of specs to run has been extracted so that it can be shared across fips and non fips integration test modes. This commit updates the non fips integration tests to use the shared function. * fix typo in helper name (kebab case, not snake) * Escape $ so buildkite upload does not try to interpolate * Wrap integration tests in shell script to avoid BK interpolation * Move entrypoint for running integration tests inside docker
* Modify FIPS test runner environment for integration tests This commit makes two small changes to the dockerfile used to define the fips test environment. Specifically it adds curl (which is required by integration tests), make (which is required by test setup), adds a c compiler (gcc and glibc for integration tests which compile a small c program) and turns off debug ssl logging as it is extremely noisy in logs and breaking some assumptions in tests about logfile content. Closes elastic/ingest-dev#5074 * Do not run test env as root The elastic stack is not meant to be run as root. This commit updates the test environment to provision a non root user and have the container context execute under that providioned user. Closes elastic/ingest-dev#5088 * Skip unit tests that reach out to rubygems for fips mode The `update` test setup reaches out to rubygems with net/http which is incompatible with our use of openssl in fips mode. This commit skips those tests when running under fips. See elastic/ingest-dev#5071 * Work around random data request limits in BCFIPS This commit changes test setup to make chunked calls to random data generation in order to work around a limit in fips mode. See elastic/ingest-dev#5072 for details. * Skip tests validating openssl defaults Openssl will not be used when running under FIPS mode. The test setup and tests themselves were failing when running in FIPS mode. This commit skips the tests that are covering behavior that will be disabled. See elastic/ingest-dev#5069 * Skip tests that require pluginmanager to install plugins This commit skips tests that rely on using the pluginmanager to install plugins during tests which require reaching out to rubygems. See elastic/ingest-dev#5108 * Skip prepare offline pack integration tests in fips mode The offline pack tests require on pluginmanager to use net-http library for resolving deps. This will not operate under fips mode. Skip when running in fips mode. See elastic/ingest-dev#5109 * Ensure a gem executible is on path for test setup This commit modifies the generate-gems script to ensure that a `gem` executable is on the path. If there is not one on the test runner, then use the one bundled with vendored jruby. * Skip webserver specs when running in FIPS mode This commit skips the existing webserver tests. We have some options and need to understand some requirements for the webserver functionality for fips mode. The elastic/ingest-dev#5110 issue has a ton of details. * Skip cli `remove` integration tests for FIPS This commit skips tests that are running `remove` action for the pluginmanager. These require reaching out to rubygems which is not available in FIPS mode. These tests were added post initial integration tests scoping work but are clearly requiring skips for FIPS mode. * Add openssl package to FIPS testing env container The setup script for filebeats requires an openssl executable. This commit updates the testing container with this tool. See elastic/ingest-dev#5107 * Re-introduce retries for FIPS tests now that we are in a passing state
* Pluginmanager clean after mutate (elastic#17203) * pluginmanager: always clean after mutate * pluginmanager: don't skip updating plugins installed with --version * pr feedback (cherry picked from commit 8c96913) * Pluginmanager install preserve (elastic#17267) * tests: integration tests for pluginmanager install --preserve * fix regression where pluginmanager's install --preserve flag didn't * Add :skip_fips to update_spec.rb
This commit adds two new CI cells to cover x-pack tests running in FIPS mode. This ensures we have coverage of these features when running existing x-pack tests.
* observabilitySRE: docker rake tasks * Apply suggestions from code review Co-authored-by: Cas Donoghue <[email protected]> * Update rakelib/plugin.rake * Update rakelib/plugin.rake * Update docker/Makefile Co-authored-by: Cas Donoghue <[email protected]> --------- Co-authored-by: Cas Donoghue <[email protected]>
…lastic#17305) The `build-from-local-observability-sre-artifacts` task depends on the `env2yaml` task. This was easy to miss in local development if other images had been built. This commit updates the makefile to properly define that dependency.
* Add a smoke test for observability SRE container Add a CI cell to ensure the observability contater is building successfully. In order to show success run a quick smoke test to point out any glaring issues. This adds some general, low risk plugins for doing quick testing. This will help developers in debugging as we work on this image. * Show what is happening when rake fails * Debug deeper in the stack Show the stdout/stderr when shelling out fails. * Debug layers of build tooling Open3 is not capturing stdout for some reason. Capture it and print to see what is wrong in CI. * Actually run ls command in docker container 🤦 * Update safe_system based on code review suggestion * Dynamically generate version for container invocation Co-authored-by: Ry Biesemeyer <[email protected]> * Refactor smoke test setup to script Avoid interpolation backflips with buildkite by extracting to a script. * Split out message surfacing improvment to separate PR. Moved to: elastic#17310 * Extract version qualifier into standalone script * Wait for version-qualifier.sh script to land upstream Use elastic#17311 once it lands and gets backported to 8.x. For now just hard code version. --------- Co-authored-by: Ry Biesemeyer <[email protected]>
This commit establishes a pattern for configuring the container to run in fips mode. - Use chainguard-fips - Copy over java properties from ls tar archive - Convert default jks to BC keystore - Configure logstash to use java properties and FIPS config NOTE: this assumes bouncycastle jars are in the tarball. The elastic/ingest-dev#5049 ticket will address that.
…SRE (elastic#17359) * Conditionally install bcfips jars when building for observabilitySRE This commit implements a pattern for performing specific gradle tasks based on a newly named "fedrampHighMode" option. This option is used to configure tests to run with additional configuration specific to the observabilitySRE use case. Similarly the additional jar dependencies for bouncycastle fips providers are conditionally installed gated on the "fedrampHighMode" option. In order to ensure the the "fedrampHighMode" option persists through the layers of sub-processes spawned between gradle and rake we store and respect an environment variable FEDRAMP_HIGH_MODE. This may be useful generally in building the docker image. Try codereview suggestion * Use gradle pattern for setting properties with env vars Gradle has a mechanism for setting properties with environment variables prefixed with `ORG_GRADLE_PROJECT`. This commit updates the gradle tasks to use that pattern. See https://docs.gradle.org/current/userguide/build_environment.html#setting_a_project_property for details.
…astic#17385) * Fix empty node stats pipelines (elastic#17185) (elastic#17197) Fixed an issue where the `/_node/stats` API displayed empty pipeline metrics when X-Pack monitoring was enabled (cherry picked from commit 8678581) Co-authored-by: kaisecheng <[email protected]> * Update z_rubycheck.rake to no longer inject Xmx1g (elastic#17211) This allows the environment variable JRUBY_OPTS to be used for setting properties like Xmx original pr: elastic#16420 (cherry picked from commit f562f37) Co-authored-by: kaisecheng <[email protected]> * Improve warning for insufficient file resources for PQ max_bytes (elastic#16656) (elastic#17222) This commit refactors the `PersistedQueueConfigValidator` class to provide a more detailed, accurate and actionable warning when pipeline's PQ configs are at risk of running out of disk space. See elastic#14839 for design considerations. The highlights of the changes include accurately determining the free resources on a filesystem disk and then providing a breakdown of the usage for each of the paths configured for a queue. (cherry picked from commit 0621544) Co-authored-by: Cas Donoghue <[email protected]> * gradle task migrate to the new artifacts-api (elastic#17232) (elastic#17236) This commit migrates gradle task to the new artifacts-api - remove dependency on staging artifacts - all builds use snapshot artifacts - resolve version from current branch, major.x, previous minor, with priority given in that order. Co-authored-by: Andrea Selva <[email protected]> (cherry picked from commit 0a74568) Co-authored-by: kaisecheng <[email protected]> * tests: ls2ls delay checking until events have been processed (elastic#17167) (elastic#17252) * tests: ls2ls delay checking until events have been processed * Make sure upstream sends expected number of events before checking the expectation with downstream. Remove unnecessary or duplicated logics from the spec. * Add exception handling in `wait_for_rest_api` to make wait for LS REST API retriable. --------- Co-authored-by: Mashhur <[email protected]> Co-authored-by: Mashhur <[email protected]> (cherry picked from commit 73ffa24) Co-authored-by: Ry Biesemeyer <[email protected]> * Additional cleanify changes to ls2ls integ tests (elastic#17246) (elastic#17255) * Additional cleanify changes to ls2ls integ tests: replace heartbeat-input with reload option, set queue drain to get consistent result. (cherry picked from commit 1e06eea) Co-authored-by: Mashhur <[email protected]> * [8.x] Reimplement LogStash::Numeric setting in Java (backport elastic#17127) (elastic#17273) This is an automatic backport of pull request elastic#17127 done by [Mergify](https://mergify.com). ---- * Reimplement LogStash::Numeric setting in Java (elastic#17127) Reimplements `LogStash::Setting::Numeric` Ruby setting class into the `org.logstash.settings.NumericSetting` and exposes it through `java_import` as `LogStash::Setting::NumericSetting`. Updates the rspec tests: - verifies `java.lang.IllegalArgumentException` instead of `ArgumentError` is thrown because the kind of exception thrown by Java code, during verification. (cherry picked from commit 07a3c8e) * Fixed reference of SettingNumeric class (on main modules were removed) --------- Co-authored-by: Andrea Selva <[email protected]> * [CI] Health report integration tests use the new artifacts-api (elastic#17274) (elastic#17277) migrate to the new artifacts-api (cherry picked from commit feb2b92) Co-authored-by: kaisecheng <[email protected]> * Backport 17203 and 17267 8.x (elastic#17270) * Pluginmanager clean after mutate (elastic#17203) * pluginmanager: always clean after mutate * pluginmanager: don't skip updating plugins installed with --version * pr feedback (cherry picked from commit 8c96913) * Pluginmanager install preserve (elastic#17267) * tests: integration tests for pluginmanager install --preserve * fix regression where pluginmanager's install --preserve flag didn't * [Backport 8.x] benchmark script (elastic#17283) This commit cherry-picked the missing becnhmark script PRs The deprecated artifacts-api is removed [CI] benchmark uses the new artifacts-api (elastic#17224) [CI] benchmark readme (elastic#16783) Introduce a new flag to explicitly permit legacy monitoring (elastic#16586) (Only take the benchmark script) [ci] fix wrong queue type in benchmark marathon (elastic#16465) [CI] fix benchmark marathon (elastic#16447) [CI] benchmark dashboard and pipeline for testing against multiple versions (elastic#16421) * Fix pqcheck and pqrepair on Windows (elastic#17210) (elastic#17259) A recent change to pqheck, attempted to address an issue where the pqcheck would not on Windows mahcines when located in a folder containing a space, such as "C:\program files\elastic\logstash". While this fixed an issue with spaces in folders, it introduced a new issue related to Java options, and the pqcheck was still unable to run on Windows. This PR attempts to address the issue, by removing the quotes around the Java options, which caused the option parsing to fail, and instead removes the explicit setting of the classpath - the use of `set CLASSPATH=` in the `:concat` function is sufficient to set the classpath, and should also fix the spaces issue Fixes: elastic#17209 (cherry picked from commit ba5f215) Co-authored-by: Rob Bavey <[email protected]> * Shareable function for partitioning integration tests (elastic#17223) (elastic#17303) For the fedramp high work https://github.com/elastic/logstash/pull/17038/files a use case for multiple scripts consuming the partitioning functionality emerged. As we look to more advanced partitioning we want to ensure that the functionality will be consumable from multiple scripts. See elastic#17219 (comment) (cherry picked from commit d916972) Co-authored-by: Cas Donoghue <[email protected]> * [8.x] Surface failures from nested rake/shell tasks (backport elastic#17310) (elastic#17317) * Surface failures from nested rake/shell tasks (elastic#17310) Previously when rake would shell out the output would be lost. This made debugging CI logs difficult. This commit updates the stack with improved message surfacing on error. (cherry picked from commit 0d931a5) # Conflicts: # rubyUtils.gradle * Extend ruby linting tasks to handle file inputs (elastic#16660) This commit extends the gradle and rake tasks to pass through a list of files for rubocop to lint. This allows more specificity and fine grained control for linting when the consumer of the tasks only wishes to lint a select few files. * Ensure shellwords library is loaded Without this depending on task load order `Shellwords` may not be available. --------- Co-authored-by: Cas Donoghue <[email protected]> * Forward Port of Release notes for `8.16.5` and `8.17.3` (elastic#17187), (elastic#17188) (elastic#17266) (elastic#17321) * Forward Port of Release notes for 8.17.3 (elastic#17187) * Update release notes for 8.17.3 --------- Co-authored-by: logstashmachine <[email protected]> Co-authored-by: Rob Bavey <[email protected]> * Forward Port of Release notes for 8.16.5 (elastic#17188) * Update release notes for 8.16.5 --------- Co-authored-by: logstashmachine <[email protected]> Co-authored-by: Rob Bavey <[email protected]> --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: logstashmachine <[email protected]> (cherry picked from commit 63e8fd1) Co-authored-by: Rob Bavey <[email protected]> * Add Deprecation tag to arcsight module (elastic#17331) * [8.x] Upgrade elasticsearch-ruby client. (backport elastic#17161) (elastic#17306) * Upgrade elasticsearch-ruby client. (elastic#17161) * Fix Faraday removed basic auth option and apply the ES client module name change. (cherry picked from commit e748488) * Apply the required changes in elasticsearch_client.rb after upgrading the elasticsearch-ruby client to 8.x * Swallow the exception and make non-connectable client when ES client raises connection refuses exception. --------- Co-authored-by: Mashhur <[email protected]> Co-authored-by: Mashhur <[email protected]> * Removed unused configHash computation that can be replaced by PipelineConfig.configHash() (elastic#17336) (elastic#17345) Removed unused configHash computation happening in AbstractPipeline and used only in tests replaced by PipelineConfig.configHash() invocation (cherry picked from commit 787fd2c) Co-authored-by: Andrea Selva <[email protected]> * Use org.logstash.common.Util to hashing by default to SHA256 (elastic#17346) (elastic#17352) Removes the usage fo Apache Commons Codec MessgeDigest to use internal Util class with embodies hashing methods. (cherry picked from commit 9c0e50f) Co-authored-by: Andrea Selva <[email protected]> * Added test to verify the int overflow happen (elastic#17353) (elastic#17354) Use long instead of int type to keep the length of the first token. The size limit validation requires to sum two integers, one with the length of the accumulated chars till now plus the next fragment head part. If any of the two sizes is close to the max integer it generates an overflow and could successfully fail the test https://github.com/elastic/logstash/blob/9c0e50faacc4700da3dc84a3ba729b84bff860a8/logstash-core/src/main/java/org/logstash/common/BufferedTokenizerExt.java#L123. To fall in this case it's required that sizeLimit is bigger then 2^32 bytes (2GB) and data fragments without any line delimiter is pushed to the tokenizer with a total size close to 2^32 bytes. (cherry picked from commit afde43f) Co-authored-by: Andrea Selva <[email protected]> * [8.x] add ci shared qualified-version script (backport elastic#17311) (elastic#17348) * add ci shared qualified-version script (elastic#17311) * ci: add shareable script for generating qualified version * ci: use shared script to generate qualified version (cherry picked from commit 10b5a84) # Conflicts: # .buildkite/scripts/dra/build_docker.sh * resolve merge conflict --------- Co-authored-by: Rye Biesemeyer <[email protected]> * tests: make integration split quantity configurable (elastic#17219) (elastic#17367) * tests: make integration split quantity configurable Refactors shared splitter bash function to take a list of files on stdin and split into a configurable number of partitions, emitting only those from the currently-selected partition to stdout. Also refactors the only caller in the integration_tests launcher script to accept an optional partition_count parameter (defaulting to `2` for backward- compatibility), to provide the list of specs to the function's stdin, and to output relevant information about the quantity of partition splits and which was selected. * ci: run integration tests in 3 parts (cherry picked from commit 3e0f488) Co-authored-by: Rye Biesemeyer <[email protected]> * Update buildkite with new patterns from 8.x This commit updates the buildkite definitions to be compatible with the upstream 8.x branch. Specificially: - Split integration tests for fips into 3 runners. - Use the new shared bash helper for computing QUALIFIED_VERSION It also continues standardization of using a "fedrampHighMode" for indicating the tests should be running in the context of our custom image for the SRE team. * Bug fix: Actually use shared integration_tests.sh file After refactoring to use the same script, I forgot to actually use it in the buildkite definition... --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: kaisecheng <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Ry Biesemeyer <[email protected]> Co-authored-by: Mashhur <[email protected]> Co-authored-by: Andrea Selva <[email protected]> Co-authored-by: Rob Bavey <[email protected]> Co-authored-by: Mashhur <[email protected]>
… (elastic#17433) The rubocop-ast gem just introduced a new dependency on prism. - https://rubygems.org/gems/rubocop-ast/versions/1.43.0 In our install default gem rake task we are seeing issues trying to build native extensions. I see that in upstream jruby they are seeing a similar problem (at least it is the same failure mode jruby/jruby#8415 This commit pins rubocop-ast to 1.42.0 which is the last version that did not have an explicit prism dependency. (cherry picked from commit 6de59f2) Co-authored-by: Cas Donoghue <[email protected]>
* net-zero-change refactor * add logstash-filter-age to observabilitySRE artifact
This commit adds licences for bouncycastle jars that are added for the observability SRE container artifact. It re-enables the previously disabled license check and adds a new one running in fips mode.
…tic#17401) * POC for publishing observability SRE images This commit adds a step to the pull_request_pipeline buildkite definition to push a docker image to the elastic container registry. It is added here to show that we have the proper creds etc in CI to push the container where it needs to go. We will likely move this into the DRA pipeline once we are confident it is pushing to the correct place with a naming convention that works for all consumers/producers. The general idea is to build the container with our gradle task, then once we have that image we can tag it with the git sha and a "latest" identifier. This would allow consumers to choose between an exact sha for a stream like 8.19.0 or the "latest". I will also need to factor in the case where we have the tag *without* the sha postfix. Obviously we will want to fold this in to the existing DRA pipeline for building/staging images but for now it seems reasonable to handle this separately. * check variable resolution * Move POC code into DRA pipeline This commit takes the POC from the pull_request_pipeline and adds it to the DRA pipeline. Noteably, we take care to not disrupt anything about the existing DRA pipeline by making this wait until after the artifacts are published and we set a soft_fail. While this is being introduced and stabilized we want to ensure the existing DRA pipeline continues to work without interruption. As we get more stability we can look at a tigther integration. * Disambiguate architectures Eventually we will want to do proper annotations with manifests but for now just add arch to the tag. * Use docker manifest for multi-architecture builds This commit refactors the POC pipeline for pushing observabilty SRE containers to handle conflicts for tags based on target architectures. Cells with respective architectures build containers and push to the container registry with a unique identifier. Once those exist we introduce a separate step to use the docker manifest command to annotate those images such that a container client can download the correct image based on architecture. As a result for every artifact there will be 2 images pushed (one for each arch) and N manifests pushed. The manifests will handle the final naming that the consumer would expect. * Refactor docker naming scheme In order to follow more closely the existing tagging scheme this commit refactors the naming for images to include the build sha BEFORE the SNAPSHOT identifier. WHile this does not exactly follow the whole system that exists today for container images in DRA it follows a pattern that is more similar. Ideally we can iterate to fold handling of this container into DRA and in that case consumers would not need to update their patterns for identifying images. * Code review refactor Rename INCLUDE_SHA to INCLUDE_COMMIT_ID in qualified-version script. Confine use of this argument to individual invocations instead at top level in scripts. * Build observabilitySRE containers after DRA is published This gates build/push for observability SRE containers on success of DRA pipeline.
* x-pack: add fips_validation plugin to be included in fips builds The `logstash-integration-fips_validation` plugin provides no runtime pipeline plugins, but instead provides hooks to ensure that the logstash process is correctly configured for compliance with FIPS 140-3. It is installed while building the observabilitySRE artifacts. * fips validation: ensure BCFIPS,BCJSSE,SUN are first 3 security providers * remove re-injection of BCFIPS jars * Update lib/bootstrap/rubygems.rb * add integration spec for fips_validation plugin * add missing logstash_plugin helper * fixup * skip non-fips spec on fips-configured artifact, add spec details
* Improve smoke tests for observability SRE image This commit adds a new rspec test to run the observability SRE container in a docker compose network with filebeat and elasticsearch. It uses some simple test data through a pipeline with plugins we expect to be used in production. The rspec tests will ensure the test data is flowing from filebeat to logstash to elasticsearch by querying elasticsearch for expected transformed data. * REVERT ME: debug whats goig on in CI :( * Run filebeat container as root * Work around strict file ownership perms for filebeat We add the filebeat config in a volume, the permissions checks fail due test runner not being a root user. This commit disables that check in filebeat as seems to be the consensus solution online for example: https://event-driven.io/en/tricks_on_how_to_set_up_related_docker_images/ * Dynaimcally generate PKI instead of checking it in Instead of checking in PKI, dynamically generate it with gradle task for starting containers and running the tests. This improvement avoids github warning of checked in keys and avoid expiration headaches. Generation is very fast and does not add any significant overhead to test setup. * Remove use of "should" in rspec docstrings see https://github.com/rubocop/rspec-style-guide?tab=readme-ov-file#should-in-example-docstrings * Ensure permissions readable for volume Now that certs are dynamically generated, ensure they are able to be read in container * Use elasticsearch-fips image for smoke testing * Add git ignore for temp certs
… merge-feature-fedramp-high-8.x
yaauie
approved these changes
Apr 10, 2025
Member
yaauie
left a comment
yaauie
left a comment
There was a problem hiding this comment.
I've left two nits:
- one accidental change of a buildkite job name to not match its peers
- one minor improvement to pass-through the provided value of
FEDRAMP_HIGH_MODEto gradle (flexibility to future change, three fewer places to track it)
Co-authored-by: Rye Biesemeyer <[email protected]>
Co-authored-by: Rye Biesemeyer <[email protected]>
Co-authored-by: Rye Biesemeyer <[email protected]>
Co-authored-by: Rye Biesemeyer <[email protected]>
|
💚 Build Succeeded
History
|
yaauie
pushed a commit
to yaauie/logstash
that referenced
this pull request
Jul 9, 2025
This is a cherry-pick of the merge-commit from the observabilitySRE feature branch into 8.x in PR elastic#17541 (0b1d299), resolving conflicts in `docker/*` and `rakelib/artifacts.rake` by manually migrating the 8.x-style docker structure to the refactored structure present on `main`.
yaauie
pushed a commit
to yaauie/logstash
that referenced
this pull request
Jul 14, 2025
This is the CLEAN subset of a cherry-pick of the merge-commit from the observabilitySRE feature branch into 8.x in PR elastic#17541 (0b1d299), OMITTING changes to `docker/*` and `rakelib/artifacts.rake` that would conflict due to substantial refactorings on `main`.
yaauie
pushed a commit
to yaauie/logstash
that referenced
this pull request
Jul 14, 2025
This is a forward-port of _functionality_ from the observabilitySRE feature branch into 8.x in PR elastic#17541 (0b1d299), wholly re-implementing the changes in `docker/*` and `rakelib/artifacts.rake` from the 8.x-style docker structure to the refactored structure present on `main`.
yaauie
pushed a commit
to yaauie/logstash
that referenced
this pull request
Jul 15, 2025
This is the CLEAN subset of a cherry-pick of the merge-commit from the observabilitySRE feature branch into 8.x in PR elastic#17541 (0b1d299), OMITTING changes to `docker/*` and `rakelib/artifacts.rake` that would conflict due to substantial refactorings on `main`.
yaauie
pushed a commit
to yaauie/logstash
that referenced
this pull request
Jul 15, 2025
This is a forward-port of _functionality_ from the observabilitySRE feature branch into 8.x in PR elastic#17541 (0b1d299), wholly re-implementing the changes in `docker/*` and `rakelib/artifacts.rake` from the 8.x-style docker structure to the refactored structure present on `main`.
yaauie
added a commit
that referenced
this pull request
Jul 16, 2025
…in (#17785) * forward-port observabilitySRE image creation into `main` This is the CLEAN subset of a cherry-pick of the merge-commit from the observabilitySRE feature branch into 8.x in PR #17541 (0b1d299), OMITTING changes to `docker/*` and `rakelib/artifacts.rake` that would conflict due to substantial refactorings on `main`. * forward-port observabilitySRE image creation into `main` (re-implament) This is a forward-port of _functionality_ from the observabilitySRE feature branch into 8.x in PR #17541 (0b1d299), wholly re-implementing the changes in `docker/*` and `rakelib/artifacts.rake` from the 8.x-style docker structure to the refactored structure present on `main`. * Fix pull request pipeline definition for buildkite (#17552) When the fedramp high feature branch was merged into 8.x the PR pipeline accidentally duplicated the top level `steps` key. This was a mistake and is causing issues generating exhaustive test pipeline definition. This commit fixes the bug by ensuring there is a single `steps` key that defines all the steps in the pipeline. * Ensure observabilitySRE image is pushed on DRA staging (#17569) The `artifactDockerObservabilitySRE` gradle task *always* produces a tag with a `SNAPSHOT` postfix. In the staging pipeline we use the shared `qualified-version` script for determining the LS version. That script correctly handles conditionally adding a `SNAPSHOT` postfix which is important for the tagging scheme for pushing to our container registry. Given the intermediate tag produced by the gradle task is never pushed anywhere we can update the build script to ensure the "local" artifact is always referenced with the `SNAPSHOT` postfix. * Use dedicated elasticsearch image for observabilitySRE smoke testing (#17627) * Use dedicated elasticsearch image for observabilitySRE smoke testing The ES team has started publishing a purpose built image for the fedramp high project. Update our smoke test stack to use this container. * Override default entrypoint into elasticsearch container The new image does not provide the stub `/app/elasticsearch.sh` file https://github.com/elastic/elasticsearch/blob/1a1763c591c4c32bf66f0df3bce2040e8f19a1a2/distribution/docker/README.md?plain=1#L16-L19 previously available. This commit overrides the entrypoint to avoid needing that file. See: https://github.com/elastic/elasticsearch/blob/1a1763c591c4c32bf66f0df3bce2040e8f19a1a2/distribution/docker/src/docker/Dockerfile.ess#L38C5-L40C37 * Remove entrypoint workaround due to fix landing upstream * Restore code review changes (#17539) * Comment to clarify why FIPS flag is not needed for smoke tests * Use full versions of docker commands for readability * Simplify grock pattern match The grok pattern is unanchored-by-default, we don't need the leading and trailing wildcards. * Add a step to exhaustive tests for observabilitySRE accetpance testing (#17623) * Add a step to exhaustive tests for observabilitySRE accetpance testing This commit shows the proposed pattern for adding acceptance testing for the observability SRE image. This will run when exhaustive tests run. A new gradle task will hook in to rspec similar to how it is done for the smoke tests. The main difference is that instead of building a container, the latest is pulled from the container registry and run on a fips configured host VM. * WIP: Idea for how to handle multipe container configs for acceptance tests This commit shows the rough structure for how I am planning on handling docker compose networks for acceptance tests. The main idea is to use interpolation in the docker compose file to point to different configuration files for filebeat/logstash/elasticsearch. This is mainly due to the nature of these tests showing behavior when the system is and is not configured properly for FIPS. The breakdown in responsibility is: 1. Gradle handles cert generation (similar to smoke test, this avoids checking in PKI) 2. Rspec handles stopping/starting docker compose and managing environment vars for intperolation in docker compose manifests (different from smoke tests where a single static docker compose is started in gradle) 3. Rspec handles deciding when containers are ready and querying state about data flowing through the system 4. Gradle cleans up certs THis is just a rough sketch, there are still bugs to be worked out but before i get too far in to it I want to get the idea out there. * Add tests describing behavior of LS -> ES with non-fips config This commit adds a test to show that data will not flow from LS to ES when weak non fips config is used. * Use latest ES image This will be handled separately in a separate PR, but taking this commit for now on this branch. * Remove custom entrypoint from new container The latest ES images do not require this workaround. * Take up code review suggestions 1. Remove rogue character from test file causing interpreter failure 2. Split out helpers for docker compose orchestration 3. Only send a single message instead of infinite through to ES * Add full prefix name for new image * Test filebeat -> LS -> ES using fips config As described in elastic/ingest-dev#5471 this commit adds a test for filebeat sending data through logstash to elasticsearch using fips config. * Test LS wont accept input from non fips configured filebeat This test ensures logstash will not accept data from filebeat when using weak tls configuration. See elastic/ingest-dev#5472 * Fix a funny typo. Crytpo is actually kind of a funny. * Ensure we are using the purpose build ES image in testing Similar to #17627 * Ensure JAVA_HOME is set etc Use the same buildkite agent script for setting up a vm based runner as other pipes --------- Co-authored-by: Cas Donoghue <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Release notes
[rn:skip]
What does this PR do?
Merge the feature branch for building the observabilitySRE aftifacts for internal distribution into 8.x. This will allow artifacts to be built/promoted at the same cadence as 8.x DRA artifacts.
Why is it important/What is the impact to the user?
N/A
Checklist