Split fips integration tests into two steps (elastic#17038)

* Split fips integration tests into two steps

The integration tests suite takes about 40 minutes. This is far too slow for
reasonable feedback on a PR. This commit follows the pattern for the non-fips
integration tests whereby the tests are split into two sections that can run in
parallel across two steps. This should halve the feedback time.

The logic for getting a list of specs files to run has been extracted to a
shared shell script for use here and in the integration tests shell script.

* Use shared function for splitting integration tests

The logic for getting a list of specs to run has been extracted so that it can
be shared across fips and non fips integration test modes. This commit updates
the non fips integration tests to use the shared function.

* fix typo in helper name (kebab case, not snake)

* Escape $ so buildkite upload does not try to interpolate

* Wrap integration tests in shell script to avoid BK interpolation

* Move entrypoint for running integration tests inside docker

Author: donoghuc

.buildkite/pull_request_pipeline.yml

@@ -125,8 +125,8 @@ steps:
       manual:
         allowed: true
 
-  - label: ":lab_coat: Integration Tests - FIPS mode"
-    key: "integration-tests-fips"
+  - label: ":lab_coat: Integration Tests - FIPS mode / part 1"
+    key: "integration-tests-fips-part-1"
     agents:
       provider: gcp
       imageProject: elastic-images-prod
@@ -141,7 +141,25 @@ steps:
       set -euo pipefail
 
       docker build -t test-runner-image -f x-pack/distributions/internal/observabilitySRE/docker/Dockerfile .
-      docker run test-runner-image ./gradlew --info --stacktrace -PrunTestsInFIPSMode=true runIntegrationTests
+      docker run test-runner-image ci/run-fips-integration-tests.sh 0
+
+  - label: ":lab_coat: Integration Tests - FIPS mode / part 2"
+    key: "integration-tests-fips-part-2"
+    agents:
+      provider: gcp
+      imageProject: elastic-images-prod
+      image: family/platform-ingest-logstash-ubuntu-2204
+      machineType: "n2-standard-4"
+      diskSizeGb: 64
+    retry:
+      automatic:
+      # dont retry on failure while they are expected
+        - limit: 0
+    command: |
+      set -euo pipefail
+
+      docker build -t test-runner-image -f x-pack/distributions/internal/observabilitySRE/docker/Dockerfile .
+      docker run test-runner-image ci/run-fips-integration-tests.sh 1
 
   - label: ":lab_coat: Integration Tests / part 1"
     key: "integration-tests-part-1"

ci/run-fips-integration-tests.sh

@@ -0,0 +1,9 @@
+#!/bin/bash
+# get_test_half returns either the first or second half of integration tests
+# Usage: get_test_half <half_number>
+# half_number: 0 for first half, 1 for second half
+
+half_number=$1
+source ci/get-test-half.sh
+specs=$(get_test_half "$half_number")
+./gradlew --info --stacktrace -PrunTestsInFIPSMode=true runIntegrationTests -PrubyIntegrationSpecs="$specs"