Skip to content

Improve Integration test documentation #10155

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Sep 25, 2025
+125 −7
Merged

Improve Integration test documentation #10155

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
f57f460
[WIP]
belimawr Sep 24, 2025
e4dbf15
Update docs
belimawr Sep 25, 2025
bf852fd
revert magefile changes
belimawr Sep 25, 2025
4d2976d
remove code changes
belimawr Sep 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
132 changes: 125 additions & 7 deletions docs/test-framework-dev-guide.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,13 +16,27 @@ provides a high level overview of the testing framework.
Go version should be at least the same than the one in [.go-version](https://github.com/elastic/elastic-agent/blob/main/.go-version) file at the root of this repository.


### GCloud CLI
#### GCloud CLI
The integration testing framework spins up resources in GCP. To achieve this, it needs the
[GCloud CLI](https://cloud.google.com/sdk/gcloud) to be installed on the system where the tests are initiated from.

### Beats
#### Beats
The Elastic Agent package that is used for integration tests packages Beats built from the Unified Release (as opposed to DRA). There is no explicit action needed for this prerequisite but just keep in mind that if any Agent integration tests rely on certain Beats features or bugfixes, they may not be available in the integration tests yet because a unified release containing those features or bugfixes may not have happened yet.

#### Helm & Helm charts
To run the Kubernets integration tests you need to install
[helm](https://helm.sh/). Then add and update some helm charts
repositories:

```
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
```

Make sure to run `helm repo update` before running K8s integration
tests otherwise the tests will fail stating a chart/version cannot be
found.

### Configuration

ESS (production) API Key to create on <https://cloud.elastic.co/account/keys>
Expand All @@ -39,6 +53,7 @@ If necessary, you can create a new serverless deployment manually; the previous

Credentials for these deployments are securely stored in Google and can only be accessed by Buildkite pipelines. The access control is set using [OpenID Connect in Google Cloud Platform](https://docs.github.com/en/actions/security-for-github-actions/security-hardening-your-deployments/configuring-openid-connect-in-google-cloud-platform). And that's managed by the Robots team.


## Running tests

Some integration and E2E tests are safe to run locally. These tests set
Expand All @@ -60,9 +75,100 @@ Go to https://docker-auth.elastic.co/ and authenticate with Okta to receive your

[elastic_docker_registry]: docker.elastic.co

### Packaging
Before you can run any test, you first need to package the Elastic
Agent version you want to test. For that you'll need to run

```
DEV=true SNAPSHOT=true EXTERNAL=true PACKAGES="tar.gz,deb,rpm" PLATFORMS=linux/amd64 mage -v package
```

The packaging process has many leavers that need to be correctly set:

- `DEV=true|false`: Build with debug symbols
- `EXTERNAL=true|false`: If `false` it will look for the `beats`
folder at the same level as `elastic-agent` and build
`x-pack/agentbeat` from there. Be aware that if you change Beats
code and there is already an Agentbeat artifact built, it will not
be re-build. You need to manually clean the Agentbeat build
folder. If `true` it will download the latest snapshot.
- `SNAPSHOT=true|false`: When downloading dependencies (like Beats)
use snapshot versions. That is required to package from the `main`
branch with `EXTERNAL=true`.
- `PLATFORMS`: Comma separated list of platforms you want to build. If
not set, it defaults to **ALL** platforms. [Selecting specific
platform](#selecting-specific-platform) contains a list of common
values. For a full list look at [`elastic-agent/dev-tools/mage/platforms.go`](https://github.com/elastic/elastic-agent/blob/main/dev-tools/mage/platforms.go#L15).
- `PACKAGES`: Comma separated list of packages you want to build. If
not set, it defaults to **ALL** packages, which takes a long time,
so make sure to *always* set it correctly. The packages are defined
on
[`elastic-agent/dev-tools/mage/pkgtypes.go`](https://github.com/elastic/elastic-agent/blob/main/dev-tools/mage/pkgtypes.go#L72).
To run Linux tests you need to package `tar.gz` ,`deb` and `rpm`.
The possible options are:
- `tar.gz`
- `zip` (Windows only)
- `deb`
- `rpm`
- `docker`
- `DOCKER_VARIANTS`: When building Docker images, which variants to
build. The variants are defined on
[`elastic-agent/dev-tools/mage/dockervariants.go`](https://github.com/elastic/elastic-agent/blob/9f6c6a79f9e19ec2090d8f7da184280412cb8663/dev-tools/mage/dockervariants.go#L30),
possible values are:
- `basic`
- `ubi`
- `wolfi`
- `complete`
- `complete-wolfi`
- `cloud`
- `service`
- `elastic-otel-collector`
- `slim`
- `elastic-otel-collector-wolfi`
- `slim-wolfi`

### Running the tests

The test are run with mage using the `integration` namespace:
The test are run with mage using the `integration` namespace, they
share similar leavers as the packaging process.

- `AGENT_VERSION`: The version to test, in the format
`9.2.0-SNAPSHOT`. It is **REQUIRED** to be set if you built the
packages with `SNAPTHOST=true`. Currently there is no way to tell
the integration tests framework to use snapshot versions if testing
on VMs.

- `SNAPSHOT=true|false`: Use snapshot build when running Kubernetes
tests.

- `INSTANCE_PROVISIONER`: Sets the provisioner used to create
instances, possible values are:
- `ogc`: Uses OGC to create VMs on GCP, if not set, that's the default.
- `multipass`: Uses [Multipass](https://canonical.com/multipass) to
create local VMs.
- `kind`: Uses [Kind](https://kind.sigs.k8s.io/) to run Kubernetes
in Docker. This needs to be set if running Kubernetes integration
tests.


### TL;DR: Packaging and running tests
**Package the Elastic Agent**
```
# If testing on VMs.
DEV=true SNAPSHOT=true EXTERNAL=true PACKAGES="tar.gz,deb,rpm" PLATFORMS=linux/amd64 mage -v package

# If running Kubernetes tests. Adjust the variants according to your tests
DEV=true SNAPSHOT=true EXTERNAL=true PACKAGES="tar.gz,deb,rpm" DOCKER_VARIANTS="basic,complete,elastic-otel-collector" PLATFORMS=linux/amd64 mage -v package
```

**Run the tests**
```
# If testing on VMs. Adjust the AGENT_VERSION according to your build.
AGENT_VERSION="9.2.0-SNAPSHOT" mage -v integration:single TestLogIngestionFleetManaged

# If testing on Kubernetes
SNAPSHOT=true INSTANCE_PROVISIONER=kind mage -v integration:testKubernetesSingle TestKubernetesJournaldInput
```

#### ESS oriented tests

Expand All @@ -80,11 +186,11 @@ The test are run with mage using the `integration` namespace:

#### Kubernetes oriented tests

- `mage integration:testKubernetes` to run kubernetes tests under the `testing/integration/k8s` folder for the default image on the default version of kubernetes (all previous commands will not run any kubernetes tests).
- `INSTANCE_PROVISIONER=kind mage integration:testKubernetes` to run kubernetes tests under the `testing/integration/k8s` folder for the default image on the default version of kubernetes (all previous commands will not run any kubernetes tests).

- `mage integration:testKubernetesMatrix` to run a matrix of kubernetes tests under the `testing/integration/k8s` folder for all image types and supported versions of kubernetes.
- `INSTANCE_PROVISIONER=kind mage integration:testKubernetesMatrix` to run a matrix of kubernetes tests under the `testing/integration/k8s` folder for all image types and supported versions of kubernetes.

- `mage integration:testKubernetesSingle [testName|all]` to execute a single test under the `testing/integration/k8s` folder. Only the selected test will be executed.
- `INSTANCE_PROVISIONER=kind mage integration:testKubernetesSingle [testName|all]` to execute a single test under the `testing/integration/k8s` folder. Only the selected test will be executed.

#### Serverless oriented tests

Expand All @@ -100,7 +206,7 @@ The test are run with mage using the `integration` namespace:

You can list all available mage targets by running `mage -l`

#### Selecting specific platform
#### Selecting specific platform<a name="selecting-specific-platform"></a>

By default, the runner will deploy to every combination of operating system and architecture that the tests define
as supporting. When working on tests and debugging an issue it's better to limit the operating system and architecture
Expand Down Expand Up @@ -493,3 +599,15 @@ using a separate env variable `AGENT_STACK_VERSION` like in this example (we use
custom package version for the agent):

```AGENT_VERSION="8.10.0-testpkgversion.1-SNAPSHOT" AGENT_STACK_VERSION="8.10.0-SNAPSHOT" mage integration:test```

### K8s: pod not read (`timeout waiting for k8s object elastic-agent-standalone-xxxxx to be ready`/`client rate limiter Wait returned an error: context deadline exceeded`)
Your Kind cluster might be in a broken state. Run:

```
% kubectl get nodes
NAME STATUS ROLES AGE VERSION
v1.33.0-kubernetes-control-plane NotReady control-plane 58m v1.33.0
```

And check the status of your node. If it is different than `Ready`,
delete the cluster and create a new one.
Loading