Improve Integration test documentation (#10155)

Author: belimawr

docs/test-framework-dev-guide.md

@@ -16,13 +16,27 @@ provides a high level overview of the testing framework.
 Go version should be at least the same than the one in [.go-version](https://github.com/elastic/elastic-agent/blob/main/.go-version) file at the root of this repository.
 
 
-### GCloud CLI
+#### GCloud CLI
 The integration testing framework spins up resources in GCP.  To achieve this, it needs the
 [GCloud CLI](https://cloud.google.com/sdk/gcloud) to be installed on the system where the tests are initiated from.
 
-### Beats
+#### Beats
 The Elastic Agent package that is used for integration tests packages Beats built from the Unified Release (as opposed to DRA).  There is no explicit action needed for this prerequisite but just keep in mind that if any Agent integration tests rely on certain Beats features or bugfixes, they may not be available in the integration tests yet because a unified release containing those features or bugfixes may not have happened yet.
 
+#### Helm & Helm charts
+To run the Kubernets integration tests you need to install
+[helm](https://helm.sh/). Then add and update some helm charts
+repositories:
+
+```
+helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
+helm repo update
+```
+
+Make sure to run `helm repo update` before running K8s integration
+tests otherwise the tests will fail stating a chart/version cannot be
+found.
+
 ### Configuration
 
 ESS (production) API Key to create on <https://cloud.elastic.co/account/keys>
@@ -39,6 +53,7 @@ If necessary, you can create a new serverless deployment manually; the previous
 
 Credentials for these deployments are securely stored in Google and can only be accessed by Buildkite pipelines. The access control is set using [OpenID Connect in Google Cloud Platform](https://docs.github.com/en/actions/security-for-github-actions/security-hardening-your-deployments/configuring-openid-connect-in-google-cloud-platform). And that's managed by the Robots team.
 
+
 ## Running tests
 
 Some integration and E2E tests are safe to run locally. These tests set
@@ -60,9 +75,100 @@ Go to https://docker-auth.elastic.co/ and authenticate with Okta to receive your
 
 [elastic_docker_registry]: docker.elastic.co
 
+### Packaging
+Before you can run any test, you first need to package the Elastic
+Agent version you want to test. For that you'll need to run
+
+```
+DEV=true SNAPSHOT=true EXTERNAL=true PACKAGES="tar.gz,deb,rpm" PLATFORMS=linux/amd64 mage -v package
+```
+
+The packaging process has many leavers that need to be correctly set:
+
+ - `DEV=true|false`: Build with debug symbols
+ - `EXTERNAL=true|false`: If `false` it will look for the `beats`
+   folder at the same level as `elastic-agent` and build
+   `x-pack/agentbeat` from there. Be aware that if you change Beats
+   code and there is already an Agentbeat artifact built, it will not
+   be re-build. You need to manually clean the Agentbeat build
+   folder. If `true` it will download the latest snapshot.
+ - `SNAPSHOT=true|false`: When downloading dependencies (like Beats)
+   use snapshot versions. That is required to package from the `main`
+   branch with `EXTERNAL=true`.
+ - `PLATFORMS`: Comma separated list of platforms you want to build. If
+  not set, it defaults to **ALL** platforms. [Selecting specific
+  platform](#selecting-specific-platform) contains a list of common
+  values. For a full list look at [`elastic-agent/dev-tools/mage/platforms.go`](https://github.com/elastic/elastic-agent/blob/main/dev-tools/mage/platforms.go#L15).
+ - `PACKAGES`: Comma separated list of packages you want to build. If
+  not set, it defaults to **ALL** packages, which takes a long time,
+  so make sure to *always* set it correctly. The packages are defined
+  on
+  [`elastic-agent/dev-tools/mage/pkgtypes.go`](https://github.com/elastic/elastic-agent/blob/main/dev-tools/mage/pkgtypes.go#L72).
+  To run Linux tests you need to package `tar.gz` ,`deb` and `rpm`.
+The possible options are:
+    - `tar.gz`
+    - `zip` (Windows only)
+    - `deb`
+    - `rpm`
+    - `docker`
+ - `DOCKER_VARIANTS`: When building Docker images, which variants to
+ build. The variants are defined on
+ [`elastic-agent/dev-tools/mage/dockervariants.go`](https://github.com/elastic/elastic-agent/blob/9f6c6a79f9e19ec2090d8f7da184280412cb8663/dev-tools/mage/dockervariants.go#L30),
+ possible values are:
+    - `basic`
+    - `ubi`
+    - `wolfi`
+    - `complete`
+    - `complete-wolfi`
+    - `cloud`
+    - `service`
+    - `elastic-otel-collector`
+    - `slim`
+    - `elastic-otel-collector-wolfi`
+    - `slim-wolfi`
+
 ### Running the tests
 
-The test are run with mage using the `integration` namespace:
+The test are run with mage using the `integration` namespace, they
+share similar leavers as the packaging process.
+
+ - `AGENT_VERSION`: The version to test, in the format
+   `9.2.0-SNAPSHOT`. It is **REQUIRED** to be set if you built the
+   packages with `SNAPTHOST=true`. Currently there is no way to tell
+   the integration tests framework to use snapshot versions if testing
+   on VMs.
+
+ - `SNAPSHOT=true|false`: Use snapshot build when running Kubernetes
+   tests.
+
+ - `INSTANCE_PROVISIONER`: Sets the provisioner used to create
+   instances, possible values are:
+     - `ogc`: Uses OGC to create VMs on GCP, if not set, that's the default.
+     - `multipass`: Uses [Multipass](https://canonical.com/multipass) to
+       create local VMs.
+     - `kind`: Uses [Kind](https://kind.sigs.k8s.io/) to run Kubernetes
+       in Docker. This needs to be set if running Kubernetes integration
+       tests.
+
+
+### TL;DR: Packaging and running tests
+**Package the Elastic Agent**
+```
+# If testing on VMs.
+DEV=true SNAPSHOT=true EXTERNAL=true PACKAGES="tar.gz,deb,rpm" PLATFORMS=linux/amd64 mage -v package
+
+# If running Kubernetes tests. Adjust the variants according to your tests
+DEV=true SNAPSHOT=true EXTERNAL=true PACKAGES="tar.gz,deb,rpm" DOCKER_VARIANTS="basic,complete,elastic-otel-collector" PLATFORMS=linux/amd64 mage -v package
+```
+
+**Run the tests**
+```
+# If testing on VMs. Adjust the AGENT_VERSION according to your build.
+AGENT_VERSION="9.2.0-SNAPSHOT" mage -v integration:single TestLogIngestionFleetManaged
+
+# If testing on Kubernetes
+SNAPSHOT=true INSTANCE_PROVISIONER=kind mage -v integration:testKubernetesSingle TestKubernetesJournaldInput
+```
 
 #### ESS oriented tests
 
@@ -80,11 +186,11 @@ The test are run with mage using the `integration` namespace:
 
 #### Kubernetes oriented tests
 
-- `mage integration:testKubernetes` to run kubernetes tests under the `testing/integration/k8s` folder for the default image on the default version of kubernetes (all previous commands will not run any kubernetes tests).
+- `INSTANCE_PROVISIONER=kind mage integration:testKubernetes` to run kubernetes tests under the `testing/integration/k8s` folder for the default image on the default version of kubernetes (all previous commands will not run any kubernetes tests).
 
-- `mage integration:testKubernetesMatrix` to run a matrix of kubernetes tests under the `testing/integration/k8s` folder for all image types and supported versions of kubernetes.
+- `INSTANCE_PROVISIONER=kind mage integration:testKubernetesMatrix` to run a matrix of kubernetes tests under the `testing/integration/k8s` folder for all image types and supported versions of kubernetes.
 
-- `mage integration:testKubernetesSingle [testName|all]` to execute a single test under the `testing/integration/k8s` folder. Only the selected test will be executed.
+- `INSTANCE_PROVISIONER=kind mage integration:testKubernetesSingle [testName|all]` to execute a single test under the `testing/integration/k8s` folder. Only the selected test will be executed.
 
 #### Serverless oriented tests
 
@@ -100,7 +206,7 @@ The test are run with mage using the `integration` namespace:
 
 You can list all available mage targets by running `mage -l`
 
-#### Selecting specific platform
+#### Selecting specific platform<a name="selecting-specific-platform"></a>
 
 By default, the runner will deploy to every combination of operating system and architecture that the tests define
 as supporting. When working on tests and debugging an issue it's better to limit the operating system and architecture
@@ -493,3 +599,15 @@ using a separate env variable `AGENT_STACK_VERSION` like in this example (we use
 custom package version for the agent):
 
 ```AGENT_VERSION="8.10.0-testpkgversion.1-SNAPSHOT" AGENT_STACK_VERSION="8.10.0-SNAPSHOT" mage integration:test```
+
+### K8s: pod not read (`timeout waiting for k8s object elastic-agent-standalone-xxxxx to be ready`/`client rate limiter Wait returned an error: context deadline exceeded`)
+Your Kind cluster might be in a broken state. Run:
+
+```
+% kubectl get nodes
+NAME                               STATUS     ROLES           AGE   VERSION
+v1.33.0-kubernetes-control-plane   NotReady   control-plane   58m   v1.33.0
+```
+
+And check the status of your node. If it is different than `Ready`,
+delete the cluster and create a new one.