nutanix upgrade 4.18 -> 4.19 #65740

skordas · 2025-06-05T16:32:20Z

https://issues.redhat.com/browse/OCPQE-28934

skordas · 2025-06-05T16:40:16Z

/pj-rehearse pull-ci-openshift-eng-ocp-qe-perfscale-ci-main-nutanix-4.19-nightly-x86-loaded-upgrade-from-4.18-loaded-upgrade-418to419-24nodes

openshift-ci-robot · 2025-06-05T16:40:19Z

@skordas: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

skordas · 2025-06-05T18:38:36Z

/pj-rehearse pull-ci-openshift-eng-ocp-qe-perfscale-ci-main-nutanix-4.19-nightly-x86-loaded-upgrade-from-4.18-loaded-upgrade-418to419-24nodes

openshift-ci-robot · 2025-06-05T18:38:39Z

@skordas: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

skordas · 2025-06-05T19:21:31Z

/pj-rehearse pull-ci-openshift-eng-ocp-qe-perfscale-ci-main-nutanix-4.19-nightly-x86-loaded-upgrade-from-4.18-loaded-upgrade-418to419-24nodes

openshift-ci-robot · 2025-06-05T19:21:34Z

@skordas: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

mehabhalodiya

Could you try re-running with the suggested changes, and we will check if the error persists? I think the same worked for an IBM profile, so let's see how it goes with a Nutanix profile.

...enshift-eng-ocp-qe-perfscale-ci-main__nutanix-4.19-nightly-x86-loaded-upgrade-from-4.18.yaml

mehabhalodiya · 2025-06-06T02:55:00Z

...enshift-eng-ocp-qe-perfscale-ci-main__nutanix-4.19-nightly-x86-loaded-upgrade-from-4.18.yaml

+      memory: 200Mi
+tests:
+- as: loaded-upgrade-418to419-24nodes
+  cluster: build01


I think we don't need this cluster field.

...enshift-eng-ocp-qe-perfscale-ci-main__nutanix-4.19-nightly-x86-loaded-upgrade-from-4.18.yaml

skordas · 2025-06-06T15:42:06Z

/pj-rehearse pull-ci-openshift-eng-ocp-qe-perfscale-ci-main-nutanix-4.19-nightly-x86-loaded-upgrade-from-4.18-loaded-upgrade-418to419-24nodes

openshift-ci-robot · 2025-06-06T15:42:08Z

@skordas: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

skordas · 2025-06-09T17:26:53Z

/pj-rehearse pull-ci-openshift-eng-ocp-qe-perfscale-ci-main-nutanix-4.19-nightly-x86-loaded-upgrade-from-4.18-loaded-upgrade-418to419-24nodes

openshift-ci-robot · 2025-06-09T17:26:56Z

@skordas: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

skordas · 2025-06-11T20:01:14Z

/pj-rehearse pull-ci-openshift-eng-ocp-qe-perfscale-ci-main-nutanix-4.19-nightly-x86-loaded-upgrade-from-4.18-loaded-upgrade-418to419-24nodes

openshift-ci-robot · 2025-06-11T20:01:16Z

@skordas: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

skordas · 2025-06-12T11:33:49Z

/pj-rehearse pull-ci-openshift-eng-ocp-qe-perfscale-ci-main-nutanix-4.19-nightly-x86-loaded-upgrade-from-4.18-loaded-upgrade-418to419-24nodes

openshift-ci-robot · 2025-06-12T11:33:52Z

@skordas: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

skordas · 2025-06-12T19:23:48Z

/pj-rehearse pull-ci-openshift-eng-ocp-qe-perfscale-ci-main-nutanix-4.19-nightly-x86-loaded-upgrade-from-4.18-loaded-upgrade-418to419-24nodes

openshift-ci-robot · 2025-06-12T19:23:51Z

@skordas: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

skordas · 2025-06-13T15:33:05Z

/pj-rehearse pull-ci-openshift-eng-ocp-qe-perfscale-ci-main-nutanix-4.19-nightly-x86-loaded-upgrade-from-4.18-loaded-upgrade-418to419-24nodes

openshift-ci-robot · 2025-06-13T15:33:07Z

@skordas: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

skordas · 2025-06-13T19:02:31Z

/pj-rehearse pull-ci-openshift-eng-ocp-qe-perfscale-ci-main-nutanix-4.19-nightly-x86-loaded-upgrade-from-4.18-loaded-upgrade-418to419-24nodes

openshift-ci-robot · 2025-06-13T19:02:37Z

@skordas: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

skordas · 2025-06-13T19:35:55Z

/pj-rehearse pull-ci-openshift-eng-ocp-qe-perfscale-ci-main-nutanix-4.19-nightly-x86-loaded-upgrade-from-4.18-loaded-upgrade-418to419-24nodes

openshift-ci-robot · 2025-06-13T19:35:58Z

@skordas: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

skordas · 2025-06-18T14:26:48Z

/pj-rehearse pull-ci-openshift-eng-ocp-qe-perfscale-ci-main-nutanix-4.19-nightly-x86-loaded-upgrade-from-4.18-loaded-upgrade-418to419-24nodes

openshift-ci-robot · 2025-06-18T14:26:51Z

@skordas: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

skordas · 2025-06-18T19:31:55Z

/pj-rehearse pull-ci-openshift-eng-ocp-qe-perfscale-ci-main-nutanix-4.19-nightly-x86-loaded-upgrade-from-4.18-loaded-upgrade-418to419-24nodes

openshift-ci-robot · 2025-06-18T19:31:57Z

@skordas: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

mehabhalodiya · 2025-06-19T10:31:13Z

...enshift-eng-ocp-qe-perfscale-ci-main__nutanix-4.19-nightly-x86-loaded-upgrade-from-4.18.yaml

+    env:
+      COMPUTE_CPU: "8"
+      COMPUTE_MEMORY: "32000"
+      COMPUTE_REPLICAS: "3"


Please add GC: "false" as we want to keep the workload during the upgrade.

Also, maybe you can add these params and try

OPENSHIFT_INFRA_NODE_INSTANCE_MEMORYSIZE: 64Gi OPENSHIFT_INFRA_NODE_INSTANCE_VCPU: "16" SET_ENV_BY_PLATFORM: custom

Because I see the error

Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedCreate 136m nutanixcontroller ci-op-gmvftkmm-53c18-vs8np-worker-1-77gnn: reconciler failed to Create machine: failed to create VM: ci-op-gmvftkmm-53c18-vs8np-worker-1-77gnn failed to create the vm: error_detail: INVALID_ARGUMENT: Invalid Argument: 6 :No host has enough available resources for VM bd737d9f-14b3-47d2-9528-7ec3eaa8622f., progress_message: create_vm Warning FailedUpdate 98s (x273 over 135m) nutanixcontroller ci-op-gmvftkmm-53c18-vs8np-worker-1-77gnn: reconciler failed to Update machine: The retrieved VM "ci-op-gmvftkmm-53c18-vs8np-worker-1-77gnn" has ERROR state. error: [{"message": "Invalid Argument: 6\n :No host has enough available resources for VM bd737d9f-14b3-47d2-9528-7ec3eaa8622f.", "reason": "INVALID_ARGUMENT"}] error: all 24 nodes didn't become READY in time, failing

@mehabhalodiya
This test is very unstable - every time is fails on different step. Last time was with no enough resources for scaling from 3 to 24 - but that is issue of not enough resources. - not of step.
I added GC variable, but the rest are for additional infra nodes - I'm not adding this for this test (test configuration will not pass make jobs make update verification.

mehabhalodiya · 2025-06-19T10:32:07Z

...enshift-eng-ocp-qe-perfscale-ci-main__nutanix-4.19-nightly-x86-loaded-upgrade-from-4.18.yaml

+    - ref: workers-scale
+    - chain: openshift-qe-cluster-density-v2
+    - chain: openshift-upgrade-qe-sanity
+    - ref: openshift-qe-connectivity-check


You can remove - ref: openshift-qe-connectivity-check, as it is not required; this step is typically used in an IPsec cluster.

Removed this step as suggested.

This reverts commit bad7647.

openshift-ci-robot · 2025-06-20T19:41:14Z

[REHEARSALNOTIFIER]
@skordas: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name	Repo	Type	Reason
pull-ci-openshift-eng-ocp-qe-perfscale-ci-main-nutanix-4.19-nightly-x86-loaded-upgrade-from-4.18-loaded-upgrade-418to419-24nodes	openshift-eng/ocp-qe-perfscale-ci	presubmit	Presubmit changed

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

skordas · 2025-06-20T19:50:34Z

/pj-rehearse pull-ci-openshift-eng-ocp-qe-perfscale-ci-main-nutanix-4.19-nightly-x86-loaded-upgrade-from-4.18-loaded-upgrade-418to419-24nodes

openshift-ci-robot · 2025-06-20T19:50:36Z

@skordas: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

mehabhalodiya · 2025-06-21T02:37:29Z

Now it's passing: https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_release/65740/rehearse-65740-pull-ci-openshift-eng-ocp-qe-perfscale-ci-main-nutanix-4.19-nightly-x86-loaded-upgrade-from-4.18-loaded-upgrade-418to419-24nodes/1936150509450694656 :)

mehabhalodiya

/lgtm

Thank you!

openshift-ci · 2025-06-23T04:55:23Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mehabhalodiya, skordas

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~ci-operator/config/openshift-eng/ocp-qe-perfscale-ci/OWNERS~~ [skordas]
~~ci-operator/jobs/openshift-eng/ocp-qe-perfscale-ci/OWNERS~~ [skordas]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

mehabhalodiya · 2025-06-23T04:59:04Z

/pj-rehearse ack

openshift-ci-robot · 2025-06-23T04:59:06Z

@mehabhalodiya: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

openshift-ci · 2025-06-23T05:20:41Z

@skordas: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

* nutanix upgrade 4.18 -> 4.19 * Mulitzone istallation * New flow for upgrade * Debugging - something is wrong here * Revert "Debugging - something is wrong here" This reverts commit bad7647. * adding GC variable to test, removing step * fix for metadata

openshift-ci bot requested review from mehabhalodiya and SachinNinganure June 5, 2025 16:35

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 5, 2025

mehabhalodiya suggested changes Jun 6, 2025

View reviewed changes

openshift-ci bot assigned mehabhalodiya Jun 6, 2025

skordas force-pushed the nutanix418-419 branch from c30ca19 to 3b70ac6 Compare June 6, 2025 15:36

skordas force-pushed the nutanix418-419 branch from 3b70ac6 to 1aca6d4 Compare June 13, 2025 18:55

openshift-ci bot removed the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 13, 2025

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 13, 2025

mehabhalodiya suggested changes Jun 19, 2025

View reviewed changes

skordas added 5 commits June 20, 2025 15:19

nutanix upgrade 4.18 -> 4.19

473fa33

Mulitzone istallation

0d2518c

New flow for upgrade

1de2056

Debugging - something is wrong here

e8120e6

Revert "Debugging - something is wrong here"

b199678

This reverts commit bad7647.

skordas force-pushed the nutanix418-419 branch from 6ca8e92 to b199678 Compare June 20, 2025 19:20

skordas added 2 commits June 20, 2025 15:29

adding GC variable to test, removing step

23206c7

fix for metadata

e7b4531

mehabhalodiya approved these changes Jun 23, 2025

View reviewed changes

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jun 23, 2025

openshift-ci-robot added the rehearsals-ack Signifies that rehearsal jobs have been acknowledged label Jun 23, 2025

openshift-merge-bot bot merged commit a889caf into openshift:master Jun 23, 2025
16 checks passed

nutanix upgrade 4.18 -> 4.19 #65740

nutanix upgrade 4.18 -> 4.19 #65740

Uh oh!

Conversation

skordas commented Jun 5, 2025

Uh oh!

skordas commented Jun 5, 2025

Uh oh!

openshift-ci-robot commented Jun 5, 2025

Uh oh!

skordas commented Jun 5, 2025

Uh oh!

openshift-ci-robot commented Jun 5, 2025

Uh oh!

skordas commented Jun 5, 2025

Uh oh!

openshift-ci-robot commented Jun 5, 2025

Uh oh!

mehabhalodiya left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mehabhalodiya Jun 6, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

skordas commented Jun 6, 2025

Uh oh!

openshift-ci-robot commented Jun 6, 2025

Uh oh!

skordas commented Jun 9, 2025

Uh oh!

openshift-ci-robot commented Jun 9, 2025

Uh oh!

skordas commented Jun 11, 2025

Uh oh!

openshift-ci-robot commented Jun 11, 2025

Uh oh!

skordas commented Jun 12, 2025

Uh oh!

openshift-ci-robot commented Jun 12, 2025

Uh oh!

skordas commented Jun 12, 2025

Uh oh!

openshift-ci-robot commented Jun 12, 2025

Uh oh!

skordas commented Jun 13, 2025

Uh oh!

openshift-ci-robot commented Jun 13, 2025

Uh oh!

skordas commented Jun 13, 2025

Uh oh!

openshift-ci-robot commented Jun 13, 2025

Uh oh!

skordas commented Jun 13, 2025

Uh oh!

openshift-ci-robot commented Jun 13, 2025

Uh oh!

skordas commented Jun 18, 2025

Uh oh!

openshift-ci-robot commented Jun 18, 2025

Uh oh!

skordas commented Jun 18, 2025

Uh oh!

openshift-ci-robot commented Jun 18, 2025

Uh oh!

mehabhalodiya Jun 19, 2025

Choose a reason for hiding this comment

Uh oh!

skordas Jun 20, 2025

Choose a reason for hiding this comment

Uh oh!

mehabhalodiya Jun 19, 2025

Choose a reason for hiding this comment

Uh oh!

skordas Jun 20, 2025

Choose a reason for hiding this comment

Uh oh!

openshift-ci-robot commented Jun 20, 2025

Uh oh!