PodGroup isn't triggering scaling up in Kubernetes, when using Cluster Autoscaler

[JgardemeRelex]

What happened:
PodGroup which isn't fitting to the current resource capacity of Kubernetes won't trigger scale-up for node pool, even if Cluster Autoscaler is enabled in Kubernetes.
What you expected to happen:
I except that Kubernetes Cluster Autoscaler will detect the increased workload, and then trigger the scaling up.
How to reproduce it (as minimally and precisely as possible):
Apply a PodGroup which isn't fitting the current resource capacity of node pool.

Anything else we need to know?:
Microsoft has decided to use Volcano in their Azure ML platform for scheduling training jobs in Kubernetes, so this will an issue with lot of people in the future.
Environment:

• Kubernetes version (use kubectl version): v1.22.15

[nature1995]

+1

[william-wang]

@JgardemeRelex Would you clarify this case more detail? We can have an investigation.
As far as I know, currently, there two features are not working with cluster autoscaler. They are the pod delay creation and gang.

[Wennn]

Hi @william-wang , we have met a similar issue. I created a job with 3 pods, and it should trigger the cluster to scale up three more nodes. We have set the overcommit factor to a large number, so the pods can be created when using the gang plugin. From the events, auto scaler is only triggered by one of the pods, so at last only one node are created, the job is still stuck at pending and not triggering more nodes.

Could you briefly explain why gang is not working with autoscaler? Thanks.

[JgardemeRelex]

@william-wang My issue is also related to the gang plugin. In my case, PodGroup or pods aren't triggering scale up at all with Cluster Autoscaler. I created gang job with 3 pods, and 2 of them are fitted into the current node pools, but the third one isn't. Here is the PodGroup yaml, what it looks like in the Kubernetes:

`

Name:         azureml-dev-78c8878c58fb623e0c3bf0ccfa3cc834
Namespace:    azureml-dev
Labels:       jobId=*
Annotations:  volcano.sh/task-topology-affinity: worker
API Version:  scheduling.volcano.sh/v1beta1
Kind:         PodGroup
Metadata:
  Creation Timestamp:  2022-11-03T13:06:56Z
  Generation:          8
  Managed Fields:
    API Version:  scheduling.volcano.sh/v1beta1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:volcano.sh/task-topology-affinity:
        f:labels:
          .:
          f:jobId:
        f:ownerReferences:
          .:
          k:{"uid":"0073953d-5684-4a72-90e5-3f528890db26"}:
      f:spec:
        .:
        f:minMember:
        f:minResources:
          .:
          f:cpu:
          f:memory:
        f:minTaskMember:
          .:
          f:worker:
        f:queue:
      f:status:
    Manager:      vc-controller-manager
    Operation:    Update
    Time:         2022-11-03T13:06:56Z
    API Version:  scheduling.volcano.sh/v1beta1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        f:conditions:
        f:phase:
    Manager:    vc-scheduler
    Operation:  Update
    Time:       2022-11-03T13:06:56Z
  Owner References:
    API Version:           batch.volcano.sh/v1alpha1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Job
    Name:                  azureml-dev-78c8878c58fb623e0c3bf0ccfa3cc834
    UID:                   0073953d-5684-4a72-90e5-3f528890db26
  Resource Version:        963126
  UID:                     db3d1e0c-ff7a-40d9-a86e-908fb6a7f73e
Spec:
  Min Member:  3
  Min Resources:
    Cpu:     1800m
    Memory:  4831838208
  Min Task Member:
    Worker:  3
  Queue:     default
Status:
  Conditions:
    Last Transition Time:  2022-11-03T13:12:29Z
    Message:               1/3 tasks in gang unschedulable: pod group is not ready, 3 Pending, 3 minAvailable; Pending: 1 Unschedulable, 2 Schedulable
    Reason:                NotEnoughResources
    Status:                True
    Transition ID:         0b88e106-9778-4af7-9efb-b85bbb8aa569
    Type:                  Unschedulable
  Phase:                   Inqueue
Events:
  Type     Reason         Age                    From     Message
  ----     ------         ----                   ----     -------
  Warning  Unschedulable  5m47s                  volcano  0/0 tasks in gang unschedulable: pod group is not ready, 3 minAvailable
  Warning  Unschedulable  47s (x296 over 5m46s)  volcano  3/3 tasks in gang unschedulable: pod group is not ready, 3 P
```ending, 3 minAvailable; Pending: 1 Unschedulable, 2 Schedulable
`

[william-wang]

@william-wang My issue is also related to the gang plugin. In my case, PodGroup or pods aren't triggering scale up at all with Cluster Autoscaler. I created gang job with 3 pods, and 2 of them are fitted into the current node pools, but the third one isn't. Here is the PodGroup yaml, what it looks like in the Kubernetes:

@JgardemeRelex What is your autoscaler configuration like? In your case, there are already 3 pending pod. It's strange that no node was scaled out.

[wangyang0616]

@JgardemeRelex Is it convenient to provide the configuration file information of the Volcano?

[JgardemeRelex]

@JgardemeRelex What is your autoscaler configuration like? In your case, there are already 3 pending pod. It's strange that no node was scaled out.

@william-wang Here is the cluster autoscaler profile settings from AKS:

autoScalerProfile": {
    "balance-similar-node-groups": "false",
    "expander": "random",
    "max-empty-bulk-delete": "10",
    "max-graceful-termination-sec": "600",
    "max-node-provision-time": "15m",
    "max-total-unready-percentage": "45",
    "new-pod-scale-up-delay": "0s",
    "ok-total-unready-count": "3",
    "scale-down-delay-after-add": "10m",
    "scale-down-delay-after-delete": "10s",
    "scale-down-delay-after-failure": "3m",
    "scale-down-unneeded-time": "10m",
    "scale-down-unready-time": "20m",
    "scale-down-utilization-threshold": "0.5",
    "scan-interval": "10s",
    "skip-nodes-with-local-storage": "false",
    "skip-nodes-with-system-pods": "true"
},

[JgardemeRelex]

@JgardemeRelex Is it convenient to provide the configuration file information of the Volcano?

@wangyang0616 Idea was to help you to understand my issue, and if you see something from yaml. I can remove the it if needed.

[wangyang0616]

@JgardemeRelex Is it convenient to provide the configuration file information of the Volcano?

@wangyang0616 Idea was to help you to understand my issue, and if you see something from yaml. I can remove the it if needed.

To my understanding, in a cluster where the autoscaler policy is configured, the gang plug-in can be deleted from the plugin scheduled by the Volcano.

The gang plugin is used to perform the “All or nothing” operation when cluster resources are insufficient during scheduling to avoid resource contention and deadlock.

If the autoscaler policy is configured in the cluster, In this case, the cluster resources are large enough to avoid resource contention and deadlock. The major problems that need to be solved by the gang plugin are avoided. The gang plugin can be deleted from the configuration file.

[JgardemeRelex]

Thank you @wangyang0616! When I removed manually the gang plugin from Volcano scheduler config map, it worked! This same thing will be problem for other people in the future, since Microsoft is using gang plugin as default with their batch jobs in AKS.

[wangyang0616]

We need to consider the compatibility between the gang plug-in and auto-scaler in Volcano.

When a pod in the pending state exists in the cluster, auto-scaler triggers node capacity expansion. Before capacity expansion, auto-scaler performs a pre-scheduling verification on the pod to determine whether the existing resources in the cluster are insufficient for scheduling the pod. If the resources are insufficient, the subsequent capacity expansion process continues. Otherwise, the capacity expansion is interrupted.

During scheduling, the gang plug-in will roll back all resources when resources are insufficient and release pod resources that have been successfully scheduled in the current job. In this way, when autoscaler pre-schedules cluster resources, task resources are sufficient and capacity expansion is not required.

[wangyang0616]

If the AutoscalerWaitTime parameter is added to the gang plug-in of Volcano to control the time for the gang plug-in to release scheduled pod resources, the gang plug-in releases the scheduled pod resources after the preset time. Can this scenario be compatible?

[yuyue9284]

We need to consider the compatibility between the gang plug-in and auto-scaler in Volcano.

When a pod in the pending state exists in the cluster, auto-scaler triggers node capacity expansion. Before capacity expansion, auto-scaler performs a pre-scheduling verification on the pod to determine whether the existing resources in the cluster are insufficient for scheduling the pod. If the resources are insufficient, the subsequent capacity expansion process continues. Otherwise, the capacity expansion is interrupted.

During scheduling, the gang plug-in will roll back all resources when resources are insufficient and release pod resources that have been successfully scheduled in the current job. In this way, when autoscaler pre-schedules cluster resources, task resources are sufficient and capacity expansion is not required.

The auto scaler filtered the unscheduled pod with reason set to apiv1.PodReasonUnschedulable https://github.com/kubernetes/autoscaler/blob/cluster-autoscaler-release-1.25/cluster-autoscaler/utils/kubernetes/listers.go#L161-L175, when using the gang plugin, once one of that task failed to schedule, the rest unscheduled pod will have the reason set to undetermined instead of unschedulable, so those pods are invisible to autoscaler. If volcano change the undetermined to unschedulable, will it make gang works with autoscaler?

[wangyang0616]

Can the PR solve the current problem? #2602