Move kube-scheduler related metrics initilization to server.go to avoid panic #4422
Conversation
volcano-sh-bot
added
the
size/L
JesseStutler
force-pushed
the
fix_metrics_init
branch
2 times, most recently
from
c7d6e9b to
ac2498e
Compare
| // It is safe here to directly use the state to run plugins because we have already initialized the cycle state | ||
| // for each pending pod when open session and will not meet nil state | ||
| state := ssn.GetCycleState(task.UID) | ||
| metrics.Register() |
There was a problem hiding this comment.
What does the metrics include? Is it necessary for volcano, or volcano already has these metrics? It's exposed by kube-scheduler framework, may not useful for volcano, we should check that.
There was a problem hiding this comment.
Actually we only need Goroutines metric in kube-scheduler, so now I init this metric(override the gloabl variable ) in InitKubeSchedulerRelatedMetrics and this metrics won't be registered and exported, only to avoid panic when we need to execute kube-scheduler related plugins
| @@ -0,0 +1,141 @@ | |||
| /* | |||
There was a problem hiding this comment.
This is not related to this pr.
There was a problem hiding this comment.
It's used to judge when we disable predicates plugin, only contain the nodeorder plugin, the scheduler won't panic
…id panic Signed-off-by: jessestutler <[email protected]>
JesseStutler
force-pushed
the
fix_metrics_init
branch
from
ac2498e to
a8b0c07
Compare
| // related plugins. And there is no need to export these metrics, therefore currently initialization is enough. | ||
| func InitKubeSchedulerRelatedMetrics() { | ||
| k8smetrics.Goroutines = metrics.NewGaugeVec( | ||
| &metrics.GaugeOpts{ |
There was a problem hiding this comment.
Is the goroutine metrics already included in volcano metrics?
There was a problem hiding this comment.
I think not yet, it's used to record how many goroutines are simultaneously executing in a plugin:
Do we need to register and export it?
|
/approve |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: Monokaix The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
volcano-sh-bot
added
the
approved
|
backport to release1.12. |
|
/lgtm |
…4422-origin-release-1.12 Automated cherry pick of #4422: Move kube-scheduler related metrics initilization to server.go to avoid panic
4 participants
What type of PR is this?
/kind bug
What this PR does / why we need it:
Currently, if a user doesn't configure predicates plugin and then submit a simple VC job, the scheduler will panic. We need to fix it.
Which issue(s) this PR fixes:
Fixes # #4421
Special notes for your reviewer:
I also add a unit test to test if we only specify nodeorder plugin and doesn't configure predicates plugin whether will meet panic
Does this PR introduce a user-facing change?