Volume Not Found Errors on Previously Working Instance

[rkbennett]

Describe the bug
I had mayastor running for over a year on my k8s instance and then the cluster when down for a while and when it was brought back up one of my pods were unable to be deployed with the following error:

AttachVolume.Attach failed for volume "pvc-abace6a0-ccd8-4a92-8caf-60640359aa74" : rpc error: code = NotFound desc = error in response: status code '404 Not Found', content: 'RestJsonError { details: "Volume 'abace6a0-ccd8-4a92-8caf-60640359aa74' not found", message: "SvcError :: VolumeNotFound", kind: NotFound }'

I've tried recreating the PVC, Other deployments that use the PVC, etc and it always throws this error now.

To Reproduce
Steps to reproduce the behavior:
I'm honestly not sure what caused it.

Expected behavior
Pods should be able to mount PVCs related to mayastor's CSI provider

Screenshots
If applicable, add screenshots to help explain your problem.

** OS info (please complete the following information):**

• Distro: Talos 1.4.5
• Kernel version
• MayaStor revision or container image: v2.2.0

Additional context
Add any other context about the problem here.

[tiagolobocastro]

Could you please take a support bundle and attach it here?