VCH Upgrade and Revert not refreshing cVM status

[malikkal]

Steps to reproduce.

1. Upgrade VCH to newer build.
2. Delete cVMs using the upgraded VCH.
3. Revert snapshot; i.e. rollback using vic-machine.
4. Docker ps will continue to display previously deleted cVMs.
5. State wouldn't be refreshed regardless of wait time.

[hickeng]

@emlin Will the vic-machine rollback enforce a power cycle or is it reverting to an online snapshot?
If online #5356 will likely address this
If offline then the personality may well be failing to validate tag entries against actual inventory on reload. @cgtexmex @jzt ?

I would expect this to potentially occur from any drop in the vSphere event stream and an out-of-band deletion, even without the rollback. The testing added for #5343 should be a good starting place for asserting correct behaviour here (see tests/test-cases/Group0-Bugs/5343.robot)

@caglar10ur please check if this presents with out-of-band deletion of cVMs when doing the connection drop/session deletion followup.

[mdubya66]

marking high for now until we get a triage.

[cgtexmex]

@hickeng containerVMs are loaded from vSphere on a VCH power cycle, so I'd suspect it's reverting to the snapshot.

[jzt]

@hickeng am I understanding this correctly? The cache is returned to its previous state after the snapshot revert? If that's the case, there's no other way around this than doing a periodic refresh, or perhaps forcing a refresh/sync after a write.

[cgtexmex]

@jzt if it's a snapshot...I'd assume so...it's like pausing and resuming a vm -- vm state is saved in the snapshot.

[emlin]

@hickeng right, vic-machine rollback does not enforce a power cycle, that should be the reason to have this dated cache.
So you're right, #5356 should address this issue. Or we could easily add the power cycle into rollback as well.

[emlin]

@jzt @cgtexmex you're right. The cache and process status are all cached in the snapshot, and after vm is resumed, the old data in memory is used.

[stuclem]

An attempt at a release note:

---

• Rolling back upgrade on a VCH reverts container VMs to their pre-upgrade state. #5754
  If you upgrade a VCH, perform some operations, for example vic-machine delete, on container VMs, then use vic-machine upgrade --rollback to revert the VCH to the previous version, the container VMs revert to their previous state. For example, deleted container VMs reappear after the rollback.

---

@emlin @cgtexmex @jzt is this accurate? Thanks!