Fleet check-in should send policy_id and revision

[blakerouse]

Overview

Currently when the Elastic Agent checks-in with Fleet Server it doesn't send the policy_id or revision of the policy that it is currently running. The Fleet Server stores this information by the fact that the Elastic Agent ACK'd the policy change notification, but there are many cases where this could be come out of sync.

VM Snapshot

1. VM is snapshotted
2. new policy revision occurs
3. ACK'd by Elastic Agent (stored new revision in Fleet)
4. VM is rolled back

Now the running Elastic Agent policy is the old version, but to Fleet it is the new version.

Bad Error Case

This is just a weird case but a coding issue could result in this problem.

1. New revision is sent to Elastic Agent
2. Policy failed to be saved to disk (could be coding issue or just with filesystem problem)
3. policy revision is ACK'd anyway (shouldn't happen, but if it does...)

Elastic Agent is now running old version of policy but Fleet Server believes that its the new revision

Backup/restore of fleet.enc

In the case of backup/restore of fleet.enc.

1. fleet.enc is backed up
2. new policy revision occurs
3. ACK'd by Elastic Agent (stored new revision in Fleet)
4. fleet.enc is replaced with backup from 1
5. Elastic Agent restarted

Elastic Agent is now running old version of policy but Fleet Server believes that its the new revision

How to solve it?

Upon check-in the Elastic Agent should be sending its current policy ID and revision. That is then compared to what Fleet Server expects and if it is not correct then it sends the correct policy.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[cmacknz]

The Fleet Server stores this information by the fact that the Elastic Agent ACK'd the policy change notification, but there are many cases where this could be come out of sync.

This is another example of a place where we don't actually need explicit ACKs like upgrades, because the actual state of the agent can be detected from the checkin payload.

More and more I think the way ACKs are used in action processing needs to be completely revisited or just designed out of the system. We frequently have an ACK and an actual state change or action result that are two separate updates to the system state, allowing one of them to not happen regardless of the other. We should eliminate this problem.

[michel-laterman]

Just as a note, we will also need some way to determine if a revision has been received by the agent, but has failed to apply

[blakerouse]

I believe We already have that with ACK. Elastic Agent should not ACK a revision that it cannot apply. Or are you thinking of something different than me?

This is more like it did ACK but it has been rollback out of control of the Elastic Agent.

We need to send the policy_id and revision on checkin so Fleet Server can check if it's on the correct policy and correct revision. If any of those are incorrect it should send a new policy update action with the policy information.

[michel-laterman]

When a policy change fails to apply, no ack is sent; the agent instead checks in with an error status and the fleet-server will basically resend the same policy_change action.

If we are OK with the same behaviour then just adding the ID + revision will do.

However we may also want to be specific about what the agent is currently running and what the agent has tried (and failed) to run, for example:

...
"policy_id": "agent-policy",
"revision_idx": 2,
"policy_error_state": {
  "policy_id": "agent-policy",
  "revision_idx: 3,
  "message": "..."
}

This way the fleet-server knows that the agent is running the policy with revision 2, but revision 3 has been sent (and failed).

[blakerouse]

@michel-laterman I think it would be great to provide more feedback. +1

[cmacknz]

Quoting myself from Slack to move conversation here:

I like this but makes me think this is special casing something that should be a generic mechanism. You are building a NACK into the checkin but specialized to policy changes, when it could be generic.

The intent of Fleet is basically everything is actions, policy change, upgrade, diagnostics, etc so we should try to do this in generic ways. Upgrade details we cheated a bit and created a way to communicate about long running actions both because upgrades are important and it’s the only use case for that so far.

...
"policy_id": "agent-policy",
"revision_idx": 2,
"policy_error_state": {
  "policy_id": "agent-policy",
  "revision_idx: 3,
  "message": "..."
}

What you are encoding here is a policy change action request for revision 3, with a policy change action response with an error.

Reporting the current state with policy_id and revision_idx is fine. I think we have “current state”, “request to change state”, and “response to state change request” as the moving pieces here.

I think the error reporting makes a lot of sense but I think for error reporting we should just fix the action handling to be able to report this.

[cmacknz]

Summarizing more recent discussion on how to handle this:

• We are converging on the idea that all actions that are a mutation of the agent state should be handled separately from one off actions.
  • State mutations are policy change, upgrade, migrate, unprivileged, settings, etc
  • One off actions are request diagnostics, most input actions
• The state changes should likely not be actions that require an ack anymore
• The effect of state changes should be observable in agent's checkin payload, for example the agent indicating the revision and ID as described here, upgrade details indicating upgrade is complete (which was an earlier step in this direction).
• The state changes should use a reconcilation process where agent is always told what it's current state should be e.g. revision X version Y and Fleet can tell if it is up to date based on the current state in the checkin payload.

For this issue, we should aim to put the plumbing an design changes in to handle this but only specifically handle the policy change action as the first action to work this way. One reason to do this is that fleet-server already dynamically generates policy change actions so there is no need to change Fleet itself. To fully move upgrades to this model, we would require more changes and verification with the Fleet UI.

For the automatic upgrades feature in 9.2.0 Fleet does something similar itself where the target agent version is in the policy, but removed in Fleet server and translated into an upgrade action. This does not apply to all upgrades however, only auto upgrades.

Other action types besides policy change and upgrade do not work this way at all yet.

[cmacknz]

I think we will want a short design document for the changes here that outlines how the changes can be supported for all of the relevant action types beyond just policy change.