Agent should attempt a POLICY_CHANGE action before checking in

[michel-laterman]

The current action handling behaviour dispatches actions completely asynchrously from the checkin-loop:

elastic-agent/internal/pkg/agent/application/gateway/fleet/fleet_gateway.go

Lines 161 to 176 in ab68480

	case <-f.scheduler.WaitTick():
	f.log.Debug("FleetGateway calling Checkin API")
	// Execute the checkin call and for any errors returned by the fleet-server API
	// the function will retry to communicate with fleet-server with an exponential delay and some
	// jitter to help better distribute the load from a fleet of agents.
	resp, err := f.doExecute(ctx, requestBackoff)
	if err != nil {
	continue
	}
	actions := make([]fleetapi.Action, len(resp.Actions))
	copy(actions, resp.Actions)
	if len(actions) > 0 {
	f.actionCh <- actions
	}

After fleet checkin, the agent sends all actions through a channel to the dispatcher. They are executed concurrently with the checkin loop; the ticker by default has a 1s duration with up to 500ms jitter.
There is no guarantee that the POLICY_CHANGE action is executed before the next checkin.

We should adjust this to attempt to run a POLICY_CHANGE action before the next checkin is sent.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[kaanyalti]

@michel-laterman Can you please explain the need for this change? As you mentioned in your comments in the action handling improvements design doc this may affect the design so I would like to understand the reason why policy changes should be attempted before a checkin

[michel-laterman]

Newer agents will include agent_policy_id and policy_revision_idx in the checkin request. If the checkin runs before the action is handled, it will use the old values and fleet-server will immediately return a new (duplicate) POLICY_CHANGE action.

[kaanyalti]

Perfect, thank you