Shall we introduce engine modes?

[xiaomi7732]

The problem to address

• When application performance issue happens, adding profiler is impossible today without restarting the application.
• If the profiler is added beforehand, there's a relative heavy overhead and that's a waste of resources until there's a problem.

Modes can address that issue by delivering the binaries (package) ahead of time but has close to zero overhead. And can be activated without requiring restart of the app.

Modes proposals

1. Inactive:

1. Profiler introduces very minor (ideally none) overhead - there's no actual profiling at all.
2. It can then be activated into other mode without requiring restarting of the app.
   1. Notes: it is important to not require restarts. For example, change an environment variable would cause a restart to make it in effect.
   2. Disable background services to reduce overheads. For examples:
      1. Disable CPU/Memory monitoring.
      2. Disable FileSavengers.

1. Active:

1. Sampling happens.
2. CPU / Memory monitoring and triggers works.
3. Profile now works.

Considerations

• Start simple - having 2 modes to make the story simple.
• How to tell that the profiler is successfully on in inactive mode? Shall we issue a heartbeat in Inactive mode? Is that acceptable?
• Design the activating protocol carefully for scaled apps:
  • For example, is the activate signal going against a specific instance or going against all agents for a given application insights resource?
• There's also a traditional user setting of IsDisabled to fully disable profiler. Consider the status carefully when that's in effective.
• There are also portal settings to turn on/off specific trigger and sampling.