[Beats receivers] Add exporter translation support for all currently unsupported Elasticsearch output features

[cmacknz]

• Relates Fall back to process runtime if otel runtime is unsupported elastic-agent#10087

The list of Elasticsearch output parameters that currently cannot be translated to the Elasticsearch exporter is currently defined below. These parameters will cause Elastic Agent to fall back to the sub-process runtime when they are used, preventing adoption of Beats receivers.

beats/libbeat/otelbeat/oteltranslate/outputs/elasticsearch/config_otel.go

Lines 166 to 187 in 4ba000d

	func checkUnsupportedConfig(cfg *config.C, logger *logp.Logger) error {
	if cfg.HasField("indices") {
	return fmt.Errorf("indices is currently not supported: %w", errors.ErrUnsupported)
	} else if cfg.HasField("pipelines") {
	return fmt.Errorf("pipelines is currently not supported: %w", errors.ErrUnsupported)
	} else if cfg.HasField("parameters") {
	return fmt.Errorf("parameters is currently not supported: %w", errors.ErrUnsupported)
	} else if value, err := cfg.Bool("allow_older_versions", -1); err == nil && !value {
	return fmt.Errorf("allow_older_versions:false is currently not supported: %w", errors.ErrUnsupported)
	} else if cfg.HasField("loadbalance") {
	return fmt.Errorf("loadbalance is currently not supported: %w", errors.ErrUnsupported)
	} else if cfg.HasField("non_indexable_policy") {
	return fmt.Errorf("non_indexable_policy is currently not supported: %w", errors.ErrUnsupported)
	} else if cfg.HasField("escape_html") {
	return fmt.Errorf("escape_html is currently not supported: %w", errors.ErrUnsupported)
	} else if cfg.HasField("kerberos") {
	return fmt.Errorf("kerberos is currently not supported: %w", errors.ErrUnsupported)
	}
	return nil
	}

Documented Elastic Agent Elasticsearch output parameters

• allow_older_versions
• Fleet https://www.elastic.co/docs/reference/fleet/es-output-settings#es-output-settings-yaml-config
• Standalone https://www.elastic.co/docs/reference/fleet/elasticsearch-output#output-elasticsearch-compatibility-setting
• Filebeat https://www.elastic.co/docs/reference/beats/filebeat/elasticsearch-output#_allow_older_versions
• pipelines
  • Standalone https://www.elastic.co/docs/reference/fleet/elasticsearch-output#output-elasticsearch-data-parsing-settings
  • Filebeat https://www.elastic.co/docs/reference/beats/filebeat/elasticsearch-output#pipelines-option-es
• kerberos
  • Standalone https://www.elastic.co/docs/reference/fleet/elasticsearch-output#output-elasticsearch-kerberos-authentication-settings
  • Filebeat https://www.elastic.co/docs/reference/beats/filebeat/elasticsearch-output#_kerberos_2
• parameters
  • Standalone https://www.elastic.co/docs/reference/fleet/elasticsearch-output#output-elasticsearch-http-settings
  • Filebeat https://www.elastic.co/docs/reference/beats/filebeat/elasticsearch-output#_parameters
• escape_html
  • Standalone https://www.elastic.co/docs/reference/fleet/elasticsearch-output#output-elasticsearch-data-parsing-settings
  • Filebeat https://www.elastic.co/docs/reference/beats/filebeat/elasticsearch-output#_escape_html

Undocumented Elastic Agent Elasticsearch output parameters

• loadbalance
  • Filebeat https://www.elastic.co/docs/reference/beats/filebeat/elasticsearch-output#_loadbalance
  • This one is definitely used with some frequency.
• indices
  • Filebeat https://www.elastic.co/docs/reference/beats/filebeat/elasticsearch-output#indices-option-es
  • Conflicts with the idea of the data stream naming scheme
• non_indexable_policy
  • Filebeat https://www.elastic.co/docs/reference/beats/filebeat/elasticsearch-output#_non_indexable_policy
  • Dead letter index should probably be deprecated, removed, and replaced with the failure store.
• ssl.restart_on_cert_change.*
  • Filebeat https://www.elastic.co/docs/reference/beats/filebeat/configuration-ssl#exit_on_cert_change_enabled

Undocumented Elastic Agent Elasticsearch output parameters

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)