Add Multiple Metrics About Search Performance to splunkenterprisereceiver

[macolby42]

Description

Added a new splunk.search.initiation, splunk.search.duration, splunk.search.status, and splunk.search.success metrics.

Link to tracking issue

Fixes #39566

Testing

Ran on a test stack and observed the metrics in Splunk Observability

Documentation

The only documentation added was the generated documentation about the newly added metrics.

[atoulme]

Almost there - some lint issues:

Error: receiver/splunkenterprisereceiver/scraper.go:2031:25: var-naming: method setSearchJobTTLById should be setSearchJobTTLByID (revive)
func (s *splunkScraper) setSearchJobTTLById(sid string) error {
                        ^
Error: receiver/splunkenterprisereceiver/scraper.go:1924:33: QF1008: could remove embedded field "ControllerConfig" from selector (staticcheck)
		if time.Since(start) > s.conf.ControllerConfig.Timeout {
		                              ^
Error: receiver/splunkenterprisereceiver/scraper.go:1946:77: QF1008: could remove embedded field "ControllerConfig" from selector (staticcheck)
	for entry.Content.DispatchState != StateDone || time.Since(start) < s.conf.ControllerConfig.Timeout {
	                                                                           ^
Error: receiver/splunkenterprisereceiver/scraper.go:2051:23: "200" can be replaced by http.StatusOK (usestdlibvars)
	if res.StatusCode != 200 {
	                     ^

[atoulme]

Moving to draft while this is being worked on. Please move back to ready when done.

[MovieStoreGuy]

Is this no the same thing? One is just the count of the other?

[macolby42]

The dispatchState property can have many different states so we capture that in splunk.search.status while splunk.search.success is more of a convience metrics that is a quick signal to knowing that the submitted searches have readed the dispatchState of DONE