[receiver/splunkenterprise] Adding Search Artifact Cache Metrics (open-telemetry#36181)

<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description

This PR adds search artifact cache metrics to the
splunkenterprisereceiver.

<!--Describe the documentation added.-->
#### Documentation

`metadata.yaml`

Author: michael-burt

.chloggen/splunkenterprisereceiver-search-artifact-cache.yaml

@@ -0,0 +1,27 @@
+# Use this changelog template to create an entry for release notes.
+
+# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
+change_type: enhancement
+
+# The name of the component, or a single word describing the area of concern, (e.g. filelogreceiver)
+component: splunkenterprisereceiver
+
+# A brief description of the change.  Surround your text with quotes ("") if it needs to start with a backtick (`).
+note: Add new metrics for Splunk Enterprise dispatch artifacts caches
+
+# Mandatory: One or more tracking issues related to the change. You can use the PR number here if no issue exists.
+issues: [36181]
+
+# (Optional) One or more lines of additional information to render under the primary note.
+# These lines will be padded with 2 spaces and then inserted directly into the document.
+# Use pipe (|) for multiline entries.
+subtext:
+
+# If your change doesn't affect end users or the exported elements of any package,
+# you should instead start your pull request title with [chore] or use the "Skip Changelog" label.
+# Optional: The change log or logs in which this entry should be included.
+# e.g. '[user]' or '[user, api]'
+# Include 'user' if the change is relevant to end users.
+# Include 'api' if there is a change to a library API.
+# Default: '[user]'
+change_logs: [user]

receiver/splunkenterprisereceiver/documentation.md

@@ -444,8 +444,8 @@ This is the overall status of the kvstore for the given deployment.
 
 | Name | Description | Values |
 | ---- | ----------- | ------ |
-| splunk.kvstore.storage.engine | The backend storage used by the KV store. | Any Str |
-| splunk.kvstore.external | Value denoting if the KV store is using an external service. | Any Str |
+| splunk.kvstore.storage.engine | The backend storage used by the KV store | Any Str |
+| splunk.kvstore.external | Value denoting if the KV store is using an external service | Any Str |
 | splunk.kvstore.status.value | The string value of the status returned when reporting on KV store using the introspection endpoint | Any Str |
 
 ### splunk.server.introspection.queues.current
@@ -478,7 +478,7 @@ Gauge tracking current bytes waiting in queue. *Note:** Must be pointed at speci
 
 ### splunk.server.searchartifacts.adhoc
 
-Gauge tracking number of ad hoc search artifacts currently on disk. Note:* Must be pointed at specific Search Head endpoint and gathers metrics from only that Search Head.
+Gauge tracking number of ad hoc search artifacts currently on disk. Note:* Must be pointed at specific Search Head endpoint and gathers metrics from only that Search Head. Available in builds 9.1.2312.207+ and 9.3.x+.
 
 | Unit | Metric Type | Value Type |
 | ---- | ----------- | ---------- |
@@ -492,7 +492,7 @@ Gauge tracking number of ad hoc search artifacts currently on disk. Note:* Must
 
 ### splunk.server.searchartifacts.completed
 
-Gauge tracking number of artifacts currently on disk that belong to finished searches. Note:* Must be pointed at specific Search Head endpoint and gathers metrics from only that Search Head.
+Gauge tracking number of artifacts currently on disk that belong to finished searches. Note:* Must be pointed at specific Search Head endpoint and gathers metrics from only that Search Head. Available in builds 9.1.2312.207+ and 9.3.x+.
 
 | Unit | Metric Type | Value Type |
 | ---- | ----------- | ---------- |
@@ -506,7 +506,7 @@ Gauge tracking number of artifacts currently on disk that belong to finished sea
 
 ### splunk.server.searchartifacts.incomplete
 
-Gauge tracking number of artifacts currently on disk that belong to unfinished/running searches. Note:* Must be pointed at specific Search Head endpoint and gathers metrics from only that Search Head.
+Gauge tracking number of artifacts currently on disk that belong to unfinished/running searches. Note:* Must be pointed at specific Search Head endpoint and gathers metrics from only that Search Head. Available in builds 9.1.2312.207+ and 9.3.x+.
 
 | Unit | Metric Type | Value Type |
 | ---- | ----------- | ---------- |
@@ -520,7 +520,7 @@ Gauge tracking number of artifacts currently on disk that belong to unfinished/r
 
 ### splunk.server.searchartifacts.invalid
 
-Gauge tracking number of artifacts currently on disk that are not in a valid state, such as missing info.csv file, etc. Note:* Must be pointed at specific Search Head endpoint and gathers metrics from only that Search Head.
+Gauge tracking number of artifacts currently on disk that are not in a valid state, such as missing info.csv file, etc. Note:* Must be pointed at specific Search Head endpoint and gathers metrics from only that Search Head. Available in builds 9.1.2312.207+ and 9.3.x+.
 
 | Unit | Metric Type | Value Type |
 | ---- | ----------- | ---------- |
@@ -532,9 +532,38 @@ Gauge tracking number of artifacts currently on disk that are not in a valid sta
 | ---- | ----------- | ------ |
 | splunk.host | The name of the splunk host | Any Str |
 
+### splunk.server.searchartifacts.job.cache.count
+
+Gauge tracking number search artifacts metadata stored in memory, available in builds 9.1.2312.207+ and 9.3.x+.
+
+| Unit | Metric Type | Value Type |
+| ---- | ----------- | ---------- |
+| {search_artifacts} | Gauge | Int |
+
+#### Attributes
+
+| Name | Description | Values |
+| ---- | ----------- | ------ |
+| splunk.host | The name of the splunk host | Any Str |
+
+### splunk.server.searchartifacts.job.cache.size
+
+Gauge tracking, in megabytes, memory used to cache job status and job info of all search artifacts, available in builds 9.1.2312.207+ and 9.3.x+.
+
+| Unit | Metric Type | Value Type |
+| ---- | ----------- | ---------- |
+| {mb} | Gauge | Int |
+
+#### Attributes
+
+| Name | Description | Values |
+| ---- | ----------- | ------ |
+| splunk.host | The name of the splunk host | Any Str |
+| splunk.searchartifacts.cache.type | The search artifacts cache type | Any Str |
+
 ### splunk.server.searchartifacts.savedsearches
 
-Gauge tracking, for the `splunk.server.searchartifacts.scheduled` number of scheduled search artifacts, how many different saved-searches they belong to. Note:* Must be pointed at specific Search Head endpoint and gathers metrics from only that Search Head.
+Gauge tracking, for the `splunk.server.searchartifacts.scheduled` number of scheduled search artifacts, how many different saved-searches they belong to. Note:* Must be pointed at specific Search Head endpoint and gathers metrics from only that Search Head. Available in builds 9.1.2312.207+ and 9.3.x+.
 
 | Unit | Metric Type | Value Type |
 | ---- | ----------- | ---------- |
@@ -548,7 +577,7 @@ Gauge tracking, for the `splunk.server.searchartifacts.scheduled` number of sche
 
 ### splunk.server.searchartifacts.scheduled
 
-Gauge tracking number of scheduled search artifacts currently on disk. Note:* Must be pointed at specific Search Head endpoint and gathers metrics from only that Search Head.
+Gauge tracking number of scheduled search artifacts currently on disk. Note:* Must be pointed at specific Search Head endpoint and gathers metrics from only that Search Head. Available in builds 9.1.2312.207+ and 9.3.x+.
 
 | Unit | Metric Type | Value Type |
 | ---- | ----------- | ---------- |