Skip to content

Added support for persistent buffering in receiver/gateway #1342

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 13 commits into from
Closed

Added support for persistent buffering in receiver/gateway #1342

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
058014d
Added support for persistent buffering in receiver/gateway
sbylica-splunk Jul 11, 2024
503912f
Fixed receiver tpl
sbylica-splunk Jul 15, 2024
57a97b4
Update helm-charts/splunk-otel-collector/templates/config/_otel-k8s-c…
sbylica-splunk Jul 18, 2024
25b578e
Update helm-charts/splunk-otel-collector/templates/config/_otel-k8s-c…
sbylica-splunk Jul 18, 2024
9a8b3f7
changed dir to clusterReceiver
sbylica-splunk Jul 18, 2024
4240a93
Update helm-charts/splunk-otel-collector/templates/config/_otel-k8s-c…
sbylica-splunk Mar 26, 2025
738fbd2
Update helm-charts/splunk-otel-collector/templates/config/_otel-k8s-c…
sbylica-splunk Mar 26, 2025
1cd8627
Refactoring after the review
sbylica-splunk Mar 27, 2025
9bd5818
.
sbylica-splunk Mar 27, 2025
85cd7f7
.
sbylica-splunk Mar 27, 2025
88d6940
.
sbylica-splunk Mar 27, 2025
be0b1fa
.
sbylica-splunk Mar 27, 2025
b74f748
Remoed persistent queue from gateway
sbylica-splunk Mar 27, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 16 additions & 6 deletions helm-charts/splunk-otel-collector/templates/config/_otel-k8s-cluster-receiver-config.tpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,6 @@ The values can be overridden in .Values.clusterReceiver.config
extensions:
health_check:


{{- if eq (include "splunk-otel-collector.distribution" .) "eks/fargate" }}
# k8s_observer w/ pod and node detection for eks/fargate deployment
k8s_observer:
Expand All @@ -16,6 +15,12 @@ extensions:
observe_nodes: true
{{- end }}

{{- if .Values.splunkPlatform.sendingQueue.persistentQueue.enabled }}
file_storage/persistent_queue:
directory: {{ .Values.splunkPlatform.sendingQueue.persistentQueue.storagePath }}/clusterReceiver
timeout: 0
{{- end }}

receivers:
# Prometheus receiver scraping metrics from the pod itself
{{- include "splunk-otel-collector.prometheusInternalMetrics" "k8s-cluster-receiver" | nindent 2}}
Expand Down Expand Up @@ -190,6 +195,7 @@ exporters:
disable_compression: true
{{- end }}

{{- $_ := set . "addPersistentStorage" .Values.splunkPlatform.sendingQueue.persistentQueue.enabled }}
{{- if (eq (include "splunk-otel-collector.platformMetricsEnabled" .) "true") }}
{{- include "splunk-otel-collector.splunkPlatformMetricsExporter" . | nindent 2 }}
{{- end }}
Expand All @@ -200,16 +206,20 @@ exporters:
sourcetype: kube:events
{{- end }}
{{- end }}
{{- $_ := unset . "addPersistentStorage" }}

service:
telemetry:
metrics:
address: 0.0.0.0:8889
{{- if eq (include "splunk-otel-collector.distribution" .) "eks/fargate" }}
extensions: [health_check, k8s_observer]
{{- else }}
extensions: [health_check]
{{- end }}
extensions:
- health_check
{{- if eq (include "splunk-otel-collector.distribution" .) "eks/fargate" }}
- k8s_observer
{{- end }}
{{- if .Values.splunkPlatform.sendingQueue.persistentQueue.enabled }}
- file_storage/persistent_queue
{{- end }}
pipelines:
{{- if or (eq (include "splunk-otel-collector.o11yMetricsEnabled" $) "true") (eq (include "splunk-otel-collector.platformMetricsEnabled" $) "true") }}
# k8s metrics pipeline
Expand Down
31 changes: 29 additions & 2 deletions helm-charts/splunk-otel-collector/templates/deployment-cluster-receiver.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,8 +82,8 @@ spec:
securityContext:
{{- include "splunk-otel-collector.securityContext" (dict "isWindows" .Values.isWindows "securityContext" $clusterReceiver.securityContext) | nindent 8 }}
{{- end }}
{{- if eq (include "splunk-otel-collector.distribution" .) "eks/fargate" }}
initContainers:
{{- if eq (include "splunk-otel-collector.distribution" .) "eks/fargate" }}
- name: cluster-receiver-node-discoverer
image: public.ecr.aws/amazonlinux/amazonlinux:latest
imagePullPolicy: IfNotPresent
Expand All @@ -104,7 +104,24 @@ spec:
mountPath: /splunk-messages
- mountPath: /conf
name: collector-configmap
{{- end }}
{{- end }}
{{- if .Values.splunkPlatform.sendingQueue.persistentQueue.enabled }}
- name: patch-dirs
image: {{ template "splunk-otel-collector.image.initPatchLogDirs" . }}
imagePullPolicy: {{ .Values.image.initPatchLogDirs.pullPolicy }}
command: ['sh', '-c', '
mkdir -p {{ .Values.splunkPlatform.sendingQueue.persistentQueue.storagePath }}/clusterReceiver;
chown -Rv {{ $clusterReceiver.securityContext.runAsUser | default 999 }}:{{ $clusterReceiver.securityContext.runAsGroup | default 999 }} {{ .Values.splunkPlatform.sendingQueue.persistentQueue.storagePath }}/clusterReceiver;
chmod -v g+rwxs {{ .Values.splunkPlatform.sendingQueue.persistentQueue.storagePath }}/clusterReceiver;
setfacl -n -Rm d:m::rx,m::rx,d:g:{{ $clusterReceiver.securityContext.runAsGroup | default 999 }}:rx,g:{{ $clusterReceiver.securityContext.runAsGroup | default 999 }}:rx {{ .Values.splunkPlatform.sendingQueue.persistentQueue.storagePath }}/clusterReceiver;']
securityContext:
runAsUser: 0
Copy link
Contributor

@jvoravong jvoravong Jul 16, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: We probably need to document this runAsUser 0 and above used permissions in https://github.com/signalfx/splunk-otel-collector-chart/blob/main/docs/advanced-configuration.md#data-persistence as well as customer documentation. This would be done for the agent, cluster receiver, and gateway. A number of customers request information like this.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe a note in here:
https://github.com/signalfx/splunk-otel-collector-chart/blob/main/docs/advanced-configuration.md#running-the-container-in-non-root-user-mode
?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd suggest adding documentation primarily in the Advanced Configuration: Data Persistence section. Then, include a brief note with a hyperlink in the Running the Container in Non-Root User Mode section that points to the data persistence section.

resources:
{{- toYaml $clusterReceiver.resources | nindent 12 }}
volumeMounts:
- name: persistent-queue
mountPath: {{ .Values.splunkPlatform.sendingQueue.persistentQueue.storagePath }}/clusterReceiver
{{- end }}
containers:
- name: otel-collector
command:
Expand Down Expand Up @@ -189,6 +206,10 @@ spec:
mountPath: /otel/etc
readOnly: true
{{- end }}
{{- if .Values.splunkPlatform.sendingQueue.persistentQueue.enabled }}
- name: persistent-queue
mountPath: {{ .Values.splunkPlatform.sendingQueue.persistentQueue.storagePath }}/clusterReceiver
{{- end }}
- mountPath: {{ .Values.isWindows | ternary "C:\\conf" "/conf" }}
name: collector-configmap
{{- if eq (include "splunk-otel-collector.distribution" .) "eks/fargate" }}
Expand Down Expand Up @@ -217,6 +238,12 @@ spec:
secret:
secretName: {{ template "splunk-otel-collector.secret" . }}
{{- end }}
{{- if .Values.splunkPlatform.sendingQueue.persistentQueue.enabled }}
- name: persistent-queue
hostPath:
path: {{ .Values.splunkPlatform.sendingQueue.persistentQueue.storagePath }}/clusterReceiver
type: DirectoryOrCreate
{{- end }}
{{- if eq (include "splunk-otel-collector.distribution" .) "eks/fargate" }}
- name: init-eks-fargate-cluster-receiver-script
configMap:
Expand Down