Skip to content

Commit bbfd013

Browse files
flandsBogdan Drutubogdandrutu
authored
Make distribution more configurable (#32)
* Make distribution more configuration Changes to main.go - Leverage const for env vars and defaults - Ensure all env vars are properly checked - Add support for non-linux Configuration: - Add support for non-linux (memory percentage versus mib) - Enable http_forwarder extension Update README: - Add config and memory env vars - Update docker ports - Add more advanced configuration options * Fix tests and lint * Fix RPM/DEB packages * Set memory env vars at end of function * Fix lint * Refactor * Address feedback * Address feedback * Update version * Update buildscripts with latest env Signed-off-by: Bogdan Drutu <[email protected]> * Remove setting config from example Signed-off-by: Bogdan Drutu <[email protected]> * Fix default config file * Consistent use SPLUNK_ACCESS_TOKEN Signed-off-by: Bogdan Drutu <[email protected]> Co-authored-by: Bogdan Drutu <[email protected]> Co-authored-by: Bogdan Drutu <[email protected]>
1 parent b19e1e9 commit bbfd013

File tree

9 files changed

+415
-35
lines changed

9 files changed

+415
-35
lines changed

‎README.md

Lines changed: 42 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -18,30 +18,41 @@ This distribution comes with a [default
1818
configuration](https://github.com/signalfx/splunk-otel-collector/blob/main/cmd/otelcol/config/collector/splunk_config.yaml)
1919
which requires the following environment variables:
2020

21-
- `${SPLUNK_REALM}`: Which realm to send the data to (for example: `us0`)
22-
- `${SPLUNK_ACCESS_TOKEN}`: Access token to authenticate requests
23-
- `${SPLUNK_BALLAST_SIZE_MIB}`: How much memory to allocate to the ballast. This should be set to 1/3 to 1/2 of configured memory.
21+
- `SPLUNK_REALM` (no default): Which realm to send the data to (for example: `us0`)
22+
- `SPLUNK_ACCESS_TOKEN` (no default): Access token to authenticate requests
23+
- `SPLUNK_BALLAST_SIZE_MIB` (no default): How much memory to allocate to the ballast. This should be set to 1/3 to 1/2 of configured memory.
24+
25+
In addition, the following environment variables are optional:
26+
27+
- `SPLUNK_CONFIG` (default = `/etc/otel/collector/splunk-config_linux.yaml`): Which configuration to load.
28+
- `SPLUNK_MEMORY_LIMIT_PERCENTAGE` (default = `90`): Maximum amount of total memory targeted to be allocated by the process heap.
29+
- `SPLUNK_MEMORY_SPIKE_PERCENTAGE` (default = `20`): Maximum spike expected between the measurements of memory usage.
30+
31+
When running on a non-linux system, the following environment variables are required:
32+
33+
- `SPLUNK_CONFIG` (default = `/etc/otel/collector/splunk-config_non_linux.yaml`): Which configuration to load.
34+
- `SPLUNK_MEMORY_LIMIT_MIB` (no default): Maximum amount of total memory targeted to be allocated by the process heap.
35+
- `SPLUNK_MEMORY_SPIKE_MIB` (no default): Maximum spike expected between the measurements of memory usage.
2436

2537
Deploy the collector as outlined in the below. More information
2638
about deploying and configuring the collector can be found
2739
[here](https://docs.signalfx.com/en/latest/apm/apm-getting-started/apm-opentelemetry-collector.html)
2840

2941
### Docker
3042

31-
Deploy from a Docker container (replace `0.1.0-otel-0.11.0` with the latest
32-
stable version number if necessary):
43+
Deploy from a Docker container (replace `0.1.0` with the latest stable version number if necessary):
3344

3445
```bash
3546
$ SPLUNK_REALM=us0 SPLUNK_ACCESS_TOKEN=12345 SPLUNK_BALLAST_SIZE_MIB=683 \
36-
docker run -p 7276:7276 -p 8888:8888 -p 9943:9943 -p 55679:55679 -p 55680:55680 -p 9411:9411 \
37-
--name otelcol signalfx/splunk-otel-collector:0.1.0-otel-0.11.0
47+
docker run -p 13133 -p 14250 -p 14268 -p 55678-55680 -p 6060 -p 7276 -p 8888 -p 9411 -p 9943 \
48+
--name otelcol signalfx/splunk-otel-collector:0.1.0
3849
```
3950

4051
### Kubernetes
4152

42-
To deploy the OpenTelemetry Collector in Kubernetes, create a configuration
43-
file that defines a ConfigMap, Service, and Deployment for the cluster. For
44-
more information about creating a configuration file, see the example
53+
To deploy in Kubernetes, create a configuration file that defines a ConfigMap,
54+
Service, and Deployment for the cluster. For more information about creating a
55+
configuration file, see the example
4556
[signalfx-k8s.yaml](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/master/exporter/sapmexporter/examples/signalfx-k8s.yaml)
4657
file on GitHub.
4758

@@ -53,19 +64,35 @@ $ SPLUNK_REALM=us0 SPLUNK_ACCESS_TOKEN=12345 SPLUNK_BALLAST_SIZE_MIB=683 \
5364
./bin/otelcol
5465
```
5566

56-
## Custom Configuration
67+
## Advanced Configuration
68+
69+
### Command Line Arguments
70+
71+
Following the binary command or Docker container command line arguments can be
72+
specified. Command line arguments take priority over environment variables.
73+
74+
For example in Docker:
75+
76+
```bash
77+
$ SPLUNK_REALM=us0 SPLUNK_ACCESS_TOKEN=12345 SPLUNK_BALLAST_SIZE_MIB=683 \
78+
docker run -p 13133 -p 14250 -p 14268 -p 55678-55680 -p 6060 -p 7276 -p 8888 -p 9411 -p 9943 \
79+
-v collector.yaml:/etc/collector.yaml:ro \
80+
--name otelcol signalfx/splunk-otel-collector:0.1.0 \
81+
--log-level=DEBUG
82+
```
83+
84+
### Custom Configuration
5785

5886
In addition to using the default configuration, a custom configuration can also
5987
be provided.
6088

6189
For example in Docker:
6290

6391
```bash
64-
$ SPLUNK_REALM=us0 SPLUNK_ACCESS_TOKEN=12345 SPLUNK_BALLAST_SIZE_MIB=683 \
65-
docker run -p 7276:7276 -p 8888:8888 -p 9943:9943 -p 55679:55679 -p 55680:55680 -p 9411:9411 \
92+
$ SPLUNK_REALM=us0 SPLUNK_ACCESS_TOKEN=12345 SPLUNK_BALLAST_SIZE_MIB=683 SPLUNK_CONFIG=/etc/collector.yaml \
93+
docker run -p 13133 -p 14250 -p 14268 -p 55678-55680 -p 6060 -p 7276 -p 8888 -p 9411 -p 9943 \
6694
-v collector.yaml:/etc/collector.yaml:ro \
67-
--name otelcol signalfx/splunk-otel-collector:0.1.0-otel-0.11.0 \
68-
--config /etc/collector.yaml
95+
--name otelcol signalfx/splunk-otel-collector:0.1.0
6996
```
7097

7198
Note that if the configuration includes a memorylimiter processor then it must set the

‎cmd/otelcol/Dockerfile

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -10,8 +10,9 @@ RUN chmod 755 /otelcol
1010
FROM scratch
1111
COPY --from=certs /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
1212
COPY --from=otelcol /otelcol /
13-
COPY config/collector/splunk_config.yaml /etc/otel/collector/splunk_config.yaml
14-
COPY config/collector/otlp_config.yaml /etc/otel/collector/otlp_config.yaml
13+
COPY config/collector/splunk_config_linux.yaml /etc/otel/collector/splunk_config_linux.yaml
14+
COPY config/collector/splunk_config_non_linux.yaml /etc/otel/collector/splunk_config_non_linux.yaml
15+
COPY config/collector/otlp_config_linux.yaml /etc/otel/collector/otlp_config_linux.yaml
16+
COPY config/collector/otlp_config_non_linux.yaml /etc/otel/collector/otlp_config_non_linux.yaml
1517
ENTRYPOINT ["/otelcol"]
16-
CMD ["--config", "/etc/otel/collector/splunk_config.yaml"]
17-
EXPOSE 13133 14250 14268 55678 55679 55680 7276 9411 9443
18+
EXPOSE 13133 14250 14268 55678 55679 55680 6060 7276 8888 9411 9443

‎cmd/otelcol/config/collector/otlp_config_linux.yaml

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
# Configuration file that uses the OTLP exporters to push data to Splunk products.
2+
# Currently supports only tracing.
3+
4+
extensions:
5+
health_check:
6+
http_forwarder:
7+
egress:
8+
endpoint: "https://api.${SPLUNK_REALM}.signalfx.com"
9+
zpages:
10+
11+
receivers:
12+
jaeger:
13+
protocols:
14+
grpc:
15+
thrift_binary:
16+
thrift_compact:
17+
thrift_http:
18+
opencensus:
19+
otlp:
20+
protocols:
21+
grpc:
22+
http:
23+
# This section is used to collect the OpenTelemetry Collector metrics
24+
# Even if just a Splunk µAPM customer, these metrics are included
25+
prometheus:
26+
config:
27+
scrape_configs:
28+
- job_name: 'otel-collector'
29+
scrape_interval: 10s
30+
static_configs:
31+
- targets: ['0.0.0.0:8888']
32+
metric_relabel_configs:
33+
- source_labels: [ __name__ ]
34+
regex: '.*grpc_io.*'
35+
action: drop
36+
sapm:
37+
signalfx:
38+
zipkin:
39+
40+
processors:
41+
batch:
42+
# Enabling the memory_limiter is strongly recommended for every pipeline.
43+
# Configuration is based on the amount of memory allocated to the collector.
44+
# The configuration below assumes 2GB of memory. In general, the ballast
45+
# should be set to 1/3 of the collector's memory, the limit should be 90% of
46+
# the collector's memory up to 2GB, and the spike should be 25% of the
47+
# collector's memory up to 2GB. The simplest way to specify the ballast size is
48+
# set the value of SPLUNK_BALLAST_SIZE_MIB env variable. This will overrides
49+
# the value of --mem-ballast-size-mib command line flag. If SPLUNK_BALLAST_SIZE_MIB
50+
# is not defined then --mem-ballast-size-mib command line flag must be manually specified.
51+
# For more information about memory limiter, see
52+
# https://github.com/open-telemetry/opentelemetry-collector/blob/master/processor/memorylimiter/README.md
53+
memory_limiter:
54+
ballast_size_mib: ${SPLUNK_BALLAST_SIZE_MIB}
55+
check_interval: 2s
56+
limit_percentage: ${SPLUNK_MEMORY_LIMIT_PERCENTAGE}
57+
spike_limit_percentage: ${SPLUNK_MEMORY_SPIKE_PERCENTAGE}
58+
59+
exporters:
60+
# Traces
61+
otlp:
62+
endpoint: "https://ingest.${SPLUNK_REALM}.signalfx.com/v2/trace/otlp"
63+
insecure: true
64+
headers:
65+
"X-SF-Token": "${SPLUNK_ACCESS_TOKEN}"
66+
# Metrics + Events
67+
signalfx:
68+
access_token: "${SPLUNK_ACCESS_TOKEN}"
69+
realm: "${SPLUNK_REALM}"
70+
71+
service:
72+
pipelines:
73+
traces:
74+
receivers: [jaeger, opencensus, otlp, sapm, zipkin]
75+
processors: [memory_limiter, batch]
76+
exporters: [otlp]
77+
metrics:
78+
receivers: [opencensus, otlp, signalfx, prometheus]
79+
processors: [memory_limiter, batch]
80+
exporters: [signalfx]
81+
logs:
82+
receivers: [signalfx]
83+
processors: [memory_limiter, batch]
84+
exporters: [signalfx]
85+
86+
extensions: [health_check, http_forwarder, zpages]

‎cmd/otelcol/config/collector/otlp_config.yaml renamed to ‎cmd/otelcol/config/collector/otlp_config_non_linux.yaml

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,9 @@
33

44
extensions:
55
health_check:
6+
http_forwarder:
7+
egress:
8+
endpoint: "https://api.${SPLUNK_REALM}.signalfx.com"
69
zpages:
710

811
receivers:
@@ -50,8 +53,8 @@ processors:
5053
memory_limiter:
5154
ballast_size_mib: ${SPLUNK_BALLAST_SIZE_MIB}
5255
check_interval: 2s
53-
limit_percentage: 90
54-
spike_limit_percentage: 25
56+
limit_mib: ${SPLUNK_MEMORY_LIMIT_MIB}
57+
spike_limit_mib: ${SPLUNK_MEMORY_SPIKE_MIB}
5558

5659
exporters:
5760
# Traces
@@ -80,4 +83,4 @@ service:
8083
processors: [memory_limiter, batch]
8184
exporters: [signalfx]
8285

83-
extensions: [health_check, zpages]
86+
extensions: [health_check, http_forwarder, zpages]

‎cmd/otelcol/config/collector/splunk_config_linux.yaml

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
# Configuration file that uses the Splunk exporters (SAPM, SignalFX) to push
2+
# data to Splunk products.
3+
4+
extensions:
5+
health_check:
6+
http_forwarder:
7+
egress:
8+
endpoint: "https://api.${SPLUNK_REALM}.signalfx.com"
9+
zpages:
10+
11+
receivers:
12+
jaeger:
13+
protocols:
14+
grpc:
15+
thrift_binary:
16+
thrift_compact:
17+
thrift_http:
18+
opencensus:
19+
otlp:
20+
protocols:
21+
grpc:
22+
http:
23+
# This section is used to collect the OpenTelemetry Collector metrics
24+
# Even if just a Splunk µAPM customer, these metrics are included
25+
prometheus:
26+
config:
27+
scrape_configs:
28+
- job_name: 'otel-collector'
29+
scrape_interval: 10s
30+
static_configs:
31+
- targets: ['0.0.0.0:8888']
32+
metric_relabel_configs:
33+
- source_labels: [ __name__ ]
34+
regex: '.*grpc_io.*'
35+
action: drop
36+
sapm:
37+
signalfx:
38+
zipkin:
39+
40+
processors:
41+
batch:
42+
# Enabling the memory_limiter is strongly recommended for every pipeline.
43+
# Configuration is based on the amount of memory allocated to the collector.
44+
# The configuration below assumes 2GB of memory. In general, the ballast
45+
# should be set to 1/3 of the collector's memory, the limit should be 90% of
46+
# the collector's memory up to 2GB, and the spike should be 25% of the
47+
# collector's memory up to 2GB. The simplest way to specify the ballast size is
48+
# set the value of SPLUNK_BALLAST_SIZE_MIB env variable. This will overrides
49+
# the value of --mem-ballast-size-mib command line flag. If SPLUNK_BALLAST_SIZE_MIB
50+
# is not defined then --mem-ballast-size-mib command line flag must be manually specified.
51+
# For more information about memory limiter, see
52+
# https://github.com/open-telemetry/opentelemetry-collector/blob/master/processor/memorylimiter/README.md
53+
memory_limiter:
54+
ballast_size_mib: ${SPLUNK_BALLAST_SIZE_MIB}
55+
check_interval: 2s
56+
limit_percentage: ${SPLUNK_MEMORY_LIMIT_PERCENTAGE}
57+
spike_limit_percentage: ${SPLUNK_MEMORY_SPIKE_PERCENTAGE}
58+
59+
exporters:
60+
# Traces
61+
sapm:
62+
access_token: "${SPLUNK_ACCESS_TOKEN}"
63+
endpoint: "https://ingest.${SPLUNK_REALM}.signalfx.com/v2/trace"
64+
# Metrics + Events
65+
signalfx:
66+
access_token: "${SPLUNK_ACCESS_TOKEN}"
67+
realm: "${SPLUNK_REALM}"
68+
69+
service:
70+
pipelines:
71+
traces:
72+
receivers: [jaeger, opencensus, otlp, sapm, zipkin]
73+
processors: [memory_limiter, batch]
74+
exporters: [sapm]
75+
metrics:
76+
receivers: [opencensus, otlp, signalfx, prometheus]
77+
processors: [memory_limiter, batch]
78+
exporters: [signalfx]
79+
logs:
80+
receivers: [signalfx]
81+
processors: [memory_limiter, batch]
82+
exporters: [signalfx]
83+
84+
extensions: [health_check, http_forwarder, zpages]

‎cmd/otelcol/config/collector/splunk_config.yaml renamed to ‎cmd/otelcol/config/collector/splunk_config_non_linux.yaml

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,9 @@
33

44
extensions:
55
health_check:
6+
http_forwarder:
7+
egress:
8+
endpoint: "https://api.${SPLUNK_REALM}.signalfx.com"
69
zpages:
710

811
receivers:
@@ -50,8 +53,8 @@ processors:
5053
memory_limiter:
5154
ballast_size_mib: ${SPLUNK_BALLAST_SIZE_MIB}
5255
check_interval: 2s
53-
limit_percentage: 90
54-
spike_limit_percentage: 25
56+
limit_mib: ${SPLUNK_MEMORY_LIMIT_MIB}
57+
spike_limit_mib: ${SPLUNK_MEMORY_SPIKE_MIB}
5558

5659
exporters:
5760
# Traces
@@ -78,4 +81,4 @@ service:
7881
processors: [memory_limiter, batch]
7982
exporters: [signalfx]
8083

81-
extensions: [health_check, zpages]
84+
extensions: [health_check, http_forwarder, zpages]

0 commit comments

Comments
 (0)