Skip to content

Commit cb4e27e

Browse files
rmfitzpatrickrmfitzpatrick
authored
Add discoverybundler, initial embedded bundle.d, and enabled properties (#2601)
1 parent 631a752 commit cb4e27e

29 files changed

+1003
-97
lines changed

‎Makefile

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -149,6 +149,10 @@ migratecheckpoint:
149149
GO111MODULE=on CGO_ENABLED=0 go build -trimpath -o ./bin/migratecheckpoint_$(GOOS)_$(GOARCH)$(EXTENSION) $(BUILD_INFO) ./cmd/migratecheckpoint
150150
ln -sf migratecheckpoint_$(GOOS)_$(GOARCH)$(EXTENSION) ./bin/migratecheckpoint
151151

152+
.PHONY: bundle.d
153+
bundle.d:
154+
go generate -tags bundle.d ./...
155+
152156
.PHONY: add-tag
153157
add-tag:
154158
@[ "${TAG}" ] || ( echo ">> env var TAG is not set"; exit 1 )

‎go.mod

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -469,7 +469,7 @@ require (
469469
gopkg.in/natefinch/lumberjack.v2 v2.0.0 // indirect
470470
gopkg.in/square/go-jose.v2 v2.6.0 // indirect
471471
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 // indirect
472-
gopkg.in/yaml.v3 v3.0.1 // indirect
472+
gopkg.in/yaml.v3 v3.0.1
473473
k8s.io/api v0.26.1 // indirect
474474
k8s.io/apimachinery v0.26.1 // indirect
475475
k8s.io/client-go v0.26.1 // indirect

‎internal/confmapprovider/discovery/README.md

Lines changed: 64 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ graph LR
1616
2 --> 2a1>otlp.yaml]
1717
2a1 --> 2b1[[otlp:<br>endpoint: 1.2.3.4:2345]]
1818
2 --> 2a2>logging.yaml]
19-
2a2 --> 2b2[[logging:<br>loglevel: debug]]
19+
2a2 --> 2b2[[logging:<br>verbosity: detailed]]
2020
end
2121
config.d --> 3[/extensions/]
2222
subgraph 3a[extensions]
@@ -35,17 +35,70 @@ graph LR
3535
config.d --> 5[/receivers/]
3636
subgraph 5a[receivers]
3737
5 --> 5a1>otlp.yaml]
38-
5a1 --> 5b1[[otlp:<br>protocols:<br>grpc]]
38+
5a1 --> 5b1[[otlp:<br>protocols:<br>grpc:]]
3939
end
4040
```
4141

42-
This component is currently exposed in the Collector via the `--configd` option with corresponding
43-
`--config-dir <config.d path>` and `SPLUNK_CONFIG_DIR` option and environment variable to load
44-
additional components and service configuration from the specified `config.d` directory (`/etc/otel/collector/config.d`
45-
by default).
42+
This component is currently supported in the Collector settings via the following commandline options:
4643

47-
This component is also exposed in the Collector via the `--discovery [--dry-run]` option that also uses the
48-
`--config-dir <config.d path>` and `SPLUNK_CONFIG_DIR` option and environment variable that attempts to
49-
instantiate any `.discovery.yaml` receivers using corresponding `.discovery.yaml` observers in a "preflight"
50-
Collector service, using any successfully discovered entities in the final config, or writing it to stdout
51-
if `--dry-run` was specified.
44+
| option | environment variable | default | description |
45+
|----------------|----------------------|--------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------|
46+
| `--configd` | none | disabled | Whether to enable `config.d` functionality for final Collector config content. |
47+
| `--config-dir` | `SPLUNK_CONFIG_DIR` | `/etc/otel/collector/config.d` | The root `config.d` directory to walk for component directories and yaml mapping files. |
48+
| `--dry-run` | none | disabled | Whether to report the final assembled config contents to stdout before immediately exiting. This can be used with or without `config.d` |
49+
50+
To source only `config.d` content and not an additional or default configuration file, the `--config` option or
51+
`SPLUNK_CONFIG` environment variable must be set to `/dev/null` or an arbitrary empty file:
52+
53+
```bash
54+
$ # run the Collector without a config file using components from a local ./config.d config directory,
55+
$ # printing the config to stdout before exiting instead of starting the Collector service:
56+
$ bin/otelcol --config /dev/null --configd --config-dir ./config.d --dry-run
57+
2023/02/24 19:54:23 settings.go:331: Set config to [/dev/null]
58+
2023/02/24 19:54:23 settings.go:384: Set ballast to 168 MiB
59+
2023/02/24 19:54:23 settings.go:400: Set memory limit to 460 MiB
60+
exporters:
61+
logging:
62+
verbosity: detailed
63+
otlp:
64+
endpoint: 1.2.3.4:2345
65+
extensions:
66+
health_check:
67+
path: /health
68+
zpages:
69+
endpoint: 0.0.0.0:1234
70+
processors:
71+
batch: {}
72+
resourcedetection:
73+
detectors:
74+
- system
75+
receivers:
76+
otlp:
77+
protocols:
78+
grpc: null
79+
service:
80+
pipelines:
81+
metrics:
82+
exporters:
83+
- logging
84+
receivers:
85+
- otlp
86+
```
87+
88+
## Discovery Mode
89+
90+
This component also provides a `--discovery [--dry-run]` option compatible with `config.d` that attempts to instantiate
91+
any `.discovery.yaml` receivers using corresponding `.discovery.yaml` observers in a "preflight" Collector service.
92+
Discovery mode will:
93+
94+
1. Load and attempt to start any observers in `config.d/extensions/<name>.discovery.yaml`.
95+
1. Load and attempt to start any receiver blocks in `config.d/receivers/<name>.discovery.yaml` in a
96+
[Discovery Receiver](../../receiver/discoveryreceiver/README.md) instance to receive discovery events from all
97+
successfully started observers.
98+
1. Wait 10s or the configured `SPLUNK_DISCOVERY_DURATION` environment variable [`time.Duration`](https://pkg.go.dev/time#ParseDuration).
99+
1. Embed any receiver instances' configs resulting in a `discovery.status` of `successful` inside a `receiver_creator/discovery` receiver's configuration to be passed to the final Collector service config (or outputted w/ `--dry-run`).
100+
1. Log any receiver resulting in a `discovery.status` of `partial` with the configured guidance for setting any relevant discovery properties.
101+
1. Stop all temporary components before continuing on to the actual Collector service (or exiting early with `--dry-run`).
102+
103+
104+
By default, the Discovery mode is provided with pre-made discovery config components in [`bundle.d`](./bundle/README.md).

‎internal/confmapprovider/discovery/bundle/README.md

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
## bundle.d
2+
3+
`bundle.d` refers to the [`embed.FS`](https://pkg.go.dev/embed#hdr-File_Systems) config directory made available by the
4+
[`bundle.BundledFS`](./bundle.go). It currently consists of all `./bundle.d/extensions/*.discovery.yaml` and
5+
`./bundle.d/receivers/*.discovery.yaml` files that are generated by the `discoverybundler` cmd as used by `go:generate`
6+
directives in [bundle_gen.go](./bundle_gen.go).
7+
8+
To construct the latest bundle.d contents before building the collector run:
9+
10+
```bash
11+
$ make bundle.d
12+
```
13+
14+
### *.discovery.yaml.tmpl
15+
16+
All discovery config component discovery.yaml files are generated from [`text/template`](https://pkg.go.dev/text/template)
17+
`discovery.yaml.tmpl` files using built-in validators and property guidance helpers:
18+
19+
Example `redis.discovery.yaml.tmpl`:
20+
21+
```yaml
22+
{{ receiver "redis" }}:
23+
rule:
24+
docker_observer: type == "container" and port == 6379
25+
<...>
26+
status:
27+
<...>
28+
statements:
29+
partial:
30+
- regexp: 'ERR AUTH.*'
31+
first_only: true
32+
log_record:
33+
severity_text: info
34+
body: >-
35+
Please ensure your redis password is correctly specified with
36+
`--set {{ configProperty "password" "<password>" }}` or
37+
`{{ configPropertyEnvVar "password" "<username>" }}` environment variable.
38+
```
39+
40+
After adding the required generate directive to `bundle_gen.go` and running `make bundle.d`:
41+
42+
```go
43+
//go:generate discoverybundler -r -t bundle.d/receivers/redis.discovery.yaml.tmpl
44+
```
45+
46+
There is now a corresponding `bundle.d/receiver/redis.discovery.yaml`:
47+
48+
```yaml
49+
#####################################################################################
50+
# This file is generated by the Splunk Distribution of the OpenTelemetry Collector. #
51+
#####################################################################################
52+
redis:
53+
rule:
54+
docker_observer: type == "container" and port == 6379
55+
<...>
56+
status:
57+
<...>
58+
statements:
59+
partial:
60+
- regexp: 'ERR AUTH.*'
61+
first_only: true
62+
log_record:
63+
severity_text: info
64+
body: >-
65+
Please ensure your redis password is correctly specified with
66+
`--set splunk.discovery.receivers.redis.config.password="<password>"` or
67+
`SPLUNK_DISCOVERY_RECEIVERS_redis_CONFIG_password="<username>"` environment variable.
68+
```
69+
70+
When building the collector afterward, this redis receiver discovery config is now made available to discovery mode, and
71+
it can be disabled by `--set splunk.discovery.receivers.redis.enabled=false` or
72+
`SPLUNK_DISCOVERY_RECEIVERS_redis_ENABLED=false`.

‎internal/confmapprovider/discovery/bundle/bundle.d/extensions/docker-observer.discovery.yaml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
#####################################################################################
2+
# This file is generated by the Splunk Distribution of the OpenTelemetry Collector. #
3+
#####################################################################################
4+
docker_observer:

‎internal/confmapprovider/discovery/bundle/bundle.d/extensions/docker-observer.discovery.yaml.tmpl

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
{{ extension "docker_observer" }}:

‎internal/confmapprovider/discovery/bundle/bundle.d/extensions/host-observer.discovery.yaml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
#####################################################################################
2+
# This file is generated by the Splunk Distribution of the OpenTelemetry Collector. #
3+
#####################################################################################
4+
host_observer:

‎internal/confmapprovider/discovery/bundle/bundle.d/extensions/host-observer.discovery.yaml.tmpl

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
{{ extension "host_observer" }}:

‎internal/confmapprovider/discovery/bundle/bundle.d/extensions/k8s-observer.discovery.yaml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
#####################################################################################
2+
# This file is generated by the Splunk Distribution of the OpenTelemetry Collector. #
3+
#####################################################################################
4+
k8s_observer:
5+
auth_type: serviceAccount

‎internal/confmapprovider/discovery/bundle/bundle.d/extensions/k8s-observer.discovery.yaml.tmpl

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
{{ extension "k8s_observer" }}:
2+
auth_type: serviceAccount

0 commit comments

Comments
 (0)