Skip to content

Commit 53c15ed

Browse files
rmfitzpatrickrmfitzpatrick
committed
Add discoverybundler, initial embedded bundle.d, and enabled properties
1 parent a809186 commit 53c15ed

29 files changed

+999
-95
lines changed

‎Makefile

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -149,6 +149,10 @@ migratecheckpoint:
149149
GO111MODULE=on CGO_ENABLED=0 go build -trimpath -o ./bin/migratecheckpoint_$(GOOS)_$(GOARCH)$(EXTENSION) $(BUILD_INFO) ./cmd/migratecheckpoint
150150
ln -sf migratecheckpoint_$(GOOS)_$(GOARCH)$(EXTENSION) ./bin/migratecheckpoint
151151

152+
.PHONY: bundle.d
153+
bundle.d:
154+
go generate -tags bundle.d ./...
155+
152156
.PHONY: add-tag
153157
add-tag:
154158
@[ "${TAG}" ] || ( echo ">> env var TAG is not set"; exit 1 )

‎go.mod

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -471,7 +471,7 @@ require (
471471
gopkg.in/natefinch/lumberjack.v2 v2.0.0 // indirect
472472
gopkg.in/square/go-jose.v2 v2.6.0 // indirect
473473
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 // indirect
474-
gopkg.in/yaml.v3 v3.0.1 // indirect
474+
gopkg.in/yaml.v3 v3.0.1
475475
k8s.io/api v0.26.1 // indirect
476476
k8s.io/apimachinery v0.26.1 // indirect
477477
k8s.io/client-go v0.26.1 // indirect

‎internal/confmapprovider/discovery/README.md

Lines changed: 60 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -35,17 +35,68 @@ graph LR
3535
config.d --> 5[/receivers/]
3636
subgraph 5a[receivers]
3737
5 --> 5a1>otlp.yaml]
38-
5a1 --> 5b1[[otlp:<br>protocols:<br>grpc]]
38+
5a1 --> 5b1[[otlp:<br>protocols:<br>grpc:]]
3939
end
4040
```
4141

42-
This component is currently exposed in the Collector via the `--configd` option with corresponding
43-
`--config-dir <config.d path>` and `SPLUNK_CONFIG_DIR` option and environment variable to load
42+
This component is currently supported in the Collector settings via the `--configd` option with corresponding
43+
`--config-dir <config.d path>` option and `SPLUNK_CONFIG_DIR` environment variable to load
4444
additional components and service configuration from the specified `config.d` directory (`/etc/otel/collector/config.d`
45-
by default).
45+
by default). You can use the terminating `--dry-run` flag to report the final assembled Collector config contents to
46+
stdout before exiting.
4647

47-
This component is also exposed in the Collector via the `--discovery [--dry-run]` option that also uses the
48-
`--config-dir <config.d path>` and `SPLUNK_CONFIG_DIR` option and environment variable that attempts to
49-
instantiate any `.discovery.yaml` receivers using corresponding `.discovery.yaml` observers in a "preflight"
50-
Collector service, using any successfully discovered entities in the final config, or writing it to stdout
51-
if `--dry-run` was specified.
48+
At this time, if you want to only use `config.d` for Collector config content and not an additional configuration file,
49+
you must set the `--config` option or `SPLUNK_CONFIG` environment variable to `/dev/null` or any empty file:
50+
51+
```bash
52+
$ # run the Collector without a config file using components from a local ./config.d config directory,
53+
$ # printing the config to stdout before exiting immediately:
54+
$ bin/otelcol --config /dev/null --configd --config-dir ./config.d --dry-run
55+
2023/02/24 19:54:23 settings.go:331: Set config to [/dev/null]
56+
2023/02/24 19:54:23 settings.go:384: Set ballast to 168 MiB
57+
2023/02/24 19:54:23 settings.go:400: Set memory limit to 460 MiB
58+
exporters:
59+
logging:
60+
loglevel: debug
61+
otlp:
62+
endpoint: 1.2.3.4:2345
63+
extensions:
64+
health_check:
65+
path: /health
66+
zpages:
67+
endpoint: 0.0.0.0:1234
68+
processors:
69+
batch: {}
70+
resourcedetection:
71+
detectors:
72+
- system
73+
receivers:
74+
otlp:
75+
protocols:
76+
grpc: null
77+
service:
78+
pipelines:
79+
metrics:
80+
exporters:
81+
- logging
82+
receivers:
83+
- otlp
84+
```
85+
86+
## Discovery Mode
87+
88+
This component also provides a `--discovery [--dry-run]` option compatible with `config.d` that attempts to instantiate
89+
any `.discovery.yaml` receivers using corresponding `.discovery.yaml` observers in a "preflight" Collector service.
90+
Discovery mode will:
91+
92+
1. Load and attempt to start any observers in `config.d/extensions/<name>.discovery.yaml`.
93+
1. Load and attempt to start any receiver blocks in `config.d/receivers/<name>.discovery.yaml` in a
94+
[Discovery Receiver](../../receiver/discoveryreceiver/README.md)` instance receiver events from all
95+
successfully started observers.
96+
1. Wait 10s or the configured `SPLUNK_DISCOVERY_DURATION` environment variable [`time.Duration`](https://pkg.go.dev/time#ParseDuration).
97+
1. Embed any receiver instances' configs resulting in a `discovery.status` of `successful` inside a `receiver_creator/discovery` receiver's configuration to be passed to the final Collector service config (or outputted w/ `--dry-run`).
98+
1. Log any receiver resulting in a `discovery.status` of `partial` with the configured guidance for setting any relevant discovery properties.
99+
1. Stop all temporary components before continuing on to the actual Collector service (or exiting early with `--dry-run`).
100+
101+
102+
By default the Discovery mode is provided with premade discovery config components in [`bundle.d`](./bundle/README.md).

‎internal/confmapprovider/discovery/bundle/README.md

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
## bundle.d
2+
3+
`bundle.d` refers to the [`embed.FS`](https://pkg.go.dev/embed#hdr-File_Systems) config directory made available by the
4+
[`bundle.BundledFS`](./bundle.go). It currently consists of all `./bundle.d/extensions/*.discovery.yaml` and
5+
`./bundle.d/receivers/*.discovery.yaml` files that are generated by the `discoverybundler` cmd as used by `go:generate`
6+
directives in [bundle_gen.go](./bundle_gen.go).
7+
8+
To construct the latest bundle.d contents before building the collector run:
9+
10+
```bash
11+
$ make bundle.d
12+
```
13+
14+
### *.discovery.yaml.tmpl
15+
16+
All discovery config component discovery.yaml files are generated from [`text/template`](https://pkg.go.dev/text/template)
17+
`discovery.yaml.tmpl` files using built-in validators and property guidance helpers:
18+
19+
Example `redis.discovery.yaml.tmpl`:
20+
21+
```yaml
22+
{{ receiver "redis" }}:
23+
rule:
24+
docker_observer: type == "container" and port == 6379
25+
<...>
26+
status:
27+
<...>
28+
statements:
29+
partial:
30+
- regexp: 'ERR AUTH.*'
31+
first_only: true
32+
log_record:
33+
severity_text: info
34+
body: >-
35+
Please ensure your redis password is correctly specified with
36+
`--set {{ configProperty "password" "<password>" }}` or
37+
`{{ configPropertyEnvVar "password" "<username>" }}` environment variable.
38+
```
39+
40+
After adding the required generate directive to `bundle_gen.go` and running `make bundle.d`:
41+
42+
```go
43+
//go:generate discoverybundler -r -t bundle.d/receivers/redis.discovery.yaml.tmpl
44+
```
45+
46+
There is now a corresponding `bundle.d/receiver/redis.discovery.yaml`:
47+
48+
```yaml
49+
#####################################################################################
50+
# This file is generated by the Splunk Distribution of the OpenTelemetry Collector. #
51+
#####################################################################################
52+
redis:
53+
rule:
54+
docker_observer: type == "container" and port == 6379
55+
<...>
56+
status:
57+
<...>
58+
statements:
59+
partial:
60+
- regexp: 'ERR AUTH.*'
61+
first_only: true
62+
log_record:
63+
severity_text: info
64+
body: >-
65+
Please ensure your redis password is correctly specified with
66+
`--set splunk.discovery.receivers.redis.config.password="<password>"` or
67+
`SPLUNK_DISCOVERY_RECEIVERS_redis_CONFIG_password="<username>"` environment variable.
68+
```
69+
70+
When building the collector afterward, this redis receiver discovery config is now made available to discovery mode, and
71+
it can be disabled by `--set splunk.discovery.receivers.redis.enabled=false` or
72+
`SPLUNK_DISCOVERY_RECEIVERS_redis_ENABLED=false`.

‎internal/confmapprovider/discovery/bundle/bundle.d/extensions/docker-observer.discovery.yaml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
#####################################################################################
2+
# This file is generated by the Splunk Distribution of the OpenTelemetry Collector. #
3+
#####################################################################################
4+
docker_observer:

‎internal/confmapprovider/discovery/bundle/bundle.d/extensions/docker-observer.discovery.yaml.tmpl

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
{{ extension "docker_observer" }}:

‎internal/confmapprovider/discovery/bundle/bundle.d/extensions/host-observer.discovery.yaml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
#####################################################################################
2+
# This file is generated by the Splunk Distribution of the OpenTelemetry Collector. #
3+
#####################################################################################
4+
host_observer:

‎internal/confmapprovider/discovery/bundle/bundle.d/extensions/host-observer.discovery.yaml.tmpl

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
{{ extension "host_observer" }}:

‎internal/confmapprovider/discovery/bundle/bundle.d/extensions/k8s-observer.discovery.yaml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
#####################################################################################
2+
# This file is generated by the Splunk Distribution of the OpenTelemetry Collector. #
3+
#####################################################################################
4+
k8s_observer:
5+
auth_type: serviceAccount

‎internal/confmapprovider/discovery/bundle/bundle.d/extensions/k8s-observer.discovery.yaml.tmpl

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
{{ extension "k8s_observer" }}:
2+
auth_type: serviceAccount

‎internal/confmapprovider/discovery/bundle/bundle.d/receivers/smartagent-postgresql.discovery.yaml

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
#####################################################################################
2+
# This file is generated by the Splunk Distribution of the OpenTelemetry Collector. #
3+
#####################################################################################
4+
smartagent/postgresql:
5+
rule:
6+
docker_observer: type == "container" and port == 5432
7+
host_observer: type == "hostport" and command contains "pg" and port == 5432
8+
config:
9+
default:
10+
type: postgresql
11+
connectionString: 'sslmode=disable user={{.username}} password={{.password}}'
12+
params:
13+
username: bundle.default
14+
password: bundle.default
15+
masterDBName: postgres
16+
status:
17+
metrics:
18+
successful:
19+
- strict: postgres_block_hit_ratio
20+
first_only: true
21+
log_record:
22+
severity_text: info
23+
body: postgresql SA receiver working!
24+
statements:
25+
failed:
26+
- regexp: '.* connect: connection refused'
27+
first_only: true
28+
log_record:
29+
severity_text: info
30+
body: container appears to not be accepting postgres connections
31+
partial:
32+
- regexp: '.*pq: password authentication failed for user.*'
33+
first_only: true
34+
log_record:
35+
severity_text: info
36+
body: >-
37+
Please ensure your user credentials are correctly specified with
38+
`--set splunk.discovery.receivers.smartagent/postgresql.config.params::username="<username>"` and
39+
`--set splunk.discovery.receivers.smartagent/postgresql.config.params::password="<password>"` or
40+
`SPLUNK_DISCOVERY_RECEIVERS_smartagent_x2f_postgresql_CONFIG_params_x3a__x3a_username="<username>"` and
41+
`SPLUNK_DISCOVERY_RECEIVERS_smartagent_x2f_postgresql_CONFIG_params_x3a__x3a_password="<password>"` environment variables.
42+
- regexp: '.*pq: database ".*" does not exist.*'
43+
first_only: true
44+
log_record:
45+
severity_text: info
46+
body: >-
47+
Please ensure your target database is correctly specified with
48+
`--set splunk.discovery.receivers.smartagent/postgresql.config.masterDBName="<db>"` or
49+
`SPLUNK_DISCOVERY_RECEIVERS_smartagent_x2f_postgresql_CONFIG_masterDBName="<db>"` environment variable.

‎internal/confmapprovider/discovery/bundle/bundle.d/receivers/smartagent-postgresql.discovery.yaml.tmpl

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
{{ receiver "smartagent/postgresql" }}:
2+
rule:
3+
docker_observer: type == "container" and port == 5432
4+
host_observer: type == "hostport" and command contains "pg" and port == 5432
5+
config:
6+
default:
7+
type: postgresql
8+
connectionString: 'sslmode=disable user={{ "{{.username}}" }} password={{ "{{.password}}" }}'
9+
params:
10+
username: bundle.default
11+
password: bundle.default
12+
masterDBName: postgres
13+
status:
14+
metrics:
15+
successful:
16+
- strict: postgres_block_hit_ratio
17+
first_only: true
18+
log_record:
19+
severity_text: info
20+
body: postgresql SA receiver working!
21+
statements:
22+
failed:
23+
- regexp: '.* connect: connection refused'
24+
first_only: true
25+
log_record:
26+
severity_text: info
27+
body: container appears to not be accepting postgres connections
28+
partial:
29+
- regexp: '.*pq: password authentication failed for user.*'
30+
first_only: true
31+
log_record:
32+
severity_text: info
33+
body: >-
34+
Please ensure your user credentials are correctly specified with
35+
`--set {{ configProperty "params" "username" "<username>" }}` and
36+
`--set {{ configProperty "params" "password" "<password>" }}` or
37+
`{{ configPropertyEnvVar "params" "username" "<username>" }}` and
38+
`{{ configPropertyEnvVar "params" "password" "<password>" }}` environment variables.
39+
- regexp: '.*pq: database ".*" does not exist.*'
40+
first_only: true
41+
log_record:
42+
severity_text: info
43+
body: >-
44+
Please ensure your target database is correctly specified with
45+
`--set {{ configProperty "masterDBName" "<db>" }}` or
46+
`{{ configPropertyEnvVar "masterDBName" "<db>" }}` environment variable.

‎internal/confmapprovider/discovery/bundle/bundle.go

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
// Copyright Splunk, Inc.
2+
//
3+
// Licensed under the Apache License, Version 2.0 (the "License");
4+
// you may not use this file except in compliance with the License.
5+
// You may obtain a copy of the License at
6+
//
7+
// http://www.apache.org/licenses/LICENSE-2.0
8+
//
9+
// Unless required by applicable law or agreed to in writing, software
10+
// distributed under the License is distributed on an "AS IS" BASIS,
11+
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
// See the License for the specific language governing permissions and
13+
// limitations under the License.
14+
15+
package bundle
16+
17+
import (
18+
"embed"
19+
)
20+
21+
// BundledFS is the in-executable filesystem that contains all bundled discovery config.d components.
22+
//
23+
// If you are bootstrapping bundle_gen.go or the `discoverybundler` cmd without any rendered files in bundle.d,
24+
// comment out the below embed directives before installing to prevent "no matching files found"
25+
// build errors.
26+
//
27+
//go:embed bundle.d/extensions/*.discovery.yaml
28+
//go:embed bundle.d/receivers/*.discovery.yaml
29+
var BundledFS embed.FS

‎internal/confmapprovider/discovery/bundle/bundle_gen.go

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
// Copyright Splunk, Inc.
2+
//
3+
// Licensed under the Apache License, Version 2.0 (the "License");
4+
// you may not use this file except in compliance with the License.
5+
// You may obtain a copy of the License at
6+
//
7+
// http://www.apache.org/licenses/LICENSE-2.0
8+
//
9+
// Unless required by applicable law or agreed to in writing, software
10+
// distributed under the License is distributed on an "AS IS" BASIS,
11+
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
// See the License for the specific language governing permissions and
13+
// limitations under the License.
14+
15+
//go:build bundle.d
16+
17+
// These are the discovery config component generating statements.
18+
// In order to update run go generate -tags bundle.d ./...
19+
//go:generate go install github.com/signalfx/splunk-otel-collector/internal/confmapprovider/discovery/bundle/cmd/discoverybundler
20+
21+
//go:generate discoverybundler -r -t bundle.d/receivers/smartagent-postgresql.discovery.yaml.tmpl
22+
23+
//go:generate discoverybundler -r -t bundle.d/extensions/docker-observer.discovery.yaml.tmpl
24+
//go:generate discoverybundler -r -t bundle.d/extensions/host-observer.discovery.yaml.tmpl
25+
//go:generate discoverybundler -r -t bundle.d/extensions/k8s-observer.discovery.yaml.tmpl
26+
27+
package bundle

‎internal/confmapprovider/discovery/bundle/bundle_test.go

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
// Copyright Splunk, Inc.
2+
//
3+
// Licensed under the Apache License, Version 2.0 (the "License");
4+
// you may not use this file except in compliance with the License.
5+
// You may obtain a copy of the License at
6+
//
7+
// http://www.apache.org/licenses/LICENSE-2.0
8+
//
9+
// Unless required by applicable law or agreed to in writing, software
10+
// distributed under the License is distributed on an "AS IS" BASIS,
11+
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
// See the License for the specific language governing permissions and
13+
// limitations under the License.
14+
15+
package bundle
16+
17+
import (
18+
"io/fs"
19+
"testing"
20+
21+
"github.com/stretchr/testify/require"
22+
)
23+
24+
func TestBundleDir(t *testing.T) {
25+
receivers, err := fs.Glob(BundledFS, "bundle.d/receivers/*.discovery.yaml")
26+
require.NoError(t, err)
27+
require.Equal(t, []string{
28+
"bundle.d/receivers/smartagent-postgresql.discovery.yaml",
29+
}, receivers)
30+
31+
extensions, err := fs.Glob(BundledFS, "bundle.d/extensions/*.discovery.yaml")
32+
require.NoError(t, err)
33+
require.Equal(t, []string{
34+
"bundle.d/extensions/docker-observer.discovery.yaml",
35+
"bundle.d/extensions/host-observer.discovery.yaml",
36+
"bundle.d/extensions/k8s-observer.discovery.yaml",
37+
}, extensions)
38+
}

0 commit comments

Comments
 (0)