Skip to content

Commit d0dfb3d

Browse files
dloucasfxpjanotti
dloucasfx
and
pjanotti
authored
Add a new discovery bundle for weaviate vector search engine (#6884)
* Add a new discovery bundle for weaviate vector search engine Signed-off-by: Dani Louca <[email protected]> * Fix Integration test Signed-off-by: Dani Louca <[email protected]> --------- Signed-off-by: Dani Louca <[email protected]> Co-authored-by: Paulo Janotti <[email protected]>
1 parent 1e72bbe commit d0dfb3d

File tree

14 files changed

+683
-3
lines changed

14 files changed

+683
-3
lines changed

‎.chloggen/weaviate-discovery.yaml‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
2+
change_type: enhancement
3+
4+
# The name of the component, or a single word describing the area of concern, (e.g. crosslink)
5+
component: discovery
6+
7+
# A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`).
8+
note: Add a new discovery bundle for weaviate vector search engine
9+
10+
# One or more tracking issues related to the change
11+
issues: [6884]
12+
13+
# (Optional) One or more lines of additional information to render under the primary note.
14+
# These lines will be padded with 2 spaces and then inserted directly into the document.
15+
# Use pipe (|) for multiline entries.
16+
subtext: The discovery rule targets Weaviate instances exposing Prometheus metrics on the default port 2112.

‎.github/workflows/integration-test.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -315,7 +315,7 @@ jobs:
315315
id: get-matrix
316316
run: |
317317
includes=""
318-
for service in "apache" "jmx/cassandra" "kafkametrics" "mongodb" "nginx" "envoy" "oracledb" "mysql" "redis"; do
318+
for service in "apache" "jmx/cassandra" "kafkametrics" "mongodb" "nginx" "envoy" "oracledb" "mysql" "redis" "weaviate"; do
319319
for arch in "amd64" "arm64"; do
320320
if [ "$service" = "mongodb" ]; then
321321
# tests for mongo "6.0" and "7.0" are flaky, skipping for now

‎Makefile‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -163,6 +163,14 @@ integration-test-nginx-discovery-with-cover:
163163
integration-test-redis-discovery:
164164
@make integration-test-target TARGET='discovery_integration_redis'
165165

166+
.PHONY: integration-test-weaviate-discovery-with-cover
167+
integration-test-weaviate-discovery-with-cover:
168+
@make integration-test-cover-target TARGET='discovery_integration_weaviate'
169+
170+
.PHONY: integration-test-weaviate-discovery
171+
integration-test-weaviate-discovery:
172+
@make integration-test-target TARGET='discovery_integration_weaviate'
173+
166174
.PHONY: integration-test-redis-discovery-with-cover
167175
integration-test-redis-discovery-with-cover:
168176
@make integration-test-cover-target TARGET='discovery_integration_redis'

‎cmd/discoverybundler/metadata/receivers/weaviate.yaml‎

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
receiver_id: prometheus/weaviate
2+
service_type: weaviate
3+
properties_tmpl: |
4+
enabled: true
5+
rule:
6+
docker_observer: type == "container" and port == 2112 and any([name, image, command], {# matches "(?i)weaviate"}) and not (command matches "splunk.discovery")
7+
k8s_observer: type == "port" and port == 2112 and pod.name matches "(?i)weaviate"
8+
config:
9+
default:
10+
config:
11+
scrape_configs:
12+
- job_name: 'weaviate'
13+
metrics_path: /metrics
14+
scrape_interval: 10s
15+
static_configs:
16+
- targets: ['`endpoint`']
17+
metric_relabel_configs:
18+
- source_labels: [__name__]
19+
action: keep
20+
regex: "(async_operations_running|\
21+
batch_delete_durations_ms|\
22+
batch_durations_ms|\
23+
index_queue_delete_duration_ms|\
24+
index_queue_paused|\
25+
index_queue_preload_count|\
26+
index_queue_preload_duration_ms|\
27+
index_queue_push_duration_ms|\
28+
index_queue_search_duration_ms|\
29+
index_queue_size|\
30+
index_queue_stale_count|\
31+
index_queue_vectors_dequeued|\
32+
index_queue_wait_duration_ms|\
33+
lsm_active_segments|\
34+
lsm_bloom_filter_duration_ms|\
35+
lsm_segment_count|\
36+
lsm_segment_objects|\
37+
lsm_segment_size|\
38+
object_count|\
39+
objects_durations_ms|\
40+
requests_total|\
41+
startup_diskio_throughput|\
42+
startup_durations_ms|\
43+
vector_index_durations_ms|\
44+
vector_index_maintenance_durations_ms|\
45+
vector_index_operations|\
46+
vector_index_size|\
47+
vector_index_tombstone_cleaned|\
48+
vector_index_tombstone_cleanup_threads|\
49+
vector_index_tombstones|\
50+
weaviate_build_info|\
51+
weaviate_runtime_config_hash|\
52+
weaviate_runtime_config_last_load_success|\
53+
weaviate_schema_collections|\
54+
weaviate_schema_shards|\
55+
weaviate_internal_sample_memberlist_queue_broadcasts|\
56+
weaviate_internal_timer_memberlist_gossip|\
57+
weaviate_internal_counter_raft_apply|\
58+
weaviate_internal_counter_raft_state_candidate|\
59+
weaviate_internal_counter_raft_state_follower|\
60+
weaviate_internal_counter_raft_state_leader|\
61+
weaviate_internal_counter_raft_transition_heartbeat_timeout|\
62+
weaviate_internal_gauge_raft_commitNumLogs|\
63+
weaviate_internal_gauge_raft_leader_dispatchNumLogs|\
64+
weaviate_internal_gauge_raft_leader_oldestLogAge|\
65+
weaviate_internal_gauge_raft_peers|\
66+
weaviate_internal_sample_raft_boltdb_logBatchSize|\
67+
weaviate_internal_sample_raft_boltdb_logSize|\
68+
weaviate_internal_sample_raft_boltdb_logsPerBatch|\
69+
weaviate_internal_sample_raft_boltdb_writeCapacity|\
70+
weaviate_internal_sample_raft_thread_fsm_saturation|\
71+
weaviate_internal_sample_raft_thread_main_saturation|\
72+
weaviate_internal_timer_raft_boltdb_getLog|\
73+
weaviate_internal_timer_raft_boltdb_storeLogs|\
74+
weaviate_internal_timer_raft_commitTime|\
75+
weaviate_internal_timer_raft_fsm_apply|\
76+
weaviate_internal_timer_raft_fsm_enqueue|\
77+
weaviate_internal_timer_raft_leader_dispatchLog|\
78+
weaviate_usage_gcs_operations_total|\
79+
weaviate_usage_gcs_operation_latency_seconds|\
80+
weaviate_usage_gcs_resource_count|\
81+
weaviate_usage_gcs_uploaded_file_size_bytes|\
82+
weaviate_usage_s3_operations_total|\
83+
weaviate_usage_s3_operation_latency_seconds|\
84+
weaviate_usage_s3_resource_count|\
85+
weaviate_usage_s3_uploaded_file_size_bytes)"
86+
status:
87+
metrics:
88+
- status: successful
89+
strict: weaviate_build_info
90+
message: weaviate prometheus receiver is working!
91+
statements:
92+
- status: failed
93+
regexp: "connection refused"
94+
message: The container is not serving http connections.
95+
- status: failed
96+
regexp: "dial tcp: lookup"
97+
message: Unable to resolve weaviate prometheus tcp endpoint

‎cmd/otelcol/config/collector/config.d.linux/receivers/weaviate.discovery.yaml‎

Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
#####################################################################################
2+
# This file is generated by the Splunk Distribution of the OpenTelemetry Collector. #
3+
# #
4+
# It reflects the default configuration bundled in the Collector executable for use #
5+
# in discovery mode (--discovery) and is provided for reference or customization. #
6+
# Please note that any changes made to this file will need to be reconciled during #
7+
# upgrades of the Collector. #
8+
#####################################################################################
9+
# prometheus/weaviate:
10+
# enabled: true
11+
# rule:
12+
# docker_observer: type == "container" and port == 2112 and any([name, image, command], {# matches "(?i)weaviate"}) and not (command matches "splunk.discovery")
13+
# k8s_observer: type == "port" and port == 2112 and pod.name matches "(?i)weaviate"
14+
# config:
15+
# default:
16+
# config:
17+
# scrape_configs:
18+
# - job_name: 'weaviate'
19+
# metrics_path: /metrics
20+
# scrape_interval: 10s
21+
# static_configs:
22+
# - targets: ['`endpoint`']
23+
# metric_relabel_configs:
24+
# - source_labels: [__name__]
25+
# action: keep
26+
# regex: "(async_operations_running|\
27+
# batch_delete_durations_ms|\
28+
# batch_durations_ms|\
29+
# index_queue_delete_duration_ms|\
30+
# index_queue_paused|\
31+
# index_queue_preload_count|\
32+
# index_queue_preload_duration_ms|\
33+
# index_queue_push_duration_ms|\
34+
# index_queue_search_duration_ms|\
35+
# index_queue_size|\
36+
# index_queue_stale_count|\
37+
# index_queue_vectors_dequeued|\
38+
# index_queue_wait_duration_ms|\
39+
# lsm_active_segments|\
40+
# lsm_bloom_filter_duration_ms|\
41+
# lsm_segment_count|\
42+
# lsm_segment_objects|\
43+
# lsm_segment_size|\
44+
# object_count|\
45+
# objects_durations_ms|\
46+
# requests_total|\
47+
# startup_diskio_throughput|\
48+
# startup_durations_ms|\
49+
# vector_index_durations_ms|\
50+
# vector_index_maintenance_durations_ms|\
51+
# vector_index_operations|\
52+
# vector_index_size|\
53+
# vector_index_tombstone_cleaned|\
54+
# vector_index_tombstone_cleanup_threads|\
55+
# vector_index_tombstones|\
56+
# weaviate_build_info|\
57+
# weaviate_runtime_config_hash|\
58+
# weaviate_runtime_config_last_load_success|\
59+
# weaviate_schema_collections|\
60+
# weaviate_schema_shards|\
61+
# weaviate_internal_sample_memberlist_queue_broadcasts|\
62+
# weaviate_internal_timer_memberlist_gossip|\
63+
# weaviate_internal_counter_raft_apply|\
64+
# weaviate_internal_counter_raft_state_candidate|\
65+
# weaviate_internal_counter_raft_state_follower|\
66+
# weaviate_internal_counter_raft_state_leader|\
67+
# weaviate_internal_counter_raft_transition_heartbeat_timeout|\
68+
# weaviate_internal_gauge_raft_commitNumLogs|\
69+
# weaviate_internal_gauge_raft_leader_dispatchNumLogs|\
70+
# weaviate_internal_gauge_raft_leader_oldestLogAge|\
71+
# weaviate_internal_gauge_raft_peers|\
72+
# weaviate_internal_sample_raft_boltdb_logBatchSize|\
73+
# weaviate_internal_sample_raft_boltdb_logSize|\
74+
# weaviate_internal_sample_raft_boltdb_logsPerBatch|\
75+
# weaviate_internal_sample_raft_boltdb_writeCapacity|\
76+
# weaviate_internal_sample_raft_thread_fsm_saturation|\
77+
# weaviate_internal_sample_raft_thread_main_saturation|\
78+
# weaviate_internal_timer_raft_boltdb_getLog|\
79+
# weaviate_internal_timer_raft_boltdb_storeLogs|\
80+
# weaviate_internal_timer_raft_commitTime|\
81+
# weaviate_internal_timer_raft_fsm_apply|\
82+
# weaviate_internal_timer_raft_fsm_enqueue|\
83+
# weaviate_internal_timer_raft_leader_dispatchLog|\
84+
# weaviate_usage_gcs_operations_total|\
85+
# weaviate_usage_gcs_operation_latency_seconds|\
86+
# weaviate_usage_gcs_resource_count|\
87+
# weaviate_usage_gcs_uploaded_file_size_bytes|\
88+
# weaviate_usage_s3_operations_total|\
89+
# weaviate_usage_s3_operation_latency_seconds|\
90+
# weaviate_usage_s3_resource_count|\
91+
# weaviate_usage_s3_uploaded_file_size_bytes)"

‎docker/docker-compose.yml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -249,6 +249,14 @@ services:
249249
ports:
250250
- "6379:6379"
251251
# Kafka test, requires multiple Kafka containers and Zookeeper:
252+
weaviate:
253+
image: cr.weaviate.io/semitechnologies/weaviate:1.33.2
254+
profiles:
255+
- integration-test-weaviate-discovery
256+
ports:
257+
- "2112:2112"
258+
environment:
259+
PROMETHEUS_MONITORING_ENABLED: "true"
252260
zookeeper:
253261
image: zookeeper:3.5
254262
profiles:

‎internal/confmapprovider/discovery/bundle.d/receivers/weaviate.discovery.yaml‎

Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
##############################################################################################
2+
# Do not edit manually! #
3+
# All changes must be made to associated .yaml metadata file before running 'make bundle.d'. #
4+
##############################################################################################
5+
prometheus/weaviate:
6+
enabled: true
7+
rule:
8+
docker_observer: type == "container" and port == 2112 and any([name, image, command], {# matches "(?i)weaviate"}) and not (command matches "splunk.discovery")
9+
k8s_observer: type == "port" and port == 2112 and pod.name matches "(?i)weaviate"
10+
config:
11+
default:
12+
config:
13+
scrape_configs:
14+
- job_name: 'weaviate'
15+
metrics_path: /metrics
16+
scrape_interval: 10s
17+
static_configs:
18+
- targets: ['`endpoint`']
19+
metric_relabel_configs:
20+
- source_labels: [__name__]
21+
action: keep
22+
regex: "(async_operations_running|\
23+
batch_delete_durations_ms|\
24+
batch_durations_ms|\
25+
index_queue_delete_duration_ms|\
26+
index_queue_paused|\
27+
index_queue_preload_count|\
28+
index_queue_preload_duration_ms|\
29+
index_queue_push_duration_ms|\
30+
index_queue_search_duration_ms|\
31+
index_queue_size|\
32+
index_queue_stale_count|\
33+
index_queue_vectors_dequeued|\
34+
index_queue_wait_duration_ms|\
35+
lsm_active_segments|\
36+
lsm_bloom_filter_duration_ms|\
37+
lsm_segment_count|\
38+
lsm_segment_objects|\
39+
lsm_segment_size|\
40+
object_count|\
41+
objects_durations_ms|\
42+
requests_total|\
43+
startup_diskio_throughput|\
44+
startup_durations_ms|\
45+
vector_index_durations_ms|\
46+
vector_index_maintenance_durations_ms|\
47+
vector_index_operations|\
48+
vector_index_size|\
49+
vector_index_tombstone_cleaned|\
50+
vector_index_tombstone_cleanup_threads|\
51+
vector_index_tombstones|\
52+
weaviate_build_info|\
53+
weaviate_runtime_config_hash|\
54+
weaviate_runtime_config_last_load_success|\
55+
weaviate_schema_collections|\
56+
weaviate_schema_shards|\
57+
weaviate_internal_sample_memberlist_queue_broadcasts|\
58+
weaviate_internal_timer_memberlist_gossip|\
59+
weaviate_internal_counter_raft_apply|\
60+
weaviate_internal_counter_raft_state_candidate|\
61+
weaviate_internal_counter_raft_state_follower|\
62+
weaviate_internal_counter_raft_state_leader|\
63+
weaviate_internal_counter_raft_transition_heartbeat_timeout|\
64+
weaviate_internal_gauge_raft_commitNumLogs|\
65+
weaviate_internal_gauge_raft_leader_dispatchNumLogs|\
66+
weaviate_internal_gauge_raft_leader_oldestLogAge|\
67+
weaviate_internal_gauge_raft_peers|\
68+
weaviate_internal_sample_raft_boltdb_logBatchSize|\
69+
weaviate_internal_sample_raft_boltdb_logSize|\
70+
weaviate_internal_sample_raft_boltdb_logsPerBatch|\
71+
weaviate_internal_sample_raft_boltdb_writeCapacity|\
72+
weaviate_internal_sample_raft_thread_fsm_saturation|\
73+
weaviate_internal_sample_raft_thread_main_saturation|\
74+
weaviate_internal_timer_raft_boltdb_getLog|\
75+
weaviate_internal_timer_raft_boltdb_storeLogs|\
76+
weaviate_internal_timer_raft_commitTime|\
77+
weaviate_internal_timer_raft_fsm_apply|\
78+
weaviate_internal_timer_raft_fsm_enqueue|\
79+
weaviate_internal_timer_raft_leader_dispatchLog|\
80+
weaviate_usage_gcs_operations_total|\
81+
weaviate_usage_gcs_operation_latency_seconds|\
82+
weaviate_usage_gcs_resource_count|\
83+
weaviate_usage_gcs_uploaded_file_size_bytes|\
84+
weaviate_usage_s3_operations_total|\
85+
weaviate_usage_s3_operation_latency_seconds|\
86+
weaviate_usage_s3_resource_count|\
87+
weaviate_usage_s3_uploaded_file_size_bytes)"

‎internal/confmapprovider/discovery/generated_bundledfs.go‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,4 +34,5 @@ import "embed"
3434
//go:embed bundle.d/receivers/rabbitmq.discovery.yaml
3535
//go:embed bundle.d/receivers/redis.discovery.yaml
3636
//go:embed bundle.d/receivers/sqlserver.discovery.yaml
37+
//go:embed bundle.d/receivers/weaviate.discovery.yaml
3738
var BundledFS embed.FS

‎internal/receiver/discoveryreceiver/generated_metadata.go‎

Lines changed: 24 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)