Skip to content

Commit a685e72

Browse files
KiranNadupuruKiranNadupuru
authored
Merge pull request #4 from oracle-quickstart/v2.0.3
Release v2.0.3
2 parents 65967d9 + 4610238 commit a685e72

File tree

6 files changed

+370
-124
lines changed

6 files changed

+370
-124
lines changed

‎CHANGELOG.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,11 @@
11
# Change Log
22

3+
## 2.0.3 - 2022-04-20
4+
### Added
5+
- Added Prometheus-client Api based internal metrics functionality.
6+
### Bug fix
7+
- Fixing minimum required version for OCI Ruby SDK (oci) runtime dependency
8+
39
## 2.0.2 - 2022-02-17
410
### Added
511
- Added required ruby version in gemspec.

‎README.md

Lines changed: 42 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -20,13 +20,15 @@ Refer [Prerequisites](https://docs.oracle.com/en/learn/oci_logging_analytics_flu
2020
<th>ruby</th>
2121
<th>rubyzip</th>
2222
<th>oci</th>
23+
<th>prometheus-client</th>
2324
</tr>
2425
<tr>
2526
<td>>= 2.0.0</td>
2627
<td>>= 0.14.10, < 2 </td>
2728
<td>>= 2.6</td>
2829
<td>~> 2.3.2 </td>
29-
<td>~> 2.13</td>
30+
<td>~> 2.16</td>
31+
<td>~> 2.1.0</td>
3032
</tr>
3133
</table>
3234

@@ -81,12 +83,49 @@ Use filter plugin (record_transformer) to transform the input log events to add
8183

8284
- Example configuration that can be used for monitoring [kafka log](examples/kafka.conf)
8385

84-
85-
8686
## Start Viewing the Logs in Logging Analytics
8787

8888
Refer [Viewing the Logs in Logging Analytics](https://docs.oracle.com/en/learn/oci_logging_analytics_fluentd/#start-viewing-the-logs-in-logging-analytics)
8989

90+
## Metrics
91+
92+
The plugin emits following metrics in Prometheus format, which provides stats/insights about the data being collected and processed by the plugin. Refer [monitoring-prometheus](https://docs.fluentd.org/monitoring-fluentd/monitoring-prometheus) for details on how to expose these and other various Fluentd metrics to Prometheus (*If the requirement is to collect and monitor core Fluentd and this plugin metrics alone using Prometheus then Step1 and Step2 from the referred document can be skipped*).
93+
94+
Metric Name: oci_la_fluentd_output_plugin_records_received
95+
labels: [:tag,:oci_la_log_group_id,:oci_la_log_source_name,:oci_la_log_set]
96+
Description: Number of records received by the OCI Logging Analytics Fluentd output plugin.
97+
Type : Gauge
98+
99+
Metric Name: oci_la_fluentd_output_plugin_records_valid
100+
labels: [:tag,:oci_la_log_group_id,:oci_la_log_source_name,:oci_la_log_set]
101+
Description: Number of valid records received by the OCI Logging Analytics Fluentd output plugin.
102+
Type : Gauge
103+
104+
Metric Name: oci_la_fluentd_output_plugin_records_invalid
105+
labels: [:tag,:oci_la_log_group_id,:oci_la_log_source_name,:oci_la_log_set,:reason]
106+
Description: Number of invalid records received by the OCI Logging Analytics Fluentd output plugin.
107+
Type : Gauge
108+
109+
Metric Name: oci_la_fluentd_output_plugin_records_post_error
110+
labels: [:tag,:oci_la_log_group_id,:oci_la_log_source_name,:oci_la_log_set,:error_code, :reason]
111+
Description: Number of records failed posting to OCI Logging Analytics by the Fluentd output plugin.
112+
Type : Gauge
113+
114+
Metric Name: oci_la_fluentd_output_plugin_records_post_success
115+
labels: [:tag,:oci_la_log_group_id,:oci_la_log_source_name,:oci_la_log_set]
116+
Description: Number of records posted by the OCI Logging Analytics Fluentd output plugin.
117+
Type : Gauge
118+
119+
Metric Name: oci_la_fluentd_output_plugin_chunk_time_to_receive
120+
labels: [:tag]
121+
Description: Average time taken by Fluentd to deliver the collected records from Input plugin to OCI Logging Analytics output plugin.
122+
Type : Histogram
123+
124+
Metric Name: oci_la_fluentd_output_plugin_chunk_time_to_post
125+
labels: [:oci_la_log_group_id]
126+
Description: Average time taken for posting the received records to OCI Logging Analytics by the Fluentd output plugin.
127+
Type : Histogram
128+
90129

91130
## Changes
92131

‎fluent-plugin-oci-logging-analytics.gemspec

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
66

77
Gem::Specification.new do |spec|
88
spec.name = "fluent-plugin-oci-logging-analytics"
9-
spec.version = "2.0.2"
9+
spec.version = "2.0.3"
1010
spec.authors = ["Oracle","OCI Observability: Logging Analytics"]
1111
spec.email = ["[email protected]"]
1212

@@ -29,6 +29,7 @@ Gem::Specification.new do |spec|
2929
spec.add_development_dependency "test-unit", "~> 3.0"
3030
spec.add_runtime_dependency "fluentd", [">= 0.14.10", "< 2"]
3131
spec.add_runtime_dependency 'rubyzip', '~> 2.3.2'
32-
spec.add_runtime_dependency "oci", "~>2.13"
32+
spec.add_runtime_dependency "oci", "~>2.16"
33+
spec.add_runtime_dependency "prometheus-client", "~>2.1.0"
3334

3435
end

‎lib/fluent/metrics/metricsLabels.rb

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
class MetricsLabels
2+
attr_accessor :tag, :logGroupId, :logSourceName, :logSet, :invalid_reason, :records_valid, :records_per_tag, :latency
3+
def initialize
4+
@tag = nil
5+
@logGroupId = nil
6+
@logSourceName = nil
7+
@logSet = nil
8+
@invalid_reason = nil
9+
@records_valid = 0
10+
@records_per_tag = 0
11+
@latency = 0
12+
end
13+
end

‎lib/fluent/metrics/prometheusMetrics.rb

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
require 'prometheus/client'
2+
require 'prometheus/client/registry'
3+
require 'prometheus/client/gauge'
4+
require 'prometheus/client/histogram'
5+
require 'singleton'
6+
7+
class PrometheusMetrics
8+
include Singleton
9+
attr_accessor :records_received, :records_valid, :records_invalid, :records_error, :records_posted,
10+
:bytes_received, :bytes_posted, :chunk_time_to_receive, :chunk_time_to_upload
11+
def initialize
12+
createMetrics
13+
registerMetrics
14+
end
15+
def createMetrics
16+
gauge = Prometheus::Client::Gauge
17+
@records_received = gauge.new(:oci_la_fluentd_output_plugin_records_received, docstring: 'Number of records received by the OCI Logging Analytics Fluentd output plugin.', labels: [:tag,:oci_la_log_group_id,:oci_la_log_source_name,:oci_la_log_set])
18+
@records_valid = gauge.new(:oci_la_fluentd_output_plugin_records_valid, docstring: 'Number of valid records received by the OCI Logging Analytics Fluentd output plugin.', labels: [:tag,:oci_la_log_group_id,:oci_la_log_source_name,:oci_la_log_set])
19+
@records_invalid = gauge.new(:oci_la_fluentd_output_plugin_records_invalid, docstring: 'Number of invalid records received by the OCI Logging Analytics Fluentd output plugin.', labels: [:tag,:oci_la_log_group_id,:oci_la_log_source_name,:oci_la_log_set,:reason])
20+
@records_error = gauge.new(:oci_la_fluentd_output_plugin_records_post_error, docstring: 'Number of records failed posting to OCI Logging Analytics by the Fluentd output plugin.', labels: [:tag,:oci_la_log_group_id,:oci_la_log_source_name,:oci_la_log_set,:error_code, :reason])
21+
@records_posted = gauge.new(:oci_la_fluentd_output_plugin_records_post_success, docstring: 'Number of records posted by the OCI Logging Analytics Fluentd output plugin.', labels: [:tag,:oci_la_log_group_id,:oci_la_log_source_name,:oci_la_log_set])
22+
#@bytes_received = gauge.new(:oci_la_bytes_received, docstring: '...', labels: [:tag])
23+
#@bytes_posted = gauge.new(:oci_la_bytes_posted, docstring: '...', labels: [:oci_la_log_group_id])
24+
histogram = Prometheus::Client::Histogram
25+
@chunk_time_to_receive = histogram.new(:oci_la_fluentd_output_plugin_chunk_time_to_receive, docstring: 'Average time taken by Fluentd to deliver the collected records from Input plugin to OCI Logging Analytics output plugin.', labels: [:tag])
26+
@chunk_time_to_upload = histogram.new(:oci_la_fluentd_output_plugin_chunk_time_to_post, docstring: 'Average time taken for posting the received records to OCI Logging Analytics by the Fluentd output plugin.', labels: [:oci_la_log_group_id])
27+
end
28+
29+
def registerMetrics
30+
registry = Prometheus::Client.registry
31+
registry.register(@records_received) unless registry.exist?('oci_la_fluentd_output_plugin_records_received')
32+
registry.register(@records_valid) unless registry.exist?('oci_la_fluentd_output_plugin_records_valid')
33+
registry.register(@records_invalid) unless registry.exist?('oci_la_fluentd_output_plugin_records_invalid')
34+
registry.register(@records_error) unless registry.exist?('oci_la_fluentd_output_plugin_records_post_error')
35+
registry.register(@records_posted) unless registry.exist?('oci_la_fluentd_output_plugin_records_post_success')
36+
#registry.register(@bytes_received) unless registry.exist?('oci_la_bytes_received')
37+
#registry.register(@bytes_posted) unless registry.exist?('oci_la_bytes_valid')
38+
registry.register(@chunk_time_to_receive) unless registry.exist?('oci_la_fluentd_output_plugin_chunk_time_to_receive')
39+
registry.register(@chunk_time_to_upload) unless registry.exist?('oci_la_fluentd_output_plugin_chunk_time_to_post')
40+
end
41+
end

0 commit comments

Comments
 (0)