x509 otel certificate refresh events in cert_refresher library. #3146

balamanova · 2025-12-09T17:26:19Z

Description

Added OpenTelemetry metrics for X.509 certificate refresh events in cert_refresher library.

Implementation follows SIA OTel pattern

This implementation mirrors the existing SIA (Go) OTel metrics pattern from libs/go/sia/otel/metricset.go:

Metrics (aligned with SIA)

This PR (Java)	SIA (Go)
`athenz_cert_refresher.refresh.result_total{function,result}`	`sia.agent_command.result_total{function,result}`
`athenz_cert_refresher.service_cert.validity.remaining_secs{name}`	`sia.service_cert.validity.remaining_secs{cname}`
`athenz_cert_refresher.refresh.result_last_timestamp{function,result}`	New - tracks when context was last updated

Attributes (same as SIA)

function - identifies the operation (e.g., "cert_refresh")
result - "success" or "failure" (same values as SIA)
name - certificate subject name (similar to SIA's cname)

Configuration

Disable with: -Dathenz.cert_refresher.otel_disabled=true
Component name: -Dathenz.cert_refresher.otel_component_name=my-service

Contribution Checklist:

The pull request does not introduce any breaking changes
I have read the contribution guidelines.
Create an issue and link to the pull request.

Attach Screenshots (Optional)

gemini-code-assist · 2025-12-09T17:26:40Z

Summary of Changes

Hello @balamanova, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances the cert_refresher library by integrating OpenTelemetry for comprehensive monitoring of X.509 certificate refresh operations. The changes introduce new metrics to track the outcomes of certificate reloads (success or failure) and provide real-time insights into the remaining validity and expiry timestamps of the currently loaded certificates, thereby improving the operational visibility and reliability of certificate management within Athenz.

Highlights

OpenTelemetry Integration: Introduces OpenTelemetry (OTel) metrics to the cert_refresher library for improved observability of certificate management.
New Metric Emitter: Adds OpenTelemetryCertReloadEventEmitter to manage and emit various certificate refresh and validity metrics.
Configurable Metrics: Allows disabling OTel metrics via a system property (athenz.cert_refresher.otel_disabled) and configuring the component name.
Enhanced Certificate Reload Logic: Modifies KeyRefresher to record successful/failed certificate reloads and export certificate validity metrics using the new OTel emitter.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces OpenTelemetry metrics for X.509 certificate refresh events by adding a new OpenTelemetryCertReloadEventEmitter class and integrating it into the KeyRefresher. The implementation is solid, adding valuable observability. My review includes several suggestions for the new OpenTelemetryCertReloadEventEmitter class to improve metric consistency, code clarity, and the utility of the emitted metrics. These changes will make the new telemetry data more robust and easier to consume.

libs/java/cert_refresher/src/main/java/com/oath/auth/OpenTelemetryCertReloadEventEmitter.java

ATHENS-8722 adding cert refresh metrics Signed-off-by: abalamanova <[email protected]>

libs/java/cert_refresher/src/main/java/com/oath/auth/KeyRefresher.java

libs/java/cert_refresher/src/main/java/com/oath/auth/OpenTelemetryCertReloadEventEmitter.java

psasidhar · 2025-12-12T16:31:25Z

libs/java/cert_refresher/src/main/java/com/oath/auth/OpenTelemetryCertReloadEventEmitter.java

+                .build();
+
+        refreshResultCounter.add(1, attrs);
+        resultLastTimestampGauge.set(timestamp, attrs);


do we need to send timestamp explicity, won't that be automatically be available with refreshResultCounter?

I'm using OTel Metrics API (counter + gauge), not OTel Events/Logs API. Prometheus is a pull-based time-series database.

Prometheus only stores scrape timestamps, not event timestamps. I can detect that a refresh happened using increase(), but the timestamp is approximate (within scrape interval).

With this timestamp I save exact timestamp when the refresh occurred

there is no need to generate a timestamp attribute. the metric already tells you that there is a failure in the given time period and the exact timestamp is not really needed and will not be used.

…tMetric Signed-off-by: abalamanova <[email protected]>

havetisyan · 2025-12-15T16:18:43Z

libs/java/cert_refresher/src/main/java/com/oath/auth/KeyRefresher.java

+     */
+    private OpenTelemetryCertReloadEventEmitter initOtelMetrics() {
+        String otelDisabledProp = System.getProperty(PROP_OTEL_DISABLED);
+        boolean otelDisabled = "true".equalsIgnoreCase(otelDisabledProp);


The default behavior here should be false. We can't introduce new functionality which could cause problems and generate more metrics if nobody is looking at them.

Instead of using the "true" (or "false") , please specify a static string and use that. Alternatively you can convert to Boolean and use Boolean.TRUE/Boolean.FALSE

havetisyan · 2025-12-15T16:22:36Z

libs/java/cert_refresher/src/main/java/com/oath/auth/KeyRefresher.java

-                        //Signal key change event
-                        if (keyRefresherListener != null) {
-                            keyRefresherListener.onKeyChangeEvent();
+                        try {


why did we add a new try/catch block here? we already have one in place where we log any errors?

havetisyan · 2025-12-15T16:26:00Z

libs/java/cert_refresher/src/main/java/com/oath/auth/OpenTelemetryCertReloadEventEmitter.java

+                .build();
+
+        refreshResultCounter.add(1, attrs);
+        resultLastTimestampGauge.set(timestamp, attrs);


there is no need to generate a timestamp attribute. the metric already tells you that there is a failure in the given time period and the exact timestamp is not really needed and will not be used.

havetisyan · 2025-12-15T16:29:12Z

libs/java/cert_refresher/pom.xml

+      <groupId>io.opentelemetry</groupId>
+      <artifactId>opentelemetry-api</artifactId>
+      <version>${opentelemetry.version}</version>
+      <optional>true</optional>


have we verified that if we enable otel metrics and the jars are not available, the code works as expected without throwing any errors?

gemini-code-assist bot reviewed Dec 9, 2025

View reviewed changes

balamanova force-pushed the ATHENS-8722-x509_otel branch 2 times, most recently from bc60e29 to c9e32ba Compare December 9, 2025 17:46

ATHENS-8722 adding cert refresh metrics

522931a

ATHENS-8722 adding cert refresh metrics Signed-off-by: abalamanova <[email protected]>

balamanova force-pushed the ATHENS-8722-x509_otel branch from c9e32ba to 522931a Compare December 9, 2025 17:46

psasidhar reviewed Dec 11, 2025

View reviewed changes

libs/java/cert_refresher/src/main/java/com/oath/auth/KeyRefresher.java Outdated Show resolved Hide resolved

libs/java/cert_refresher/src/main/java/com/oath/auth/OpenTelemetryCertReloadEventEmitter.java Outdated Show resolved Hide resolved

balamanova requested a review from psasidhar December 11, 2025 18:07

psasidhar reviewed Dec 12, 2025

View reviewed changes

balamanova changed the title ~~Athens 8722 x509 otel~~ x509 otel certificate refresh events in cert_refresher library. Dec 12, 2025

removed expiry time on and changed exportServiceCertMetric->exportCer…

8adce7d

…tMetric Signed-off-by: abalamanova <[email protected]>

balamanova force-pushed the ATHENS-8722-x509_otel branch from 9b54ae6 to 8adce7d Compare December 12, 2025 20:38

havetisyan requested changes Dec 15, 2025

View reviewed changes

x509 otel certificate refresh events in cert_refresher library. #3146

Are you sure you want to change the base?

x509 otel certificate refresh events in cert_refresher library. #3146

Conversation

balamanova commented Dec 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Implementation follows SIA OTel pattern

Metrics (aligned with SIA)

Attributes (same as SIA)

Configuration

Contribution Checklist:

Attach Screenshots (Optional)

Uh oh!

gemini-code-assist bot commented Dec 9, 2025

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

psasidhar Dec 12, 2025

Choose a reason for hiding this comment

Uh oh!

balamanova Dec 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

havetisyan Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

havetisyan Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

havetisyan Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

havetisyan Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

havetisyan Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

havetisyan Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

balamanova commented Dec 9, 2025 •

edited

Loading

balamanova Dec 12, 2025 •

edited

Loading