Skip to content

otelcol-contrib rpm floods journalctl logs on install #885

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
palash3 opened this issue Mar 19, 2025 · 3 comments
Open

otelcol-contrib rpm floods journalctl logs on install #885

palash3 opened this issue Mar 19, 2025 · 3 comments
Labels
artifact:rpm Issues related to RPM packages bug Something isn't working stale Issue has not had recent activity or appears to be solved. Stale issues will be automatically closed

Comments

@palash3
Copy link

palash3 commented Mar 19, 2025

Installed latest otelcol-contrib-0.122.0-1.x86_64 on Rocky 8.10 vm.

It started to flood journalctl logs. Ideally service should not be started by default.

Mar 19 16:32:14 ai400-006c-vm00 otelcol-contrib[2734]:      -> Name: scrape_series_added
Mar 19 16:32:14 ai400-006c-vm00 otelcol-contrib[2734]:      -> Description: The approximate number of new series in this scrape
Mar 19 16:32:14 ai400-006c-vm00 otelcol-contrib[2734]:      -> Unit:
Mar 19 16:32:14 ai400-006c-vm00 otelcol-contrib[2734]:      -> DataType: Gauge
Mar 19 16:32:14 ai400-006c-vm00 otelcol-contrib[2734]: NumberDataPoints #0
Mar 19 16:32:14 ai400-006c-vm00 otelcol-contrib[2734]: StartTimestamp: 1970-01-01 00:00:00 +0000 UTC
Mar 19 16:32:14 ai400-006c-vm00 otelcol-contrib[2734]: Timestamp: 2025-03-19 16:32:14.657 +0000 UTC
Mar 19 16:32:14 ai400-006c-vm00 otelcol-contrib[2734]: Value: 40.000000
Mar 19 16:32:14 ai400-006c-vm00 otelcol-contrib[2734]: Metric #17
Mar 19 16:32:14 ai400-006c-vm00 otelcol-contrib[2734]: Descriptor:
Mar 19 16:32:14 ai400-006c-vm00 otelcol-contrib[2734]:      -> Name: otelcol_process_cpu_seconds
Mar 19 16:32:14 ai400-006c-vm00 otelcol-contrib[2734]:      -> Description: Total CPU user and system time in seconds [alpha]
Mar 19 16:32:14 ai400-006c-vm00 otelcol-contrib[2734]:      -> Unit:
Mar 19 16:32:14 ai400-006c-vm00 otelcol-contrib[2734]:      -> DataType: Sum
Mar 19 16:32:14 ai400-006c-vm00 otelcol-contrib[2734]:      -> IsMonotonic: true
Mar 19 16:32:14 ai400-006c-vm00 otelcol-contrib[2734]:      -> AggregationTemporality: Cumulative
Mar 19 16:32:14 ai400-006c-vm00 otelcol-contrib[2734]: NumberDataPoints #0
Mar 19 16:32:14 ai400-006c-vm00 otelcol-contrib[2734]: Data point attributes:
Mar 19 16:32:14 ai400-006c-vm00 otelcol-contrib[2734]:      -> service.instance.id: Str(7d4fe0b9-b278-49e6-a4a6-babbb0851abc)
Mar 19 16:32:14 ai400-006c-vm00 otelcol-contrib[2734]:      -> service.name: Str(otelcol-contrib)
Mar 19 16:32:14 ai400-006c-vm00 otelcol-contrib[2734]:      -> service.version: Str(0.122.0)
Mar 19 16:32:14 ai400-006c-vm00 otelcol-contrib[2734]: StartTimestamp: 2025-03-19 13:17:24.657 +0000 UTC
Mar 19 16:32:14 ai400-006c-vm00 otelcol-contrib[2734]: Timestamp: 2025-03-19 16:32:14.657 +0000 UTC
Mar 19 16:32:14 ai400-006c-vm00 otelcol-contrib[2734]: Value: 15.820000
Mar 19 16:32:14 ai400-006c-vm00 otelcol-contrib[2734]: Metric #18
Mar 19 16:32:14 ai400-006c-vm00 otelcol-contrib[2734]: Descriptor:
Mar 19 16:32:14 ai400-006c-vm00 otelcol-contrib[2734]:      -> Name: promhttp_metric_handler_errors_total
Mar 19 16:32:14 ai400-006c-vm00 otelcol-contrib[2734]:      -> Description: Total number of internal errors encountered by the pro>
Mar 19 16:32:14 ai400-006c-vm00 otelcol-contrib[2734]:      -> Unit:
Mar 19 16:32:14 ai400-006c-vm00 otelcol-contrib[2734]:      -> DataType: Sum
Mar 19 16:32:14 ai400-006c-vm00 otelcol-contrib[2734]:      -> IsMonotonic: true
Mar 19 16:32:14 ai400-006c-vm00 otelcol-contrib[2734]:      -> AggregationTemporality: Cumulative
Mar 19 16:32:14 ai400-006c-vm00 otelcol-contrib[2734]: NumberDataPoints #0
Mar 19 16:32:14 ai400-006c-vm00 otelcol-contrib[2734]: Data point attributes:
Mar 19 16:32:14 ai400-006c-vm00 otelcol-contrib[2734]:      -> cause: Str(encoding)
Mar 19 16:32:14 ai400-006c-vm00 otelcol-contrib[2734]: StartTimestamp: 2025-03-19 13:17:24.657 +0000 UTC
Mar 19 16:32:14 ai400-006c-vm00 otelcol-contrib[2734]: Timestamp: 2025-03-19 16:32:14.657 +0000 UTC
Mar 19 16:32:14 ai400-006c-vm00 otelcol-contrib[2734]: Value: 0.000000
Mar 19 16:32:14 ai400-006c-vm00 otelcol-contrib[2734]: NumberDataPoints #1
Mar 19 16:32:14 ai400-006c-vm00 otelcol-contrib[2734]: Data point attributes:
Mar 19 16:32:14 ai400-006c-vm00 otelcol-contrib[2734]:      -> cause: Str(gathering)
Mar 19 16:32:14 ai400-006c-vm00 otelcol-contrib[2734]: StartTimestamp: 2025-03-19 13:17:24.657 +0000 UTC
Mar 19 16:32:14 ai400-006c-vm00 otelcol-contrib[2734]: Timestamp: 2025-03-19 16:32:14.657 +0000 UTC
Mar 19 16:32:14 ai400-006c-vm00 otelcol-contrib[2734]: Value: 0.000000
Mar 19 16:32:14 ai400-006c-vm00 otelcol-contrib[2734]:         {"otelcol.component.id": "debug", "otelcol.component.kind": "Export>
Mar 19 16:32:24 ai400-006c-vm00 otelcol-contrib[2734]: 2025-03-19T16:32:24.806Z        info        Metrics        {"otelcol.compon>
Mar 19 16:32:24 ai400-006c-vm00 otelcol-contrib[2734]: 2025-03-19T16:32:24.806Z        info        ResourceMetrics #0
Mar 19 16:32:24 ai400-006c-vm00 otelcol-contrib[2734]: Resource SchemaURL:
Mar 19 16:32:24 ai400-006c-vm00 otelcol-contrib[2734]: Resource attributes:
Mar 19 16:32:24 ai400-006c-vm00 otelcol-contrib[2734]:      -> service.name: Str(otelcol-contrib)
Mar 19 16:32:24 ai400-006c-vm00 otelcol-contrib[2734]:      -> service.instance.id: Str(7d4fe0b9-b278-49e6-a4a6-babbb0851abc)
Mar 19 16:32:24 ai400-006c-vm00 otelcol-contrib[2734]:      -> net.host.port: Str(8888)
Mar 19 16:32:24 ai400-006c-vm00 otelcol-contrib[2734]:      -> http.scheme: Str(http)
Mar 19 16:32:24 ai400-006c-vm00 otelcol-contrib[2734]:      -> server.port: Str(8888)
Mar 19 16:32:24 ai400-006c-vm00 otelcol-contrib[2734]:      -> url.scheme: Str(http)
Mar 19 16:32:24 ai400-006c-vm00 otelcol-contrib[2734]:      -> service.version: Str(0.122.0)
Mar 19 16:32:24 ai400-006c-vm00 otelcol-contrib[2734]: ScopeMetrics #0
Mar 19 16:32:24 ai400-006c-vm00 otelcol-contrib[2734]: ScopeMetrics SchemaURL:
Mar 19 16:32:24 ai400-006c-vm00 otelcol-contrib[2734]: InstrumentationScope github.com/open-telemetry/opentelemetry-collector-cont>
Mar 19 16:32:24 ai400-006c-vm00 otelcol-contrib[2734]: Metric #0

Almost 600 lines every 10 seconds.

It comes out from https://github.com/open-telemetry/opentelemetry-collector-releases/blob/main/distributions/otelcol-contrib/postinstall.sh snippet , where by default otelcol-contrib service is started.

if command -v systemctl >/dev/null 2>&1; then
    if [ -d /run/systemd/system ]; then
        systemctl daemon-reload
    fi
    systemctl enable otelcol-contrib.service
    if [ -f /etc/otelcol-contrib/config.yaml ]; then
        if [ -d /run/systemd/system ]; then
            systemctl restart otelcol-contrib.service
        fi
    fi
fi

Could this be removed from the rpm spec please ?
@dosubot dosubot bot added artifact:rpm Issues related to RPM packages bug Something isn't working labels Mar 19, 2025
@jgrund
Copy link

jgrund commented Mar 19, 2025

I also noticed this behavior. Could we get a quick fix that removes it? We have to mask the unit to work around it in our env.

@atoulme
Copy link
Contributor

atoulme commented Mar 19, 2025

How do you currently install your collector configuration file?

I think it's valuable to start the service on install. I don't think it needs to be outputting all its metrics in detail into journalctl. We can swap the debug exporter which is present in the configuration to a nop exporter which will stub out all output.

@palash3
Copy link
Author

palash3 commented Mar 19, 2025

How do you currently install your collector configuration file?

I think it's valuable to start the service on install. I don't think it needs to be outputting all its metrics in detail into journalctl. We can swap the debug exporter which is present in the configuration to a nop exporter which will stub out all output.

It is a secondary step, which might run after.

So there would a significant time between installing and configuring the collector.

@github-actions github-actions bot added the stale Issue has not had recent activity or appears to be solved. Stale issues will be automatically closed label May 19, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
artifact:rpm Issues related to RPM packages bug Something isn't working stale Issue has not had recent activity or appears to be solved. Stale issues will be automatically closed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@atoulme @jgrund @palash3