New component: GitLab Receiver

[adrielp]

The purpose and use-cases of the new component

The purpose of this receiver is the same as the GitHub Receiver but in the context of GItLab. It will support metrics, logs, and traces from GitLab in almost the same way as the GitLab receiver. Much of this code exists in some form or another.

Really, the logic and metrics are the same as the GitHub receiver. The differences are:

• API Calls are different
• Events that are used to build traces are different

Everything else is closely aligned. The CICD SIG will leverage this as one of the prototypes alongside the GitHub receiver helping to inform conventions and support telemetry in GitLab.

Example configuration for the component

extensions:
  bearertokenauth/gitlab:
    token: ${env:TOKEN}

receivers:
  gitlab:
    interval: 15s #the interval at which the scraper scrapes
    group: mygroup #the organization name in github
    query: <some query>
    traces:
        .... <traces webhook configuration>
    auth:
      authenticator: bearertokenauth/gitlab

Telemetry data types supported

Metrics, Logs, Traces

Is this a vendor-specific component?

• This is a vendor-specific component
• If this is a vendor-specific component, I am a member of the OpenTelemetry organization.
• If this is a vendor-specific component, I am proposing to contribute and support it as a representative of the vendor.

Code Owner(s)

adrielp

Sponsor (optional)

@atoulme

Additional context

No response

[atoulme]

I will sponsor this.

[niwoerner]

Hi all, thank you for all the great work you've been doing in context of CI/CD pipelines and OpenTelemetry.

I've been playing around with a Gitlabreceiver to create traces for CI/CD pipelines. That isn't exactly aligning with this issue, but since the Gitlabreicever was working pretty well for me, it might be interesting for more folks here in the community.

I'll be giving a talk about this next week at the Kubecon. If someone is present there, please let me know and we can catch up about this topic.

I would be happy to contribute (and maintain) this, if there would be an interest from the community. :)

[adrielp]

Hey @niwoerner , thanks for coming from LinkedIn and posting here on the issue! I think the alignment is quite nice, given the concept on the tracing end is the same. Would love to get support in contributing and maintaining this going forward. Let's sync up & figure out how we can move forward on this!

[abitrolly]

interval: 15s #the interval at which the scraper scrapes

I believe GitLab CLI can subscribe to events from running pipeline.

glab pipeline status --live

#  -l, --live            Show status in real time until the pipeline ends.

So maybe it is possible to receive events using some kind of (web)hook instead of the scraper.

The purpose of this receiver is the same as the #22028 but in the context of GItLab. It will support metrics, logs, and traces from GitLab in almost the same way as the GitLab receiver.

I would appreciate the expanded use cases without redirections to GitHub that may significantly drift away from what is it possible specifically with GitLab.

I will sponsor this.

What pain point do you want to address first?

[adrielp]

@abitrolly - the GitHub receiver is important context. Originally it was named the gitprovider receiver which I originally developed as a prototype here. It scraped both GitLab & GitHub's respective API's to build out a set of VCS Metrics.

Upon reconsideration of the implementation & incorporating other signals (tracing, logs), we as a community decided to separate things out and rename the receiver to be github receiver. While there are implementation differences, the high level concept, and core logic, is much the same. Events(logs) & Traces through Webhook + a set of metrics through scraping.

This is a good summary of the decisions and context I mentioned above.

[abitrolly]

@adrielp I am asking about specific use cases to estimate the scope of work left from https://github.com/niwoerner/gitlabreceiver. Ideally it should be a checkbox list.

• receive pipeline spans in real-time
• receive pipeline spans once run is finished
• ???

[niwoerner]

I plan to sync with @adrielp this week at the Kubecon. (or virtually afterwards)

That would probably help to determine how to proceed further in regards to the tracing functionality of my created Gitlabreceiver. I would love to hear a second opinion on a few of my implementation choices.

Based on that outcome we could define next steps if (and how) we want to add the functionality to the Giltabreceiver here

PS: If anyone else reading this, is attending the upcoming Kubecon and would like to exchange about this topic, please let me know :)

[abitrolly]

@niwoerner I am not able to attend Kubecon financially, but I found your slides https://static.sched.com/hosted_files/kccncna2024/86/Kubecon-NA-2024-Gitlab-CICD-Pipelines-Otel.pdf :D

For some reason page with blue stripe background are very slow to render, like page 22. Maybe it is just a problem with my Firefox viewer.

1. Is it possible to quickly see data from public projects like https://gitlab.com/gitlab-org/gitlab-development-kit/ without configuring anything?

2. What software is used for The Pipeline Fitness Tracker with expected and actual duration?

[niwoerner]

@abitrolly

Regarding 1:

To get the traces for pipelines with the Gitlabreceiver you need to:

1.) have a custom otel collector with the Gitlabreceiver running somewhere

2.) Configure the group/repo webhook settings to send pipeline events to the collector.

Regarding 2:

Do you mean the notification slide?

The notification is received in Microsoft teams and sent from Dynatrace

Feel free to open a discussion in the Gitlabreceiver repo or send me a message via slack/linkedin if you have more questions about the presentation :)

(I don't want to abuse this issue for it :D)

[abitrolly]

@niwoerner

1. I guess the most simple collector to run for the demo is https://github.com/CtrlSpice/otel-desktop-viewer

2. In theory it is possible to hack glab ci status --live to get steam events without web hooks https://gitlab.com/gitlab-org/cli/-/blob/main/commands/ci/status/status.go?ref_type=heads#L42

Feel free to open a discussion in the Gitlabreceiver repo or send me a message via slack/linkedin if you have more questions about the presentation :)

Sure, but I don't think it hurts to discuss how to test current state of things with GitLab tracing in the most easy way here. )

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

[TylerHelmuth]

@adrielp @niwoerner I annoyingly am planning to revert the skeleton PR because it seems to have caused problems with our CI. I can't explain why, only that the issue coincides with commit to add the skeleton and subsides when the commit is reverted. See https://github.com/open-telemetry/opentelemetry-collector-contrib/actions/workflows/build-and-test.yml?query=branch%3Amain and #37177.

After the revert please resubmit a PR with the skeleton and we can work together to figure out why it is causing an issue.

[TylerHelmuth]

It seems like I was wrong about the cause. No revert happening.

[TWpgo]

Any news on the current state? I really would appreciate this new component 👍

[niwoerner]

I plan to continue with the implementation next week, so if everything goes as planned there should be a new PR within the next weeks :)

[abitrolly]

GitLab folks developed pipeline analyzer https://gitlab.com/gitlab-org/gitlab/-/issues/508903

Maybe they could adapt intermediate data format to be Open Telementry. But the wishlist want to visualize spans as soon as they appear in realtime, which Open Telemetry supports not.

[adrielp]

@abitrolly - There's a lot of things going on. And if you don't want to wait for this contribution to be built into the OpenTelemetry collector, you can build your own and get traces today! That's one of the awesome things about OpenTelemetry and the collector, anyone is free to build their own distribution using ocb and pointing to @niwoerner 's original implementation of the gitlab receiver.

It won't be identical, but it'll be close and provide immediate value. The span times will appear in close to real time (solely latency from the originating event -> conversion -> sending to a backend which is minimal) and the actual span times as they appear in the backend map to the actual times the events occurred with no added latency.

On the GitLab side, there's been conversations for years around how to approach and solve this issue. See:

• https://gitlab.com/gitlab-org/gitlab/-/issues/338943
• https://gitlab.com/gitlab-org/opstrace/opstrace/-/issues/2926
• https://gitlab.com/groups/gitlab-org/-/epics/11835
• https://gitlab.com/gitlab-org/gitlab/-/issues/373143#note_1720730918

[nicholasdille]

Today, I tested this implementation of the GitLab receiver in PR #39123 against our non-production instance of GitLab and it worked. I added some code from @niwoerner's implementation to fill the span attributes and in the end this looks a lot like what we need.

We have just shutdown GCPE for pipeline metrics because of the load on the API. We are in need of a new solution.

So, we'd be happy to support this, e.g. by testing and providing feedback. What else can we do to help?