[mdatagen] Consider linking metrics/attributes from metadata.yaml to SemConv definitions

[ChrsMark]

mdatagen allows today to set sem_conv_version within the metdata.yaml file. However this version is only used to import the semantic conventions module at

opentelemetry-collector/cmd/mdatagen/internal/templates/metrics.go.tmpl

Line 311 in 4ca0f18

rm.SetSchemaUrl(conventions.SchemaURL)

Currently, there is no way for us to know if metrics defined in metadata.yaml follow the Semantic Conventions or not. We should consider linking metrics defined in metadata.yaml to their Semantic Convnetions' definitions, if exist.

This should be similar to how SchemaUrl is defined:

opentelemetry-collector/cmd/mdatagen/internal/templates/metrics.go.tmpl

Line 311 in 4ca0f18

rm.SetSchemaUrl(conventions.SchemaURL)

This will allow us to:

1. Leverage Semantic Conventions definitions directly where exist
2. Generated code will use imported conventions' metric names directly
3. Define per metric/attribute stability level directly from SemConv
4. Start using stable semantic conventions in components: [receiver/hostmetricsreceiver] semantic conventions transition opentelemetry-collector-contrib#35325
5. Help with components' stability discussions: Document SemConv compatibility requirements for components stability #11878

[mx-psi]

This makes sense to me, I support doing this

[ChrsMark]

note: I'd be happy to work on this after August unless somebody else want to tackle it earlier

[ChrsMark]

It seems that OpenTelemetry-Go already generates code based on SemConv (using Weaver).

It looks like there is some overlap between mdatagen and OpenTelemetry-Go.

For example https://github.com/open-telemetry/opentelemetry-go/blob/main/semconv/v1.34.0/cpuconv/metric.go#L240-L259 looks quite close to what mdatagen generates at https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/eae15340d383deef0e63ed8b8fea4ef12ead9e54/receiver/hostmetricsreceiver/internal/scraper/cpuscraper/internal/metadata/generated_metrics.go#L317-L327.

@open-telemetry/semconv-system-approvers since this is related to the discussions about stabilising components like the hostmetrics receiver, should we consider switching to what OpenTelemetry-Go already does? I haven't checked how much overlap exists there, however I guess that's something worth checking. I guess mdatagen would still be needed for what OpenTelemetry-Go does not cover.

[braydonk]

I think we can only use Weaver if Weaver codegen supported generating pdata APIs rather than using OTel SDK. We probably can't directly leverage the Go code Weaver generates today, at least not in an elegant way. But it would probably be a really beneficial feature for future SemConv transitions to allow this to be possible; the current plan of replicating SemConv definitions in mdatagen format has a lot of potential for mixups and errors.

[ChrsMark]

I think the problem here boils down to the fact that the mdatagen generates code out of the metadata.yaml files and there is no actual dependency to released Semantic Conventions. While the OpenTelemetry-Go generates code out of the Semantic Conventions directly.

We still use the semconv module of the OpenTelemetry-Go but we don't really leverage it for metrics. We use some attributes' references within components' code (example), but not for metrics. Note that in OpenTelemetry-Go, only attributes are exported directly, not metrics' names. For metrics we could reference a type like CPUTime but I'm not sure if we should.

Based on the above I think we need to shape a plan on how we are going to use the OpenTelemetry-Go. Probably just using it for referencing attributes is fine but I feel we can do better here, because right now things seem to be a bit scattered.

[ChrsMark]

Few thoughts on this for metrics case, but it can be generalised for other signals too.

First of all metadata.yml file might have metrics that are defined in SemConv already, but also metrics that are not defined in SemConv. We should allow this since introducing experimental metrics in components without first having them released as Semantic Conventions might be useful. Based on this we should consider the following:

1. we should introduce stability level per metric (prior art: Add per-metric stability guidelines #5487)
2. stable metrics should have Semantic Conventions definition/linking, otherwise mdatagen should fail
3. stable components should have at least 1 stable metric

So, regardless of how we generate code and documentation, the above could be the minimum for marking a component as stable (related to #11878).

@open-telemetry/semconv-system-approvers would love your feedback here, since the above most probably will first be applied for System metrics and hostmetricsreceiver.

[rogercoll]

1. we should introduce stability level per metric
2. stable metrics should have Semantic Conventions definition/linking, otherwise mdatagen should fail
3. stable components should have at least 1 stable metric

Sounds great to me, that would give more visibility of Semantic Conventions when defining new receivers/signals. Although not sure if I would make it a blocker for hostmetricsreceiver stability (first contrib component) but something to work in parallel.