Split pdata into multiple packages

[dmitryax]

Problem

As a result of discussion on how to handle experimental data types, a decision was made to split pdata package into multiple modules per signal and publish them as separate modules. pdata package incapsulates internal proto data structures into private fields. Given that currently all signal data structures (logs, metrics, traces) use common structures like Resource, AttributeMap, AttributeValue, they need direct access to the private proto fields of the common structures. Splitting them as is cannot be done since it removes the access to the private fields of common data types from other signal packages. There are several ways how this can be solved, all of them have some drawbacks.

Possible solutions

1. Wrap common pdata structs

Common pdata structures like Resource and others can be wrapped into another struct that is accessible in a public module while wrapped part can be placed into an internal package that all other signals have access to.

Cons:

• Complicates common data structures
• Additional computation overhead to wrap/unwrap

2. Add new exported functions to common data structures to be used only by other pdata signals

Private functions of common data types that currently used to pass private proto fields can be exported. In that case other signal modules can have access to it.

For example:

func newResource(orig *otlpresource.Resource) Resource

can be changed to

func NewResourceFromProto(orig *otlpresource.Resource) Resource {

Pros:

• Less additional complexiy

Cons:

• Introducing exported functions that are needed for internal use only.

3. Put pdata into an internal package as is and use aliases to it in exported modules

The whole pdata package can be put as is into an internal package. And exported packages will contain only aliases to the internal pdata package.

Pros:

• Easy to generate and maintain

Cons:

• Slightly complicates the module structure and godoc.

Proposal

The third solution seems to be the most appropriate. It keeps the overhead minimal while introducing no changes to existing pdata structures. It just adds another layer of reference in the code and generated documentation.

[bogdandrutu]

My first question/reaction: Considering all these problems... Do we still want to split the packages?

[dmitryax]

Also the only downside of the approach 3 can be mitigated to some extent. Signal specific parts of the pdata code that is put under internal in #4833 potentially can be moved to exported p<signal> modules and only pcommon will stay with the aliases.

[dmitryax]

@open-telemetry/collector-approvers please let me know if the options outlined in this issue need more clarifications for further discussion.

I don't think we should consider Option 1 due to the overhead. So I believe the question is whether we want to accept one of the following:

Option 2: exposing functions that are needed for internal usage only like NewResourceFromProto. There are must be several of them.

Option 3: having aliases in pcommon module pointing to an internal package.

[Aneurysm9]

I don't feel that this issue provides enough information for us to determine the best path forward. I don't see from here what the structure of independent signal modules would look like. What would be the dependencies among them and the existing modules/packages? What would the wrapping structures for option 1) look like? What would the interfaces for common types in option 2) look like? Is it only exporting initializers or is there more that needs to be exposed? Why would it be a bad thing to allow the initialization of common data types that might be used by multiple signals? It feels like they are currently internal-only but need not be going forward.

I don't think we should consider Option 1 due to the overhead.

What overhead? Is it a performance concern? Do you have benchmarks that demonstrate the magnitude of the overhead?

Option 2: exposing functions that are needed for internal usage only like NewResourceFromProto.

Why should the pdata package be the only place that a resource can be constructed from proto structs? Can external callers not already contstruct a new empy resource and then fill it with attributes from the proto struct? Why not give a direct method, particularly if we're talking about ways to enable support for more, new signal types.

Again, I think we need a lot more detail on all of these options before we can make an informed decision. Please see open-telemetry/opentelemetry-go#2526 where the Go SIG was recently presented with choices for how to structure a metrics API. Having that level of detail made it easier for us to understand the end-state we're moving toward and how we're getting there.

[codeboten]

Thanks for putting together a draft PR demonstrating option 3. I don't know how feasible it would be to have a PR for both other options, or if it would be easier to put together a rough sketch of what Options 1 & 2 would look like (as per @Aneurysm9's linked proposal in the go repo)

Looking through the PR for Option 3, I'm not sure how different from option 2 it is, sure we're hiding the internals but exposing the aliases essentially does almost the same as exposing the internals here.

2. Add new exported functions to common data structures to be used only by other pdata signals

I don't think there's a good way to enfore only pdata signals to use exported functions, so i guess the question is what's the downside in exposing those methods?

[dmitryax]

@codeboten

sure we're hiding the internals but exposing the aliases essentially does almost the same as exposing the internals here.

Not really, for Option 2 we would need to expose functions like NewResourceFromProto, but we don't need to provide aliases for them in Option 3. We don't have any internals exposed in the draft PR that I submitted

I don't think there's a good way to enfore only pdata signals to use exported functions, so i guess the question is what's the downside in exposing those methods?

I think the only downside is their actual presence. Users won't be able to do anything with them outside the Collector's model

[Aneurysm9]

I think the only downside is their actual presence. Users won't be able to do anything with them outside the Collector's model

Can you elaborate on this? I'm not sure I understand.

[dmitryax]

I think the only downside is their actual presence. Users won't be able to do anything with them outside the Collector's model

Can you elaborate on this? I'm not sure I understand.

Option 2 implies introducing publicly exported functions like the following:

func ResourceFromProto(orig *otlpresource.Resource) Resource {
	return Resource{orig: orig}
}

This function can be imported by any client, but cannot be used anywhere outside of Core Collector's model, because it takes an argument of a type defined in model/internal which is not leaked anywhere else.

[Aneurysm9]

Would that be an issue if we used the same proto structs that everyone else uses? I think that being able to interoperate with other systems at that level would be advantageous.

[tigrannajaryan]

Would that be an issue if we used the same proto structs that everyone else uses? I think that being able to interoperate with other systems at that level would be advantageous.

That would be a significant performance degradation (IIRC around 1.3x last time we measured). Also probably (?) makes impossible/more complicated the patching/manual Protobuf serialization code that @codeboten works on.

[tigrannajaryan]

My first question/reaction: Considering all these problems... Do we still want to split the packages?

+1.

I think we need to keep asking this question. I still don't see an alternate that works if we keep insisting on our formal approach to introducing experimental API (according to Go semver requirements). However if we relax this requirement then the alternate is much simpler: don't split, mark unstable parts (currently logs) as "unstable" via some sort of labelling (like gRPC does).

I have a feeling by trying to be formal we are complicating our lives disproportionately.

[mx-psi]

I still don't see an alternate that works if we keep insisting on our formal approach to introducing experimental API (according to Go semver requirements).

@tigrannajaryan How would the implementation in #4833 'not work' in this sense?