[pdata/common] Marshal `Value` to JSON bytes using the OTLP/JSON format

[zhengkezhou1]

Component(s)

pdata

Describe the issue you're reporting

Background

We aim to store the complex types (map, slice) of value within attributes as JSON safely. Currently, serializing them to JSON strings using value.AsString() leads to the risk of losing type information during deserialization. Specifically:

1. All numerical values become JSON numbers, making it impossible to distinguish whether a value was originally an int64 or a float64.
2. Parsing nested map and slice structures is particularly problematic due to their dynamic and unpredictable nesting levels.

Solution

This issue can be resolved with a minor modification. Instead of directly serializing and deserializing the high-level Value type, we will operate on the underlying otlpcommon.AnyValue type it holds. AnyValue is a Go struct generated from Protobuf definitions, which allows for a more precise representation of different data types.

Here is the code implementing this functionality:

import oteljson "go.opentelemetry.io/collector/pdata/internal/json"
import "go.opentelemetry.io/collector/pdata/internal"
import "go.opentelemetry.io/proto/otlp/common/v1"

// UnmarshalValue deserializes a JSON byte stream into a Value type.
// It uses oteljson.ReadValue to read the JSON data into an AnyValue,
// and then creates a new Value instance using the AnyValue.
func UnmarshalValue(buf []byte) Value {
	iter := jsoniter.ConfigFastest.BorrowIterator(buf)
	defer jsoniter.ConfigFastest.ReturnIterator(iter)

	var anyValue v1.AnyValue // Using the correct Protobuf-generated AnyValue type
	state := internal.StateMutable
	oteljson.ReadValue(iter, &anyValue)
	return newValue(&anyValue, &state)
}

// MarshalValue serializes a Value type into a JSON byte stream.
// It retrieves the underlying AnyValue using value.getOrig()
// and then uses oteljson.Marshal to serialize the AnyValue into JSON.
func MarshalValue(value Value) ([]byte, error) {
	var buf bytes.Buffer
	if err := oteljson.Marshal(&buf, value.getOrig()); err != nil {
		return nil, err
	}
	return buf.Bytes(), nil
}

[github-actions]

Pinging code owners:

• pdata: @bogdandrutu @dmitryax

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[mx-psi]

We should be following the spec on this one, see https://github.com/open-telemetry/opentelemetry-proto/blob/main/docs/specification.md#json-protobuf-encoding. If we are not, feel free to point the bug and we will fix it. If you think the spec does not have the right behavior, this is something we should discuss over at https://github.com/open-telemetry/opentelemetry-proto

[zhengkezhou1]

@mx-psi Thanks for your reply! There is no bug or wrong behavior here. I am just looking for some functions called MarshalValue and UnmarshalValue that could do the same things as MarshalTraces and UnmarshalTraces.

opentelemetry-collector/pdata/ptrace/json.go

Lines 21 to 43 in 568c3ce

	// MarshalTraces to the OTLP/JSON format.
	func (*JSONMarshaler) MarshalTraces(td Traces) ([]byte, error) {
	buf := bytes.Buffer{}
	pb := internal.TracesToProto(internal.Traces(td))
	err := json.Marshal(&buf, &pb)
	return buf.Bytes(), err
	}
	// JSONUnmarshaler unmarshals OTLP/JSON formatted-bytes to pdata.Traces.
	type JSONUnmarshaler struct{}
	// UnmarshalTraces from OTLP/JSON format into pdata.Traces.
	func (*JSONUnmarshaler) UnmarshalTraces(buf []byte) (Traces, error) {
	iter := jsoniter.ConfigFastest.BorrowIterator(buf)
	defer jsoniter.ConfigFastest.ReturnIterator(iter)
	td := NewTraces()
	td.unmarshalJsoniter(iter)
	if iter.Error != nil {
	return Traces{}, iter.Error
	}
	otlp.MigrateTraces(td.getOrig().ResourceSpans)
	return td, nil
	}

[yurishkuro]

+1 to support MarshalValue / UnmarshalValue (but via marshaler structs, not as global methods).

The support for complex types in attributes has always been contentious in the spec, with the common suggestion for the backends which can't support those natively to store data as JSON, but there are no methods in pdata to produce OTLP/JSON representation of individual Values.

[yurishkuro]

We can provide a PR if maintainers indicate acceptance.

[mx-psi]

Thanks for the context, I support this, but would like to confirm thart @bogdandrutu and @dmitryax are okay with this before moving ahead

[mahadzaryab1]

Hi @mx-psi! I wanted to check in on the status of this ticket. Would you be willing to accept a PR from us?