[receiver/splunkhecreceiver] timestamp overflows when not in seconds

[timannguyen]

Component(s)

https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/8e0ea012fe93a272b78e27a94e690084538b0963/receiver/splunkhecreceiver

What happened?

Description

Splunk HEC allows time to be nanosecond, microsecond, millisecond and in second. splunk to log/metric converters assumes all time is in seconds:

• opentelemetry-collector-contrib/receiver/splunkhecreceiver/splunk_to_logdata.go

  Line 57 in 8e0ea01

  logRecord.SetTimestamp(pcommon.Timestamp(event.Time * 1e9))

• opentelemetry-collector-contrib/receiver/splunkhecreceiver/splunkhec_to_metricdata.go

  Line 122 in 8e0ea01

  return pcommon.Timestamp(sec * 1e9)

The receiver should not assume all incoming time is in second and cause overflow by trying to convert time to nanosecond before confirming the time unit.

Steps to Reproduce

1. run otel with splunkhecreceiver
2. curl -X POST http://localhost:8088/services/collector -d '{"time":1732604863241,"event":"data","source":"test","sourcetype":"test","host":"test","index":"main"}'
3. timestamp for the event is now 9223372036854774

Expected Result

1732604863241000000
Tuesday, November 26, 2024 7:07:43.241 UTC

Actual Result

9223372036854774
Friday, April 11, 2262 11:47:16.854 UTC

Collector version

1.0.9

Environment information

Environment

Debian Bookworm
go 1.22

OpenTelemetry Collector configuration

receivers:
 splunk_hec:
   endpoint: 8088
   splitting: false

Log output

No response

Additional context

No response

[github-actions]

Pinging code owners:

• receiver/splunkhec: @atoulme

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[VihasMakwana]

Looking at https://docs.splunk.com/Documentation/SplunkCloud/latest/Data/FormateventsforHTTPEventCollector, It says

The event time. The default time format is UNIX time format, in the format . and depends on your local timezone.

Also, the examples uses seconds for time field.

ccing @atoulme as he might have more thoughts on this.

[timannguyen]

Splunk will accepts milli, micro, nano and will be aware to parse it to valid epoch time.

[atoulme]

IIRC I asked if you would you please update the docs, which are serving as the spec for HEC.

As @VihasMakwana pointed out the time field documentation is:

The event time. The default time format is UNIX time format, in the format . and depends on your local timezone. For example, 1433188255.500 indicates 1433188255 seconds and 500 milliseconds after epoch, or Monday, June 1, 2015, at 7:50:55 PM GMT.

If this can please be clarified, then we can go for those changes.

[timannguyen]

@atoulme @VihasMakwana HEC Documentation has been updated to reflect stated issue https://docs.splunk.com/Documentation/SplunkCloud/latest/Data/FormateventsforHTTPEventCollector#Event_metadata

[atoulme]

@timannguyen this is weird

For instance, the following values in nanoseconds and microseconds:

"time": "1733768295123456789"
"time": "1733768295123456"
can also be accepted with decimal points inserted as follows:

"time": "1733768295.123456789"
"time": "1733768295.123456"

This is different from what you are asserting in the bug description, or at the very least unclear.
You give this example:

curl -X POST http://localhost:8088/services/collector -d '{"time":1732604863241,"event":"data","source":"test","sourcetype":"test","host":"test","index":"main"}'

Are you asserting that we can somehow guess that the timestamp "1732604863241" is in milliseconds?
It looks like the doc says you should have placed a decimal point such that the timestamp is a number of seconds, with a decimal portion: 1732604863.241.

A second read of the documentation confirms my read:

The default time format is UNIX time format, in the format <sec>.<ms> and depends on your local timezone. For example, 1433188255.500 indicates 1433188255 seconds and 500 milliseconds after epoch, or Monday, June 1, 2015, at 7:50:55 PM GM.

The HEC endpoint can also accept time in microseconds or nanoseconds, interpreted by inserting a decimal at the appropriate position.

Please clarify or consider closing this issue as invalid.

[timannguyen]

@atoulme documentation missed the case and i will revisit the issue with the team.

Since Splunk HEC will accept ms, us, and ns epoch in addition to . format, I want to align the hec receiver by guessing if the time is in ms, us or ns before time * 1e9.

[atoulme]

By which mechanism can we find that "1732604863241" is a value in milliseconds? Same for microseconds and nanoseconds values - Can you draft out the algorithm for this?

[timannguyen]

@atoulme what i am thinking is a bit hacky

if t >= 10_000_000_000_000_000
  // nano
  return pcommon.Timestamp(sec)
if t >= 10_000_000_000_000
  // micro
  return pcommon.Timestamp(t * 1e3)
if t >= 10_000_000_000
  // milli
  return pcommon.Timestamp(t * 1e6)

// seconds
return pcommon.Timestamp(t * 1e9)

but this is not too different than what Splunk is doing