Skip to content

Commit 25cb194

Browse files
authored
[pkg/ottl] Add support for localized time parsing into the timeutils (#34353)
**Description:** <Describe what has changed.> Added support for localized time parsing into the `coreinternal/timeutils` package. The [tracking issue](#32977) is a following up to #32140, and the added function (`ParseLocalizedStrptime`) can be used later to [add locale support](#32978) to the ottl `Time` function. As discussed in the related issues, the plan is to have a similar support as implemented by the library [monday](https://github.com/goodsign/monday), making it possible to parse non-english time strings into `time.Time` objects. Elastic has built a new OSS library for that same purpose ([lunes](https://github.com/elastic/lunes)), that considering the discussed requirements, seems to be a better option than `monday`. Both libraries are just wrapper to the `time.Parse` and `time.ParseInLocation` features. They work by translating the foreign language value to English before calling the standard parsing functions. The main `lunes` differences are: 1 - Performance is considerably better, ~13x faster for complete .Parse operations: ``` BenchmarkParseLunes-10 2707008 429.7 ns/op 220 B/op 5 allocs/op BenchmarkParseMonday-10 212086 5630 ns/op 3753 B/op 117 allocs/op BenchmarkParseInLocationLunes-10 2777323 429.4 ns/op 220 B/op 5 allocs/op BenchmarkParseInLocationMonday-10 210357 5596 ns/op 3754 B/op 117 allocs/op ``` Given `ParseLocalizedStrptime` uses `lunes.Translate` under the hood, its performance is similar to the existing `ParseStrptime`, adding an extra overhead of ~303 ns/op for translating the value before parsing: ``` BenchmarkTranslate-10 3591234 302.4 ns/op 220 B/op 5 allocs/op ``` ``` BenchmarkParseLocalizedStrptime-10 821572 1405 ns/op 429 B/op 9 allocs/op BenchmarkParseStrptime-10 1000000 1082 ns/op 186 B/op 6 allocs/op ``` 2 - Translations are based on the [CLDR](https://cldr.unicode.org/) project, and it does support 900+ locales (vs 45+), including locales in draft stage. Those lunes translations are [generated](https://github.com/elastic/lunes/blob/main/generator.go) from a CLDR core file, and does not require manually changes. 3 - Replicates all the relevant standard `time.format_test.go` test cases, parsing foreign language values with and without layout replacements in all available locales and supported layouts (https://github.com/elastic/lunes/blob/main/lunes_test.go#L154). 4 - It is actively maintained (hosted under elastic repo). **Link to tracking Issue:** #32977 **Testing:** - Added unit tests For now, the only way of manually testing this functionality is by invoking the `ParseLocalizedStrptime` function manually through tests. **Documentation:**
1 parent 642cc35 commit 25cb194

File tree

113 files changed

+418
-143
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

113 files changed

+418
-143
lines changed
Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
# Use this changelog template to create an entry for release notes.
2+
3+
# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
4+
change_type: enhancement
5+
6+
# The name of the component, or a single word describing the area of concern, (e.g. filelogreceiver)
7+
component: pkg/ottl
8+
9+
# A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`).
10+
note: Add support for localized time parsing into the coreinternal/timeutils
11+
12+
# Mandatory: One or more tracking issues related to the change. You can use the PR number here if no issue exists.
13+
issues:
14+
- 32977
15+
16+
# (Optional) One or more lines of additional information to render under the primary note.
17+
# These lines will be padded with 2 spaces and then inserted directly into the document.
18+
# Use pipe (|) for multiline entries.
19+
subtext:
20+
21+
# If your change doesn't affect end users or the exported elements of any package,
22+
# you should instead start your pull request title with [chore] or use the "Skip Changelog" label.
23+
# Optional: The change log or logs in which this entry should be included.
24+
# e.g. '[user]' or '[user, api]'
25+
# Include 'user' if the change is relevant to end users.
26+
# Include 'api' if there is a change to a library API.
27+
# Default: '[user]'
28+
change_logs: [api]

cmd/otelcontribcol/go.mod

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -439,7 +439,7 @@ require (
439439
github.com/containerd/ttrpc v1.2.4 // indirect
440440
github.com/coreos/go-oidc/v3 v3.11.0 // indirect
441441
github.com/coreos/go-systemd/v22 v22.5.0 // indirect
442-
github.com/cyphar/filepath-securejoin v0.2.4 // indirect
442+
github.com/cyphar/filepath-securejoin v0.2.5 // indirect
443443
github.com/danieljoos/wincred v1.1.2 // indirect
444444
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
445445
github.com/dennwc/varint v1.0.0 // indirect
@@ -464,6 +464,7 @@ require (
464464
github.com/elastic/go-structform v0.0.12 // indirect
465465
github.com/elastic/go-sysinfo v1.7.1 // indirect
466466
github.com/elastic/go-windows v1.0.1 // indirect
467+
github.com/elastic/lunes v0.1.0 // indirect
467468
github.com/emicklei/go-restful/v3 v3.11.0 // indirect
468469
github.com/envoyproxy/go-control-plane v0.12.1-0.20240621013728-1eb8caab5155 // indirect
469470
github.com/envoyproxy/protoc-gen-validate v1.0.4 // indirect

cmd/otelcontribcol/go.sum

Lines changed: 4 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

cmd/oteltestbedcol/go.mod

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -89,6 +89,7 @@ require (
8989
github.com/docker/go-units v0.5.0 // indirect
9090
github.com/elastic/go-grok v0.3.1 // indirect
9191
github.com/elastic/go-structform v0.0.12 // indirect
92+
github.com/elastic/lunes v0.1.0 // indirect
9293
github.com/emicklei/go-restful/v3 v3.11.0 // indirect
9394
github.com/envoyproxy/go-control-plane v0.12.1-0.20240621013728-1eb8caab5155 // indirect
9495
github.com/envoyproxy/protoc-gen-validate v1.0.4 // indirect

cmd/oteltestbedcol/go.sum

Lines changed: 2 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

connector/countconnector/go.mod

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@ require (
2525
github.com/cespare/xxhash/v2 v2.3.0 // indirect
2626
github.com/davecgh/go-spew v1.1.1 // indirect
2727
github.com/elastic/go-grok v0.3.1 // indirect
28+
github.com/elastic/lunes v0.1.0 // indirect
2829
github.com/go-logr/logr v1.4.2 // indirect
2930
github.com/go-logr/stdr v1.2.2 // indirect
3031
github.com/go-viper/mapstructure/v2 v2.1.0 // indirect

connector/countconnector/go.sum

Lines changed: 2 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

connector/datadogconnector/go.mod

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -119,6 +119,7 @@ require (
119119
github.com/docker/go-units v0.5.0 // indirect
120120
github.com/dustin/go-humanize v1.0.1 // indirect
121121
github.com/elastic/go-grok v0.3.1 // indirect
122+
github.com/elastic/lunes v0.1.0 // indirect
122123
github.com/emicklei/go-restful/v3 v3.11.0 // indirect
123124
github.com/fatih/color v1.16.0 // indirect
124125
github.com/felixge/httpsnoop v1.0.4 // indirect

connector/datadogconnector/go.sum

Lines changed: 2 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

connector/routingconnector/go.mod

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@ require (
2121
github.com/cespare/xxhash/v2 v2.3.0 // indirect
2222
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
2323
github.com/elastic/go-grok v0.3.1 // indirect
24+
github.com/elastic/lunes v0.1.0 // indirect
2425
github.com/go-logr/logr v1.4.2 // indirect
2526
github.com/go-logr/stdr v1.2.2 // indirect
2627
github.com/go-viper/mapstructure/v2 v2.1.0 // indirect

connector/routingconnector/go.sum

Lines changed: 2 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

connector/sumconnector/go.mod

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ require (
2222
github.com/cespare/xxhash/v2 v2.3.0 // indirect
2323
github.com/davecgh/go-spew v1.1.1 // indirect
2424
github.com/elastic/go-grok v0.3.1 // indirect
25+
github.com/elastic/lunes v0.1.0 // indirect
2526
github.com/go-logr/logr v1.4.2 // indirect
2627
github.com/go-logr/stdr v1.2.2 // indirect
2728
github.com/go-viper/mapstructure/v2 v2.1.0 // indirect

connector/sumconnector/go.sum

Lines changed: 2 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

exporter/datadogexporter/go.mod

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -163,6 +163,7 @@ require (
163163
github.com/docker/go-units v0.5.0 // indirect
164164
github.com/dustin/go-humanize v1.0.1 // indirect
165165
github.com/elastic/go-grok v0.3.1 // indirect
166+
github.com/elastic/lunes v0.1.0 // indirect
166167
github.com/emicklei/go-restful/v3 v3.11.0 // indirect
167168
github.com/envoyproxy/go-control-plane v0.12.1-0.20240621013728-1eb8caab5155 // indirect
168169
github.com/envoyproxy/protoc-gen-validate v1.0.4 // indirect

exporter/datadogexporter/go.sum

Lines changed: 2 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

exporter/datadogexporter/integrationtest/go.mod

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -140,6 +140,7 @@ require (
140140
github.com/docker/go-units v0.5.0 // indirect
141141
github.com/dustin/go-humanize v1.0.1 // indirect
142142
github.com/elastic/go-grok v0.3.1 // indirect
143+
github.com/elastic/lunes v0.1.0 // indirect
143144
github.com/emicklei/go-restful/v3 v3.11.0 // indirect
144145
github.com/envoyproxy/go-control-plane v0.12.1-0.20240621013728-1eb8caab5155 // indirect
145146
github.com/envoyproxy/protoc-gen-validate v1.0.4 // indirect

exporter/datadogexporter/integrationtest/go.sum

Lines changed: 2 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

exporter/elasticsearchexporter/integrationtest/go.mod

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,7 @@ require (
4646
github.com/elastic/go-structform v0.0.12 // indirect
4747
github.com/elastic/go-sysinfo v1.14.0 // indirect
4848
github.com/elastic/go-windows v1.0.1 // indirect
49+
github.com/elastic/lunes v0.1.0 // indirect
4950
github.com/expr-lang/expr v1.16.9 // indirect
5051
github.com/felixge/httpsnoop v1.0.4 // indirect
5152
github.com/fsnotify/fsnotify v1.7.0 // indirect

exporter/elasticsearchexporter/integrationtest/go.sum

Lines changed: 2 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

exporter/honeycombmarkerexporter/go.mod

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@ require (
2727
github.com/cespare/xxhash/v2 v2.3.0 // indirect
2828
github.com/davecgh/go-spew v1.1.1 // indirect
2929
github.com/elastic/go-grok v0.3.1 // indirect
30+
github.com/elastic/lunes v0.1.0 // indirect
3031
github.com/felixge/httpsnoop v1.0.4 // indirect
3132
github.com/fsnotify/fsnotify v1.7.0 // indirect
3233
github.com/go-logr/logr v1.4.2 // indirect

exporter/honeycombmarkerexporter/go.sum

Lines changed: 2 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

exporter/rabbitmqexporter/go.mod

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -97,12 +97,12 @@ require (
9797
go.opentelemetry.io/otel/trace v1.29.0 // indirect
9898
go.uber.org/multierr v1.11.0 // indirect
9999
golang.org/x/crypto v0.26.0 // indirect
100-
golang.org/x/mod v0.17.0 // indirect
100+
golang.org/x/mod v0.19.0 // indirect
101101
golang.org/x/net v0.28.0 // indirect
102102
golang.org/x/sync v0.8.0 // indirect
103103
golang.org/x/sys v0.24.0 // indirect
104104
golang.org/x/text v0.17.0 // indirect
105-
golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d // indirect
105+
golang.org/x/tools v0.23.0 // indirect
106106
google.golang.org/genproto/googleapis/rpc v0.0.0-20240822170219-fc7c04adadcd // indirect
107107
google.golang.org/grpc v1.66.0 // indirect
108108
google.golang.org/protobuf v1.34.2 // indirect

exporter/rabbitmqexporter/go.sum

Lines changed: 4 additions & 4 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

exporter/signalfxexporter/go.mod

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,7 @@ require (
3838
github.com/beorn7/perks v1.0.1 // indirect
3939
github.com/cespare/xxhash/v2 v2.3.0 // indirect
4040
github.com/davecgh/go-spew v1.1.1 // indirect
41+
github.com/elastic/lunes v0.1.0 // indirect
4142
github.com/felixge/httpsnoop v1.0.4 // indirect
4243
github.com/fsnotify/fsnotify v1.7.0 // indirect
4344
github.com/go-logr/logr v1.4.2 // indirect
@@ -54,6 +55,7 @@ require (
5455
github.com/knadh/koanf/providers/confmap v0.1.0 // indirect
5556
github.com/knadh/koanf/v2 v2.1.1 // indirect
5657
github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0 // indirect
58+
github.com/magefile/mage v1.15.0 // indirect
5759
github.com/mitchellh/copystructure v1.2.0 // indirect
5860
github.com/mitchellh/reflectwalk v1.0.2 // indirect
5961
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect

0 commit comments

Comments
 (0)