[refactor] Rework ClickHouse schema structure #7181

mahadzaryab1 · 2025-05-31T23:02:24Z

Which problem is this PR solving?

Towards [clickhouse] Implement Trace Reader For ClickHouse Storage #7134

Description of the changes

This PR cleans up the dbmodel that we have to make it compatible with the clickhouse-go driver. It removes the nesting of types and instead stores all the fields under one struct to make it reflective of the actual schema.

How was this change tested?

CI

Checklist

I have read https://github.com/jaegertracing/jaeger/blob/master/CONTRIBUTING_GUIDELINES.md
I have signed all commits
I have added unit tests for the new functionality
I have run lint and test steps successfully
- for jaeger: make lint test
- for jaeger-ui: npm run lint and npm run test

Signed-off-by: Mahad Zaryab <[email protected]>

codecov · 2025-05-31T23:09:58Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.16%. Comparing base (e598691) to head (6593f41).
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #7181      +/-   ##
==========================================
- Coverage   96.21%   96.16%   -0.06%     
==========================================
  Files         365      366       +1     
  Lines       21964    21947      -17     
==========================================
- Hits        21133    21105      -28     
- Misses        621      629       +8     
- Partials      210      213       +3

Flag	Coverage Δ
badger_v1	`9.85% <ø> (ø)`
badger_v2	`1.90% <ø> (ø)`
cassandra-4.x-v1-manual	`14.82% <ø> (ø)`
cassandra-4.x-v2-auto	`1.89% <ø> (ø)`
cassandra-4.x-v2-manual	`1.89% <ø> (ø)`
cassandra-5.x-v1-manual	`14.82% <ø> (ø)`
cassandra-5.x-v2-auto	`1.89% <ø> (ø)`
cassandra-5.x-v2-manual	`1.89% <ø> (ø)`
elasticsearch-6.x-v1	`20.65% <ø> (ø)`
elasticsearch-7.x-v1	`20.73% <ø> (ø)`
elasticsearch-8.x-v1	`20.90% <ø> (ø)`
elasticsearch-8.x-v2	`1.90% <ø> (ø)`
grpc_v1	`11.39% <ø> (ø)`
grpc_v2	`1.90% <ø> (ø)`
kafka-3.x-v1	`10.20% <ø> (ø)`
kafka-3.x-v2	`1.90% <ø> (ø)`
memory_v2	`1.90% <ø> (ø)`
opensearch-1.x-v1	`20.78% <ø> (ø)`
opensearch-2.x-v1	`20.78% <ø> (ø)`
opensearch-2.x-v2	`1.90% <ø> (ø)`
query	`1.90% <ø> (ø)`
tailsampling-processor	`0.51% <ø> (ø)`
unittests	`94.98% <100.00%> (-0.06%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Signed-off-by: Mahad Zaryab <[email protected]>

mahadzaryab1

couple of questions

mahadzaryab1 · 2025-06-01T00:20:03Z

internal/storage/v2/clickhouse/tracestore/dbmodel/to_dbmodel.go

-type AttributesGroup struct {
-	BoolKeys     []string
-	BoolValues   []bool
-	DoubleKeys   []string
-	DoubleValues []float64
-	IntKeys      []string
-	IntValues    []int64
-	StrKeys      []string
-	StrValues    []string
-	BytesKeys    []string
-	BytesValues  []string
-}


@yurishkuro I don't believe this approach for attributes is compatible with ClickHouse. If we want to maintain typed attributes, I think using the JSON type might be the best approach here. Thoughts?

Why don't you think it's not compatible? We discussed this on the original PR, please check how it was proposed to be used.

It's using the NESTED construct, which stores the attributes as sub-columnar structures. I don't know how exactly JSON support is implemented in CH but I highly doubt it would match in efficiency with explicitly managed columnar layout. Does CH even support schema for JSON fields? Without schema it would have to treat it as a full text index (similar to ES).

@yurishkuro Ah okay sounds good! I reverted this change here to keep that struct for now. But the semantics of the conversion will need to change to follow https://clickhouse.com/docs/integrations/go#nested. I'll leave it out of the refactored model for now and bring it back cleanly in the following PRs.

mahadzaryab1 · 2025-06-01T00:22:15Z

internal/storage/v2/clickhouse/tracestore/dbmodel/dbmodel.go

-	StatusCode    string
-	StatusMessage string
-	Attributes    AttributesGroup
+	// --- Span ---


@yurishkuro There's a couple of fields missing from the OTLP type. Do we want to store these?

Flags

DroppedAttributesCount

DroppedEventsCount

DroppedLinksCount

yes we can add these too, doesn't hurt.

This PR is already quite large so I'll add these in a following PR

Signed-off-by: Mahad Zaryab <[email protected]>

mahadzaryab1 · 2025-06-01T00:30:47Z

internal/storage/v2/clickhouse/tracestore/dbmodel/from_dbmodel.go

-	}
-	resourceAttributes.CopyTo(resource.Attributes())
+	// TODO: populate attributes
+	// TODO: do we populate the service name from the span?


@yurishkuro Question for you. My guess was no and that this would already exist in the attributes we store but just wanted to double check.

Signed-off-by: Mahad Zaryab <[email protected]>

internal/storage/v2/clickhouse/tracestore/dbmodel/from_dbmodel_test.go

Signed-off-by: Mahad Zaryab <[email protected]>

Rework ClickHouse Schema Structure

fdb5df0

Signed-off-by: Mahad Zaryab <[email protected]>

mahadzaryab1 added the changelog:exprimental Change to an experimental part of the code label May 31, 2025

mahadzaryab1 added 3 commits May 31, 2025 19:38

Remove Attributes From Event

410fa45

Signed-off-by: Mahad Zaryab <[email protected]>

Remove to_dbmodel.go

f21bc14

Signed-off-by: Mahad Zaryab <[email protected]>

Cleanup Tests

b858eb0

Signed-off-by: Mahad Zaryab <[email protected]>

mahadzaryab1 commented Jun 1, 2025

View reviewed changes

Fix Lint

207d31f

Signed-off-by: Mahad Zaryab <[email protected]>

mahadzaryab1 commented Jun 1, 2025

View reviewed changes

Bring Back from_dbmodel.go Changes

10a5558

Signed-off-by: Mahad Zaryab <[email protected]>

mahadzaryab1 marked this pull request as ready for review June 1, 2025 00:56

mahadzaryab1 requested a review from a team as a code owner June 1, 2025 00:56

mahadzaryab1 requested a review from jkowall June 1, 2025 00:56

dosubot bot added the area/storage label Jun 1, 2025

Fix Lint

4a40113

Signed-off-by: Mahad Zaryab <[email protected]>

graphite-app bot reviewed Jun 1, 2025

View reviewed changes

internal/storage/v2/clickhouse/tracestore/dbmodel/from_dbmodel_test.go Outdated Show resolved Hide resolved

mahadzaryab1 added 4 commits May 31, 2025 20:58

Remove Debug Statement

7ee5184

Signed-off-by: Mahad Zaryab <[email protected]>

Add Helpers For Parsing Nested Types

664dcd6

Signed-off-by: Mahad Zaryab <[email protected]>

Add Unit Tests

be2dde8

Signed-off-by: Mahad Zaryab <[email protected]>

Remove ch Tag For Duration

6593f41

Signed-off-by: Mahad Zaryab <[email protected]>

yurishkuro approved these changes Jun 1, 2025

View reviewed changes

yurishkuro added this pull request to the merge queue Jun 1, 2025

Merged via the queue into jaegertracing:main with commit 2a12360 Jun 1, 2025
60 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[refactor] Rework ClickHouse schema structure #7181

[refactor] Rework ClickHouse schema structure #7181

Uh oh!

mahadzaryab1 commented May 31, 2025 •

edited

Loading

Uh oh!

codecov bot commented May 31, 2025 •

edited

Loading

Uh oh!

mahadzaryab1 left a comment

Uh oh!

mahadzaryab1 Jun 1, 2025

Uh oh!

yurishkuro Jun 1, 2025

Uh oh!

mahadzaryab1 Jun 1, 2025

Uh oh!

mahadzaryab1 Jun 1, 2025

Uh oh!

yurishkuro Jun 1, 2025

Uh oh!

mahadzaryab1 Jun 1, 2025

Uh oh!

mahadzaryab1 Jun 1, 2025

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

[refactor] Rework ClickHouse schema structure #7181

[refactor] Rework ClickHouse schema structure #7181

Uh oh!

Conversation

mahadzaryab1 commented May 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which problem is this PR solving?

Description of the changes

How was this change tested?

Checklist

Uh oh!

codecov bot commented May 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

mahadzaryab1 left a comment

Choose a reason for hiding this comment

Uh oh!

mahadzaryab1 Jun 1, 2025

Choose a reason for hiding this comment

Uh oh!

yurishkuro Jun 1, 2025

Choose a reason for hiding this comment

Uh oh!

mahadzaryab1 Jun 1, 2025

Choose a reason for hiding this comment

Uh oh!

mahadzaryab1 Jun 1, 2025

Choose a reason for hiding this comment

Uh oh!

yurishkuro Jun 1, 2025

Choose a reason for hiding this comment

Uh oh!

mahadzaryab1 Jun 1, 2025

Choose a reason for hiding this comment

Uh oh!

mahadzaryab1 Jun 1, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mahadzaryab1 commented May 31, 2025 •

edited

Loading

codecov bot commented May 31, 2025 •

edited

Loading