Skip to content

[fix] Allow es-index-cleaner to delete indices based on current time #6790

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Mar 9, 2025

Conversation

Asatyam
Copy link
Contributor

@Asatyam Asatyam commented Mar 2, 2025

Which problem is this PR solving?

Description of the changes

  • Changes the es-index-cleaner to remove indices older than 24 hours from the time it runs, so deletions happen hourly instead of just at midnight.

How was this change tested?

  • CI

Checklist

@Asatyam Asatyam requested a review from a team as a code owner March 2, 2025 14:13
@Asatyam Asatyam requested a review from albertteoh March 2, 2025 14:13
gemini-code-assist[bot]

This comment was marked as spam.

Copy link

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

The pull request updates the es-index-cleaner to delete indices based on the current time, resolving issue #6236. The change replaces the previous logic that calculated the deletion date based on tomorrow's midnight with a calculation based on the current time. This seems like a good change, but I have a question about it.

Summary of Findings

  • Timezone Considerations: The code uses time.Now().UTC() to determine the current time. While this ensures consistency, it's important to verify that the Elasticsearch indices are also created using UTC time. If the indices are created using a different timezone, the deletion logic might not work as expected.

Merge Readiness

The code change appears straightforward and addresses the intended issue. However, it's crucial to ensure that the Elasticsearch indices are created using UTC time to avoid any unexpected behavior. I am unable to approve this pull request, and recommend that others review and approve this code before merging.

@dosubot dosubot bot added the area/storage label Mar 2, 2025
Copy link
Member

@yurishkuro yurishkuro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change looks ok but it's a breaking change that should be handled according to our policy that is documented in CONTRIBUTING.

@Asatyam Asatyam force-pushed the fix_index_cleaner_date branch from 2b6b2bd to 585325a Compare March 3, 2025 16:51
@Asatyam Asatyam requested a review from yurishkuro March 4, 2025 02:21
@Asatyam
Copy link
Contributor Author

Asatyam commented Mar 6, 2025

@yurishkuro PTAL

@yurishkuro yurishkuro changed the title [fix] Update es-index-cleaner to delete indices based on current time [fix] Allow es-index-cleaner to delete indices based on current time Mar 7, 2025
@yurishkuro
Copy link
Member

Please include evidence of testing

Copy link

codecov bot commented Mar 7, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.04%. Comparing base (6037db2) to head (d68e036).
Report is 2 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #6790      +/-   ##
==========================================
- Coverage   96.06%   96.04%   -0.03%     
==========================================
  Files         364      365       +1     
  Lines       20712    20720       +8     
==========================================
+ Hits        19897    19900       +3     
- Misses        622      626       +4     
- Partials      193      194       +1     
Flag Coverage Δ
badger_v1 9.86% <ø> (ø)
badger_v2 1.98% <ø> (ø)
cassandra-4.x-v1-manual 14.87% <ø> (ø)
cassandra-4.x-v2-auto 1.97% <ø> (ø)
cassandra-4.x-v2-manual 1.97% <ø> (ø)
cassandra-5.x-v1-manual 14.87% <ø> (ø)
cassandra-5.x-v2-auto 1.97% <ø> (ø)
cassandra-5.x-v2-manual 1.97% <ø> (ø)
elasticsearch-6.x-v1 19.26% <ø> (ø)
elasticsearch-7.x-v1 19.34% <ø> (ø)
elasticsearch-8.x-v1 19.51% <ø> (ø)
elasticsearch-8.x-v2 1.98% <ø> (-0.12%) ⬇️
grpc_v1 10.90% <ø> (ø)
grpc_v2 7.95% <ø> (ø)
kafka-3.x-v1 10.16% <ø> (ø)
kafka-3.x-v2 1.98% <ø> (ø)
memory_v2 1.98% <ø> (ø)
opensearch-1.x-v1 19.39% <ø> (ø)
opensearch-2.x-v1 19.39% <ø> (ø)
opensearch-2.x-v2 1.98% <ø> (ø)
tailsampling-processor 0.48% <ø> (ø)
unittests 94.93% <100.00%> (-0.03%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@Asatyam Asatyam force-pushed the fix_index_cleaner_date branch from f52b9c4 to 5a9f3fe Compare March 9, 2025 12:31
@yurishkuro yurishkuro enabled auto-merge March 9, 2025 15:36
@yurishkuro yurishkuro added this pull request to the merge queue Mar 9, 2025
Merged via the queue into jaegertracing:main with commit 1d516aa Mar 9, 2025
56 checks passed
amilbcahat pushed a commit to amilbcahat/jaeger that referenced this pull request May 4, 2025
…aegertracing#6790)

## Which problem is this PR solving?
- Resolves jaegertracing#6236 
## Description of the changes
- Changes the es-index-cleaner to remove indices older than 24 hours
from the time it runs, so deletions happen hourly instead of just at
midnight.
## How was this change tested?
- CI

## Checklist
- [x] I have read
https://github.com/jaegertracing/jaeger/blob/master/CONTRIBUTING_GUIDELINES.md
- [x] I have signed all commits
- [x] I have added unit tests for the new functionality
- [x] I have run lint and test steps successfully
  - for `jaeger`: `make lint test`
  - for `jaeger-ui`: `npm run lint` and `npm run test`

---------

Signed-off-by: SatyamAgrawal <[email protected]>
Signed-off-by: Yuri Shkuro <[email protected]>
Signed-off-by: Yuri Shkuro <[email protected]>
Co-authored-by: Yuri Shkuro <[email protected]>
Co-authored-by: Yuri Shkuro <[email protected]>
amilbcahat pushed a commit to amilbcahat/jaeger that referenced this pull request May 4, 2025
…aegertracing#6790)

## Which problem is this PR solving?
- Resolves jaegertracing#6236
## Description of the changes
- Changes the es-index-cleaner to remove indices older than 24 hours
from the time it runs, so deletions happen hourly instead of just at
midnight.
## How was this change tested?
- CI

## Checklist
- [x] I have read
https://github.com/jaegertracing/jaeger/blob/master/CONTRIBUTING_GUIDELINES.md
- [x] I have signed all commits
- [x] I have added unit tests for the new functionality
- [x] I have run lint and test steps successfully
  - for `jaeger`: `make lint test`
  - for `jaeger-ui`: `npm run lint` and `npm run test`

---------

Signed-off-by: SatyamAgrawal <[email protected]>
Signed-off-by: Yuri Shkuro <[email protected]>
Signed-off-by: Yuri Shkuro <[email protected]>
Co-authored-by: Yuri Shkuro <[email protected]>
Co-authored-by: Yuri Shkuro <[email protected]>
Signed-off-by: amol-verma-allen <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bug]: Index cleaner only works once a day, even if it runs every hour.
2 participants