Improving Instrumentation for Sidekiq Iteratable Jobs

[hannahramadan]

The New Relic Ruby agent team would love to gather feedback on how the agent instruments jobs that use Sidekiq's Iteratable Job feature.

The Current Behavior

Currently, the agent's instrumentation model is optimized for traditional, short-lived jobs. When it encounters a Sidekiq job using iteration, it wraps the entire perform call in a single transaction.

This means a job that iterates over 10,000 records in two hours will appear in New Relic as one two-hour-long transaction. This behavior can:

• Skew Metrics: Your average background job response time can be skewed by these long-running (but normal) iteration jobs.
• Fragment Traces: Traces that are this long are often too large to be effectively sampled, resulting in fragmented data.
• Hide Information: You lose visibility into the individual iterations, which are the real units of work.

Potential Solutions

We've been researching this pattern and have come up with some ideas.

Option 1: Instrument Iterations as Segments
We would keep the single transaction for the parent job, but add instrumentation to the each_iteration hook to create a new span/segment for each iteration.

• Pro: This would maintain the parent-child relationship in a single trace. You could see a breakdown of Iteration 1, Iteration 2, etc., within the transaction trace (for smaller jobs where sampling isn't an issue).
• Cons: The parent transaction would still be two hours long and large jobs would face trace fragmentation.

Option 2: Instrument Iterations as Separate, Linked Transactions
This is a two-phase solution to treat iterations as distinct units of work.

• Phase 1: We would instrument the each_iteration hook to begin a new, distinct New Relic transaction for every single iteration.

  • Pro: The long-running parent job would no-longer be reported, and your metrics would reflect the performance of individual iterations.
  • Con: You would lose the logical link back to the parent process that kicked off the iterations (ex. the Controller action that kicked off a job).

• Phase 2: We would enhance Phase 1 by implementing Span Links (an OpenTelemetry concept). This would allow the parent job to create a link to each new iteration transaction it spawns.

We’re Looking for Feedback

We would love to hear from you:

• How problematic is the current "single long transaction" behavior for you?
• Looking at the options, what would you prefer?
• Any other thoughts or suggestions on how you'd like to see this data?

Thank you for helping us improve the Ruby agent!