Only report on non-handled ActiveJob retry

[smathieu]

Problem Statement

We have jobs where we explicitly want to handle an exception and retry on.

e.g.

  class ExampleJob < ApplicationJob
    queue_as :high

    retry_on ActiveRecord::LockStatementTimeout
    
    def perform
      if rand(2) == 0
         # Exception we have explicitly handled using `retry_on`, only report if this is the last retry
         raise ActiveRecord::LockStatementTimeout
      else
         # Unhandled exception, always report to Sentry
         1 / 0 
      end
    end
end

If our case, we don't want to 'polute' Sentry with expected failures, like the lock not being available, unless we exhaust all retries.

As far as I under it, the only control we have over this is to set report_after_job_retries to false, but this will only report after all retries have been exhausted, which I think defaults to a 2 week time period.

Solution Brainstorm

I think report_after_job_retries = :only_unhandled should enable the described behavior.

[linear]

RUBY-84 Only report on non-handled ActiveJob retry

[solnic]

Thanks for reporting this issues. I'm not sure if I understand this correctly though - if you have a job that retries only on a specific exception, and it crashes due to another exception, then it should be reported because that's an unhandled exception. Isn't it what the SDK does already?

[smathieu]

What we're observing (and I think is the documented expected behavior) is that in the case above, the ActiveRecord::LockStatementTimeout is handled by the ActiveJob exception logic, but the ZeroDivisionError it bubbled up to the default Sidekiq error logic, which will also retry.

Both exceptions will actually retry, but one will retry based on the ActiveJob logic and the other based on the sidekiq logic.

We would like to configure how these two cases are reported to Sentry differently.

What I'm saying is that if we've specified a retry_on handler, this is an expected exception and we only want to report if the retries are exhausted as this should never happen, but we want to report exceptions retried by a sidekiq's default handle as that's unexpected.