diff --git a/src/docs/services/queue.mdx b/src/docs/services/queue.mdx index bcc0d3d4ff..f0ed92b98f 100644 --- a/src/docs/services/queue.mdx +++ b/src/docs/services/queue.mdx @@ -11,14 +11,15 @@ Sentry relies on the [Celery](https://docs.celeryproject.org/) library for manag Sentry configures tasks with a special decorator that allows us more explicit control over the callable. ```python -from sentry.tasks.base import instrumented_task +from sentry.tasks.base import instrumented_task, retry @instrumented_task( name="sentry.tasks.do_work", queue="important_queue", + max_retries=3, default_retry_delay=60 * 5, - max_retries=None, ) +@retry(on=(), exclude=(), ignore=()) def do_work(kind_of_work, **kwargs): # ... ``` @@ -33,7 +34,7 @@ There are a few important points: names which makes the name tied to the location of the code and more brittle for future code maintenance. -- Tasks _must_ accept `\*\*kwargs`` to handle rolling compatibility. +- Tasks _must_ accept `kwargs` to handle rolling compatibility. This ensures tasks will accept any message which happens to be in the queue rather than fail for unknown arguments. It helps with @@ -47,7 +48,25 @@ There are a few important points: a migration a little and gives some more operator flexibility, but message loss because of unknown arguments is still not acceptable. -- Tasks _should_ automatically retry on failure. +- Tasks _should_ retry on exception. + + Without retries, any exceptions raised while executing the task will stop + the task from moving forward. Generally, there are three categories of + exceptions that could be raised: + + 1. Retryable exceptions: transient errors (e.g. connection timeouts, + DB unavailable, query read timeouts) + 2. Ignorable exceptions: this is task specific but an example of an + ignorable exception would be if your task is attempting to delete a row + but the row doesn't exist and raises `DoesNotExist` exception. + 3. Non-ignorable programming errors that should be re-raised and fixed + (AttributeError, TypeError, ValueError, etc) + + Use the `@retry` decorator to specify which exceptions should be retried, + re-raised, and ignored. It might not be possible to know ahead of time + all the possible exceptions that could be raised in your task, so starting + with just the `@retry(on=())` might make sense and + updating the types of exceptions to handle after encountering them. - Tasks arguments _should_ be primitive types and small. @@ -70,6 +89,9 @@ There are a few important points: containing a task must be added to the `CELERY_IMPORTS` setting in `src/sentry/conf/server.py`. +- Tasks specifying a named `queue` must be added to `CELERY_QUEUES`. + + ## Running a Worker Workers can be run by using the [Sentry CLI](https://docs.sentry.io/product/cli/).