Async Jobs

Async Jobs

Patterns for background task processing with Celery, ARQ, and Redis. Covers task queues, canvas workflows, scheduling, retry strategies, rate limiting, and production monitoring. Each category has individual rule files in references/ loaded on-demand.

Quick Reference

| Category | Rules | Impact | When to Use | |----------|-------|--------|-------------| | [Configuration](#configuration) | celery-config | HIGH | Celery app setup, broker, serialization, worker tuning | | [Task Routing](#task-routing) | task-routing | HIGH | Priority queues, multi-queue workers, dynamic routing | | [Canvas Workflows](#canvas-workflows) | canvas-workflows | HIGH | Chain, group, chord, nested workflows | | [Retry Strategies](#retry-strategies) | retry-strategies | HIGH | Exponential backoff, idempotency, dead letter queues | | [Scheduling](#scheduling) | scheduled-tasks | MEDIUM | Celery Beat, crontab, database-backed schedules | | [Monitoring](#monitoring) | monitoring-health | MEDIUM | Flower, custom events, health checks, metrics | | [Result Backends](#result-backends) | result-backends | MEDIUM | Redis results, custom states, progress tracking | | [ARQ Patterns](#arq-patterns) | arq-patterns | MEDIUM | Async Redis Queue for FastAPI, lightweight jobs | | [Temporal Workflows](#temporal-workflows) | temporal-workflows | HIGH | Durable workflow definitions, sagas, signals, queries | | [Temporal Activities](#temporal-activities) | temporal-activities | HIGH | Activity patterns, workers, heartbeats, testing |

Total: 10 rules across 9 categories

Quick Start

@app.task(bind=True, max_retries=3, default_retry_delay=60)
def process_payment(self, order_id: str):
    try:
        return gateway.charge(order_id)
    except TransientError as exc:
        raise self.retry(exc=exc, countdown=2 ** self.request.retries * 60)

Load more examples: Read("${CLAUDE_SKILL_DIR}/references/quick-start-examples.md") for Celery retry task and ARQ/FastAPI integration patterns.

Configuration

Production Celery app configuration with secure defaults and worker tuning.

Key Patterns

• JSON serialization with task_serializer="json" for safety
• Late acknowledgment with task_acks_late=True to prevent task loss on crash
• Time limits with both task_time_limit (hard) and task_soft_time_limit (soft)
• Fair distribution with worker_prefetch_multiplier=1
• Reject on lost with task_reject_on_worker_lost=True

Key Decisions

| Decision | Recommendation | |----------|----------------| | Serializer | JSON (never pickle) | | Ack mode | Late ack (task_acks_late=True) | | Prefetch | 1 for fair, 4-8 for throughput | | Time limit | soft threshold |

Result Backends

Task result storage, custom states, and progress tracking patterns.

Key Patterns

• Redis backend for task status and small results
• Custom task states (VALIDATING, PROCESSING, UPLOADING) for progress
• update_state() for real-time progress reporting
• S3/database for large result storage (never Redis)
• AsyncResult for querying task state and progress

Key Decisions

| Decision | Recommendation | |----------|----------------| | Status storage | Redis result backend | | Large results | S3 or database (never Redis) | | Progress | Custom states with update_state() | | Result query | AsyncResult with state checks |

ARQ Patterns

Lightweight async Redis Queue for FastAPI and simple background tasks.

Key Patterns

• Native async/await with arq for FastAPI integration
• Worker lifecycle with startup/shutdown hooks for resource management
• Job enqueue from FastAPI routes with enqueue_job()
• Job status tracking with Job.status() and Job.result()
• Delayed tasks with _delay=timedelta() for deferred execution

Key Decisions

| Decision | Recommendation | |----------|----------------| | Simple async | ARQ (native async) | | Complex workflows | Celery (chains, chords) | | In-process quick | FastAPI BackgroundTasks | | LLM workflows | LangGraph (not Celery) |

Tool Selection

Load: Read("${CLAUDE_SKILL_DIR}/references/quick-start-examples.md") for the full tool comparison table (ARQ, Celery, RQ, Dramatiq, FastAPI BackgroundTasks).

Anti-Patterns (FORBIDDEN)

Load details: Read("${CLAUDE_SKILL_DIR}/references/anti-patterns.md") for full list.

Key rules: never run long tasks in request handlers, never block on results inside tasks, never store large results in Redis, always use idempotency for retried tasks.

Temporal Workflows

Durable execution engine for reliable distributed applications with Temporal.io.

Key Patterns

• Workflow definitions with @workflow.defn and deterministic code
• Saga pattern with compensation for multi-step transactions
• Signals and queries for external interaction with running workflows
• Timers with workflow.wait_condition() for human-in-the-loop
• Parallel activities via asyncio.gather inside workflows

Key Decisions

| Decision | Recommendation | |----------|----------------| | Workflow ID | Business-meaningful, idempotent | | Determinism | Use workflow.random(), workflow.now() | | I/O | Always via activities, never directly |

Temporal Activities

Activity and worker patterns for Temporal.io I/O operations.

Key Patterns

• Activity definitions with @activity.defn for all I/O
• Heartbeating for long-running activities (> 60s)
• Error classification with ApplicationError(non_retryable=True) for business errors
• Worker configuration with dedicated task queues
• Testing with WorkflowEnvironment.start_local()

Key Decisions

| Decision | Recommendation | |----------|----------------| | Activity timeout | start_to_close for most cases | | Error handling | Non-retryable for business errors | | Testing | WorkflowEnvironment for integration tests |

Related Skills

• ork:python-backend - FastAPI, asyncio, SQLAlchemy patterns
• ork:langgraph - LangGraph workflow patterns (use for LLM workflows, not Celery)
• ork:distributed-systems - Resilience patterns, circuit breakers
• ork:monitoring-observability - Metrics and alerting

Capability Details

Load details: Read("${CLAUDE_SKILL_DIR}/references/capability-details.md") for full keyword index and problem-solution mapping across all 8 capabilities.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: yonatangross
• Source: yonatangross/orchestkit
• License: MIT
• Homepage: https://orchestkit.yonyon.ai

Install and usage instructions live in the source repository linked above.