Inaccurate celery task status

task_result = AsyncResult(task_id, app=celery_app)

works correctly if you have a result backend configured and tasks are defined with @app.task(bind=True). The result would be STARTED; SUCCESS; FAILURE; RETRY; REVOKED.

When staring tasks, I’d always get a PENDING status regardless of the queried task_id. The PENDING state is celery's default state for any task whose status is unknown or has no history.

The workaround: write the task_id to Redis (Celery's result backend) on task start, then only report status for task_id's that exist there. Also enable track_started=True by default, long-running tasks in chains show as PENDING until completion, which is pointless for monitoring.

def trigger_chain(task_chain):
    future = task_chain.apply_async()
    celery_app.backend.client.set(f"{future.id}", "started", ex=86400)

def get_task_result(task_id: str):
    try:
        task_result = AsyncResult(task_id, app=celery_app)
        if task_result.status == "PENDING":
            # Access the backend to check if the task key exists
            backend = celery_app.backend
            task_key, backend_client = (
                backend.get_key_for_task(task_id), gettattr(backend, "client", None
            )

            # Check if the key exists in Redis
            if backend_client and 
                if not ( backend_client.exists(key) or backend_client.exists(task_id)):
                    # Task doesn't exist in the result backend
                    return jsonify(
                        {
                            "error": "Task not found",
                            "task_id": task_id,
                            "message": "No task with this ID exists in the result backend",
                        }
                    ), 404
        # Valid task
        response = {
            "task_id": task_id,
            "status": task_result.status,
        }

Happy hackin’

Inaccurate celery task status

Comments

More from this blog

8 Simple reminders

Design a url shortner

No regrets

Design a distributed unique id generator

luck surface area

References:

Command Palette

Comments

More from this blog

References: