Gevent celery config I/O bound

When NOT using Gevent (worker_pool\=[prefork|gthread]), the number of concurrent workers defaults to the number of cores which are effectively the tasks that will be executing in “parallel”.

The prefetch multiplier (default 4) is an optimization to reduce round trips to the broker (queue), to eagerly fetch tasks at the node per available worker. This happens at the consumers not at the workers.

With 4 workers:

16 tasks consumed
12 are waiting for an available worker
4 are being concurrently processed

If you have short lived tasks this isn’t a problem. However if your workloads’ execution time varies, all task could potentially convoy on the same node and dramatically reducing the throughput per worker. It’s recommended to use a prefetch of 1 or 2 to reduce the effect.

If you strictly have IO bound tasks you should use gevent. Using cooperative multitasking via a coroutine mechanism, you can effectively have 100s of workers per worker which will all execute concurrently.

Some considerations:

Use connection pooling for database or http access to avoid herding
monkey patch correctly
if the workload isn’t IO bound, you will run into issues with the GIL.

Since python 3.14, free threaded python is officially supported (phase 2). However it’s not production ready but it brings thread level parallelism to applications.

Happy Hackin’