Gevent celery config I/O bound

PublishedDecember 27, 2025

•2 min read•View as Markdown

D

Danny Crasto

I am developer/code-reviewer/debugger/bug-fixer/architect/teacher/builder from dubai, uae

Comments

Join the discussion

No comments yet. Be the first to comment.

More from this blog

8 Simple reminders

You Should Tell Yourself

Jul 22, 20261 min read14

Design a url shortner

A URL shortener is a proxy service that provides a mapping between the short and full representation of a URL. The short URL has the advantage of being small. The service can also provide useful analy

Jul 19, 20262 min read217

No regrets

Listening to the Mel Robbins's podcast on regrets was a 'ear'-opener. One can have reqrets of action or inaction. 4 types of reqrets foundation - should've done the work boldness - should've taken t

Jul 15, 20261 min read23

Design a distributed unique id generator

We need to create a unique ID generator for our high-traffic web application, generating about 10K IDs/second. The IDs can't simply be monotonically increasing integers, which are good for data access

Jul 9, 20262 min read143

Design a distributed unique id generator

luck surface area

doing + telling

Jun 24, 20261 min read90

D

Danny's blog

75 posts

I like computers

When NOT using Gevent (worker_pool\=[prefork|gthread]), the number of concurrent workers defaults to the number of cores which are effectively the tasks that will be executing in “parallel”.

The prefetch multiplier (default 4) is an optimization to reduce round trips to the broker (queue), to eagerly fetch tasks at the node per available worker. This happens at the consumers not at the workers.

With 4 workers:

16 tasks consumed
12 are waiting for an available worker
4 are being concurrently processed

If you have short lived tasks this isn’t a problem. However if your workloads’ execution time varies, all task could potentially convoy on the same node and dramatically reducing the throughput per worker. It’s recommended to use a prefetch of 1 or 2 to reduce the effect.

If you strictly have IO bound tasks you should use gevent. Using cooperative multitasking via a coroutine mechanism, you can effectively have 100s of workers per worker which will all execute concurrently.

Some considerations:

Use connection pooling for database or http access to avoid herding
monkey patch correctly
if the workload isn’t IO bound, you will run into issues with the GIL.

Since python 3.14, free threaded python is officially supported (phase 2). However it’s not production ready but it brings thread level parallelism to applications.

Happy Hackin’

References:

#async #python #threading #event-loop

33 views

Command Palette

Comments

More from this blog

References: