Why not simply use Celery/RQ/Huey/TaskTiger?
Sadly, WakaTime has been utilizing Celery for nearly 10 years now. Throughout that point I’ve skilled many crucial bugs, some nonetheless open years after being launched. Celery was fairly good, however function bloat made the challenge troublesome to take care of. Additionally for my part, splitting the code into three separate GitHub repos made the codebase onerous to learn.
Nonetheless, the principle cause: Celery delayed duties don’t scale.
Should you use Celery delayed duties, as your web site grows finally you’ll begin seeing this error message:
QoS: Disabled: prefetch_count exceeds 65535
When that occurs the employee stops processing all duties, not simply delayed ones! As WakaTime grew, we began working into this bug extra steadily.
I attempted RQ, Huey, and TaskTiger, however they have been lacking options and processed duties slower than Celery. A distributed process queue is indispensable for an internet site like WakaTime, and I used to be bored with working into bugs. For that cause, I made a decision to construct the best distributed process queue potential whereas nonetheless offering all of the options required by WakaTime.
Introducing WakaQ
WakaQ is a brand new Python distributed process queue. Use it to run code within the background so your web site stays quick and snappy, and your customers keep completely happy.
WakaQ is straightforward
It’s just one,264 strains of code!
$ discover . -name '*.py' -not -path "./migrations*" -not -path "./venv*" | xargs wc -l | grep " complete" | awk '{print $1}' | numfmt --grouping
1,264
It solely took one week from the primary line of code till absolutely changing Celery at WakaTime. That claims one thing about it’s simplicity.
Every queue is applied utilizing a Redis listing. Delayed duties get their very own queues applied utilizing Redis sorted units. Broadcast duties share a single Redis Pub/Sub queue.
WakaQ has all the required options
- Queue priorities
- Delayed duties (run duties after a timedelta eta)
- Scheduled cron periodic duties
- Broadcast duties (run a process on all employees)
- Process comfortable and onerous timeout limits
- Optionally retry duties on comfortable timeouts
- Combats reminiscence leaks by restarting employees when max_mem_percent reached
- Tremendous minimal and maintainable
Options thought of out of scope are price limiting, unique locking, storing process outcomes, and process chaining. These are simple so as to add in your utility’s process code, and also you in all probability need to implement these particular to your app’s wants anyway.
WakaQ is able to use
WakaQ remains to be a brand new challenge, so use at your individual danger. WakaQ at present powers all background duties in prod for the WakaTime web site, together with however not restricted to:
- sending code stats electronic mail reviews
- renewing our LetsEncrypt SSL certs
- pre caching dashboards, repo badges, and embeddable charts
- the rest we don’t need holding up the online requests
It’s launched beneath a BSD license, so you should use it in open and closed supply initiatives. Should you discover any bugs please open a problem, however suppose twice earlier than requesting new options 🙂
Pleased coding!
