Declarative event-driven tasks in Python
Most distributed task frameworks give you a queue. Put work in, workers pull it out. That covers the easy half of background processing.
Most distributed task frameworks give you a queue. Put work in, workers pull it out. That covers the easy half of background processing.
Pynenc models every task invocation as a finite-state machine. The file pynenc/invocation/status.py declares the invocation states, the allowed transitions between them, ownership rules, and dedicated recovery states.
Many background jobs call an external system on behalf of separate accounts, tenants, or installations. The external system allows parallel calls across different accounts, but it does not allow two calls for the same account to run at the same time.
You have a Python function that processes one item. You call it in a loop. The list grows, and the loop slows down because each call waits on I/O: an API request, a database query, a scrape, an embedding, a model call....
I wanted a small failure test: start a few long-running tasks, kill the worker, then see whether the work still finishes.