Pynenc

Pynenc is a Python distributed task system built around the problems other task frameworks leave for you to solve: lost tasks, duplicate work, dependency deadlocks, and failures you can’t debug

100% open source. No strings attached.
No paid cloud edition, no premium tier, no enterprise upsell. Core, plugins, and monitoring are MIT-licensed.

What Pynenc solves

1. Tasks disappear when workers crash

Pynenc tracks every invocation through a strict state machine with ownership semantics and runner heartbeats. Dead runners are detected automatically; orphaned invocations are reclaimed and re-routed

2. Duplicate work runs in parallel

Built-in concurrency control with four modes: DISABLED, TASK (one per task), ARGUMENTS (one per unique args), KEYS (one per arbitrary key). Duplicates are rejected before reaching a worker.

3. Dependencies deadlock your workers

Task A waits on B, B waits on C, all workers blocked. Pynenc’s orchestrator pauses waiting tasks to free their slots, then prioritizes by dependency count — the task blocking the most others runs first. Dependency chains resolve without holding threads hostage.

4. Failures are impossible to debug

Pynmon (built-in monitoring UI) provides SVG timelines showing when each invocation started, paused, resumed, and finished across all runners. Family trees show parent-child relationships. The log explorer turns raw logs into clickable cross-references.

5. Switching backends requires rewriting code

Plugin architecture. Core ships with memory and SQLite. Redis, MongoDB, and RabbitMQ install as separate packages. Swap by config, not code.

Quick start

pip install pynenc

from pynenc import Pynenc

app = Pynenc()

@app.task
def add(x: int, y: int) -> int:
    return x + y

result = add(1, 2).result  # 3

pynenc --app=tasks.app runner start

Fine print: this runs in-memory on a single thread. For distributed execution, add a backend: pip install pynenc-redis / pynenc-mongodb / pynenc-rabbitmq

More in the Usage Guide and samples repo

Under the hood

Invocation state machine — Every task call becomes an invocation moving through REGISTERED → PENDING → RUNNING → SUCCESS/FAILED. Transitions are enforced; each change is recorded with timestamps and ownership metadata.

Recovery — Runners send heartbeats. A background atomic service detects dead runners and invocations stuck beyond configured thresholds. Orphaned work is reclaimed under a distributed lock.

Concurrency control — Before reaching a worker, the orchestrator checks the task’s concurrency policy and rejects or queues duplicates at the orchestration layer.

Workflows — Multi-step workflows with result persistence. On replay, completed steps are skipped. Failed workflows resume from the last checkpoint.

Features

🔌 Plugin Architecture

Memory & SQLite built-in. Redis, MongoDB, RabbitMQ as separate packages.

🔄 Auto Orchestration

Waiting tasks are paused, slots freed. Highest-dependency tasks run first. No deadlocks.

🚦 Concurrency Control

Four modes prevent duplicate work before it reaches a worker.

📊 Monitoring (Pynmon)

SVG timelines, family trees, log explorer, runner dashboards, workflow tracking.

⚡ Workflows

Deterministic replay, checkpoint persistence, resume from failure.

🔒 Recovery

Heartbeats + dead runner detection. Orphaned invocations reclaimed automatically.

⏰ Triggers

Cron, events, task-status — composable with AND/OR logic.

🔍 Debuggability

Every state transition recorded. Logs correlate to invocations, runners, tasks.

Plugin system

Plugin	Package	Provides
Redis	`pynenc-redis`	Broker, Orchestrator, State Backend, Trigger
MongoDB	`pynenc-mongodb`	Broker, Orchestrator, State Backend, Trigger
RabbitMQ	`pynenc-rabbitmq`	Broker

from pynenc.builder import PynencBuilder

app = (
    PynencBuilder()
    .app_id("my_app")
    .redis(url="redis://localhost:6379")
    .multi_thread_runner(min_threads=2, max_threads=8)
    .build()
)

Monitoring with Pynmon

Built-in web UI for when scattered container logs aren’t cutting it.

pynenc --app=tasks.app monitor --host 0.0.0.0 --port 8000

Dashboard — component overview, queue depth, invocation counts, runner status
Timeline — SVG visualization of invocation lifetimes across runners
Family Tree — interactive parent-child invocation hierarchies
Log Explorer — paste raw logs, get clickable cross-references and mini-timelines
Runners — heartbeat status, config, hostname, PID, uptime
Workflows — multi-step progress tracking with failure points

Trigger system

trigger = app.trigger.on_success(process_data).run(notify_admin)

scheduled = app.trigger.on_cron("*/30 * * * *").run(process_data, ...)

Status

Beta (v0.1.x). Core systems are functional and tested. API may change between minor versions.

Full Changelog