Pynenc

Pynenc is a Python distributed task system built around the problems other task frameworks leave for you to solve: lost tasks, duplicate work, dependency deadlocks, and failures you can’t debug

100% open source. No strings attached.
No paid cloud edition, no premium tier, no enterprise upsell. Core, plugins, and monitoring are MIT-licensed.

What Pynenc solves

1. Tasks disappear when workers crash

Pynenc tracks every invocation through a strict state machine with ownership semantics and runner heartbeats. Dead runners are detected automatically; orphaned invocations are reclaimed and re-routed

2. Duplicate work runs in parallel

Built-in concurrency control with four modes: DISABLED, TASK (one per task), ARGUMENTS (one per unique args), KEYS (one per arbitrary key). Duplicates are rejected before reaching a worker

3. Dependencies deadlock your workers

Task A waits on B, B waits on C, all workers blocked. Pynenc’s orchestrator pauses waiting tasks to free their slots, then prioritizes by dependency count — the task blocking the most others runs first. Dependency chains resolve without holding threads hostage

4. Failures are impossible to debug

Pynmon (built-in monitoring UI) provides SVG timelines showing when each invocation started, paused, resumed, and finished across all runners. Family trees show parent-child relationships. The log explorer turns raw logs into clickable cross-references.

Pynmon execution timeline showing task execution across multiple runners with status transitions

5. Switching backends requires rewriting code

Plugin architecture. Core ships with memory and SQLite. Redis, MongoDB, and RabbitMQ install as separate packages. Swap by config, not code

Quick start

pip install pynenc

from pynenc import Pynenc

app = Pynenc()

@app.task
def add(x: int, y: int) -> int:
    return x + y

result = add(1, 2).result  # 3

pynenc --app=tasks.app runner start

Fine print: this runs in-memory on a single thread. For distributed execution, add a backend: pip install pynenc-redis / pynenc-mongodb / pynenc-rabbitmq

More in the Usage Guide and samples repo

Under the hood

Invocation state machine — Every task call becomes an invocation moving through REGISTERED → PENDING → RUNNING → SUCCESS/FAILED. Transitions are enforced; each change is recorded with timestamps and ownership metadata.

Recovery — Runners emit heartbeats. A background atomic service detects dead runners and invocations stuck beyond configured thresholds. Orphaned work is reclaimed under a distributed lock and re-queued transparently

Workflows — Multi-step workflows with result persistence. On replay, completed steps are skipped; failed workflows resume from the last checkpoint, not from the start.

Features

🔌 Plugin Architecture

Memory & SQLite built-in. Redis, MongoDB, RabbitMQ as separate packages.

🔄 Auto Orchestration

Waiting tasks are paused, slots freed. Highest-dependency tasks run first. No deadlocks.

🚦 Concurrency Control

Four modes prevent duplicate work before it reaches a worker.

📊 Monitoring (Pynmon)

SVG timelines, family trees, log explorer, runner dashboards, workflow tracking.

⚡ Workflows

Deterministic replay, checkpoint persistence, resume from failure.

🔒 Recovery

Heartbeats + dead runner detection. Orphaned invocations reclaimed automatically.

⏰ Triggers

Cron, events, task-status — composable with AND/OR logic.

🔍 Debuggability

Every state transition recorded. Logs correlate to invocations, runners, tasks.

Plugin system

Plugin	Package	Provides
Redis	`pynenc-redis`	Broker, Orchestrator, State Backend, Trigger
MongoDB	`pynenc-mongodb`	Broker, Orchestrator, State Backend, Trigger
RabbitMQ	`pynenc-rabbitmq`	Broker

from pynenc.builder import PynencBuilder

app = (
    PynencBuilder()
    .app_id("my_app")
    .redis(url="redis://localhost:6379")
    .multi_thread_runner(min_threads=2, max_threads=8)
    .build()
)

Monitoring with Pynmon

A built-in web UI that gives real-time visibility into every invocation, runner, and workflow — no external tooling required.

pynenc --app=tasks.app monitor --host 0.0.0.0 --port 8000

Pynmon dashboard showing application overview, invocation status, component architecture, and configuration

Execution Timeline

See what ran across every runner and worker, at every moment. Status transitions are color-coded — orange for pending, green for completed, blue for running — with connections between parent and child invocations.

Pynmon timeline comparing ThreadRunner, ProcessRunner, PersistentProcessRunner, and MultiThreadRunner

Family Tree

Navigate the full hierarchy of task calls as an interactive graph. Selecting a node cross-highlights it on the timeline, and vice versa.

Pynmon family tree overlaid on timeline with cross-highlighting between graph and execution view

Log Explorer

Paste your log lines and the Log Explorer augments them with full context — parsing runner contexts, invocation IDs, and task references, resolving each to its detail page with a mini-timeline.

Pynmon Log Explorer parsing log lines with augmented context, mini-timeline, and links

Trigger system

Tasks can start automatically based on cron schedules, custom events, or the outcome of other tasks. Triggers compose with AND/OR logic for complex automation flows

trigger = app.trigger.on_success(process_data).run(notify_admin)

scheduled = app.trigger.on_cron("*/30 * * * *").run(process_data, ...)

Get involved

Docs: docs.pynenc.org
Source: github.com/pynenc/pynenc
Issues & feedback: GitHub Issues
Discussions: GitHub Discussions
Changelog: docs.pynenc.org/changelog

v0.1.x — All core features are implemented and running in production. The codebase is young; you may hit rough edges. Bug reports and contributions are very welcome MIT License