A nestable Jobserver with thread-safe futures, callbacks, and type-hinting

I have been hacking on https://github.com/rhysu/jobserver for a long time. Today I put the jobserver package up on PyPi: https://pypi.org/project/jobserver/.

My initial motivation was improving robustness over the process-pooling approaches that I was using back in August 2019. I needed something where my worker tasks could SIGSEGV or SIGPIPE without my entire pool crashing. I also needed extension hooks for a few specific applications, like advisory flocking when a worker was active and being mindful of RAM limits.

It's stayed fun to intermittently hack on. From the tag messages, you can tell I began using LLMs with this project after tag v1.12. Some features, like JobserverExecutor, were simply beyond my personal bandwidth before employing coding agents. I've tried to keep the technical quality at least as high as if I'd been hand-coding.

The timeline's been...

v1.0 — 2020-02-17

First feature-complete release of the jobserver.

v1.1 — 2020-02-22

Added when_done callbacks along with grammar and type-annotation improvements.

v1.2 — 2020-03-03

Introduced per-submission env and preexec_fn support, and adopted black/flake8/mypy tooling.

v1.3 — 2020-03-04

Added grandchild polling, configurable defaults, and a pluggable sleep_fn.

v1.4 — 2020-03-09

Simplified both the public API and the internals.

v1.5 — 2020-03-10

Wrote the README and made assorted touch-ups.

v1.6 — 2020-12-18

Removed the sentinel push/pop at work submission, since __getstate__/__setstate__ are now handled explicitly.

v1.7 — 2021-01-08

Exposed reclaim_resources on the public API.

v1.8 — 2021-02-16

Added support for CPython 3.9.

v1.9 — 2021-03-18

Exposed MinimalQueue and improved test robustness.

v1.10 — 2022-04-06

Added Python 3.10 support and the ability to unset environment variables.

v1.11 — 2024-09-29

Confirmed Python 3.11 and 3.12 work correctly.

v1.12 — 2026-01-01

Confirmed Python 3.13 works correctly.

v1.13 — 2026-03-28

Added JobserverExecutor (a concurrent.futures.Executor wrapper), Jobserver.map(), and signal-based cancellation via Future.done(..., signal=...). Also hoisted submit defaults, named worker processes, fixed re-entrant CallbackRaised double-wrapping, and shipped nine worked examples.

v2.0 — 2026-03-30

Added context-manager support to MinimalQueue and Jobserver, rewrote reclaim_resources() as O(k) using wait(), and added repr(). Fixed a Future.wait() sentinel/waitpid race and added Future.done() as a wait(timeout=0) shorthand.

v2.1 — 2026-03-31

Corrected timing, resource-leak, and type-safety issues found after v2.0, including a py.typed marker, a fixed queue-timeout overrun, and event-driven dispatcher polling. Replaced asserts at public boundaries with explicit TypeError/ValueError.

v2.2 — 2026-04-01

JobserverExecutor can now own a default-constructed Jobserver automatically, making it drop-in, and slot acquisition became O(k) via a persistent selector. Also let preexec_fn return a context manager, added resource/callback warnings, and generalized nesting examples across all start methods.

v2.3 — 2026-06-01

Child tracebacks now survive Future.result() via __cause__, escaping BaseExceptions became diagnosable, and concurrent.futures.Future.cancel() was wired through the executor. Many hardening fixes followed for map() slot reclamation, submit/shutdown races, MinimalQueue, validation, and the public API surface.

v2.4 — 2026-06-01

Fixed nested fan-out under the fork start method via a lazy per-process selector that rebuilds when the PID changes. Also deduplicated pickle-error handling and the env type signature, and added a stdlib-only benchmark harness under draft/.

v3.0 — 2026-06-20

Major API redesign: Jobserver became a thin handle over a shared, reference-counted Resources object, submission controls moved off submit()/map() into sibling handles (revise_env(), replace_preexec(), replace_sleep()), and SubmissionDied was renamed LostResult. Added a __version__, eager timeout validation, a redesigned queue hierarchy, suppressed worker traceback noise, hardened deserialization, and modernized packaging, docs, tests, and examples.