GNU Make Jobserver support?

[xmo-odoo]

I was reading https://blogs.gentoo.org/mgorny/2025/11/30/one-jobserver-to-rule-them-all/ on global / system jobservers (using the gnu make jobserver protocol as it has fairly wide support) and it noted upon xdist missing that.

It's definitely an issue I have observed when i mix workloads (e.g. multiple large test suites run in parallel, plus native compilation on the side), currently I try to eyeball my CPU graph to see how I'm doing and if I have more headroom but it would definitely be useful to be able to use jobservers to allow workers to coordinate.

https://github.com/tommilligan/pytest-jobserver exists, however it notes that it is not able to call into existing jobservers because the xdist protocol does not support sending jobserver info, so I'm assuming this requires updates to xdist itself. It also does not support the "windows jobserver" method.

Further jobserver resources:

• https://make.mad-scientist.net/papers/jobserver-implementation/

• https://www.chiark.greenend.org.uk/doc/make-doc/make.html/Integrating-make.html

  provides a unified view of https://www.gnu.org/software/make/manual/html_node/Job-Slots.html + https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html which are the "official" docs

• https://doc.rust-lang.org/rustc/jobserver.html

• Add a GNU make jobserver implementation to Cargo rust-lang/cargo#4110

Of note: as the essay calls out, while the jobserver protocol calls out for a fifo this is probably something to either not check or extend: in order to mitigate common failure modes, the system-wide jobserver implementations use other file types (regular file via FUSE or character device via CUSE). Make itself does not check the file type.

[RonnyPfannschmidt]

at first glance this needs significant re-factoring in both execnet and xdist

for context - the jobserver protocol at its core is using a pipe with N bytes as a shared semaphore by having participants do blocking reads for the number of slots they want to take and finalizing the usage by writes to the pipe

my recommendation would currently be to use pytest-split + pytest-jobserver as currently we don't have working pipe passover for execnet

my understanding is that unless exencet gains pipe passover in a sensible manner its going to be next to impossible

[RonnyPfannschmidt]

a naive implementation of this would have the managing process consume N tokens to start n subprocesses

based on my current understanding pytest-jobserver is fundamentally broken and the protocol itself is slightly unsafe as well
this comes from the understanding that any consumer of tokens that fails to return them looses them

tools like make can solve this relatively safe by consuming tokens in processes that cant fail, the running the work
bot running a crashing helper under pytest-jobserver would just destroy the world

by now id go as fas as recommending to create a wrapper that safely consumes N tokens, then runs pytest -n TOKENS, then cleans up

[mgorny]

Well, not necessarily. The hack I suggested in #1282 doesn't really require passing fds. Every pytest-xdist process can open the jobserver named pipe separately to read its token.

That said, I'll look at pytest-jobserver — it didn't occur to me that it could be implemented separately from pytest-xdist.

[RonnyPfannschmidt]

@mgorny a pytest worker that is under the possibility of crashing without returning a token is the wrong place to obtain/return a token

a sidechannel to run the request/response chain in the coordinator could be used to run the token ownership in a place thats not going up in fire when a worker crashes by using a execnet channel and/or expanding the messages the sheduler uses

[xmo-odoo]

my understanding is that unless exencet gains pipe passover in a sensible manner its going to be next to impossible

Is pipe passover really necessary? Is there no way to send the jobserver spec to the clients and for the clients to connect to the jobserver on their own? This might not be compatible with the legacy fd-based POSIX protocol (depending how the workers get run) but I wouldn't think that's a big issue.

a naive implementation of this would have the managing process consume N tokens to start n subprocesses

That has the issue that it locks all these tokens to pytest (or more generally the first orchestrator process with that strategy) until the end of the run, rather than allow jobs to yield to one another. Basically a scheduling issues. Not to mention if the test suite has unbalanced loads it will keep all the tokens locked even though e.g. only one worker is still doing something.

based on my current understanding pytest-jobserver is fundamentally broken and the protocol itself is slightly unsafe as well this comes from the understanding that any consumer of tokens that fails to return them looses them

That is correct. From what I gathered it's generally considered a minor annoyance for transient jobservers (e.g. make) and it is my understanding that it's how make behaves: the sub-makes acquire a token when they find something to do.

But it is why all the "global" jobservers cited in the gentoo essay use virtual files in FUSE/CUSE: it lets them track the state of token borrowers and recover tokens. For that reason I'd think they would prefer the actual user of the token (the worker) to do the borrowing.

[mgorny]

Okay, I took pytest-jobserver for a run and I think it's good enough. Bad news is, it seems unmaintained. Good news is, is still works with pytest-9 and all the bugs are kinda minor, so I guess fixing or forking it is still less effort than changing pytest-xdist. I was worried that the per-test approach would cause overhead, but I've tried it on numpy and there was no noticeable overhead.

[RonnyPfannschmidt]

oh - extra fun - linux has event_fd in semaphore mode as a better abstraction for this, but it shares the same issue - death of the token owner means loss of the token - gnu make avoids it for portability

based on my current understanding its "wrong" to obtain/manage tokens inside of pytests runtestloop as crashes loose tokens

the jobserver protocol is so broken you basically had to invent a own filesystem to work around its issues

right now im not willing to spend bandwidth on it in pytest/pytest-xdist but im happy to provide input on making it work

[xmo-odoo]

Okay, I took pytest-jobserver for a run and I think it's good enough. Bad news is, it seems unmaintained. Good news is, is still works with pytest-9 and all the bugs are kinda minor, so I guess fixing or forking it is still less effort than changing pytest-xdist. I was worried that the per-test approach would cause overhead, but I've tried it on numpy and there was no noticeable overhead.

I assume that's using pytest-split?

[mgorny]

I assume that's using pytest-split?

No, it's with pytest-xdist. pytest-jobserver explicitly supports being combined with it.