Nested parallelism with pytest-xdist and MPI/OpenMP

[gnikit]

I have a test suite containing N tests, meant to run some long and computationally intensive programs, each test class spins up M processes (the number of processes varies from class to class). I was wondering if there was a way to use xdist and some custom weights to load balance the work in the scheduler. Any thoughts?

MWE

This is a dummy example using MPI.

from subprocess import run


class TestParallelMyApp1:
    CORES = 4

    def test_main(self):
        run(["mpiexec", "-n", str(self.CORES), "python3", "-c", "print('App1')"])


class TestParallelMyApp2:
    CORES = 14

    def test_main(self):
        run(["mpiexec", "-n", str(self.CORES), "python3", "-c", "print('App2')"])


class TestParallelMyApp3:
    CORES = 10

    def test_main(self):
        run(["mpiexec", "-n", str(self.CORES), "python3", "-c", "print('App3')"])

Desired Outcome

On a machine with 14 cores, running pytest -n auto would run:

• TestParallelMyApp1 and TestParallelMyApp3
• and then TestParallelMyApp2

[gnikit]

Is there any idea on this from the dev team? It would be good to know if there are any solutions using pytest-xdist

[RonnyPfannschmidt]

As far as I'm aware none of us has mpi experience

My last contact with mpi was about 17 years ago in a university course

*edit *

And that involved the c variant

[gnikit]

Thanks for taking the time to reply @RonnyPfannschmidt, I appreciate it.

It's not an MPI specific issue. I'm just using pytest to spin up a bunch of expensive processes (outside of the python runtime), where each one uses different cpu resources and the only thing I would like is for pytest-xdist to be aware that test_a is using 2 cores and test_b 4, so in a machine with 4 cores doing pytest-n 2 will not oversubscribe thus increasing the test suites runtime.

You can see that if you have dozens of such tests, calculating the ordering and grouping of the tests a-priori becomes big problem and oversubscribed jobs can cause substantial a slow down.

Basically any way to assign weights to xdist is what I'm looking for.

[RonnyPfannschmidt]

Currently there's no such mechanism

We intend to add support for information passover to the sheduler

However we also lack a sheduler tgats aware of such weights/ costs

Its similar to loadscope but basically loadcost

[gnikit]

I think I might have managed to make a solution to this. I have not yet tested extensively, so I can't make any promises. If it works would you be keen for me to open a PR and merge it in?

[RonnyPfannschmidt]

im curious about the solution but without seeing it i certainly cant promise a merge

we already made the mistake of letting loadgroup modify nodeids - so i want to be very arefull about what debt/solution we bring in