Use `MPIPreferences` to automatically initialise MPI?

[giordano]

While playing with mpi4py earlier this week I realised it automatically initialises MPI at loading time. This can be controlled with the mpi4py.rc object, including the threading setup. I think we can do something similar with an option in MPIPreferences.jl. How does that sound?

[sloede]

I like the idea. However, how would this work/what would happen if multiple dependencies have different threading requests - would it be that the first one to load MPI.jl just "wins" and determines the setting for everyone?

[giordano]

Wouldn't that be a problem also without auto-initialisation? The idea is that you could choose what to do with the Preferences.toml (including specifying the threading setup for that environment), so you kinda need to know what to do anyway.

I'm just floating the idea of this feature since I found it in mpi4py (and in many occasions I'd have preferred MPI to autoinitialise, instead of killing the session with the first MPI.Comm_rank(MPI.COMM_WORLD) because I had forgot to run MPI.Init()), but I haven't thought through all the possibilities 🙂

[sloede]

Wouldn't that be a problem also without auto-initialisation?

Absolutely. But right now it is kind of accepted that it is somewhat undefined behavior, depending on which packages (and dependencies) you are loading and in which order. However, once we introduce something like a preferences-based approach, users would (rightfully, imho) assume that everything now happens deterministically - or at least that there are no silent errors anymore. I was just wondering what would happen in case multiple conflicting Preferences.toml exist for dependencies, and/or if a central Preferences.toml would (silently) overrule that.

Again, if there were a central and well-documented mechanism for auto-initing MPI, I would appreciate it!

[simonbyrne]

A couple of considerations:

• Are there cases where you would load the package without intending to initialize?
  • The only case I can think of at the moment is to call mpiexec: we could move that to a subpackage (or just make it part of MPIPreferences?)
• If you build MPI.jl into a system image, then __init__() is called at Julia load time: this could be an issue if other packages have MPI.jl as a dependency?
• Preferences.toml is a build-time configuration, not a runtime one: i.e. modifying it will trigger recompilation (or necessitate a rebuild of the system image if it is used)

[giordano]

1. good question. As far as I'm concerned, basically every time I load MPI.jl I want to initialise MPI, I'm a bit sick of killing off REPL sessions because I forgot to do that 😄
2. that's a good point, I'm honestly not sure how that's going to work
3. yes, that was kinda on purpose. But if we don't want to force recompilation we could have constant Ref variables in MPIPreferences.jl that users need to change before loading MPI.jl to tweak the behaviour, which should more closely match mpi4py behaviour. The point is that we need to have the option (either a Preferences.jl one or a variable in MPIPreferences.jl, or whatever) in a package different from MPI.jl, since in Julia we don't have separate loading times within a package.

[simonbyrne]

1. good question. As far as I'm concerned, basically every time I load MPI.jl I want to initialise MPI, I'm a bit sick of killing off REPL sessions because I forgot to do that 😄

Could add a Requires hook to your .julia/config/startup.jl?

[simonbyrne]

Okay, I've thought a bit more about this: how about the following:

• we move mpiexec (and mpiexec_wrapper.jl) into MPIPreferences.jl:
  • if you just want access to the MPI launcher, you can just `using MPIPreferences.
• we have an option for disabling auto-MPI.Init() via environment variable (MPI_JULIA_INIT=false?). We can also have an option for selecing the threadlevel (MPI_JULIA_THREADLEVEL).

Thoughts?

[simonbyrne]

• we move mpiexec (and mpiexec_wrapper.jl) into MPIPreferences.jl:

Oh, not sure that we can load packages until after MPIPreferences?

[simonbyrne]

Wow, after trying to debug an issue with this, I have to say I am much more in favor of this.

The fact that it just kills the job without giving a stacktrace is painful.

[simonbyrne]

One example where this will be a problem: on our Slurm cluster, if I get an interactive session via srun --pty bash -l, then call MPI.Init() from within an interactive MPI session, I get an error about my OpenMPI not being built with PMIx support.

[simonbyrne]

Using the default JLL binaries gives even worse errors:

julia> using MPI
[ Info: Precompiling MPI [da04e1cc-30fd-572f-bb4f-1f8673147195]

julia> MPI.Init()
[cli_0]: write_line error; fd=11 buf=:cmd=init pmi_version=1 pmi_subversion=1
:
system msg for write_line failure : Bad file descriptor
[cli_0]: Unable to write to PMI_fd
[cli_0]: write_line error; fd=11 buf=:cmd=get_appnum
:
system msg for write_line failure : Bad file descriptor
Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(60): MPI_Init_thread(argc=(nil), argv=(nil), required=2, provided=0x7ffd7abe14e0) failed
MPII_Init_thread(209)...:
MPID_Init(72)...........:
init_local(135).........: channel initialization failed
init_pg(332)............:
MPIR_pmi_init(111)......: PMI_Get_appnum returned -1
[cli_0]: write_line error; fd=11 buf=:cmd=abort exitcode=740937487
:

[giordano]

One example where this will be a problem: on our Slurm cluster, if I get an interactive session via srun --pty bash -l, then call MPI.Init() from within an interactive MPI session, I get an error about my OpenMPI not being built with PMIx support.

I don't understand this: what's the difference between a Julia process started inside an interactive shell session launched by srun and a normal Julia process on a login node? srun is driving only the bash process, right?

[luraess]

One example where this will be a problem: on our Slurm cluster, if I get an interactive session via srun --pty bash -l, then call MPI.Init() from within an interactive MPI session, I get an error about my OpenMPI not being built with PMIx support.

I don't understand why PMIx issue should be dependent on MPI.jl. On one of our test cluster, I run into these PMIx issues (with MPI.jl v0.19.2). The workaround was to srun --mpi=pmix ... and to export SLURM_MPI_TYPE=pmix.

As a general thought after playing with MPI.jl v0.20 now on various machines; it would be nice to not over-engineer the setup machinery as things get usually much more complicated on clusters and supercomputers than they are on local machines (no internet on compute nodes, missing libs/env on login nodes, etc...). So in general, the more basic and robust workflows should be preferred. Ideally, with workflows that would need as few as possible "interactive setup" steps.