SLURM: libDIP.so: cannot open shared object file: No such file or directory

[AndreZeug]

Hi, I managed to install DIPimage 3.5.1 for MATLAB on a HPC Cluster

I can use DIPimage functionality within MATLAB since I added the path to the libDIP.so to the system path
either within MATLAB by putting setenv('PATH',['/project/cnp/DIPlib/lib',':',getenv('PATH')]); into the startup.m
or by adding the path to the system path.

However, when I run DIPimage commands in a MATLAB batch job using the Parallel Computing Toolbox, I receive the following ERROR
2024-12-13 01:18:25: 1105442_createMovieXYT(results/images/202_Ch2_FRaw) CreateStart 01:18:06 (12.708 s) finished, JobStart 01:18:25 (00:00:03.063 hour with ERROR: Invalid MEX-file '/tmp/ca72-4747-af6e-954e/a/tpf9fd06bd_15cf_4937_9d8e_962248c5dd7e/@dip_image/private/dip_projection.mexa64': libDIP.so: cannot open shared object file: No such file or directory )

The command I use in the batch job is probably something simple as int99 = percentile(dip_image(d),99.99)
If I run the same function inside MATLAB, but not in a batch job, it works.

probably I did define the system path not correctly?
Or, is it some other problem?

System information:

ver
---------------------------------------------------------------------------------------------------------------
MATLAB Version: 9.14.0.2337262 (R2023a) Update 5
MATLAB License Number: 40567690
Operating System: Linux 5.14.0-427.31.1.el9_4.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Aug 14 16:15:25 UTC 2024 x86_64
Java Version: Java 1.8.0_202-b08 with Oracle Corporation Java HotSpot(TM) 64-Bit Server VM mixed mode
---------------------------------------------------------------------------------------------------------------
MATLAB                                                Version 9.14        (R2023a)
DIPimage toolbox for quantitative image analysis      Version 3.5.1               

[AndreZeug]

hmm, maybe I am wrong and it is not libDIP.so which is not found, but .../@dip_image/private/dip_projection.mexa64?

[crisluengo]

On SLURM, jobs are started on other machines that do not necessarily inherit the environment of the machine you start the batch job on. When you “add the path to the system path”, do you do this in your .profile file? I think that file should be used on all compute nodes. But check with your cluster’s documentation and/or admins.

I also see that /tmp/ca72-4747-af6e-954e/a/tpf9fd06bd_15cf_4937_9d8e_962248c5dd7e/ is the DIPimage path on the compute node. Did you install it to that path yourself, or is it somehow being copied there for access of the compute nodes? I’m asking because it’s a weird path name… If it’s being copied there, we need to ensure that the DIPlib library is copied as well.

Finally, one cause could be that the compute nodes don’t have the C++ runtime library installed. If it exists on the node where you start the batch job, it likely exists on all machines, but it’s worth double-checking that.

[AndreZeug]

To be more precise,

• I installed DIPimage 3.5.1 following your doc in my case into the folder /project/cnp/DIPlib which contains all the standard folders like /project/cnp/DIPlib/lib containing libDIP.so and the folder /project/cnp/DIPlib/share containing all DIPimage *.m files
• I did put the path to libDIP.so by putting setenv('PATH',['/project/cnp/DIPlib/lib',':',getenv('PATH')]); into the startup.m of MATLAB which make DIPimage working within ths instance of MATLAB
• alternatively I added PATH=/project/cnp/DIPlib/lib:$PATH to my .profile, but maybe I did something wrong, need to clarify with my HPC admins (on Monday)
• I was told today to better use LD_LIBRARY_PATH rather than PATH
• for tests I did also copied libDIP.so to some folder, which is surely part of the systems PATH

All of it did not solved the issue

the MATLAB batch process runs the task to some tmp folder, each task into a unique one, my example /tmp/ca72-4747-af6e-954e/a/tpf9fd06bd_15cf_4937_9d8e_962248c5dd7e/ (<-this is NOT the path to DIPimage!)

The question is, if the MATLAB batch process knows where to find all DIPimage functions, which are in the folder /project/cnp/DIPlib/share since it should know the PATH to libDIP.so at this point

What I do not understand is, what is the difference between DIPimage 2.9 and 3.5, since when using DIPimage 2.9 and dip_initialze in MATLAB startup.m I have no problem and MATLAB batch processes run without problem

Unfortunately I do not remember, if I did something special 2 years ago to make DIPimage 2.9 run smoothly

What I will do next is

• dropping a "service request" to MathWorks to better understand the MATLAB batch process
• asking my HPC admins, "unfortunately" they are on vacation (till next year), don't know, to which extent I can bother them 🙈

Any Idea?

[crisluengo]

In your error message, it refers to /tmp/ca72-4747-af6e-954e/a/tpf9fd06bd_15cf_4937_9d8e_962248c5dd7e/@dip_image/private/dip_projection.mexa64

If you installed DIPlib to /project/cnp/DIPlib/, then DIPimage would be in /project/cnp/DIPlib/share/DIPimage/, not in /tmp/ca72-4747-af6e-954e/a/tpf9fd06bd_15cf_4937_9d8e_962248c5dd7e/.

So, it is clear that the toolbox is being copied over. But it is copying over only /project/cnp/DIPlib/share/DIPimage/, not the libraries under /project/cnp/DIPlib/lib/. The toolbox won't work without those.

I don't know if there's a way to tell your system to copy over the whole of /project/cnp/DIPlib?

Are you starting a single job with SLURM, and then have MATLAB build a parallel pool across compute nodes and distributing the work that way, or are you starting many jobs using SLURM, each running on a single node? That is, is it SLURM or MATLAB that is copying over the DIPimage toolbox to a temporary directory on the nodes?

---

On Windows you'd set the PATH environment variable to the DIPlib lib directory. There, PATH is used both to find executables and DLLs.

On Linux, PATH is only used for executables, not for shared objects (the equivalent of DLLs). You'd set the LD_LIBRARY_PATH to point to a library directory. But the DIPimage MEX-files have the relative path to the DIPlib libraries built in. Normally you wouldn't need to set any paths other than the MATLAB search path to point to the DIPimage directory.

The DIPimage installation instructions say to set the PATH environment variable when installing on Windows, on Linux it doesn't have a similar step.

[AndreZeug]

Dear Cris, I am close to understand what happens 🤗
To find out what is going on I created 2 MATLAB functions:
MATLAB SLURM function testMATLABbatchJob.m

function testMATLABbatchJob
fprintf('Start test job\n');
%% ParforProgress
par.ParforProgress2 = false;
par = checkParfor(par);
%% create some dummy variable
d = rand(100,100,100);
%% present path
myPWD = pwd;
fprintf('PWD: %s\n', pwd);
%% test for DIPimage
onPath  = contains(path,'/project/cnp/DIPlib/share/DIPimage', 'IgnoreCase', ispc);
if ~onPath
  warning('DIPimage not on PATH!')
end
%% test2 for DIPimage
which percentile
int99 = percentile(dip_image(d),99.99);
fprintf('int99: %.8f\n', int99);
%% start batch job
tBatch = now;
job = batch('MATLABbatchJob',0,{d,pwd});
job.Tag = sprintf('%s_MATLABbatchJob  CreateStart %s (%s s)', par.PID,datestr(tBatch(1),'HH:MM:SS'),datestr(now-tBatch,'SS.FFF'));
fprintf('MATLABbatchJob started\n');

MATLAB batch function MATLABbatchJob.m

function MATLABbatchJob(d, calling_pwd)
myLogID = fopen(sprintf('MATLABbatchJob_out.log'),'a+');
fprintf(myLogID,'\n%s Start\n',datestr(now,'yyyy-mm-dd HH:MM:SS'));
%% present path
myPWD = pwd;
fprintf(myLogID,'PWD: %s (called from %s)\n', pwd, calling_pwd);
system(sprintf('ls -al > %s/batchDir_%s_dirList.txt', calling_pwd, strrep(pwd,'/','_')))
%fprintf(myLogID,system('ls -al'))
%% test for DIPimage
onPath  = contains(path,'/project/cnp/DIPlib/share/DIPimage', 'IgnoreCase', ispc);
if ~onPath
  fprintf(myLogID,'DIPimage not on PATH!\n');
  error('DIPimage not on PATH!')
else
  fprintf(myLogID,'DIPimage PATH found!\n');
  warning('DIPimage PATH found!')
end
%% test2 for DIPimage
fprintf(myLogID,'path percentile: %s\n', which('percentile'));
addpath('/project/cnp/DIPlib/share/DIPimage')
fprintf(myLogID,'path percentile: %s\n', which('percentile'));
fprintf(myLogID,'path libDIP.so : %s\n', which('libDIP.so'));
setenv('PATH',['/project/cnp/DIPlib/lib',':',getenv('PATH')]);
int99 = percentile(dip_image(d),99.99);
fprintf(myLogID,'int99: %.8f\n', int99);
%% close log file
fclose(myLogID);

which try to use DIPimage and create two log files. MATLABbatchJob_out.log

PWD: /mnt/cifs/cnpbigdata/zeugand/Mic/SD/2024-12-11_DarkImage (called from /mnt/cifs/cnpbigdata/zeugand/Mic/SD/2024-12-11_DarkImage)
DIPimage PATH found!

2024-12-14 15:10:00 Start
PWD: /mnt/cifs/cnpbigdata/zeugand/Mic/SD/2024-12-11_DarkImage (called from /mnt/cifs/cnpbigdata/zeugand/Mic/SD/2024-12-11_DarkImage)
DIPimage PATH found!

2024-12-14 15:13:04 Start
PWD: /mnt/cifs/cnpbigdata/zeugand/Mic/SD/2024-12-11_DarkImage (called from /mnt/cifs/cnpbigdata/zeugand/Mic/SD/2024-12-11_DarkImage)
DIPimage PATH found!

2024-12-14 15:17:31 Start
PWD: /mnt/cifs/cnpbigdata/zeugand/Mic/SD/2024-12-11_DarkImage (called from /mnt/cifs/cnpbigdata/zeugand/Mic/SD/2024-12-11_DarkImage)
DIPimage PATH found!
path: /tmp/cce7-6e18-6814-c0f1/a/tp516ec123_4ef1_4de6_88f9_ba85059da91b/@dip_image/percentile.m

2024-12-14 15:21:58 Start
PWD: /mnt/cifs/cnpbigdata/zeugand/Mic/SD/2024-12-11_DarkImage (called from /mnt/cifs/cnpbigdata/zeugand/Mic/SD/2024-12-11_DarkImage)
DIPimage PATH found!
path: /tmp/1819-989c-c07f-e5eb/a/tp4ebf9377_a5e4_4933_b48e_db5bd26854cb/@dip_image/percentile.m

2024-12-14 15:36:28 Start
PWD: /mnt/cifs/cnpbigdata/zeugand/Mic/SD/2024-12-11_DarkImage (called from /mnt/cifs/cnpbigdata/zeugand/Mic/SD/2024-12-11_DarkImage)
DIPimage PATH found!
path: /tmp/bb2d-7701-fb0c-5890/a/tp7ffaf28b_6da1_4e79_9244_5c6328a2c595/@dip_image/percentile.m

2024-12-14 15:38:07 Start
PWD: /mnt/cifs/cnpbigdata/zeugand/Mic/SD/2024-12-11_DarkImage (called from /mnt/cifs/cnpbigdata/zeugand/Mic/SD/2024-12-11_DarkImage)
DIPimage PATH found!
path percentile: /tmp/deef-92a4-777c-e6f1/a/tp64c94d8c_3f14_4c29_af7a_0aad195d6219/@dip_image/percentile.m
path libDIP.so   : /hpc/project/cnp/DIPlib/lib/libDIP.so

2024-12-14 15:49:14 Start
PWD: /mnt/cifs/cnpbigdata/zeugand/Mic/SD/2024-12-11_DarkImage (called from /mnt/cifs/cnpbigdata/zeugand/Mic/SD/2024-12-11_DarkImage)
DIPimage PATH found!
path percentile: /tmp/17ee-295d-e1ff-1a2d/a/tpcecedb82_45ca_47e3_b20a_71e2f4b85c97/@dip_image/percentile.m
path percentile: /hpc/project/cnp/DIPlib/share/DIPimage/@dip_image/percentile.m
path libDIP.so   : /hpc/project/cnp/DIPlib/lib/libDIP.so
int99: 0.99988528

2024-12-14 17:16:26 Start
PWD: /mnt/cifs/cnpbigdata/zeugand/Mic/SD/2024-12-11_DarkImage (called from /mnt/cifs/cnpbigdata/zeugand/Mic/SD/2024-12-11_DarkImage)

2024-12-14 17:25:28 Start
PWD: /mnt/cifs/cnpbigdata/zeugand/Mic/SD/2024-12-11_DarkImage (called from /mnt/cifs/cnpbigdata/zeugand/Mic/SD/2024-12-11_DarkImage)
DIPimage PATH found!
path percentile: /tmp/1a5a-9215-cea9-5b21/a/tp0bb10ed0_2621_4579_90b7_b492b76b968d/@dip_image/percentile.m
path percentile: /hpc/project/cnp/DIPlib/share/DIPimage/@dip_image/percentile.m
path libDIP.so : /hpc/project/cnp/DIPlib/lib/libDIP.so
int99: 0.99988528

2024-12-14 17:58:17 Start
PWD: /mnt/cifs/cnpbigdata/zeugand/Mic/SD/2024-12-11_DarkImage (called from /mnt/cifs/cnpbigdata/zeugand/Mic/SD/2024-12-11_DarkImage)
DIPimage PATH found!
path percentile: /tmp/851e-3e52-a0ec-fc94/a/tpa56e5913_1c51_496d_9c58_28f45dc82943/@dip_image/percentile.m
path percentile: /hpc/project/cnp/DIPlib/share/DIPimage/@dip_image/percentile.m
path libDIP.so : /hpc/project/cnp/DIPlib/lib/libDIP.so
int99: 0.99988528

2024-12-14 18:06:51 Start
PWD: /mnt/cifs/cnpbigdata/zeugand/Mic/SD/2024-12-11_DarkImage (called from /mnt/cifs/cnpbigdata/zeugand/Mic/SD/2024-12-11_DarkImage)
DIPimage PATH found!
path percentile: /tmp/cec4-5d3d-0b95-04fa/a/tp7d9f1866_fc64_4b9e_8d05_1c43a14183c5/@dip_image/percentile.m
path percentile: /tmp/cec4-5d3d-0b95-04fa/a/tp7d9f1866_fc64_4b9e_8d05_1c43a14183c5/@dip_image/percentile.m
path libDIP.so : /hpc/project/cnp/DIPlib/lib/libDIP.so

and some log of the batch processes parallelJobs.log

2024-12-14 14:47:48: 928100_MATLABbatchJob  CreateStart 14:47:29 (03.635 s) finished,  JobStart 14:47:48 (00:00:02.798 hour with WARNINGS: DIPimage PATH found!  )
2024-12-14 14:54:41: 4071066_MATLABbatchJob  CreateStart 14:54:18 (08.372 s) finished,  JobStart 14:54:41 (00:00:03.064 hour with ERROR: Invalid MEX-file '/tmp/6df2-66f2-96f9-1597/a/tp962b4a19_a176_4175_b483_d70e7537c7b1/@dip_image/private/dip_projection.mexa64': libDIP.so: cannot open shared object file: No such file or directory  with WARNINGS: DIPimage PATH found!  )
2024-12-14 14:58:24: 4079731_MATLABbatchJob  CreateStart 14:57:39 (18.006 s) finished,  JobStart 14:58:24 (00:00:03.231 hour with ERROR: Invalid MEX-file '/tmp/9eeb-21db-be50-930c/a/tp2867154e_0bf8_4ac5_b902_dcee3e03c7a3/@dip_image/private/dip_projection.mexa64': libDIP.so: cannot open shared object file: No such file or directory  with WARNINGS: DIPimage PATH found!  )
2024-12-14 15:09:58: 4127137_MATLABbatchJob  CreateStart 15:09:34 (07.908 s) finished,  JobStart 15:09:58 (00:00:02.772 hour with ERROR: Invalid MEX-file '/tmp/290b-c552-eab7-df73/a/tp0f71678d_3307_4bdd_a7d5_c2328a0c6958/@dip_image/private/dip_projection.mexa64': libDIP.so: cannot open shared object file: No such file or directory  with WARNINGS: DIPimage PATH found!  )
2024-12-14 15:13:02: 4136799_MATLABbatchJob  CreateStart 15:12:39 (08.976 s) finished,  JobStart 15:13:02 (00:00:02.939 hour with ERROR: Invalid MEX-file '/tmp/1c59-20d1-2607-bce3/a/tpa99d8081_e9b4_4c8c_9bbd_60496bed2aff/@dip_image/private/dip_projection.mexa64': libDIP.so: cannot open shared object file: No such file or directory  with WARNINGS: DIPimage PATH found!  )
2024-12-14 15:17:29: 4147918_MATLABbatchJob  CreateStart 15:17:07 (08.023 s) finished,  JobStart 15:17:29 (00:00:03.023 hour with ERROR: Invalid MEX-file '/tmp/cce7-6e18-6814-c0f1/a/tp516ec123_4ef1_4de6_88f9_ba85059da91b/@dip_image/private/dip_projection.mexa64': libDIP.so: cannot open shared object file: No such file or directory  with WARNINGS: DIPimage PATH found!  )
2024-12-14 15:21:56: 4158389_MATLABbatchJob  CreateStart 15:21:31 (09.436 s) finished,  JobStart 15:21:56 (00:00:02.698 hour with ERROR: Invalid MEX-file '/tmp/1819-989c-c07f-e5eb/a/tp4ebf9377_a5e4_4933_b48e_db5bd26854cb/@dip_image/private/dip_projection.mexa64': libDIP.so: cannot open shared object file: No such file or directory  with WARNINGS: DIPimage PATH found!  )
2024-12-14 15:36:26: 1644816_MATLABbatchJob  CreateStart 15:36:09 (06.844 s) finished,  JobStart 15:36:26 (00:00:02.112 hour with ERROR: Invalid MEX-file '/tmp/bb2d-7701-fb0c-5890/a/tp7ffaf28b_6da1_4e79_9244_5c6328a2c595/@dip_image/private/dip_projection.mexa64': libDIP.so: cannot open shared object file: No such file or directory  with WARNINGS: DIPimage PATH found!  )
2024-12-14 15:38:05: 1650532_MATLABbatchJob  CreateStart 15:37:49 (06.787 s) finished,  JobStart 15:38:05 (00:00:02.369 hour with ERROR: Invalid MEX-file '/tmp/deef-92a4-777c-e6f1/a/tp64c94d8c_3f14_4c29_af7a_0aad195d6219/@dip_image/private/dip_projection.mexa64': libDIP.so: cannot open shared object file: No such file or directory  with WARNINGS: DIPimage PATH found!  )
2024-12-14 15:49:11: 24761_MATLABbatchJob  CreateStart 15:48:46 (09.939 s) finished,  JobStart 15:49:11 (00:00:03.230 hour with WARNINGS: DIPimage PATH found!  )
2024-12-14 17:16:23: 229245_MATLABbatchJob  CreateStart 17:15:58 (09.614 s) finished,  JobStart 17:16:23 (00:00:03.266 hour with ERROR: Invalid format.  )
2024-12-14 17:25:25: 250176_MATLABbatchJob  CreateStart 17:25:03 (07.973 s) finished,  JobStart 17:25:25 (00:00:03.458 hour with WARNINGS: DIPimage PATH found!  )
2024-12-14 17:58:15: 318975_MATLABbatchJob  CreateStart 17:57:49 (09.535 s) finished,  JobStart 17:58:15 (00:00:03.014 hour with WARNINGS: DIPimage PATH found!  )
2024-12-14 18:06:49: 338343_MATLABbatchJob  CreateStart 18:06:23 (09.019 s) finished,  JobStart 18:06:49 (00:00:03.199 hour with ERROR: Invalid MEX-file '/tmp/cec4-5d3d-0b95-04fa/a/tp7d9f1866_fc64_4b9e_8d05_1c43a14183c5/@dip_image/private/dip_projection.mexa64': libDIP.so: cannot open shared object file: No such file or directory  with WARNINGS: DIPimage PATH found!  )

The important key part is in MATLABbatchJob.m right after %% test2 for DIPimage
Only when I enable the command addpath('/project/cnp/DIPlib... and with this overwrite or shadow some intermediate path or softlink of the batch function I can use DIPimage unless I can always find the DIPimage functions, even if in a strange temporary directory.
The command setenv('PATH',['/project/cnp/DIPlib/lib',':',getenv('PATH')]); has no impact!
So it is not enough to (re)set the path to the library libDIP.so.

I further tried to set a softlink to libDIP.so by
zeugand@leinelogin01:/project/cnp/DIPlib/share/DIPimage » ln -s /project/cnp/DIPlib/lib/libDIP.so libDIP.so
But the last call 2024-12-14 18:06:51 shows, the softlink has no effect ( addpath('/project/cnp/DIPlib... was commented out!)

You explained "...DIPimage MEX-files have the relative path to the DIPlib libraries built in...".
So, a solution could be that I get the chance to provide DIPimage MEX-files the absolute path to the DIPlib libraries manually, or while compilation.

Do you see any alternative?

(sorry for the long text blocks, hope this is not too confusing 🙈)

[crisluengo]

Interesting indeed! It looks like the tmp dir is being added to the path, you manually adding the original DIPimage path to the top shadows the one in tmp.

So, adding this path at the top of your script solves the problem?

Setting PATH from within MATLAB has no effect on Linux, that only works on Windows. On Linux it would have to be LD_LIBRARY_PATH, and it would have to be set before starting MATLAB. For example by setting it in .profile. But I’d rather find a solution that doesn’t require the user to set environment variables, it’s ugly and annoying.

One thing we could do is copy the DIPlib .so files into DIPimage/private, and have the MEX-files link to those copies. This needs to be done during compilation, in the CMake script. I’m not entirely sure how to do this yet. But it seems that it would be the most robust way of deploying DIPimage, having everything contained in the same directory. I also need to think about what else this could break.

[AndreZeug]

YES , adding this path at the top of my script(s) solves the problem!
but I have many 😨
Indeed I tried to set LD_LIBRARY_PATH but without success, maybe I did it wrong, or, maybe before calling MATLAB is not enough, but it would also require before calling the MATLAB batch job?
Copying or linking the DIPlib.so files into DIPimage/private, and have the MEX-files link to those copies should solve the problem - as far as I understand the issue.
How was it solved within DIPimage 2.9? This I can use without any problem.

[AndreZeug]

UPDATE
adding export LD_LIBRARY_PATH="/project/cnp/DIPlib/lib:$LD_LIBRARY_PATH" to my ~/.profile does not work.
BUT if I place export LD_LIBRARY_PATH="/project/cnp/DIPlib/lib:$LD_LIBRARY_PATH" to the calling SLURM script testMATLABbatchJob_slurm.sh it works.

So I might be able to solve the problem "by myself" by asking my HPC admins how to define LD_LIBRARY_PATH correctly. 😅
copying export LD... to all my *_slurm.sh files would be a workaround.

[crisluengo]

Setting LD_LIBRARY_PATH before starting MATLAB should do the trick, but it has to be set on each of the nodes before the workers are started too. I think .profile should do that, but I’m not sure. [Your new post suggests it doesn’t.]

I also remember some time ago running into an issue where the MATLAB startup script clobbered the LD_LIBRARY_PATH, but that was a long time ago, I haven’t had a need to set it in many years, so maybe I’m misremembering that issue.

---

If setting the LD_LIBRARY_PATH in your SLURM scripts is easier than adding the path to the MATLAB scripts your run, then you should do that for now. I hope to find a better solution though.

I want to find out, for example, why MATLAB copies the toolbox to a temp directory. There's no need to do so. Maybe there’s a way to configure MATLAB to not do that. Maybe it is because it is in your user directory, and it assumes that it is not available on the compute nodes?

Another solution might be to install DIPimage system-wide? (I.e. have the admins install it.) Also a shitty solution though. :(

[crisluengo]

Where do you add the DIPimage directory to your MATLAB path? Is it in startup.m, or did you save a pathdef.m?

[AndreZeug]

LD_LIBRARY_PATH: With the help of my admins I will surely find out what I did wrong in my ~/.profile

MATLAB: oh, batch jobs are rather complicate, it stores each job in a folder before executing, in my case
project/cnp/zeug/.matlab/local_cluster_jobs/R2023a, later it is probably executed somewhere (in /tmp)
Idea behind, batch jobs would be re-executed when restarting MATLAB and the Jobs are not (marked as) finished

side-note: on SLURM systems, when running many instances of MATLAB in parallel, batch jobs get overwritten and deleted before finishing which is a disaster 🙈 , therefore I implemented a Workaround to define a separate subfolder for each instance of MATLAB for the batch jobs according to MATLABs ProcessID project/cnp/zeug/.matlab/local_cluster_jobs/R2022a/PID_277265,
99% the cases this workaround run smooth, but sometimes PIDs repeat, sometimes jobs get not deleted
Leftover look like this:

 /project/cnp/zeug/.matlab/local_cluster_jobs/R2022a/PID_873831/Job6/Task1.common.mat
/project/cnp/zeug/.matlab/local_cluster_jobs/R2022a/PID_873831/Job6/Task1.log
/project/cnp/zeug/.matlab/local_cluster_jobs/R2022a/PID_873831/Job6/Task1.diary.txt
/project/cnp/zeug/.matlab/local_cluster_jobs/R2022a/PID_873831/Job6/Task1.in.mat
/project/cnp/zeug/.matlab/local_cluster_jobs/R2022a/PID_873831/Job6/Task1.out.mat
/project/cnp/zeug/.matlab/local_cluster_jobs/R2022a/PID_873831/Job6/Task1.state.mat

I need to delete these folders on a regular basis

DIPimage directory: startup.m but let my try pathdef.m I forgot to test

[AndreZeug]

DIPimage directory: startup.m and pathdef.m, both work when I define export LD... in my *_slurm.sh but do not work when not - as expected

[crisluengo]

Thanks for checking!

By the way, as an aside, you should probably use either batch in MATLAB or start multiple single-node jobs with SLURM, but not so both. I’m guessing also that batch assumes that /tmp is a local drive, only visible to the one compute node. In your case it seems to be a system-wide directory, which is why different MATLAB instances will overwrite each other’s temporary files.

It looks like batch sends all local files (home directory) and data to each of the compute nodes, which store the files in their local temp directory, and at the end send the resulting data back to the main MATLAB instance. It assumes the compute nodes don’t have access to your home directory, which is weird? Still reading up on this.

[AndreZeug]

either/or: good point, but my tasks are way more complex since I shifted my complete processing pipeline to HPC cluster processing, happy to share my strategy

SOLUTION
I placed

# Needed for MATLAB with DIPimage 3.x (at least till DIPimage ver 3.5.1)
if [[ ":$LD_LIBRARY_PATH:" != *":/project/cnp/DIPlib/lib:"* ]]; then
 export LD_LIBRARY_PATH="/project/cnp/DIPlib/lib:$LD_LIBRARY_PATH"
fi

to all my "profile start scripts" such as ~/.profile, ~/.bash_profile, ~/.bashrc, ~/.zshrc ...
This is way less then all my *.sh or all my *.m files, as discussed above

🙈 I know you do not like this and try to avoid it, but for now it works and I am happy 😅