Cluster Cosmology

{{toc}}

*Please read through this entire wikipage so everyone can make efficient use of this cluster*

h1. Hardware overview

You access the Euclid cluster through cosmogw.kosmo.physik.uni-muenchen.de

* cosmogw is a gateway machines and should *not* be used for computing

* there are 21 compute nodes named euclides01--euclides11 and euclides12--euclides21;

* euclides01-euclides11 have each 32 logical CPUs and 64GB of RAM;

* euclides12-euclides21 have each 56 logical CPUs and 128GB of RAM;

h1. How to run jobs on the euclides nodes (using Slurm)

Use slurm to submit jobs or login to the euclides nodes (euclides01-21).

h2. Control node cosmogw

The machine cosmogw is the login node and submit nodes for the slurm queue, so please do not use them as a simple compute nodes - it's hardware is different from the nodes. It hosts our file server and other services that are important to us.

You should use cosmogw to:

* transfer files;

* develop your code;

* compile your code;

* submit jobs to the nodes via the slurm queues;

If you need to debug and would like to login to a node, please start an interactive job to one of the nodes using slurm. For instructions see below.

h2. euclides nodes

Job submission to the euclides nodes is handled by the slurm jobmanager (see http://slurm.schedmd.com and https://computing.llnl.gov/linux/slurm/). 

*Important: In order to run jobs, you need to be added to the slurm accounting system - please contact the admin*

All slurm commands listed below have very helpful man pages (e.g. 'man slurm', 'man squeue', ...). 

If you are already familiar with another jobmanager the following information may be helpful to you http://slurm.schedmd.com/rosetta.pdf‎.

h3. Scheduling of Jobs

At this point there are four queues, called partitions in slurm:

* on cosmogw:

** *normal* which is the default partition your jobs will be sent to if you do not specify it otherwise. At this point there is a time limit of

four days; this queue comprises the computing nodes euclides01-18;

** the *lowpri* partition also comprises the computing nodes euclides01-18; it is a so called preempty queue, allowing more resources for the users; however jobs are re-queued (canceled and re-scheduled) if the resources are demanded on the normal queue;

** *eucliddevel* which is intended for software development, especially if he 'normal' is full; this queue comprises the computing nodes euclides19-21; people from the Euclid group have an account on this queue; each user is allowed to use up to 56 cpus;

** *cosmodevel* which is intended for software development, especially if he 'normal' is full; this queue comprises the computing nodes euclides19-21; people from the cosmology group have an account on this queue; each user is allowed to use up to 4 cpus; note that this queue is preempt, meaning that the users of the queue eucliddevel precedence;

The default memory per core used is 2GB, if you need more or less, please specify with the --mem or --mem-per-cpu option.

We have also set up a scheduler that goes beyond the first come first serve - some jobs will be favoured over others depending

on how much you or your group have been using euclides in the past 2 weeks, how long the job has been queued and how much

resources it will consume.

Job scheduling is a complex issue and we can and have to adjust to the users need whenever possible. Please feel free to speak out if

there is something that can be improved without creating an unfair disadvantage for other users.

You can run interactive jobs on all partitions.

h3. Running an interactive job with slurm (a.k.a. logging in)

To run an interactive job with slurm in the default partition, use

<pre>

srun -u --pty bash

</pre>

If you want to use tcsh use

<pre>

srun -u --pty tcsh

</pre>

If you want to use a larger memory per job do

<pre>

srun -u --mem-per-cpu=8000 --pty tcsh

</pre>

In case you want to open x11 applications, use the --x11=first option, e.g.

<pre>

srun --x11=first -u --pty  bash

</pre>

Opening an interactive session on one of the development partitions is done with:

<pre>

srun --account=<euclid_dev/cosmo_dev> --partition=<eucliddevel/cosmodevel> --x11=first -u --pty  bash

</pre>

h3. limited ssh access

If you have an active job (batch or interactive), you can login to the node the job is running on. Your ssh session will be killed if the job terminates. Your ssh session will be restricted to the same resources as your job (so you cannot accidentally bypass the job scheduler and harm other user's jobs).

h3. Running a simple one core batch job with slurm using the default partition

* To see what queues are available to you (called partitions in slurm), run:

<pre>

sinfo

</pre>

* To run slurm, create a myjob.slurm containing the following information:

<pre>

#!/bin/bash

#SBATCH --output=slurm.out

#SBATCH --error=slurm.err

#SBATCH --mail-user <put your email address here>

#SBATCH --mail-type=BEGIN

#SBATCH -p normal

#SBATCH --ntasks=1

/bin/hostname

</pre>

* To submit a batch job use:

<pre>

sbatch myjob.slurm

</pre>

* To see the status of you job, use 

<pre>

squeue

</pre>

* To kill a job use:

<pre>

scancel <jobid>

</pre> the <jobid> you can get from using squeue.

* For some more information on your job use

<pre>

scontrol show job <jobid>

</pre>the <jobid> you can get from using squeue.

h3. Running a simple once core batch job with slurm using the lowpri partition

Change the partition to lowpri and add the appropriate account depending if you're part of

the euclid or cosmology group.

<pre>

#!/bin/bash

#SBATCH --output=slurm.out

#SBATCH --error=slurm.err

#SBATCH --mail-user <put your email address here>

#SBATCH --mail-type=BEGIN

#SBATCH --account=[euclid_lowpri/cosmo_lowpri]

#SBATCH --partition=lowpri

#SBATCH --ntasks=1

/bin/hostname

</pre>

h3. Accessing a node where a job is running or starting additional processes on a node

You can attach an srun command to an already existing job (batch or interactive). This

means you can start an interactive session on a node where a job of yours is running

or start an additional process.

First determine the jobid of the desired job using squeue, then use 

<pre>

srun  --jobid <jobid> [options] <executable> 

</pre>

Or more concrete

<pre>

srun  --jobid <jobid> -u --pty  bash # to start an interactive session

srun  --jobid <jobid> ps -eaFAl  # to start get detailed process information 

</pre>

The processes will only run on cores that have been allocated to you. This works 

for batch as well as interactive jobs. 

*Important: If the original job that was submitted is finished, any process 

attached in this fashion will be killed.*

h3. Batch script for running a multi-core job

mpi is installed on cosmofs1.

To run a 4 core job for an executable compiled with mpi you can use

<pre>

#!/bin/bash

#SBATCH --output=slurm.out

#SBATCH --error=slurm.err

#SBATCH --mail-user <put your email address here>

#SBATCH --mail-type=BEGIN

#SBATCH --ntasks=4

mpirun <programname>

</pre>

and it will automatically start on the number of nodes specified.

To ensure that the job is being executed on only one node, add

<pre>

#SBATCH -n 4

</pre>

to the job script.

If you would like to run a program that itself starts processes, you can use the

environment variable $SLURM_NPROCS that is automatically defined for slurm

jobs to explicitly pass the number of cores the program can run on.

To check if your job is acutally running on the specified number of cores, you can check

the PSR column of

<pre>

ps -eaFAl

# or ps -eaFAl | egrep "<yourusername>|UID" if you just want to see your jobs

</pre>

h3. environment for jobs

By default, slurm does not initialize the environment (using .bashrc, .profile, .tcshrc, ...)

To use your usual system environment, add the following line in the submission script:

<pre>

#SBATCH --get-user-env

</pre>

h3. Slurm reporting and accounting

For information on job usage and cluster utilization for slurm jobs the slurm command "sreport" can be used. E.g. the command:

<pre>

sreport user topusage start=01/15/18 -t percent

</pre>

shows the top ten users in percent since January 15th 2018. For more information please look at "man sreport".

For accounting on specific jobs the slurm command "sacct" can be used. E.g. the command:

<pre>

sacct -j 18551 --format=JobID,JobName,MaxRSS,Elapsed

</pre>

displays information (elapsed time, memory usage, ...) on the job number "18551". For more details please  use "man sacct".

h3. Some points on the 'normal' versus 'lowpri' queue on cosmogw

The allowances for each user on the *normal* partition are 304CPU's and 661335MB, which corresponds to about 1/3 of the entire cluster (euclides01-19). In short, every user is allowed to use up to 1/3 of the cluster in the normal partition.

On the partition *lowpri* (for low priority) there are no limits on the CPU numbers or RAM consumption, meaning the user can take all available resources up to the *entire* cluster! However, jobs on the partition "lowpri" have a lower priority through the so called preemption mechanism. This means if all nodes are busy (partially through the lowpri queue) and an additional job is submitted to the "normal" partition, slurm will re-queue (meaning cancel and re-schedule to the lowpri-queue) job(s) on the "lowpri" partition to get the job on the "normal" partition running.

Here is an example scenario to illustrate the opportunities the "lowpri" partition offers:

I want to submit a number of jobs for in total 744cpu's. The entire cluster has 744 cpu's in total, this means in the optimal case I get 1/3 of the cluster on the "normal" partition, and it takes at least three cycles to get all my jobs finished. However, if I submit to the "lowpri" partition, in the case of an empty cluster I can use the *entire* cluster and finish in only one cycle. Of course it may happen that other users submit lots of jobs to the "normal" partition afterwards and many of my jobs are being re-queued. That would then delay the finishing of my jobs on the "lowpri" partition correspondingly. To highlight some aspects of using the "lowpri" partition:

* it is relevant especially when you want to submit several jobs that significantly exceed the user allowance on the "normal" partition and need the entire cluster to get finished;

* on average, the available ressources on the "lowpri" partition are much *larger* than on the "normal" partition, especially during the night or on the weekend;

* please not that *no job gets ever lost* at the "lowpri" partition; if re-queuing occurs, the user gets an email (Subject: "SLURM Job_id=2563 Name=test_mpi_gather.slurm Failed, Run time 00:01:58, PREEMPTED, ExitCode 0") when the job is stopped and subsequently when it starts again and when it finishes (see 1.);

* also on the "lowpri" partition there is a queue which decides which job comes first (of course only in the case of an oversubscription);

* the preemption mechanism tries to minimize the number of re-queued jobs necessary to get the job in the "normal" partition going; so, if 8 cpus are requested and the "lowpri" partion contains one job using 8 cpus, three jobs using 4 cpus and several dozens jobs using 1 cpu, only the job with 8 cpus is re-scheduled independent on the run times and other parameters.

To submit a job to the "lowpri" partition please insert the following lines into the slurm batch script (see also example above):

<pre>

#SBATCH --account=<your account>

#SBATCH -p lowpri

</pre>

with <your_acount> being either "cosmo_lowpri" or "euclid_lowpri".

There are two typical scenarios where a user can gain from the lowpri queue:

* if a job stores intermediate results at regular intervals and picks up from there once started again; then even a long job looses only the computing time since the last storage point if a job is re-scheduled;

* if a single job needs only a small amount of computing time (perhaps <12h) but a lot of jobs need to be run; then the loss of computing time is rather small if a job is re-scheduled;

h2. desdb node

Some specific jobs in cosmodb, such as the "catalog ingest", need to be performed on the machines desdb1/2. For those jobs there is the slurm account "euclid_cat_ing" with the partition "cat_ing". Only selected persons from the Euclid group have access to this node. Please specify "-p cat_ing" and "--account euclid_cat_ing" on the command line or in the slurm script.

h2. Software specific setup

h3. Python environment 

You can use the python 2.7.3 installed on the euclides cluster by using

<pre>

source /data2/users/ccsoft/etc/setup_all

source  /data2/users/ccsoft/etc/setup_python2.7.3

</pre>

h2. Notes For Euclid users

For those submitting jobs to euclides* nodes through Cosmo DM pipeline  here are some things which need to be specified for customized job submissions,

since a different interface to slurm is used.

* To use larger memory per block , specify max_memory = 6000 (for 6G) and so on. inside block definition or in the submit file (in

case you want to use it for all blocks)

* If you want to run on multiple cores/cores then use 

nodes='<number of nodes>:ppn=<number of cores> inside the block definition of a particular block or in the submit file in case you want

to use it for all blocks.

* If you want to use a larger wall time then specify wall_mod=<wall time in minutes> inside the module definition

* note that queue=serial does not work on cosmofs1 (we usually use it for c2pap)

h1. Admin

There is a user "slurm" which however is not really necessary for the administration work. The slurm administrator needs sudo access. Some scripts re-starting slurm, adding a user and similar things are in "/data1/users/slurm/cosmo". With the sudo access the admin can execute those scripts. In the mysql database there is the username "slurmdb" with password.

h2. Slurm configuration

h3. Slurm configuration file

The currently valid version of the configuration file is "/data1/users/slurm/cosmo/slurm.conf" on cosmogw, respectively. To apply a modified slurm configuration, the script "newconfig.sh" can be used. 

The script 

* copies the configuration file to the submit node and restarts the submit service;

* copies the configuration file to all computing nodes and triggers the reconfiguration there;

Then the slurm daemon needs to be started on the submit node and all computing nodes with the script "restart.sh". 

*Note:* Right now the slurmd deamons do not properly start on cosmogw. Even if the start fails, the slurmd daemon is there and working.

h2. User management

h3. Overview over users, accounts, etc.

No sudo access needed:

<pre>

/usr/local/bin/sacctmgr show account withassoc

</pre>

h3. Adding a new user

As root on @cosmofs1@,

<pre>

cd /data1/users/slurm/

./add_user.sh UserName account(cosmo or euclid)

/usr/local/bin/.scontrol reconfigure

</pre>

h3. To increase memory, cores etc for a user

Inside script above, various commands for changing user settings, e.g.

<pre>

/usr/local/bin/sacctmgr -i modify user  name=$1 set GrpCPUs=32

/usr/local/bin/sacctmgr -i modify user  name=$1 set GrpMem=128000

</pre>

h2. Trouble shooting

h3. Information on a particular node

The command "/usr/local/bin/scontrol show node <nodename>" gives detailed information on a particular node (status, reason for being down and so on)

h3. Node in state "drain"

When a node is in "drain" state when calling <pre>sinfo</pre>

run

<pre>

/usr/local/bin/scontrol update nodename=NODE_NAME state=resume

</pre>

to put it back to operation.

h3. Restart

A full running of slurm requires:

* running the data base mysql;

* running the slurm data base daemon slurmdpd (for accounting);

* running slurmctld on cosmogw;

* slurmd on all nodes;

h4. mysql

Mysql is started with 'systemctl start mysql'. The log is in '/var/log/mysqld.log' At one re-start (January 2019) the log said "/usr/sbin/mysqld: Can't create/write to file '/var/run/mysqld/mysqld.pid'". Then '/var/run/mysqld' did not exist (somehow disappeared). It had to be created and given to the owner 'mysql'. Then the file mysql.pid is created and mysql seems to be working fine.

h4. slurmdbd

Should be started with 'systemctl status slurmdbd'. However this does not seem to work always (at least not on January 2019). It is possible to start the daemon directly with '/usr/local/sbin/slurmdbd'. The log of slurm is in '/var/log/slurm/slurmdbd.log'.

h4. slurmctld and slurmd

A re-start of the slurm daemons ('slurmctld' on cosmogw and 'slurmd' on the nodes) is done bye executing the script:

/data1/users/slurm/cosmo/restart.sh

h2. Nodes down

Sometimes nodes are reported as "down". This seems to happen as a result of network problems. Here is some "troubleshooting":https://computing.llnl.gov/linux/slurm/troubleshoot.html#nodes for this situation. Also after a re-boot of cosmofs1 some manual work on slurm might be necessary to get going again.

If a job does not finish and remains int eh state "CG" then the sequence:

<pre>

/usr/local/bin/scontrol update NodeName=euclides01 State=down Reason=hung_proc

/usr/local/bin/scontrol update NodeName=euclides01 State=resume Reason=hung_proc

</pre>

brings the node back again.

h2. History

* January 2nd 2020: one user (Thomas) got the error:

<pre>

[cosmogw][~] $ srun -u --mem-per-cpu=2000 --x11=first --cpus-per-task=56 --pty bash 

srun: error: plugin_load_from_file: dlopen(/usr/local/lib/slurm/select_cons_res.so): /usr/local/lib/slurm/select_cons_res.so: undefined symbol: powercap_get_cluster_current_cap

srun: error: Couldn't load specified plugin name for select/cons_res: Dlopen of plugin file failed

srun: fatal: Can't find plugin for select/cons_res

</pre>

Turns out that the problem was to set "export LD_BIND_NOW=1" for another issue. After unsetting this slurm worked normally.

* December 30th 2019: somehow the controler job slurmctld on cosmogw was down. Agter re-star everything was okay.

* January 23rd 2018: Jobs on euclides12 are no longer finishing. They end up in the state "CG" and hang there forever. In the slurmd log there is the entry "[2018-01-23T10:12:17.477] [18153] error: Unable to establish controller machine" basically every 15mins or so. ssh from euclides12 to cosmogw via name and IP address was possible, so it is difficult to interpret this error message. At the end the problem was solved by:

** stopping slurmd

** removing /var/run/slurmd.pid

** creating /var/run/slurmd.pid via touch

** re-starting slurmd again

** euclides12 had before this sometimes created problems, maybe this was the culmination now.

* May 18th 2017: On cosmogw, three nodes were reported as "DOWN" despite running the slurmd daemon and having connections to the slurmctl daemon on the control node; turns out that with a normal "/etc/init.d/slurm start" on the control machine only nodes are considered that are *not* DOWN; "/etc/init.d/slurm startclean" must be used to establish new connections to all nodes to take them back into the queue;

* May 2nd 2017: the control daemon on cosmofs1 was no longer working; also it could not e re-started; the corresponding commands "/etc/init.d/slurm status/start" were not giving back any kind of feedback, the log files were empty; the relevant daemon on the nodes "slurmd", was running smoothly; a comparison revealed that the difference was whether the command  "/usr/local/bin/scontrol show daemon" does return the daemon name or nothing, and in the later case nothing happens and the daemon does not run well; further investigation showed that the machine name given in "slurm.conf" as "ControlMachine=" needs to be identical to the name returned of the command "hostname"; this was no longer the case, likely induced due to moving the machines to the new sub-net (the exact mechanism is unclear);

* April 24th 2017: taking euclides11 out of the queues to free it for the new OS and the slurm test on it; euclides10 is now the development node;

* April 07th 2017: Applying "/usr/local/bin/scontrol show node euclides11" for the debug partition euclides11 says "Reason=Node unexpectedly rebooted [root@2016-12-14T13:25:01]"; internet research suggested to change "ReturnToService=" from 1 to 2 in the configuration file; after applying and restarting the new configuration file the debug nodes works again.;

* April 06th 2017: After the reconfiguration of the cluster the slurm confguration file was adjusted (to reflect the new machine names); also minor changes had to be applied to the scripts "newconfig.sh" and "restart.sh" to loop over the new names; the new configuration files were applied and slurm restarted; all computing nodes for the normal partition came up, the debug partition stayed down;

* March 29th 2017: euclides7 is in drain state;  "/usr/local/bin/scontrol show node euclides2" says "Reason=Epilog error"; when resumed, seems to work normal;

* March 28th 2017: euclides2 is in drain state; when resumed, it goes into drain state when using it the next time; "/usr/local/bin/scontrol show node euclides2" says "Reason=Prolog error"; after a reboot the machine was in status "idle*"; when resumed, it worked again;