FAQ

Because I'm tired of explaining the same things over and over again

Using the Cluster

• How do I get my code on the cluster?

  1. From a workstation with your code on it:

     scp -rp folder_with_your_code/ infoprism:~

  2. Your code is now on infoprism

  3. Hopefully your code should automatically sync between the workstations and cluster, but this is not always the case.

• Where do I go to run code on the cluster?

  • From RAS or a workstation:

    ssh infoprism

• How do I compile code on the cluster?

  • If using a single C/C++ file like a pleb:

    mpicc your_file.c # or mpicxx if using C++

  • If using CMake like a boss:

    cmake . -DCMAKE_C_COMPILER=mpicc -DCMAKE_CXX_COMPILER=mpicxx
    make

• Do I need to compile my code on the cluster?

  • YES, there is no other option, binaries will never be compatible. This is not a challenge, this is a statement of fact. Do not complain if your a.out from the workstation doesn't work.

• Now that I did all that, how do I run code on the cluster?

  • If using an MPI binary (most parallel computing classes), or just running a non-MPI binary n number of times:

    srun -N <number_of_machines> -n <number_of_processes> --mpi=pmix_v2 ./your_binary --your-binary-flags

  • Alternative (non-preferred) method of running an MPI binary:

    salloc -N <number_of_machines> -n <number_of_processes> mpiexec ./your_binary --your-binary-flags

• That gives me some error about not having an account! What gives?

  • Ask a sysadmin if they can make you an account on the cluster. Point them to this document if they don't know how.

• My job just hangs forever!

  • Either someone is inconsiderately using the entire cluster, or the cluster is broken. If this is the case, please email us.

  • To check the former, use the sinfo and squeue commands.

• When I access the cluster, I'm not in my home directory, I'm in the root directory!

  • This is a problem with mounting, sometimes you just have to wait a bit.

  • In other cases, you might just not have a home directory because that machine is being used for something important.

Managing the Cluster

Or, "I was put in charge of your mess Jack you better have documented everything." Don't worry, I didn't. Keeps you on your toes.

Note: everything here expects you have root access on infosphere, and are running all these commands as root.

• What are the basic troubleshooting steps?

  • sinfo to view the state of the cluster

  • squeue to see what jobs are holding things up

  • Checking files in /var/log/slurm on infosphere and the affected nodes

• Someone wants me to make an account for them!

  • Run this in /root on infosphere:

    cat your_user_list.txt | ./make_new_users.sh

• How do I run the ansible play?

  • From infosphere (this is important):

    cd ansible
    ansible-playbook -i hosts -f 50 --ask-vault-pass cluster.yml --skip-tags=install

  • The above command basically runs the default, updating-only play on all the cluster nodes, including infosphere.

• I want to upgrade a specific part of the cluster, how do?

  • There are a couple partitions in ansible:

    • hpc: All the HPC nodes

    • hpcgpu: Just Zoidberg

    • ibm: All the Borg nodes

    • ibmgpu: All the Borg nodes with GPUs in them

    • clustermaster: infosphere itself

  • Run the corresponding ansible .yml playbooks from infosphere

• I would like to install a new version of something on the cluster, how do?

  • The current software names and versions are under [fullcluster:vars] in the hosts file in the ansible repository.

  • Change the version number to the latest available, and run the following command replacing <software_name> with something like abinit or openmpi.

    ansible-playbook -i hosts -f 50 cluster.yml --tags=install_<software_name>

  • The plays should automatically uninstall the old version for you when updating.

  • IMPORTANT NOTE: If you update pmix, you also need to reinstall slurm and openmpi.

• After a slurm upgrade, everything went down. How fix?

  • Check that the slurm version is the same everywhere

  • ansible fullcluster -i hosts -m command -a "bash -c 'systemctl disable slurmd; systemctl enable slurmd; systemctl restart slurmd'" (disable slurmd on infosphere afterwards)

  • Really, just check the logs yourself, listen to error messages, Google your way to victory.

• How to recreate the slurmdbd database (last resort)?

  • systemctl stop slurmdbd slurmctld && systemctl restart mariadb && mysql

    1. CREATE DATABASE slurm;

       • If this command fails, skip to the end of this list

    2. USE slurm;

    3. SHOW TABLES;

    4. For every table in the database, DROP TABLE <table_name>;

    5. exit

  • systemctl start slurmdbd slurmctld

  • sacctmgr add cluster hpc

  • find /cluster -type d | /root/make_new_users.sh