CommunityData:Klone
Klone is the latest version of hyak, the UW super computing system. We will soon have a larger allocation of machines on Klone than on Mox. The Klone machines have 40 cores and either 384GB or 768GB of RAM. You can check storage allocation usage with the 'hyakstorage' command.
Setting up SSH[edit]
When you connect to SSH, it will ask you for a key from your token. Typing this in every time you start a connection be a pain. One approach is to create an .ssh config file that will create a "tunnel" the first time you connect and send all subsequent connections to Hyak over that tunnel. Some details in the Hyak documentation.
I've added the following config to the file ~/.ssh/config
on my laptop (you will want to change the username):
Host klone klone.hyak.uw.edu User <YOURNETID> HostName klone.hyak.uw.edu ControlPath ~/.ssh/master-%r@%h:%p ControlMaster auto ControlPersist yes Compression yes
Note: If your SSH connection becomes stale or disconnected (e.g., if you change networks) it may take some time for the connection to time out. Until that happens, any connections you make to hyak will silently hang. If your connections to ssh hyak are silently hanging but your Internet connection seems good, look for ssh processes running on your local machine with:
ps ax|grep klone
If you find any, kill them with kill <PROCESSID>
. Once that is done, you should have no problem connecting to Hyak.
Setting up your Environment[edit]
The recommended way to manage software for your research projects on Klone is to use Apptainer containers (formerly known as Singularity). At first, you probably do not need to know much about containers because we maintain a shared setup described below. However, before getting to work on Klone, you'll need to set up an environment that provides our containerized commands and a few other conveniences. You do this by creating the following .bashrc
file in your home directory (i.e., /mmfs1/home/{your_username}
) where you land when you connect to klone.
Initial .Bashrc[edit]
Before we get started using our apptainer package on klone, we need to start with a .bashrc
. Using a text editor (nano is a good choice if you don't already have a preference), create your .bashrc
by pasting in the following code. Then run the command source ~/.bashrc
to run the .bashrc and enable the environment.
# .bashrc
export LOGIN_NODE=$(hostname | grep -q '^klone-login01' ; echo $?)
export SBATCH_EXPORT=BASH_ENV='~/.bashrc'
export SLURM_EXPORT_ENV=BASH_ENV='~/.bashrc'
export SLURM_EXPORT_ENV=BASH_ENV='~/.bashrc'
if [ -f ~/.bash_aliases ]; then
. ~/.bash_aliases
fi
# User specific environment
if ! [[ "$PATH" =~ "$HOME/.local/bin:$HOME/bin:" ]]
then
PATH="$HOME/.local/bin:$HOME/bin:$PATH"
fi
export PATH
# Source global definitions
if [ -f /etc/bashrc ]; then
. /etc/bashrc
fi
source "/gscratch/comdata/env/cdsc_klone_bashrc"
if [[ "$LOGIN_NODE" == 0 ]]; then
:
else
# Uncomment the following line if you don't like systemctl's auto-paging feature:
# export SYSTEMD_PAGER=
# User specific aliases and functions
umask 007
export APPTAINER_BIND="/gscratch:/gscratch,/mmfs1:/mmfs1,/gpfs:/gpfs,/sw:/sw,/usr:/kloneusr,/bin:/klonebin"
export OMP_THREAD_LIMIT=40
export OMP_NUM_THREADS=40
export PATH="$PATH:/gscratch/comdata/users/$(whoami)/bin:/gscratch/comdata/local/spark:/gscratch/comdata/local/bin"
source "/gscratch/comdata/users/nathante/spark_env.sh"
export _JAVA_OPTIONS="-Xmx362g"
fi
Connect to a Compute Node[edit]
When you first SSH into Klone, you will be on your login node. Before you can do computational work, or use software installed in our containers (see below), you will need to log into a compute node from your login node. After your ~/.bashrc
file is setup and run, you can do so by running a SLURM job or use one of the aliases described in https://wiki.communitydata.science/CommunityData:Hyak_tutorial#Interactive_Jobs.
About Containers[edit]
We use Apptainer (formerly known as, and sometimes still referred to as Singularity) containers to install software on klone. Klone provides a very minimal operating system so without these containers, installing software can be quite labor-intensive. Our goal has been to make using software installed through apptainer as seamless as possible. For the most part, once you have your environment configured as above, you shouldn't have to think about the containers unless you need to install something new.
We created commands (e.g., python3
, Rscript
, jupyter-console
) that run the containerized version of the program. The full list of such commands is in /gscratch/comdata/containers/bin
.
Importantly, installing packages in R, Python (e.g., using pip) or other programming languages should usually work normally because the containers already have the most common dependencies. Installing packages this way will not update the container. Instead the packages will be installed in your user directory. This is desirable so that different container users do not break each other's environments. It may happen that an installation fails because it requires a missing dependency from the operating system. If this happens you can try to add the dependency to the container as described below. If this seems challenging or complicated or you need many changes to the container, or changes you don't understand, reach out to the IT team.
We will use multiple different apptainter containers for different applications to avoid incidentally breaking existing versions of packages during upgrades. We want containers that include "soft dependencies" that R or Python libraries might want.
To make a new container alias[edit]
For example, let's say you want to make a command to run jupyter-console
for interactive python work and let's say you know that you want to run this from the cdsc_python.sif
container located in /gscratch/comdata/containers/cdsc_python
.
1. Ensure that the software you want to execute is installed in the container. Test this by running apptainer exec /gscratch jupyter-console
.
2. Create an executable file in /gscratch/comdata/containers/bin. The file should look like:
#!/usr/bin/env bash
apptainer exec /gscratch/comdata/containers/cdsc_python/cdsc_python.sif jupyter-console.
Installing apptainer on your local computer[edit]
You might find it more convenient to develop your apptainer container on your local machine. You'll want apptainer version 3.4.2. which is the version installed on klone. Follow these instructions for installing apptainer on your local linux machine.
Creating a apptainer container[edit]
Our goal is to write a apptainer definition file that will install the software that we want to work with. The definition file contains instructions for building a more reproducible environment. For example, the file cdsc_base.def
contains instructions for installing an environment based on debian 11 (bullseye). Once we have the definition file, we just have to run:
NOTE: For some reason building a container doesn't work on the /gscratch
filesystem. Instead build containers on the /mmfs1
filesystem and then copy them to their eventual homes on /gscratch
.
apptainer build --fakeroot cdsc_base.sif cdsc_base.def
On a klone compute node to create the apptainer container cdsc_base.sif
. This can take quite awhile to run as it downloads and installs a lot of software!
You can start a shell in the container using:
apptainer shell cdsc_base.sif
You can also just execute a single command using:
apptainer exec cdsc_base.sif echo "my command"
Sandbox containers don't seem to work consistently. It's better to just update the definition file and rebuild the container. It's a hassle, but it works.
The .sif
container is immutable, but you can modify it by converting it to a sandbox.
apptainer build --sandbox cdsc_base_sandbox cdsc_base.sif
You might run into trouble with exceeding space in your temporary file path. If you do, run
sudo export APPTAINER_TMPDIR=/my/large/tmp
sudo export APPTAINER_CACHEDIR=/my/large/apt_cache
sudo export APPTAINER_LOCALCACHEDIR=/my/large/apt_cache
before running the build.
For developing a container it's useful to use a sandbox
container, which is mutable
so you can continue installing software on it. However, you should add your changes to the definition file so you can build immutable containers that are as reproducible as possible.
The
So in summary, the workflow is:
cdsc_base_sandbox
is mutable, so we can continue working on that environment and installing more software as we like. We just need to build it as a .sif
file to use it on klone. It's also possible to convert the container back into sandbox mode and then modify non-root parts of the container on klone, but this requires running the container in a way that makes the outside klone system invisible! This is useful for installing R or Python packages in userspace within the container. It's not that useful for working with data outside of the container.
- Develop a definition file (
cdsc_base.dev
) to setup your desired environment. - Keep the definition file up to date with any modifications you make to the container in sandbox mode so your container is reproducible.
- Run programs in the container to work with files outside of it (possibly including other packages, allowing us to use debian to bootstrap klone-optimzed binaries).
- If you want to work on you local machine you can use the same definition file to install the container on your local machine.
Spark[edit]
To set up a spark cluster using apptainer the first step to "run" the container on each node in the cluster:
# on the first node
apptainer instance start --fakeroot cdsc_base.sif spark-boss
export SPARK_BOSS=$(hostname)
# on the first worker node (typically same as boss node)
apptainer instance start --fakeroot cdsc_base.sif spark-worker-1
# second worker node
apptainer instance start --fakeroot cdsc_base.sif spark-worker-2
The second step is to start the spark services on the instances
apptainer exec instance://spark-boss /opt/spark/sbin/start_master.sh
apptainer exec instance://spark-worker-1 /opt/spark/sbin/start-worker.sh $SPARK_BOSS:7077
That should be it. Though in practice it might make more sense to have special containers for the spark boss and workers.
You can now submit spark jobs by running spark-submit.sh
.
# replace n3078 with the master hostname
apptainer exec instance://spark-boss /opt/spark/bin/spark --master spark://n3078.hyak.local:7077
Nate's working on wrapping the above nonsense in friendlier scripts.