CommunityData:Hyak software installation

You will often find that Hyak is missing software that you need to do you work. This page includes documentation on building those:

R packages
To install a R package that's not available globally, you can check out a build node, and install the package locally. Here's how to do it:

This will start R, where you can install a package in the usual way. The build node has access to the Internet, so it will be able to download the required source packages, etc.

Python Packages
Our setup on Mox provides the Anaconda python distribution.

Using an anaconda python distribution has important implications for how you install packages. While in normal python, you would install python packages using `pip`, when you use an anaconda distribution you should use  to install packages. Conda also has some fancy features like virtual environments for using different versions of python or different versions of packages in different projects. The problem with using conda is that it does not include all the packages you might want to use. If you want to install a python package that is missing from conda, you can use pip. Importantly, you should prefer to install software using conda over pip.

Details on both are here:


 * Conda Documentation
 * Pip Documentation

It's often a good idea to create new environments for installing software you will use in each project. This way you can use the latest and greatest versions of packages, but you'll still be able to run code in your old projects. To create a new conda environment:

Then add the following to your .bashrc to use this environment automatically by default:

Conda doesn't like it when you try to activate an environment that is already active.

Conda modifies your prompt in a possibly annoying way. To disable this behavior run the command:

Custom modules
Software on Hyak can be outdated, or in some cases, not available at all. In some of these situations, it may be possible to use environment modules to install and run software without necessitating administrative (root) privileges. For example, it is possible to have and run the newest version of R that is installed in a central, shared directory, and it is even possible to have multiple versions of R available in parallel. The following subsection shows how to do this. Ordinarily, this should not be necessary on a day-to-day basis.

Installing and making available a custom module
If you are using  to run and manage your builds, keep in mind that   drops a few environment variables such as , which may mess up your build process. You should check that all the relevant environment variables are set before starting your build.

The first step toward installing and making available a custom module (in this case, slang 2.3.2) is to spin up the build node, download slang, compile it with a specific prefix, and install it.

The  option to   tells the build scripts that slang is going to be installed in. This follows a convention that we picked—software in modules should go into. The  option is the most important flag for  —any other flag or option will be specific to the software being installed.

The second step is to write a. This contains the metadata about our module. Edit the file  to contain the following

Note that the filename follows a similar convention as  earlier. This file sets up the,  ,  , and  , environment variables appropriately so that the specified version of slang can be accessed and run as needed. There are many more directives that can go into the —see   for details on those directives. If your program is a simple binary, you will likely only need to append to the, and   environment variables.

Once this file is written out, the  command should list   as an available module. This is because the module system is set up to look inside  for module files, thanks to the   variable that is set through. The command  should make slang available and ready for use. To avoid running  whenever you log in, you can add the command at the end of your   file (after the section that sets  ).