Editing CommunityData:Hyak software installation

Our setup on Mox provides the Anaconda python distribution.  

Using an anaconda python distribution has important implications for how you install packages. While in normal python, you would install python packages using `pip`, when you use an anaconda distribution you should use <code>conda</code> to install packages. Conda also has some fancy features like virtual environments for using different versions of python or different versions of packages in different projects. The problem with using conda is that it does not include all the packages you might want to use. If you want to install a python package that is missing from conda, you can use pip. ''Importantly, you should prefer to install software using conda over pip.''

* [https://conda.io/docs/ Conda Documentation]

* [https://pip.pypa.io/en/stable/ Pip Documentation]

It's often a good idea to create new environments for installing software you will use in each project. This way you can use the latest and greatest versions of packages, but you'll still be able to run code in your old projects. To create a new conda environment:

$ conda create -n my_root

Then add the following to your .bashrc to use this environment automatically by default:

if [ -z $(conda info --env | grep my_root | grep \*) ]; then

    source activate my_root

fi

Conda doesn't like it when you try to activate an environment that is already active.  

<syntaxhighlight lang='bash'>

$ conda config --set changeps1 False

</syntaxhighlight>

The first step toward installing and making available a custom module (in this case, [https://www.jedsoft.org/releases/slang/ slang 2.3.2]) is to spin up the build node, download slang, compile it with a specific prefix, and install it.

$ wget https://www.jedsoft.org/releases/slang/slang-2.3.2.tar.bz2

$ tar jxvf slang-2.3.2.tar.bz2

$ cd slang-2.3.2

$ ./configure --prefix=/gscratch/comdata/modules/sw/slang/2.3.2

The <code>--prefix</code> option to <code>./configure</code> tells the build scripts that slang is going to be installed in <code>/gscratch/comdata/modules/sw/R/3.5.0</code>. This follows a convention that we picked—software in modules should go into <code>/gscratch/comdata/modules/sw/{SOFTWARE_NAME}/{SOFTWARE_VERSION}</code>. The <code>--prefix</code> option is the most important flag for <code>./configure</code>—any other flag or option will be specific to the software being installed.

The second step is to write a <code>modulefile</code>. This contains the metadata about our module. Edit the file <code>/gscratch/comdata/modules/modulefiles/slang/2.3.2</code> to contain the following

         puts stderr "\tModule providing slang 2.3.2"

module-whatis "Module providing slang 2.3.2"

Note that the filename follows a similar convention as <code>--prefix</code> earlier (<code>/gscratch/comdata/modules/modulefiles/{SOFTWARE_NAME}/{SOFTWARE_VERSION}</code>). This file sets up the <code>PATH</code>, <code>MANPATH</code>, <code>LD_LIBRARY_PATH</code>, and <code>PKG_CONFIG_PATH</code>, environment variables appropriately so that the specified version of slang can be accessed and run as needed. There are many more directives that can go into the <code>modulefile</code>—see <code>man modulefile</code> for details on those directives. If your program is a simple binary, you will likely only need to append to the <code>PATH</code>, and <code>MANPATH</code> environment variables.

Once this file is written out, the <code>module avail</code> command should list <code>slang/2.3.2</code> as an available module. This is because the module system is set up to look inside <code>/gscratch/comdata/modules/modulefiles</code> for module files, thanks to the <code>MODULEPATH</code> variable that is set through <code>.bashrc</code>. The command <code>module load slang/2.3.2</code> should make slang available and ready for use. To avoid running <code>module load slang/2.3.2</code> whenever you log in, you can add the command at the end of your <code>.bashrc</code> file (after the section that sets <code>MODULEPATH</code>).