CommunityData:Hyak software installation: Difference between revisions

From CommunityData
 
(3 intermediate revisions by 2 users not shown)
Line 18: Line 18:
=== Python Packages ===
=== Python Packages ===


DO NOT TRUST THIS SECTION. Intel python appears to have some issues.  
Our setup on Mox provides the Anaconda python distribution.  


The recommended python to use on hyak is the intel-python. This is a customized anaconda distribution with a magical optimization of python that really increases the performance of numpy.  
Using an anaconda python distribution has important implications for how you install packages. While in normal python, you would install python packages using `pip`, when you use an anaconda distribution you should use <code>conda</code> to install packages. Conda also has some fancy features like virtual environments for using different versions of python or different versions of packages in different projects. The problem with using conda is that it does not include all the packages you might want to use. If you want to install a python package that is missing from conda, you can use pip. ''Importantly, you should prefer to install software using conda over pip.''


Using an anaconda python distribution has important implications for how you install packages. While in normal python, you would install python packages using `pip`, when you use an anaconda distribution you should use `conda` to install packages. Conda also has some fancy features like virtual environments for using different versions of python or different versions of packages in different projects. The problem with using conda is that it does not include all the packages you might want to use. If you want to install a python package that is missing from conda, you can use pip.
Details on both are here:


Importantly, when using intel-python, you should prefer to install software using conda over pip.  
* [https://conda.io/docs/ Conda Documentation]
* [https://pip.pypa.io/en/stable/ Pip Documentation]


[https://conda.io/docs/ Conda Documentation]
It's often a good idea to create new environments for installing software you will use in each project. This way you can use the latest and greatest versions of packages, but you'll still be able to run code in your old projects. To create a new conda environment:  
[https://pip.pypa.io/en/stable/ Pip Documentation]


The first time you use intel-python you need to create a custom environment for installing software:  
<syntaxhighlight lang='bash'>
$ conda create -n my_root
</syntaxhighlight>
Then add the following to your .bashrc to use this environment automatically by default:


    conda create -n my_root
<syntaxhighlight lang='bash'>
 
if [ -z $(conda info --env | grep my_root | grep \*) ]; then
Then add the following to your .bashrc to use this environment.
    source activate my_root
    if [ -z $(conda info --env | grep my_root | grep \*) ]; then
fi
        source activate my_root
</syntaxhighlight>
    fi
Conda doesn't like it when you try to activate an environment that is already active.  
 
Conda doesn't like it when you try to activate an environment that is already active. T


Conda modifies your prompt in a possibly annoying way. To disable this behavior run the command:  
Conda modifies your prompt in a possibly annoying way. To disable this behavior run the command:  
    $ conda config --set changeps1 False
<!-- If you need python libraries that are not installed in the shared environment:
$ pip3 install --user YOURLIBHERE
...replacing YOURLIBHERE with the name of the library you need, e.g. 'pandas'. The --user option will install it for just you.


If you have a lot of dependencies for a specific project, consider using [[#Python Virtual Environments |Python Virtual Environments]] -->
<syntaxhighlight lang='bash'>
$ conda config --set changeps1 False
</syntaxhighlight>


=== Custom modules ===
=== Custom modules ===
Line 62: Line 55:
{{note}} If you are using <code>screen</code> to run and manage your builds, keep in mind that <code>screen</code> [https://superuser.com/a/235773 drops a few environment variables] such as <code>LD_LIBRARY_PATH</code>, which may mess up your build process. You should check that all the relevant environment variables are set before starting your build.  
{{note}} If you are using <code>screen</code> to run and manage your builds, keep in mind that <code>screen</code> [https://superuser.com/a/235773 drops a few environment variables] such as <code>LD_LIBRARY_PATH</code>, which may mess up your build process. You should check that all the relevant environment variables are set before starting your build.  


The first step toward installing and making available a custom module (in this case, R 3.5.0) is to spin up the build node, download R, compile it with a specific prefix, and install it.
The first step toward installing and making available a custom module (in this case, [https://www.jedsoft.org/releases/slang/ slang 2.3.2]) is to spin up the build node, download slang, compile it with a specific prefix, and install it.


<source lang='bash'>
<source lang='bash'>
$ build_machine
$ build_machine
$ module load contrib/texlive/2017  # loads the texlive module that is helpful for generating R documentation
$ wget https://www.jedsoft.org/releases/slang/slang-2.3.2.tar.bz2
$ module load contrib/openblas/0.2.20  # loads the openblas library, which speeds up some R operations significantly
$ tar jxvf slang-2.3.2.tar.bz2
$ wget https://cran.r-project.org/src/base/R-3/R-3.5.0.tar.gz
$ cd slang-2.3.2
$ tar xzvf R-3.5.0.tar.gz
$ ./configure --prefix=/gscratch/comdata/modules/sw/slang/2.3.2
$ cd R-3.5.0
$ ./configure --prefix=/gscratch/comdata/modules/sw/R/3.5.0  --without-x --enable-R-shlib --with-lapack --with-blas="-L/sw/contrib/openblas/0.2.20/lib -lopenblas"
$ make
$ make
$ make install
$ make install
</source>
</source>


The <code>--prefix</code> option to <code>./configure</code> tells the build scripts that R is going to be installed in <code>/gscratch/comdata/modules/sw/R/3.5.0</code>. This follows a convention that we picked—software in modules should go into <code>/gscratch/comdata/modules/sw/{SOFTWARE_NAME}/{SOFTWARE_VERSION}</code>. The <code>--prefix</code> option is the most important flag for <code>./configure</code>—any other flag or option will be specific to the software being installed.
The <code>--prefix</code> option to <code>./configure</code> tells the build scripts that slang is going to be installed in <code>/gscratch/comdata/modules/sw/R/3.5.0</code>. This follows a convention that we picked—software in modules should go into <code>/gscratch/comdata/modules/sw/{SOFTWARE_NAME}/{SOFTWARE_VERSION}</code>. The <code>--prefix</code> option is the most important flag for <code>./configure</code>—any other flag or option will be specific to the software being installed.


The second step is to write a <code>modulefile</code>. This contains the metadata about our module. Edit the file <code>/gscratch/comdata/modules/modulefiles/R/3.5.0</code> to contain the following
The second step is to write a <code>modulefile</code>. This contains the metadata about our module. Edit the file <code>/gscratch/comdata/modules/modulefiles/slang/2.3.2</code> to contain the following


<source lang='tcl'>
<source lang='tcl'>
Line 84: Line 75:
##
##
proc ModulesHelp { } {
proc ModulesHelp { } {
         puts stderr "\tModule providing R 3.5.0."
         puts stderr "\tModule providing slang 2.3.2"
}
}


module-whatis "Module providing R 3.5.0."
module-whatis "Module providing slang 2.3.2"


module load contrib/openblas/0.2.20 # this loads any module dependencies
prepend-path    PATH            /gscratch/comdata/modules/sw/slang/2.3.2/bin
prepend-path    PATH            /gscratch/comdata/modules/sw/R/3.5.0/bin
prepend-path    MANPATH        /gscratch/comdata/modules/sw/slang/2.3.2/share/man
prepend-path    MANPATH        /gscratch/comdata/modules/sw/R/3.5.0/share/man
prepend-path    LD_LIBRARY_PATH /gscratch/comdata/modules/sw/slang/2.3.2/lib
prepend-path    PKG_CONFIG_PATH /gscratch/comdata/modules/sw/slang/2.3.2/lib/pkgconfig
</source>
</source>


Note that the filename follows a similar convention as <code>--prefix</code> earlier (<code>/gscratch/comdata/modules/modulefiles/{SOFTWARE_NAME}/{SOFTWARE_VERSION}</code>). This file sets up the <code>PATH</code> and <code>MANPATH</code> environment variables appropriately so that the specified version of R can be accessed and run as needed. There are many more directives that can go into the <code>modulefile</code>—see <code>man modulefile</code> for details on those directives.
Note that the filename follows a similar convention as <code>--prefix</code> earlier (<code>/gscratch/comdata/modules/modulefiles/{SOFTWARE_NAME}/{SOFTWARE_VERSION}</code>). This file sets up the <code>PATH</code>, <code>MANPATH</code>, <code>LD_LIBRARY_PATH</code>, and <code>PKG_CONFIG_PATH</code>, environment variables appropriately so that the specified version of slang can be accessed and run as needed. There are many more directives that can go into the <code>modulefile</code>—see <code>man modulefile</code> for details on those directives. If your program is a simple binary, you will likely only need to append to the <code>PATH</code>, and <code>MANPATH</code> environment variables.


Once this file is written out, the <code>module avail</code> command should list <code>R/3.5.0</code> as an available module. This is because the module system is set up to look inside <code>/gscratch/comdata/modules/modulefiles</code> for module files, thanks to the <code>MODULEPATH</code> variable that is set through <code>.bashrc</code>. The command <code>module load R/3.5.0</code> should make R available and ready for use. To avoid running <code>module load R/3.5.0</code> whenever you log in, you can add the command at the end of your <code>.bashrc</code> file (after the section that sets <code>MODULEPATH</code>).
Once this file is written out, the <code>module avail</code> command should list <code>slang/2.3.2</code> as an available module. This is because the module system is set up to look inside <code>/gscratch/comdata/modules/modulefiles</code> for module files, thanks to the <code>MODULEPATH</code> variable that is set through <code>.bashrc</code>. The command <code>module load slang/2.3.2</code> should make slang available and ready for use. To avoid running <code>module load slang/2.3.2</code> whenever you log in, you can add the command at the end of your <code>.bashrc</code> file (after the section that sets <code>MODULEPATH</code>).

Latest revision as of 00:26, 20 May 2020

You will often find that Hyak is missing software that you need to do you work. This page includes documentation on building those:

R packages[edit]

To install a R package that's not available globally, you can check out a build node, and install the package locally. Here's how to do it:

$ build_machine
$ R

This will start R, where you can install a package in the usual way. The build node has access to the Internet, so it will be able to download the required source packages, etc.

> install.packages('lme4')

Python Packages[edit]

Our setup on Mox provides the Anaconda python distribution.

Using an anaconda python distribution has important implications for how you install packages. While in normal python, you would install python packages using `pip`, when you use an anaconda distribution you should use conda to install packages. Conda also has some fancy features like virtual environments for using different versions of python or different versions of packages in different projects. The problem with using conda is that it does not include all the packages you might want to use. If you want to install a python package that is missing from conda, you can use pip. Importantly, you should prefer to install software using conda over pip.

Details on both are here:

It's often a good idea to create new environments for installing software you will use in each project. This way you can use the latest and greatest versions of packages, but you'll still be able to run code in your old projects. To create a new conda environment:

$ conda create -n my_root

Then add the following to your .bashrc to use this environment automatically by default:

if [ -z $(conda info --env | grep my_root | grep \*) ]; then
    source activate my_root
fi

Conda doesn't like it when you try to activate an environment that is already active.

Conda modifies your prompt in a possibly annoying way. To disable this behavior run the command:

$ conda config --set changeps1 False

Custom modules[edit]

Software on Hyak can be outdated, or in some cases, not available at all. In some of these situations, it may be possible to use environment modules to install and run software without necessitating administrative (root) privileges. For example, it is possible to have and run the newest version of R that is installed in a central, shared directory, and it is even possible to have multiple versions of R available in parallel. The following subsection shows how to do this. Ordinarily, this should not be necessary on a day-to-day basis.

Installing and making available a custom module[edit]

Note Note: If you are using screen to run and manage your builds, keep in mind that screen drops a few environment variables such as LD_LIBRARY_PATH, which may mess up your build process. You should check that all the relevant environment variables are set before starting your build.

The first step toward installing and making available a custom module (in this case, slang 2.3.2) is to spin up the build node, download slang, compile it with a specific prefix, and install it.

$ build_machine
$ wget https://www.jedsoft.org/releases/slang/slang-2.3.2.tar.bz2
$ tar jxvf slang-2.3.2.tar.bz2
$ cd slang-2.3.2
$ ./configure --prefix=/gscratch/comdata/modules/sw/slang/2.3.2
$ make
$ make install

The --prefix option to ./configure tells the build scripts that slang is going to be installed in /gscratch/comdata/modules/sw/R/3.5.0. This follows a convention that we picked—software in modules should go into /gscratch/comdata/modules/sw/{SOFTWARE_NAME}/{SOFTWARE_VERSION}. The --prefix option is the most important flag for ./configure—any other flag or option will be specific to the software being installed.

The second step is to write a modulefile. This contains the metadata about our module. Edit the file /gscratch/comdata/modules/modulefiles/slang/2.3.2 to contain the following

#%Module1.0####################################################################
##
proc ModulesHelp { } {
        puts stderr "\tModule providing slang 2.3.2"
}

module-whatis "Module providing slang 2.3.2"

prepend-path    PATH            /gscratch/comdata/modules/sw/slang/2.3.2/bin
prepend-path    MANPATH         /gscratch/comdata/modules/sw/slang/2.3.2/share/man
prepend-path    LD_LIBRARY_PATH /gscratch/comdata/modules/sw/slang/2.3.2/lib
prepend-path    PKG_CONFIG_PATH /gscratch/comdata/modules/sw/slang/2.3.2/lib/pkgconfig

Note that the filename follows a similar convention as --prefix earlier (/gscratch/comdata/modules/modulefiles/{SOFTWARE_NAME}/{SOFTWARE_VERSION}). This file sets up the PATH, MANPATH, LD_LIBRARY_PATH, and PKG_CONFIG_PATH, environment variables appropriately so that the specified version of slang can be accessed and run as needed. There are many more directives that can go into the modulefile—see man modulefile for details on those directives. If your program is a simple binary, you will likely only need to append to the PATH, and MANPATH environment variables.

Once this file is written out, the module avail command should list slang/2.3.2 as an available module. This is because the module system is set up to look inside /gscratch/comdata/modules/modulefiles for module files, thanks to the MODULEPATH variable that is set through .bashrc. The command module load slang/2.3.2 should make slang available and ready for use. To avoid running module load slang/2.3.2 whenever you log in, you can add the command at the end of your .bashrc file (after the section that sets MODULEPATH).