CommunityData:Hyak migration (ikt to mox): Difference between revisions
Groceryheist (talk | contribs) (remove extra new line) |
Groceryheist (talk | contribs) (Should we backup other data?) |
||
| Line 2: | Line 2: | ||
# Copy data (only raw data, data that we are using in current and future projects) | # Copy data (only raw data, data that we are using in current and future projects) | ||
# Backup other data? | |||
# Copy code (Everyone copy their own user directory) | # Copy code (Everyone copy their own user directory) | ||
# Create a shared .bashrc that everyone will load. This will provide us with a shared environment (python, R, other packages, useful aliases). | # Create a shared .bashrc that everyone will load. This will provide us with a shared environment (python, R, other packages, useful aliases). | ||
Revision as of 23:51, 13 November 2019
This page is a list of things that we want to do to migrate from ikt to mox.
- Copy data (only raw data, data that we are using in current and future projects)
- Backup other data?
- Copy code (Everyone copy their own user directory)
- Create a shared .bashrc that everyone will load. This will provide us with a shared environment (python, R, other packages, useful aliases).
Hyak Migration Working Group
- Mako
- Kaylea
- Nate
- Sayamindu
- Jeremy
- Jim?
We will use custom modules to maintain installations of software that we use. Sometimes the hyak team already provides a module that we need (i.e. up-to-date R and Python) then we should prefer these packages so we don't have to do the work of compiling and packaging the modules. But if we want to be on the cutting edge of python and R we'll be in the business.
Since I (Nate) typically develop code on my laptop before running it on hyak. I think it is ideal if our Hyak environment maintains versions of software that are equivalent to those included in Debian Buster whenever possible. Ideally we will even provide modules for important R and Python packages (e.g. spark, ggplot, pandas) so that we can keep versions consistent and stable over time.
We'll create a list of packages that people can expect to be loaded in their environments and load them in the shared .bashrc.
We'll also provide a shared .bash_aliases that provide common commands for interacting with slurm.
List of modules we'll maintain on Hyak (WIP)
We can get a list of packages from /gscratch/com/local/bin on Ikt.
Add packages you want below!
- zsh
- Spark 2.4
- Python 3.7
- R 3.5.2
- moreutils 0.62
- emacs 25
- p7zip 16.02
- htop 2.2.0
- pandoc 2.2.1
RStudio Server
It could be nice to run an RStudio server on the interactive node to provide a nicer IDE for working interactively on hyak compared to Jupyter notebooks or editing in the terminal. If this isn't feasible then we should use kibo for this purpose instead.
Scheduler Options
It might be a good idea to ask the hyak folks to configure the scheduler for our partition so that we can request specific quantities of memory or cpus in our jobs. (Hyak Wiki)
