CommunityData:Hyak tutorial

This file provides a complete, step-by-step walk-through for how to parse a list of Wikia wikis with wikiq. The same principles can be followed for other tasks.

Things you should know before you start

• Computing paradigms: wikipedia:High performance computing versus MapReduce and the Hadoop
• ikt versus mox and the transition
  • This material will cover getting setup on the older ikt cluster
  • Our mox cluster is online and we will migrating to it in late 2019/early 2020

Setup steps (only need to be done once)

Create a users directory for yourself in /com/users:

You will want to store the output of your script in /com/, or you will run out of space in your personal filesystem (/usr/lusers/...)

$ mkdir /com/users/USERNAME  # Replace USERNAME with your user name

2. Create a batch_jobs directory

$ mkdir /com/users/USERNAME/batch_jobs

3. Create a symlink from your home directory to this directory (this lets you use the /com storage from the more convenient home directory)

$ ln -s /com/users/USERNAME/batch_jobs ~/batch_jobs

4. Create a user in parallel SQL

$ module load parallel_sql
$ sudo pssu --initial
$ [sudo] password for USERID: <Enter your UW NetID password>