CommunityData:Hyak walkthrough

This file provides a complete, step-by-step walkthrough for how to parse a list of Wikia wikis with wikiq. The same principles can be followed for other tasks.

Setup steps (only need to be done once)

 * 1) Create a users directory for yourself in /gscratch/comdata/users
 * 2) * You will want to store the output of your script in /gscratch/comdata/, or you will run out of space in your personal filesystem (/usr/lusers/...)
 * 3) Create a batch_jobs directory
 * 4) Create a symlink from your home directory to this directory (this lets you use the /gscratch/comdata storage from the more convenient home directory)
 * 5) Create a user in parallel SQL
 * 1) Create a symlink from your home directory to this directory (this lets you use the /gscratch/comdata storage from the more convenient home directory)
 * 2) Create a user in parallel SQL
 * 1) Create a user in parallel SQL

Project-specific steps (done for each project)

 * 1) Create a new project in your batch_jobs directory
 * 2) Create a symlink to the data that you will be using as an input (in this case, the 2010 wikia dump)
 * 3) Create an output directory
 * 4) Test to make sure everything is working well, and everything is where it should be, run wikiq on one file
 * 5) * This should provide some output in the terminal, and should create a file at ~/batch_jobs/wikiq_test/output/012thfurryarmybrigade.tsv. You should examine this file to make sure it looks as expected
 * 6) When you're done, remove it
 * 7) Now we'll use that command as a template for creating a task_list. This is a file with a line for each command we would like our job to run. In this case, we'll use the terminal to find a list of all of the wiki files, which we will pipe to xargs. xargs takes each file name, and uses echo to insert it into the command. Each line is then written to the task_list file.
 * 8) * This will create a file named task_list. * Note: this will take a while - approx. 1 minute.
 * 9) Make sure it is as large as expected (it should have 76471 lines)
 * 10) * You can also visually inspect it, to make sure that it looks like it should.
 * 11) Copy this job_script to your wikiq_test directory
 * 12) Edit the job_script. https://sig.washington.edu/itsigs/Hyak_parallel-sql has a good example script, with explanations for what each piece does. For our project, you should just change where it says USERNAME to your user name
 * 13) * You can do this with vim, or you can just run the following:
 * 14) *   The other part of this file that you will often have to change is the walltime. This is how long you want to have the node assigned to your job. For long jobs, you will need to increase this parameter.
 * 15) Load up 100 tasks into Parallel SQL, as a test. You want to make sure that everything is working end-to-end before trying it on the whole set of files.
 * 16) *   Check to make sure that they loaded correctly (they should show up as 100 available jobs)
 * 17) Check to make sure there are available nodes
 * 18) *   We have 8 nodes currently, so subtract the number of active jobs from 8, and that is the number of available nodes.
 * 19) Run the jobs on the available nodes.
 * 20) Make sure things are working correctly
 * 21) *   This lets you watch to make sure that your jobs are assigned to nodes correctly. Once they are assigned, Ctrl+c gets you out of watch, and you can watch the task list in Parallel SQL
 * 22) *	This lets you watch the task list. You should see the tasks move from available to completed. When they are all completed, ru
 * 23) *	This checks to make sure all 100 files were written to the output folder. You probably want to also look at a few files, to make sure they look as expected.
 * 24) *	If everything looks good, then remove the output files
 * 25) *	and clean up the parallel SQL DB
 * 26) Finally, run the jobs over the full set of files
 * 27) *	Keep an eye on the tasks with
 * and
 * 1) *   Check to make sure that they loaded correctly (they should show up as 100 available jobs)
 * 2) Check to make sure there are available nodes
 * 3) *   We have 8 nodes currently, so subtract the number of active jobs from 8, and that is the number of available nodes.
 * 4) Run the jobs on the available nodes.
 * 5) Make sure things are working correctly
 * 6) *   This lets you watch to make sure that your jobs are assigned to nodes correctly. Once they are assigned, Ctrl+c gets you out of watch, and you can watch the task list in Parallel SQL
 * 7) *	This lets you watch the task list. You should see the tasks move from available to completed. When they are all completed, ru
 * 8) *	This checks to make sure all 100 files were written to the output folder. You probably want to also look at a few files, to make sure they look as expected.
 * 9) *	If everything looks good, then remove the output files
 * 10) *	and clean up the parallel SQL DB
 * 11) Finally, run the jobs over the full set of files
 * 12) *	Keep an eye on the tasks with
 * and
 * 1) *	This checks to make sure all 100 files were written to the output folder. You probably want to also look at a few files, to make sure they look as expected.
 * 2) *	If everything looks good, then remove the output files
 * 3) *	and clean up the parallel SQL DB
 * 4) Finally, run the jobs over the full set of files
 * 5) *	Keep an eye on the tasks with
 * and
 * 1) *	Keep an eye on the tasks with
 * and
 * 1) *	Keep an eye on the tasks with
 * and
 * 1) *	Keep an eye on the tasks with
 * and