Editing CommunityData:Hyak
From CommunityData
The edit can be undone. Please check the comparison below to verify that this is what you want to do, and then publish the changes below to finish undoing the edit.
Latest revision | Your text | ||
Line 149: | Line 149: | ||
Hyak has a special way of scheduling jobs using the '''checkpoint queue'''. When you run jobs on the checkpoint queue, they run on someone else's hyak node that they aren't using right now. This is awesome as it gives us a huge amount of free (as in beer) computing. But using the checkpoint queue does take some effort, mainly because your jobs can get killed at any time if the owner of the node checks it out. So if you want to run a job for more than a few minutes on the checkpoint queue it will need to be able to "checkpoint" by saving it's state periodically and then restarting. | Hyak has a special way of scheduling jobs using the '''checkpoint queue'''. When you run jobs on the checkpoint queue, they run on someone else's hyak node that they aren't using right now. This is awesome as it gives us a huge amount of free (as in beer) computing. But using the checkpoint queue does take some effort, mainly because your jobs can get killed at any time if the owner of the node checks it out. So if you want to run a job for more than a few minutes on the checkpoint queue it will need to be able to "checkpoint" by saving it's state periodically and then restarting. | ||
This would be a pain to do manually, fortunately, we have <code>[http://dmtcp.sourceforge.net/FAQ.html dmtcp] </code> which can automatically checkpoint and resume most programs. | |||
Nate's working got dmtcp working for arbitrary scripts, and also with wikiq using parallel_sql. | |||
dmtcp 3.0 is installed on Mox. | |||
This will make more sense if you know that dmtcp works by starting a '''coordinator''' process which is responsible for pausing and saving the checkpointed process. A [https://hpcc.usc.edu/support/documentation/checkpointing/ tutorial on dmtcp with slurm from USC] has a bash function for starting the coordinator called <code>start_dmtcp_coordinator</code>. Nate added this function to the shared .bashrc. So it should be available in your environment on Mox. | |||
==== Starting a checkpoint queue job ==== | ==== Starting a checkpoint queue job ==== |