Editing CommunityData:Hyak (section)

=== Parallelization Tips ===

The nodes on Mox have 28 CPU cores.  Our nodes on Klone have 40.  These may help in speeding up your analysis ''significantly''. If you are using R functions such as <code>lapply</code>, there are parallelized equivalents (e.g. <code>mclappy</code>) which can take advantage of all the cores and give you a 2800% or (4000)% boost! However, something to be aware of here is your code's memory requirement—if you are running 28 processes in parallel, your memory needs can also go up to 28x, which may be more than the ~200GB that the <code>big_machine</code> node on mox will have. In such cases, you may want to dial down the number of CPU cores being used—a way to do that globally in your code is to run the following snippet of code before calling any of the parallelized functions.

If you find yourself doing this often, consider if it is possible to reduce your memory usage via streaming, databases (like sqlite; parquet files; or duckdb), or lower-precision data types (i.e., use 32bit or even 16bit floating point numbers instead of the standard 64bit). 

<source lang="r">
library(parallel)
options(mc.cores=20)  ## tell the mc* functions to use 20 cores unless otherwise specified
mcaffinity(1:20)
</source> 

More information on parallelizing your R code can be found in the [https://stat.ethz.ch/R-manual/R-devel/library/parallel/doc/parallel.pdf <code>parallel</code> package documentation].