Editing CommunityData:Klone (section)

== Spark == 

To set up a spark cluster using apptainer the first step to "run" the container on each node in the cluster:

<syntaxhighlight lang='bash'>
# on the first node
apptainer instance start --fakeroot cdsc_base.sif spark-boss
export SPARK_BOSS=$(hostname)
# on the first worker node (typically same as boss node)
apptainer instance start --fakeroot cdsc_base.sif spark-worker-1
# second worker node
apptainer instance start --fakeroot cdsc_base.sif spark-worker-2
</syntaxhighlight>

The second step is to start the spark services on the instances

<syntaxhighlight lang='bash'>
apptainer exec instance://spark-boss /opt/spark/sbin/start_master.sh

apptainer exec instance://spark-worker-1 /opt/spark/sbin/start-worker.sh $SPARK_BOSS:7077
</syntaxhighlight>

That should be it. Though in practice it might make more sense to have special containers for the spark boss and workers.

You can now submit spark jobs by running <code>spark-submit.sh</code>.

<syntaxhighlight lang='bash'>
# replace n3078 with the master hostname
apptainer exec instance://spark-boss /opt/spark/bin/spark --master spark://n3078.hyak.local:7077
</syntaxhighlight>

Nate's working on wrapping the above nonsense in friendlier scripts.