Editing CommunityData:TACC (section)

==== Starting the Cluster and Submitting Your Script ====

Now that we have configured Spark, we are ready to start the cluster and run jobs. 

{{note}} It is not recommended to run clusters through the Jupyter notebook or other interactive workflow.  Instead, develop and test your spark code with a sample of your data. Run the cluster when you're ready to scale out to an entire large dataset. So you'll need to connect via <code>ssh</code> before proceeding. 

Here are the steps to start the cluster and run your <code>pyspark</code> script on it. 

# Run <code>tmux</code> so that your cluster stays alive if your laptop disconnects from Stampede3.
# Use <code>idev</code> to checkout nodes. For example to checkout 2 icx nodes you can run <code>idev -N 2 -p icx -t 48:00:00</code>. When the job begins you'll land on the node that will be the Spark "master" node. This node runs the "driver" process which coordinates the cluster. It also runs a worker just like the other nodes.      
# Next, we need to tell spark which nodes to use. The <code>slurm_workers.sh</code> makes a list of nodes have been assigned to your job. Spark will read the <code>$SPARK_CONF_DIR/workers</code> file to know where to start the executors. Run <code>slurm_workers.sh > workers</code> to update the <code>workers</code> file with the nodes you have checked out.
# Start the cluster! Run <code>$SPARK_HOME/sbin/start-all.sh</code>
# Finally, you're ready to run your pyspark script. <code>$SPARK_HOME/bin/spark-submit/ --master spark://$(hostname):7077 my_script.py</code>.

{{note}} You can work with the spark cluster interactively by installing pyspark in your environment via <code>pip</code> or <code>uv</code> and then running <code>pyspark --master spark://$(hostname):7077</code>.