Editing CommunityData:Hyak Spark
From CommunityData
Warning: You are not logged in. Your IP address will be publicly visible if you make any edits. If you log in or create an account, your edits will be attributed to your username, along with other benefits.
The edit can be undone. Please check the comparison below to verify that this is what you want to do, and then publish the changes below to finish undoing the edit.
Latest revision | Your text | ||
Line 64: | Line 64: | ||
== Spark Walkthrough == | == Spark Walkthrough == | ||
Spark | Spark programs is somewhat different from normal python programming. This section will walk you through a script to help you learn how to work with Spark. You may find this script useful as a template for building variables on top of [[wikiq]] data. | ||
This section presents a pyspark program that | This section presents a pyspark program that | ||
Line 316: | Line 316: | ||
=== Monitoring the cluster === | === Monitoring the cluster === | ||
From a login node ( | From a login node (ikt): | ||
<syntaxhighlight lang="bash"> | <syntaxhighlight lang="bash"> | ||
Line 325: | Line 325: | ||
<syntaxhighlight lang="bash"> | <syntaxhighlight lang="bash"> | ||
ssh -L localhost:8989:localhost:8989 | ssh -L localhost:8989:localhost:8989 ikt -N -f && ssh -L localhost:4040:localhost:4040 ikt -n -F | ||
</syntaxhighlight> | </syntaxhighlight> | ||