CommunityData:Hyak Spark

Apache Spark is a powerful system for writing programs that deal with large datasets. The most likely reason for using Spark on Hyak is that you run into memory limitations when building variables. For example suppose you want to compute:

• The number of prior edits a Wikipedia editor has made, for every editor and every edit.
• The number of views for every Wikipedia page for every month and every edit.
• The number of times every Reddit user has commented on every thread on every subreddit for every week.

You might try writing a program to build these variables using a data science tool like pandas in Python or data.table, or plyr in R. These common data science tools are powerful, expressive, and fast, but do not work when data does not fit in memory. When a table does not fit in memory, but the computation you want to do only requires operating on one row at a time (such as in a simple transformation or aggregation), you can often work around this limitation by writing a simple custom program that operates in a streaming fashion. However, when computation cannot be done one row at a time, such as in a sort, group by, or join a streaming solution will not work. In this case your options are limited. One option is writing bespoke code to perform the required operations and building variables. However, this can be technically challenging and time consuming work. Moreover, your eventual solution is likely to be relatively slow and difficult to extend or maintain compared to a solution build using Spark. A number of us (Nate, Jeremy, Kaylea) have all at some point written bespoke code for computing user-level variables on Wikipedia data. The infamous "million file problem" is a result from abusing the filesystem to perform a massive group by.

This page will help you decide if you should use Spark on Hyak for your problem and provide instructions on how to get started.

Pros and Cons of Spark

The main advantages of Spark on Hyak:

1. Work with "big data" without ever running out of memory.
2. You get very good parallelism for free.
3. Distribute computational work across many hyak nodes so your programs run faster.
4. Common database operations (select, join, groupby, filter) are pretty easy.
5. Spark supports common statistical and analytical tasks (stratified sampling, summary and pairwise statistics, common and simple models).
6. Spark is a trendy technology that lots of people know or want to learn.

The main disadvantages of Spark are

1. It takes several steps to get the cluster up and running.
2. The programming paradigm is not super intuitive, especially if you are not familiar with SQL databases or lazy evaluation.
3. Effectively developing your spark code means getting it setup on your own laptop, which isn't trivial.
4. Doing more advanced things requires programming in Scala.