Editing CommunityData:Hyak Datasets
From CommunityData
Warning: You are not logged in. Your IP address will be publicly visible if you make any edits. If you log in or create an account, your edits will be attributed to your username, along with other benefits.
The edit can be undone. Please check the comparison below to verify that this is what you want to do, and then publish the changes below to finish undoing the edit.
Latest revision | Your text | ||
Line 6: | Line 6: | ||
We maintain an archive of Reddit submissions and comments going back to Reddit's early history that is up-do-date with January 2019 (for comments) and August 2019 (for submissions). We have copies of dumps collected and published by [https://Pushshift.io pushshift] and tabular datasets derived from them. Compared to obtaining data from the Reddit (or pushshift) APIs, working with these archival datasets will be faster and less work for you. The tabular datasets in particular are quite fast thanks to the parquet file format making it possible to pull subsets of the data (e.g. complete history of a subreddit) in as little as 15 minutes. In contrast it takes about a day to extract and parse the dumps on a mox node. | We maintain an archive of Reddit submissions and comments going back to Reddit's early history that is up-do-date with January 2019 (for comments) and August 2019 (for submissions). We have copies of dumps collected and published by [https://Pushshift.io pushshift] and tabular datasets derived from them. Compared to obtaining data from the Reddit (or pushshift) APIs, working with these archival datasets will be faster and less work for you. The tabular datasets in particular are quite fast thanks to the parquet file format making it possible to pull subsets of the data (e.g. complete history of a subreddit) in as little as 15 minutes. In contrast it takes about a day to extract and parse the dumps on a mox node. | ||
For computational efficiency it is best to parse the dumps as little as possible. So if it is possible for you to work with the tabular datasets, please do so. The tabular datasets currently have the variables that most projects will want to use, but there are many other metadata variables including ones related to moderation, media, Reddit gold and more. If you want a variable from the pushshift json that isn't in parquet tables, don't fret! It will not be too much work to add it. Reach out to [[User:Groceryheist|Nate]]. | For computational efficiency it is best to parse the dumps as little as possible. So if it is possible for you to work with the tabular datasets, please do so. The tabular datasets currently have the variables that most projects will want to use, but there are many other metadata variables including ones related to moderation, media, Reddit gold and more. If you want a variable from the pushshift json that isn't in parquet tables, don't fret! It will not be too much work to add it. Reach out to [[User:Groceryheist|Nate]]. | ||
Line 23: | Line 21: | ||
</code> | </code> | ||
" | "`by_author`" and "`by_subreddit`" refer to how the data is sorted. Sorting the data makes filtering by the sorted column fast and determines the order that the data will be read. Sorting by author makes it possible to stream the dataset one user at a time to build user-level variables without resorting to Spark. | ||
=== Reading Reddit parquet datasets === | === Reading Reddit parquet datasets === | ||
The recommended way to pull data from parquet on Hyak is to use [https://arrow.apache.org/docs/python/ pyarrow], which makes it relatively easy to filter the data and load it into Pandas. The main alternative is [[CommunityData:Hyak_Spark| Spark]], which is a more complex | The recommended way to pull data from parquet on Hyak is to use [https://arrow.apache.org/docs/python/ pyarrow], which makes it relatively easy to filter the data and load it into Pandas. The main alternative is [[CommunityData:Hyak_Spark| Spark]], which is a more complex system, but can read and write parquet and is useful for working with data that is too large to fit in memory. | ||
This example loads all comments to the Seattle subreddit. You should try it out! | This example loads all comments to the Seattle subreddit. You should try it out! | ||
<syntaxhighlight lang='python'> | <syntaxhighlight lang='python'> | ||
import pyarrow as pa | |||
import pyarrow.parquet as pq | |||
import pyarrow.dataset as ds | import pyarrow.dataset as ds | ||
import pathlib | |||
# A pyarrow dataset abstracts reading, writing, or filtering a parquet file | # A pyarrow dataset abstracts reading, writing, or filtering a parquet file | ||
dataset = ds.dataset(pathlib.Path('/gscratch/comdata/output/reddit_comments_by_subreddit.parquet/'), format='parquet', partitioning='hive') | |||
dataset = ds.dataset('/gscratch/comdata/output/ | |||
# let's get all the comments to two subreddits: | # let's get all the comments to two subreddits: | ||
subreddits_to_pull = ['seattle','seattlewa'] | subreddits_to_pull = ['seattle','seattlewa'] | ||
# a table is a low-level structured data format. | # a table is a low-level structured data format. The data still has not been read into memory. | ||
table = dataset.to_table(filter = ds.field('subreddit').isin( | table = dataset.to_table(filter = ds.field('subreddit').isin(subreddits_to_track), columns=['id','subreddit','link_id','parent_id','CreatedAt','author','ups','downs','score','subreddit_id','stickied','is_submitter','body']) | ||
# Since data from just these 2 subreddits fits in memory we can just turn our table into a pandas dataframe. | # Since data from just these 2 subreddits fits in memory we can just turn our table into a pandas dataframe. | ||
df = table.to_pandas() | df = table.to_pandas() | ||
# We should save this smaller dataset so we don't have to wait 15 min to pull from parquet next time. | |||
df.to_csv("mydataset.csv") | |||
</syntaxhighlight> | </syntaxhighlight> | ||
Parquet is a [[:en:Column-oriented_DBMS|column-oriented format]] which means that it is capable of reading each column independently of others. This confers two key advantages compared to unstructured formats that can make it very fast. First, the `filter` runs only on the `subreddit` column to figure out what rows need to be read for the other fields. Second, only the columns that are selected in `columns=` need to be read at all. This is how arrow can pull data from parquet so fast. | |||