Editing CommunityData:CDSC Reddit
From CommunityData
Warning: You are not logged in. Your IP address will be publicly visible if you make any edits. If you log in or create an account, your edits will be attributed to your username, along with other benefits.
The edit can be undone. Please check the comparison below to verify that this is what you want to do, and then publish the changes below to finish undoing the edit.
Latest revision | Your text | ||
Line 1: | Line 1: | ||
The reddit_cdsc [https://code.communitydata.science/cdsc_reddit.git/ git repository] contains tools for working with Reddit data. | The reddit_cdsc [https://code.communitydata.science/cdsc_reddit.git/ git repository] contains tools for working with Reddit data. | ||
The project is designed for the hyak super computing system at The University of Washington. It consists of a set of python and bash scripts and uses [https://spark.apache.org/docs/latest/api/python/index.html pyspark] and [https://arrow.apache.org/docs/python/ pyarrow] to process large datasets. As of March 1st 2021, the project is under active development by [https://wiki.communitydata.science/People#Nathan_TeBlunthuis_.28University_of_Washington.29 Nate TeBlunthuis] and provides scripts for: | The project is designed for the hyak super computing system at The University of Washington. It consists of a set of python and bash scripts and uses the [https://spark.apache.org/docs/latest/api/python/index.html pyspark] and [https://arrow.apache.org/docs/python/ pyarrow] to process large datasets. As of March 1st 2021, the project is under active development by [https://wiki.communitydata.science/People#Nathan_TeBlunthuis_.28University_of_Washington.29 Nate TeBlunthuis] and provides scripts for: | ||
* Pulling and updating dumps from [https://pushshift.io Pushshift] in <code>dumps/pull_pushshift_comments.sh</code> and <code>dumps/pull_pushshift_submissions.sh</code>. | * Pulling and updating dumps from [https://pushshift.io Pushshift] in <code>dumps/pull_pushshift_comments.sh</code> and <code>dumps/pull_pushshift_submissions.sh</code>. |