Editing CommunityData:CDSC Reddit
From CommunityData
The edit can be undone. Please check the comparison below to verify that this is what you want to do, and then publish the changes below to finish undoing the edit.
Latest revision | Your text | ||
Line 1: | Line 1: | ||
The reddit_cdsc [https://code.communitydata.science/cdsc_reddit.git/ git repository] contains tools for working with Reddit data. | The reddit_cdsc [https://code.communitydata.science/cdsc_reddit.git/ git repository] contains tools for working with Reddit data. | ||
The project is designed for the hyak super computing system at The University of Washington. It consists of a set of python and bash scripts and uses [https://spark.apache.org/docs/latest/api/python/index.html pyspark] and [https://arrow.apache.org/docs/python/ pyarrow] to process large datasets. As of March 1st 2021, the project is under active development by [https://wiki.communitydata.science/People#Nathan_TeBlunthuis_.28University_of_Washington.29 Nate TeBlunthuis] and provides scripts for: | The project is designed for the hyak super computing system at The University of Washington. It consists of a set of python and bash scripts and uses the [https://spark.apache.org/docs/latest/api/python/index.html pyspark] and [https://arrow.apache.org/docs/python/ pyarrow] to process large datasets. As of March 1st 2021, the project is under active development by [https://wiki.communitydata.science/People#Nathan_TeBlunthuis_.28University_of_Washington.29 Nate TeBlunthuis] and provides scripts for: | ||
* Pulling and updating dumps from [https://pushshift.io Pushshift] in <code>dumps/pull_pushshift_comments.sh</code> and <code>dumps/pull_pushshift_submissions.sh</code>. | * Pulling and updating dumps from [https://pushshift.io Pushshift] in <code>dumps/pull_pushshift_comments.sh</code> and <code>dumps/pull_pushshift_submissions.sh</code>. |