Editing Community Data Science Workshops (Spring 2016)/Day 3 Projects/Twitter
From CommunityData
The edit can be undone. Please check the comparison below to verify that this is what you want to do, and then publish the changes below to finish undoing the edit.
Latest revision | Your text | ||
Line 8: | Line 8: | ||
== Analyzing Tweet Data with Python == | == Analyzing Tweet Data with Python == | ||
Last week, the [[Community_Data_Science_Workshops_%28Spring_2016%29/Day_2_projects/Twitter|Twitter API Session]] covered accessing data from the Twitter API, including accessing streaming data. After the session, we set up a streaming collection from the code you modified in the workshop to track all earthquake-related tweets. That stream captured 3.5GB of tweets (!). We've given you a sample of those tweets in the zipped file above. | |||
Our goal in this workshop is to use those tweets to answer questions. As always, we've suggested questions below, but the real goal is for you to find something that you find interesting from the data. | |||
Our goal in this workshop is to | |||
Your data consists of 1 Tweet per line encoded in JSON exactly as Twitter returns it. | Your data consists of 1 Tweet per line encoded in JSON exactly as Twitter returns it. | ||
Line 78: | Line 74: | ||
Since the raw data is too large for everyone to download locally, we had to down sample to a reasonable sub-set. Down sampling usually involves choosing a random subset of the raw data, but it's important to think about how that sample is constructed. For instance, consider two sampling criteria for a 1% sample: by tweet and by user. Sampling by tweet means that every single tweet has a 1 in 100 chance of being in the sample. However, what if I want to count the number of tweets per user? Since I don't have all the tweets for any particular user, my estimate is going to be wrong. Sampling by user lets me capture all of the tweets by an account or none of them. This way I can still estimate the number of tweets in a time period and I can also make valid measurements of per-user metrics. | Since the raw data is too large for everyone to download locally, we had to down sample to a reasonable sub-set. Down sampling usually involves choosing a random subset of the raw data, but it's important to think about how that sample is constructed. For instance, consider two sampling criteria for a 1% sample: by tweet and by user. Sampling by tweet means that every single tweet has a 1 in 100 chance of being in the sample. However, what if I want to count the number of tweets per user? Since I don't have all the tweets for any particular user, my estimate is going to be wrong. Sampling by user lets me capture all of the tweets by an account or none of them. This way I can still estimate the number of tweets in a time period and I can also make valid measurements of per-user metrics. | ||
'''Full source code''' | '''Full source code''' | ||
The full source code of this exercise is available on github at [https://github.com/offbyone/tweetstream]. The examples are in the section <code>students</code> | The full source code of this exercise is available on github at [https://github.com/offbyone/tweetstream]. The examples are in the section <code>students</code> |