Editing Twitter analysis (CDSW) (section)

== Details == 

'''Sampling'''

Since the raw data is too large for everyone to download locally, we had to down sample to a reasonable sub-set. Down sampling usually involves choosing a random subset of the raw data, but it's important to think about how that sample is constructed. For instance, consider two sampling criteria for a 1% sample: by tweet and by user. Sampling by tweet means that every single tweet has a 1 in 100 chance of being in the sample. However, what if I want to count the number of tweets per user? Since I don't have all the tweets for any particular user, my estimate is going to be wrong. Sampling by user lets me capture all of the tweets by an account or none of them. This way I can still estimate the number of tweets in a time period and I can also make valid measurements of per-user metrics.

'''Full source code'''

The full source code of this exercise is available on github at [https://github.com/offbyone/tweetstream]. The examples are in the section <code>students</code>