Editing Twitter analysis (CDSW) (section)

== Hints and New Material ==

This section lists a few new topics.

'''CSV Files'''

Comma-Separated Values (CSV) is a common way to store structured data in a text file. An example CSV file would be:

 Day,NumUsers
 2015-11-03,345
 2015-11-04,451
 ...

The nice thing about CSV files is that spreadsheet programs like Microsoft Excel can import them, so it's easy to plot columns. 

'''Dates and times'''

Python provides a rich datetime functionality available via <code>import datetime</code>. We encourage you to look at the documentation at [https://github.com/offbyone/tweetstream]. In our examples, the only function we'll use is the ability to create strings from datetimes. Everything we need is in this example.


<syntaxhighlight lang="python">
import datetime
for line in tweets:
    tweet_as_dictionary = json.loads(line)
    tweet_daytime = datetime.datetime.fromtimestamp(int(tweet_as_dictionary['timestamp_ms']) / 1000)
    tweet_day = tweet_daytime.strftime('%Y-%m-%d')
</syntaxhighlight>

The line <code>int(tweet_as_dictionary['timestamp_ms']) / 1000</code> does a few things:
* extract the timestamp_ms field from the tweet. This is the tweet time measured in number of milliseconds since Jan 1, 1970. See [https://en.wikipedia.org/wiki/Unix_time] for more details.
* divide the timestamp in milliseconds by 1000 to get the number of seconds since Jan 1, 1970.
* convert it to a datetime in python.

The other line that matters is <code>tweet_day = tweet_daytime.strftime('%Y-%m-%d')</code>, which takes the datetime and converts it to a string. The characters <code>%Y-%m-%d</code> describe the string format. The codes can be found in the documentation [https://docs.python.org/2/library/datetime.html#strftime-strptime-behavior]. For our purposes, here are the codes and the portion of the date they represent for the date "2015-11-06 14:52:12":

* %Y - 2015
* %m - 11
* %d - 06
* %H - 14 (note 24 hour time)
* %M - 52
* %S - 12