Community Data Science Workshops (Fall 2015)/Day 3 Lecture

Welcome to the Saturday lecture section of the Community Data Science Workshop Session 3! For about 140 minutes, we'll work through an example of a Python program end-to-end that answers of simple questions using data from the Wikipedia API via both a lecture and hand-on exercises.

Resources

• Lecture Recording/Screencast — The file is in OGV/Theora format. If you have trouble playing it, you can install the free software VLC software which runs on Mac OSX, Windows and Linux and should be able to play the video. Keep in mind that there is lots of "dead" time where folks are working on things and getting help from mentors.

Material for the lecture

For the lecture, you will need two files. Download both of these to your computer by using right or control click on the link and then using Save as or Save link as. Keep track of where you put the files.

• http://mako.cc/teaching/2015/cdsw-autumn/harrypotter-wikipedia-cdsw.zip
• http://communitydata.cc/~mako/hp_wiki.tsv

Overview of the day

• Lecture
  • Our philosophy around data visualization
  • Introduce some new programming tools!
  • We're going to walk through some analysis of edits to Harry Potter in Wikipedia, start to finish
  • We'll focus on manipulating data in Python
  • Visualizing things in Google Docs
• Lunch (vegetarian Greek!)
• Project based work
  • Wikipedia
  • Twitter
  • Data.seattle.gov
  • Matplotlib
• Wrap-up!

Lecture outline

Step 1: Pre-Requisites

• My philosophy about data analysis: use the tools you have
• Four things in Python I have to teach you now and one more thing later):
  • while loops
    • infinite loops
    • loops with a greater than or less than
  • break / continue
  • "\t".join()
  • defining your own functions with def foo(argument): and return bar
  • The .update() function that is associated with dictionaries.

Step 2: Walking through a Program

• Walk-through of get_hpwp_dataset.py
• Look at dataset with more and/or in spreadsheet

Step 3: Loading Data Back In

• Load data into Python
  • review of opening files
    • we can also open them for reading with open('file', 'r', encoding="utf-8")
  • csv.DictReader()
• Basic counting: hpwp-minor.py
  • Answer question: What proportion of edits to Wikipedia Harry Potter articles are minor?
    • Count the number of minor edits and calculate proportion
• Looking at time series data hpwp-trend.py
  • "Bin" data by day to generate the trend line
• Exporting and visualizing data
  • Export dataset on edits over time
  • Export dataset on articles over users
  • Load data into Google Docs