Editing Community Data Science Course (Spring 2015)

From CommunityData

Warning: You are not logged in. Your IP address will be publicly visible if you make any edits. If you log in or create an account, your edits will be attributed to your username, along with other benefits.

The edit can be undone. Please check the comparison below to verify that this is what you want to do, and then publish the changes below to finish undoing the edit.

Latest revision Your text
Line 5: Line 5:
:'''Course Catalog Description:'''
:'''Course Catalog Description:'''
:This course will introduce basic programming and data science tools to give students the skills to use data to answer questions about social media and online communities. The class will cover the basics of the Python programming language, an introduction to web APIs including APIs from Wikipedia and Twitter, and will teach basic tools and techniques for data analysis and visualization. As part of the class, participants will learn to write software in Python to collect data from web APIs and process that data to produce numbers, hypothesis tests, tables, and graphical visualizations that answer questions like: Are new contributors in Wikipedia this year sticking around longer or contributing more than people who joined last year? Who are the most active or influential users of a particular Twitter hashtag? Are people who join through a Wikipedia outreach event staying involved? How do they compare to people who decide to join the project outside of the event? The class will be built around student-designed independent projects and is targeted at students with absolutely no previous programming experience.
:This course will introduce basic programming and data science tools to give students the skills to use data to answer questions about social media and online communities. The class will cover the basics of the Python programming language, an introduction to web APIs including APIs from Wikipedia and Twitter, and will teach basic tools and techniques for data analysis and visualization. As part of the class, participants will learn to write software in Python to collect data from web APIs and process that data to produce numbers, hypothesis tests, tables, and graphical visualizations that answer questions like: Are new contributors in Wikipedia this year sticking around longer or contributing more than people who joined last year? Who are the most active or influential users of a particular Twitter hashtag? Are people who join through a Wikipedia outreach event staying involved? How do they compare to people who decide to join the project outside of the event? The class will be built around student-designed independent projects and is targeted at students with absolutely no previous programming experience.
== Overview and Learning Objectives ==
<div style="float:right;">__TOC__</div>
In a world that is increasingly driven by software and data, developing a basic level of fluency with programming and the basic tools of data analysis is a crucial skill. This course will introduce basic programming and data science tools to give students the skills to use data to answer questions about social media and online communities.
In particular, the class will cover the basics of the Python programming language, an introduction to web APIs including APIs from Wikipedia and Twitter, and will teach basic tools and techniques for data analysis and visualization. As part of the class, participants will learn to write software in Python to collect data from web APIs and process that data to produce numbers, hypothesis tests, tables, and graphical visualizations that answer real questions. The class will be built around student-designed independent projects. Every student will pick a question or issue they are interested in pursuing in the first week and will work with the instructor to build from that question toward a completed analysis of data that the student has collected using software they have written.
This is not a computer science class and I am not going to be training you to become professional programmers. This introduction to programming is intentionally quick and dirty and is focused on what you need to get things done. If you want to become a professional programmers, this is probably not the right class. If you want to learn about programming so that you can more effectively answer questions about social media by writing your own software and by managing and communicating more effectively with programmers, you are in the right place.
I will consider this class a complete success if, at the end, every student can:
* Write or modify a program to collect a dataset from the Wikipedia and Twitter APIs.
* Effectively read web API documentation and write Python software to parse and understand a new and unfamiliar JSON-based web API.
* Use both Python based tools like MatPlotLib as well as tools like LibreOffice, Google Docs, or Microsoft Excel to effectively graph and analyze data.
* Use web-based data to effective answer a substantively interesting question and to present this data effectively in the context of both a formal presentation and a written report.
* Work effectively with a professional programmer (e.g., on Elance/Odesk) to define a set of realistic requirements for an API-based data collection programming project.


== Note About This Syllabus ==  
== Note About This Syllabus ==  
Line 31: Line 14:
# I will ask the class for voluntary anonymous feedback frequently — especially toward the beginning of the quarter. Please let me know what is working and what can be improved. In the past, I have made many adjustments based on this feedback.
# I will ask the class for voluntary anonymous feedback frequently — especially toward the beginning of the quarter. Please let me know what is working and what can be improved. In the past, I have made many adjustments based on this feedback.


== Books ==
== Overview and Learning Objectives ==  
 
This class is going to be a studio and project based class. Although we will not rely very heavily on readings or discussing them in depth in class, I'm strongly recommending two books that will cover the material we go over in class and which will provide a reference work for you to refer to:
 
# '''[http://www.pythonlearn.com/book.php Python for Informatics: Exploring Information]''' by Charles Severance. The book is available online for free but you can also buy a physical copy of the book [http://www.amazon.com/gp/product/1492339245/ref=as_li_ss_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=1492339245&linkCode=as2&tag=drchu02-20 from Amazon] or get an electronic copy from the [http://www.amazon.com/dp/B00K0O8HFQ Kindle Store]. According to the book's website: "The goal of this book is to provide an Informatics-oriented introduction to programming. The primary difference between a computer science approach and the Informatics approach taken in this book is a greater focus on using Python to solve data analysis problems common in the world of Informatics."
# '''[http://shop.oreilly.com/product/0636920023784.do Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython]''' by Wes McKinney. According to the website: "Python for Data Analysis is concerned with the nuts and bolts of manipulating, processing, cleaning, and crunching data in Python. It is also a practical, modern introduction to scientific computing in Python, tailored for data-intensive applications." The book is available for purchase in print and non-DRM ebook directly from the O'Reilly website.
 
== General Notes ==
 
* I expect you to come to class every day ''with your own laptop''. Windows, Mac OS and Linux are all fine but a tablet is not going to cut it. We're going to install software during the class and you'll be working on projects for homework so please bring the same laptop each time.
* Finally, William Hale will be acting as an in-class student assistant and mentor for the class. William is an undergraduate student in communication who is working with me on research into online communities and who has a strong substantive interest in social media and the kinds of communities we'll study. Much of the class will be project-based and William and I will be available to help you through challenges you encounter in this work during class. If you have questions and need to reach to somebody outside of class, however, please reach out to me!


== Assignments ==
The assignments in this class are designed to give you an opportunity to try your hand at using the technical skills that we're covering in the class. There will be no exams or quizzes. There will be weekly assignments that I will ask you to hand-in but will only be graded as ''complete/incomplete''.
=== Final Project Idea ===
:'''Maximum Length:''' 600 words (~2 pages double spaced)
:'''Due Date:''' April 13
:'''Drop box:''' [[https://canvas.uw.edu/courses/963931/assignments/2816619 Turn in on Canvas]]
In this assignment, you should concisely identify an community that you are interested in a source of data and/or and a list of at least 3-4 questions you might be interested in answering in the context of your final project. I am hoping that each of you will pick an area or domain that you are intellectually committed to and invested in (e.g., in your business or personal life). You will be successful if you describe the scope of the problem and explain why you are interested in using community data science methods.
If you are unsure, asking a question about Wikipedia is probably among the safer paths.
I will give you feedback on these write-ups and will let you each know if I think you have identified a questions that might be too ambitious, too trivial, too broad, too narrow, etc.
=== Final Project Proposal ===
:'''Maximum Length:''' 1500 words (~5 pages)
:'''Due Date:''' May 4th (at 6pm)
Building on your project idea assignment, you should describe the specific types of data you will collect, the steps you will take to collect the dataset, the limits and strength of these data for answering the question you have selected, and a description of the kinds of report and visualization you will make.
And important step here is going to be ''framing'' your analysis. Why is this is an important question? Why do you care? What do we need to know (e.g., about the question, about underlying theories, about your business, about the topic, about the community) to understand this analysis? This will all need to be part of your final project and it's good opportunity to do this, or at least being doing it, here.
I will give you feedback on these proposals and suggest changes or modifications that are more likely to make them successful or compelling and to work with you to make sure that you have the resources and support necessary to carry out your project successfully.
=== Final Project ===
:'''Presentation Date:''' June 2
:'''Paper Due Date:''' June 12
For your final project, I expect you to build on the first two assignments to describe what they have done and what you have found. I'll expect every student to give both:
# A short presentation to the class (10 minutes)
# A final report that is not more than 4500 words (~18 pages)
I expect that your reports will include text from the first two assignments and reflect comprehensive documentation of your project. Each project should include: (a) the description of the question and community you have identified and information necessary to frame your question, (b) a description of the how you collected your data, (c) the results.
A successful project will tell a compelling story and will engage with, and improve upon, the course material to teach an audience that includes me, your classmates, and Comm Lead students taking this class in future years, how to take advantage of community data science more effectively. The very best papers will give us all a new understanding of some aspect of course material and change the way I teach some portion of this course in the future.
==== Paper and Code ====
Your final project should include detailed information on:
* The problem or area you have identified and enough background to understand the rest of your work and its importance or relevance.
* Your research question(s) and/or hypotheses.
* The methods, data, and approach that you used to collect the data plus information on why you think this was appropriate way to approach your question(s).
* The results and findings including numbers, tables, graphics, and figures.
* A discussion of limitations for your work and how you might improve them.
If you want inspiration for how people use data science to communicate this kinds of findings broadly and effectively, take a look at great sources of data journalism including [http://fivethirtyeight.com/ Five Thirty Eight] or [http://www.nytimes.com/upshot/ The Upshot at the New York Times]. Both of these publish an large amount of excellent examples of data analysis aimed at broader non-technical audiences like the ones you'll be communicating with and quite a bit of their work is actually done using Python and web APIs!  A simple Five Thirty Eight story will include a clear question, a brief overview of the data sources and method, a figure or two plus several paragraphs walking through the results, followed by a nice conclusion. I'm asking you to try to produce something roughly like this.
Keep in mind that most stories on Five Thirty Eight are under 1000 words and I'm giving up to 4,5000 words to show me what you've learned. As a result, you should do ''more'' than FiveThirtyEight does in a single story. You can ask and answer more questions, you can provide more background, context, and justification, you can provide more details on your methods and data sources, you can show us more graphs, you can discuss the implications of your findings more. You to use the space I've given you to show off what you've done and what you've learned!
Finally, you should also share with me the full Python source code you used to collect the data and the dataset itself. Keep in mind that I will not be judging the quality or quantity of your code but rather the degree to which you have been successful at answering the ''substantive'' questions you have identified.
==== Presentation ====
Your presentation should do everything that your paper does and should provide me with a very clear idea of what to expect in your final paper. I'm going to give you all at least a paragraph of feedback after your talk. This will be an opportunity for me to see a preview of your paper and give you a sense for what I think you can improve. It's too your advantage to both give a compelling talk and to give me a sense for your project.
;Timing: All presentations will need to be '''a maximum of 7 minutes long'' with additional 2-3 minutes for questions and answers. Timing is going to be tight and I'm going to set an alarm and stop presentations that go too long.
;Presentation Order: You '''must''' sign up for a presentation slot by editing [https://docs.google.com/spreadsheets/d/1P_saUgq1UEjg42KXRDXPa5pdaLTOKVUf2evEwmQWrUY/edit#gid=0 this spreadsheet]. If you are not on the sheet by Monday June 1st at 12:00pm, I will add you.
;Slides: You are encouraged to use slides for your talk but I will need your slides ahead of class. If you want to submit slides, you must upload slides in PDF format to [https://canvas.uw.edu/courses/963931/assignments/2816622 the assignment page in Canvas] by 12:00pm on Monday June 1st. I'm going to get everything in order on my laptop before class so we can make quick transitions. Because time will be very tight, if you do not submit slides, or if you submit them late, you will not be able to use slides for your talk. There will not be time in class for me to able to load your slides onto the computer.
=== Participation ===
The course relies heavily on participation. The material we're going to be covering is difficult and we're going to be covering it quickly. It is going to be extremely difficult to make up any missed classes. Attendance will be the most important part of participation and missing more than 1 class is going got make it extremely difficult to excel in our class.
Nearly every week, we will begin by discussing challenges and problem sets that we'll define as a group at the end of the previous class. Please speak up and engage in this part of the class as well as asking questions anytime there is anything confusing.
The "Participation Rubric" section of my [http://mako.cc/teaching/assessment.html detailed page on assessment] gives the rubric I tend to use in evaluating participation.
=== Grading ===
I have put together a very detailed page that describes [the grading rubric] we will be using in this course. Please read it carefully I will assign grades for each of following items on the UW 4.0 grade scale according to the weights below:
* Participation: 30%
* Final project idea: 5%
* Final project proposal 10%
* Final project presentation: 15%
* Final paper: 40%
=== Weekly Coding Challenges ===
Each week I will give you all a set of weekly coding challenges before the end of class that will involve changing or adding to code that I've given you as part of the projects in the final parts of class to solve new problems. These coding challenges '''will not be turned in''' and '''will not be graded'''.
I will share my solutions answers to each of the coding challenges by Monday morning of class in a Canvas discussion threads. As you will see over the course of the quarter, there are many possible solutions to many programming problems and my own approaches will often be different than yours. That's completely fine! Coding is a creative act!
Please do not share answers to challenges before midnight on Sunday so that everybody has a chance to work through answers on their own. After midnight on Sunday, you are all welcome to share your solutions and/or to discuss different approaches.  We will discuss the coding challenges for a short period of time at the beginning of each class.


== Schedule ==
== Schedule ==
Line 137: Line 21:
=== Week 1: March 30 ===
=== Week 1: March 30 ===


'''Readings:'''
''Daily Goals:'''
 
* Python for Informatics: [http://www.pythonlearn.com/html-009/book001.html Preface] and [http://www.pythonlearn.com/html-009/book002.html Chapter 1  Why should you learn to write programs?]
 
'''Class Schedule:'''


* Quick introductions — Be ready to introduce yourself and describe your interest and goals in the class.
* Quick introductions — Be ready to introduce yourself and describe your interest and goals in the class.
* Class overview and expectations — We'll walk through this syllabus.
* Class overview and expectations —  
* [[Community Data Science Course (Spring 2015)/Day 1 Exercise|Installation and setup]] — You'll install software including the Python programming language and run through a series of exercises.
* [[Community Data Science Course (Spring 2015)/Day 1 Exercise|Self-guided tutorial and exercises]] — You'll work through a self-guided tutorial introducing you to some basic concepts. When you're done, you'll meet with a member of the teaching team and we'll check you off.
 
'''Resources:'''


* [[Community Data Science Course (Spring 2015)/Day 1 Plan|Day 1 Plan]]
introduction, setup. installation, variables, looping, conditionals


=== Week 2: April 6 ===
=== Week 2: April 6 ===


'''Assignment Due:'''
lecture overview, dictionaries, using functions, for loops;


* Finish setup, tutorial and code academy in the [[Community Data Science Course (Spring 2015)/Day 1 Exercise|week 01 exercises]].
programming practice, baby names projects


'''Readings:'''
due: one paragraph community suggestion
 
* Python for Informatics: [http://www.pythonlearn.com/html-009/book003.html Chapter 2  Variables, expressions and statements] and [http://www.pythonlearn.com/html-009/book004.html Chapter 3  Conditional execution]
 
'''Class Schedule:'''
 
* [[Community Data Science Course (Spring 2015)/Day 2 Lecture|Day 2 Lecture]] — Interactive class lecture including a review of material from last week and new material including dictionaries, loops, functions, and modules.
* Project time — We'll begin working on the [[baby names]] independent projects independently or in small groups with assistance from the teaching team.
 
'''Resources:'''
 
* [[Community Data Science Course (Spring 2015)/Day 2 Plan|Day 2 Plan]]
* [[Community Data Science Course (Spring 2015)/Day 2 Coding Challenges|Day 2 Coding Challenges]]
* [[Community Data Science Course (Spring 2015)/Day 2 Followup|Day 2 Followup]]


=== Week 3: April 13 ===
=== Week 3: April 13 ===


'''Assignment Due:'''
week 3: lecture: web apis; getting pages with requests, json, writing to files; wikipedia api
 
* [[#Final_Project_Ideas|Final Project Ideas]] [[https://canvas.uw.edu/courses/963931/assignments/2816619 Turn in on Canvas]]
* Code solving challenges in [[Baby names]] project.
 
'''Class Schedule:'''
 
* Review and Lecture — We'll walk through answers to the assignments for last week as a group.
* Project time — We'll begin working on a series of project based on the [[Wordplay]] project.
 
'''Resources:'''
 
* [[Community Data Science Course (Spring 2015)/Day 3 Plan|Day 3 Plan]]
* [[Community Data Science Course (Spring 2015)/Day 3 Coding Challenges|Day 3 Coding Challenges]]


=== Week 4: April 20 ===
=== Week 4: April 20 ===


<!-- '''Readings:'''
week 4: more advanced looping; defining functions, full run through of big function
 
* Python for Informatics: [http://www.pythonlearn.com/html-009/book013.html Chapter 12  Networked programs] and [http://www.pythonlearn.com/html-009/book014.html Chapter 13  Using Web Services] (Moved from the previous week)
-->
 
'''Class Schedule''':
 
* Review: We'll walk through answers to the assignment and code challenges from last week as a group.
* Lecture — Interactive class lecture including background into web APIs; requesting web pages with <code>requests</code>, JSON, and writing to files.
* Project time — We'll begin working on [[Community Data Science Course (Spring 2015)/Wikipedia API projects|a series of projects using the Wikipedia API]].
 
'''Resources''':
 
* [[Community Data Science Course (Spring 2015)/Day 4 Lecture|Day 4 Lecture]]
* [[Community Data Science Course (Spring 2015)/Wikipedia API projects|Wikipedia API projects]]
* [[Community Data Science Course (Spring 2015)/Day 4 Coding Challenges|Day 4 Coding Challenges]]


=== Week 5: April 27 ===
=== Week 5: April 27 ===


'''Assignment Due:''' Code solving challenges in in the Wikipedia API project from last week.
week 5: objects and classes; using tweepy: example projects using twitter
 
'''Readings:'''
 
* Python for Informatics: [http://www.pythonlearn.com/html-009/book006.html Chapter 5 Iteration] and [http://www.pythonlearn.com/html-009/book008.html Chapter 7 Files]
 
'''Class Schedule:'''
 
* Review — We'll walk through answers to the assignments for last week as a group.
* Lecture — [[Community Data Science Course (Spring 2015)/Day 5 Lecture|Interactive class lecture]] covering user-defined functions, debugging, filesystem input, and putting things together into a "real" program.
* Project time — We'll begin modifying the program we walk through in class to adapt it toward our needs and we'll pick out ideas for next steps and challenges for the coming week..


'''Resources:'''
due: project proposal
 
* [[Community Data Science Course (Spring 2015)/Day 5 Lecture|Day 5 Lecture]]
* '''Day 5 Project:''' [[Harry Potter on Wikipedia]]
* [[Community Data Science Course (Spring 2015)/Day 5 Coding Challenges|Day 5 Coding Challenges]]


=== Week 6: May 4 ===
=== Week 6: May 4 ===


'''Assignment Due:'''
week 6: freqency counting; defaultdicts; simple visualization and analysis using google docs
 
* Code solving challenges in created at the end of class the previous week.
* Finish the [[Twitter authentication setup]] to request keys necessary to begin using the Twitter API.
* [[#Final Project Proposal|Final project proposal]]
 
'''Readings:'''
 
* [[:w:Object-oriented_programming|Object-oriented programming article on Wikipedia]]
* Browse the [http://docs.tweepy.org/en/v3.2.0/ Tweepy API Documentation]
 
'''Class Schedule:'''
 
* Review — We'll walk through answers to the assignments for last week as a group.
* Lecture — Interactive class lecture covering Python objects and classes and using Tweepy to collect data from Twitter.
* Project time — [[Community Data Science Course (Spring 2015)/Day 6 Project|Twitter API project]]
 
'''Resources:'''
 
* '''Day 6 Project:''' [[Community Data Science Course (Spring 2015)/Day 6 Project|Day 6 Project]]
* [[Community Data Science Course (Spring 2015)/Day 6 Coding Challenges|Day 6 Coding Challenges]]
* [[Twitter words of warning]]


=== Week 7: May 11 ===
=== Week 7: May 11 ===


'''Assignment Due:'''
week 7: visualization using matplotlib: example projects using matplotlib; independent project time
 
* Code solving challenges in created at the end of previous class.
 
'''Readings:'''
 
* Python for Informatics: [http://www.pythonlearn.com/html-009/book005.html Chapter 4 Functions] and [http://www.pythonlearn.com/html-009/book012.html Chapter 11 Regular expressions]
 
'''Class Schedule:'''
 
* Review — We'll walk through answers to the assignments for last week as a group.
* Lecture — Interactive class lecture on regular expressions and pattern matching
* Project time — Working on regular expressions and independent projects


=== Week 8: May 18 ===
=== Week 8: May 18 ===


'''Class Schedule:'''
week 8: num.py and pandas: example projects using pandas; basic statistical tests; independent project time
 
* Final Project — We'll through expectations for final projects.
* Lecture — We'll walk through a series of common challenges people are having on their projects.
* Project time — We'll spend the majority of class focused on creating space for students to work on their individual final projects.
 
'''Optional Readings:'''
 
* Python for Informatics: [http://www.pythonlearn.com/html-009/book016.html Chapter 15 Visualizing Data]
* Python for Data Analysis: ''Chapter 8 Plotting and Visualization''


=== Week 9: May 25 ===
=== Week 9: May 25 ===


{{divbox|Note|May 25th is Memorial day and is a University Holiday. Because UW policy requires that we meet 10 times, we will meeting as scheduled. That said, because the building will lock at 6pm, we will be meeting half an hour early at '''5:30pm'''. Please do not be late!}}
week 9: web scraping; working outside of APIs; indepdent project time
 
'''Class Schedule:'''
 
* Final Project — We'll through expectations for final projects.
* Lecture — We'll walk through a series of common challenges people are having on their projects.
* Project time — We'll spend the majority of class focused on creating space for students to work on their individual final projects.
 
'''Optional Readings:'''
 
* Python for Data Analysis: ''Chapter 4 NumPy Basics: Arrays and Vectorized Computation'' and ''Chapter 5 Getting Started with pandas''


=== Week 10: June 1 ===
=== Week 10: June 1 ===


The full length of class will be devoted to final presentations of your data collection, your initial visualizations, and your results.
week 10: final presentations


== Administrative Notes ==
== Administrative Notes ==
Line 359: Line 128:
* Engage as much as possible with colleagues and the instructor.
* Engage as much as possible with colleagues and the instructor.
* Stay current with the latest developments in the field of communications and digital media.
* Stay current with the latest developments in the field of communications and digital media.
<!--  LocalWords:  TOC JSON MatPlotLib Elance Odesk MCCN MCDM Tweepy
-->
<!--  LocalWords:  num py Statmodels th Scrapy TTY ing analytical
-->
Please note that all contributions to CommunityData are considered to be released under the Attribution-Share Alike 3.0 Unported (see CommunityData:Copyrights for details). If you do not want your writing to be edited mercilessly and redistributed at will, then do not submit it here.
You are also promising us that you wrote this yourself, or copied it from a public domain or similar free resource. Do not submit copyrighted work without permission!

To protect the wiki against automated edit spam, we kindly ask you to solve the following CAPTCHA:

Cancel Editing help (opens in new window)

Template used on this page: