Editing DS4UX (Spring 2016)/Panama Papers
From CommunityData
The edit can be undone. Please check the comparison below to verify that this is what you want to do, and then publish the changes below to finish undoing the edit.
Latest revision | Your text | ||
Line 1: | Line 1: | ||
[[File:Wikipedia.png|right|250px]] | [[File:Wikipedia.png|right|250px]] | ||
In this project, we will explore a few ways to gather data using two Wikipedia APIs: one provides data related to edits, and the other provides data related to pageviews. Once we've done that, we will extend this to code to create our own datasets of Wikipedia edits or other data that | In this project, we will explore a few ways to gather data using two Wikipedia APIs: one provides data related to edits, and the other provides data related to pageviews. Once we've done that, we will extend this to code to create our own datasets of Wikipedia edits or other data that we might be able to use to ask and answer questions in the final session. | ||
''[[w:Panama_Papers| Panama Papers]]'' | |||
== Goals == | |||
* Understand the basic anatomy of an API request | * Understand the basic anatomy of an API request | ||
Line 18: | Line 12: | ||
* Practice reading and extending other people's code | * Practice reading and extending other people's code | ||
== Download and test the Wikipedia API project == | |||
# Right click the following file, click "Save Target as..." or "Save link as...", and save it to your Desktop directory: https://jtmorgan.net/ds4ux/week6/panama-papers.zip | # Right click the following file, click "Save Target as..." or "Save link as...", and save it to your Desktop directory: | ||
https://jtmorgan.net/ds4ux/week6/panama-papers.zip | |||
# Find <code>panama-papers.zip</code> on your Desktop and double-click on it to "unzip" it. That will create a folder called <code>panama-papers</code> containing several files. | # Find <code>panama-papers.zip</code> on your Desktop and double-click on it to "unzip" it. That will create a folder called <code>panama-papers</code> containing several files. | ||
# In PowerShell or Terminal, navigate to the <code>panama-papers</code> directory and type: | # In PowerShell or Terminal, navigate to the <code>panama-papers</code> directory and type: | ||
Line 28: | Line 23: | ||
at the command prompt to execute the <code>wikipedia-1.py</code> Python script. Wait a little while while your computer connects to Wikipedia. You should see the result of a query from Wikipedia API on your screen. If you don't, let Jonathan or Ray know . | at the command prompt to execute the <code>wikipedia-1.py</code> Python script. Wait a little while while your computer connects to Wikipedia. You should see the result of a query from Wikipedia API on your screen. If you don't, let Jonathan or Ray know . | ||
== | == Panama Papers data sources == | ||
;Wikipedia Edit API: An API that provides data on edits to Wikipedia pages. Use the [https://en.wikipedia.org/wiki/Special:ApiSandbox API sandbox] to build and test queries with this API. | ;Wikipedia Edit API: An API that provides data on edits to Wikipedia pages. Use the [https://en.wikipedia.org/wiki/Special:ApiSandbox API sandbox] to build and test queries with this API. | ||
;Wikipedia Page View API: Use [https://wikimedia.org/api/rest_v1/?doc#!/Pageviews_data the (experimental!) Wikipedia page view API] to find out how many views a given Wikipedia article gets on a daily basis. | ;Wikipedia Page View API: Use [https://wikimedia.org/api/rest_v1/?doc#!/Pageviews_data the (experimental!) Wikipedia page view API] to find out how many views a given Wikipedia article gets on a daily basis. | ||
* This API is a little different than the Wikipedia edit API because it uses relative paths (<code>/.../.../</code>) instead of parameters (<code>somekey=someval&otherkey=otherval</code>). | |||
* Note: This API is a little different than the Wikipedia edit API because it uses relative paths (<code>/.../.../</code>) instead of parameters (<code>somekey=someval&otherkey=otherval</code>). | |||
* This API is case-sensitive, and space-sensitive (replace spaces in page titles with underscores). It also requires you to [http://www.w3schools.com/tags/ref_urlencode.asp URL encode] any special characters (like !, $, # and %) that might appear in a page title. | * This API is case-sensitive, and space-sensitive (replace spaces in page titles with underscores). It also requires you to [http://www.w3schools.com/tags/ref_urlencode.asp URL encode] any special characters (like !, $, # and %) that might appear in a page title. | ||
* Example request for Panama Papers | * Example request for Panama Papers: https://wikimedia.org/api/rest_v1/metrics/pageviews/per-article/en.wikipedia/all-access/user/Panama_Papers/daily/20160401/20160420 | ||
== Exercises == | == Exercises == | ||
Line 48: | Line 43: | ||
* [https://en.wikipedia.org/wiki/Special:ApiSandbox#action=query&format=json&prop=revisions&list=&meta=&titles=Panama_Papers&rvprop=ids%7Ctimestamp%7Cflags%7Ccomment%7Cuser&rvlimit=1&rvdir=newer View query in sandbox] | * [https://en.wikipedia.org/wiki/Special:ApiSandbox#action=query&format=json&prop=revisions&list=&meta=&titles=Panama_Papers&rvprop=ids%7Ctimestamp%7Cflags%7Ccomment%7Cuser&rvlimit=1&rvdir=newer View query in sandbox] | ||
* [https://en.wikipedia.org/w/api.php?action=query&format=json&prop=revisions&list=&meta=&titles=Panama_Papers&rvprop=ids%7Ctimestamp%7Cflags%7Ccomment%7Cuser&rvlimit=1&rvdir=newer View result in browser] | * [https://en.wikipedia.org/w/api.php?action=query&format=json&prop=revisions&list=&meta=&titles=Panama_Papers&rvprop=ids%7Ctimestamp%7Cflags%7Ccomment%7Cuser&rvlimit=1&rvdir=newer View result in browser] | ||
Line 55: | Line 49: | ||
* [https://en.wikipedia.org/wiki/Special:ApiSandbox#action=query&format=json&prop=revisions&list=&meta=&titles=Panama_Papers&rvprop=ids%7Ctimestamp%7Cflags%7Ccomment%7Cuser&rvlimit=1&rvdir=older View query in sandbox] | * [https://en.wikipedia.org/wiki/Special:ApiSandbox#action=query&format=json&prop=revisions&list=&meta=&titles=Panama_Papers&rvprop=ids%7Ctimestamp%7Cflags%7Ccomment%7Cuser&rvlimit=1&rvdir=older View query in sandbox] | ||
* [https://en.wikipedia.org/w/api.php?action=query&format=json&prop=revisions&list=&meta=&titles=Panama_Papers&rvprop=ids%7Ctimestamp%7Cflags%7Ccomment%7Cuser&rvlimit=1&rvdir=newer View result in browser] | * [https://en.wikipedia.org/w/api.php?action=query&format=json&prop=revisions&list=&meta=&titles=Panama_Papers&rvprop=ids%7Ctimestamp%7Cflags%7Ccomment%7Cuser&rvlimit=1&rvdir=newer View result in browser] | ||
; How many edits has the creator of Panama Papers made to Wikipedia? | ; How many edits has the creator of Panama Papers made to Wikipedia? | ||
* [https://en.wikipedia.org/wiki/Special:ApiSandbox#action=query&format=json&list=users&usprop=editcount%7Cregistration&ususers=Czar View query in sandbox] | :* [https://en.wikipedia.org/wiki/Special:ApiSandbox#action=query&format=json&list=users&usprop=editcount%7Cregistration&ususers=Czar View query in sandbox] | ||
* [https://en.wikipedia.org/w/api.php?action=query&format=json&list=users&usprop=editcount%7Cregistration&ususers=Czar View result in browser | :* [https://en.wikipedia.org/w/api.php?action=query&format=json&list=users&usprop=editcount%7Cregistration&ususers=Czar View result in browser] | ||
Line 69: | Line 61: | ||
* [https://en.wikipedia.org/wiki/Special:ApiSandbox#action=query&format=json&prop=revisions&titles=Panama+Papers&rvprop=ids%7Ctimestamp%7Ccontent&rvstart=2016-04-04T17%3A58%3A00.000Z&rvend=2016-04-04T17%3A59%3A05.000Z&rvdir=newer View query in sandbox] | * [https://en.wikipedia.org/wiki/Special:ApiSandbox#action=query&format=json&prop=revisions&titles=Panama+Papers&rvprop=ids%7Ctimestamp%7Ccontent&rvstart=2016-04-04T17%3A58%3A00.000Z&rvend=2016-04-04T17%3A59%3A05.000Z&rvdir=newer View query in sandbox] | ||
* [https://en.wikipedia.org/w/api.php?action=query&format=json&prop=revisions&titles=Panama+Papers&rvprop=ids%7Ctimestamp%7Ccontent&rvstart=2016-04-04T17%3A58%3A00.000Z&rvend=2016-04-04T17%3A59%3A05.000Z&rvdir=newer View result in browser] | * [https://en.wikipedia.org/w/api.php?action=query&format=json&prop=revisions&titles=Panama+Papers&rvprop=ids%7Ctimestamp%7Ccontent&rvstart=2016-04-04T17%3A58%3A00.000Z&rvend=2016-04-04T17%3A59%3A05.000Z&rvdir=newer View result in browser] | ||
* [https://en.wikipedia.org/w/index.php?title=Panama_Papers&oldid=713548359 revision on Wikipedia] | |||
* [https://en.wikipedia.org/w/index.php?title=Panama_Papers&oldid=713548359 | * [https://en.wikipedia.org/w/index.php?title=Panama_Papers&diff=713548359&oldid=713548357 "diff" view of revision] | ||
* [https://en.wikipedia.org/w/index.php?title=Panama_Papers&diff=713548359&oldid=713548357 | |||
Line 109: | Line 100: | ||
=== Building queries with Python <code>requests</code> === | === Building queries with Python <code>requests</code> === | ||
<!-- ''Note:'' this query returns ''data'' about each edit, but it doesn't count them for us. We need to use Python for that! --> | |||
;1. How many edits did Panama Papers receive in its first 24 hours? | ;1. How many edits did Panama Papers receive in its first 24 hours? | ||
Completing this exercise requires you to | Completing this exercise requires you to firist query the Wikipedia API to get revisions to the article that have a timestamp between 2016/4/3 17:59:05 and 2016/4/4 17:59:05, and then use Python to count the number of revisions in the resulting JSON. | ||
Line 128: | Line 113: | ||
;3. How many times was Panama Papers viewed in the first week? What proportion of those views came from mobile devices? | ;3. How many times was Panama Papers viewed in the first week? What proportion of those views came from mobile devices? | ||
Line 148: | Line 131: | ||
* [https://en.wikipedia.org/w/api.php?action=help&modules=query API documentation for the query module] | * [https://en.wikipedia.org/w/api.php?action=help&modules=query API documentation for the query module] | ||
* [https://en.wikipedia.org/wiki/Special:ApiSandbox API Sandbox] | * [https://en.wikipedia.org/wiki/Special:ApiSandbox API Sandbox] | ||
* [[Sample | * [[Sample Wikipedia API queries]] | ||
* [https://github.com/ben-zen/wikipedia-session The session lecture notes (in Markdown) and python sources.] | |||
* | |||
[[Category:DS4UX (Spring 2016)]] | [[Category:DS4UX (Spring 2016)]] |