Editing DS4UX (Spring 2016)/Panama Papers
From CommunityData
The edit can be undone. Please check the comparison below to verify that this is what you want to do, and then publish the changes below to finish undoing the edit.
Latest revision | Your text | ||
Line 1: | Line 1: | ||
[[File:Wikipedia.png|right|250px]] | [[File:Wikipedia.png|right|250px]] | ||
In this project, we will explore a few ways to gather data using two Wikipedia APIs: one provides data related to edits, and the other provides data related to pageviews. Once we've done that, we will extend this to code to create our own datasets of Wikipedia edits or other data that | In this project, we will explore a few ways to gather data using two Wikipedia APIs: one provides data related to edits, and the other provides data related to pageviews. Once we've done that, we will extend this to code to create our own datasets of Wikipedia edits or other data that we might be able to use to ask and answer questions in the final session. | ||
== Overview == | == Overview == | ||
In this project we will look at the viewing and editing history of a recently created Wikipedia article about a breaking news event—''[[w:Panama_Papers| Panama Papers]]''. When events of global significance occur, Wikipedia is often among the first places that people look for information about these events. By examining both the editing and viewing history of this article, we can learn a lot about how people create ''and'' consume information on Wikipedia. | In this project we will look at the viewing and editing history of a recently created Wikipedia article about a breaking news event—''[[w:Panama_Papers| Panama Papers]]''. When events of global significance occur, Wikipedia is often among the first places that people look for information about these events. By examining both the editing and viewing history of this article, we can learn a lot about how people create ''and'' consume information on Wikipedia. | ||
=== Goals === | === Goals === | ||
Line 28: | Line 24: | ||
at the command prompt to execute the <code>wikipedia-1.py</code> Python script. Wait a little while while your computer connects to Wikipedia. You should see the result of a query from Wikipedia API on your screen. If you don't, let Jonathan or Ray know . | at the command prompt to execute the <code>wikipedia-1.py</code> Python script. Wait a little while while your computer connects to Wikipedia. You should see the result of a query from Wikipedia API on your screen. If you don't, let Jonathan or Ray know . | ||
=== | === Panama Papers data sources === | ||
;Wikipedia Edit API: An API that provides data on edits to Wikipedia pages. Use the [https://en.wikipedia.org/wiki/Special:ApiSandbox API sandbox] to build and test queries with this API. | ;Wikipedia Edit API: An API that provides data on edits to Wikipedia pages. Use the [https://en.wikipedia.org/wiki/Special:ApiSandbox API sandbox] to build and test queries with this API. | ||
;Wikipedia Page View API: Use [https://wikimedia.org/api/rest_v1/?doc#!/Pageviews_data the (experimental!) Wikipedia page view API] to find out how many views a given Wikipedia article gets on a daily basis. | |||
* Note: This API is a little different than the Wikipedia edit API because it uses relative paths (<code>/.../.../</code>) instead of parameters (<code>somekey=someval&otherkey=otherval</code>). | |||
* This API is case-sensitive, and space-sensitive (replace spaces in page titles with underscores). It also requires you to [http://www.w3schools.com/tags/ref_urlencode.asp URL encode] any special characters (like !, $, # and %) that might appear in a page title. | * This API is case-sensitive, and space-sensitive (replace spaces in page titles with underscores). It also requires you to [http://www.w3schools.com/tags/ref_urlencode.asp URL encode] any special characters (like !, $, # and %) that might appear in a page title. | ||
* Example request for Panama Papers | * Example request for Panama Papers: https://wikimedia.org/api/rest_v1/metrics/pageviews/per-article/en.wikipedia/all-access/user/Panama_Papers/daily/20160401/20160420 | ||
== Exercises == | == Exercises == | ||
Line 48: | Line 43: | ||
* [https://en.wikipedia.org/wiki/Special:ApiSandbox#action=query&format=json&prop=revisions&list=&meta=&titles=Panama_Papers&rvprop=ids%7Ctimestamp%7Cflags%7Ccomment%7Cuser&rvlimit=1&rvdir=newer View query in sandbox] | * [https://en.wikipedia.org/wiki/Special:ApiSandbox#action=query&format=json&prop=revisions&list=&meta=&titles=Panama_Papers&rvprop=ids%7Ctimestamp%7Cflags%7Ccomment%7Cuser&rvlimit=1&rvdir=newer View query in sandbox] | ||
* [https://en.wikipedia.org/w/api.php?action=query&format=json&prop=revisions&list=&meta=&titles=Panama_Papers&rvprop=ids%7Ctimestamp%7Cflags%7Ccomment%7Cuser&rvlimit=1&rvdir=newer View result in browser] | * [https://en.wikipedia.org/w/api.php?action=query&format=json&prop=revisions&list=&meta=&titles=Panama_Papers&rvprop=ids%7Ctimestamp%7Cflags%7Ccomment%7Cuser&rvlimit=1&rvdir=newer View result in browser] | ||
Line 55: | Line 49: | ||
* [https://en.wikipedia.org/wiki/Special:ApiSandbox#action=query&format=json&prop=revisions&list=&meta=&titles=Panama_Papers&rvprop=ids%7Ctimestamp%7Cflags%7Ccomment%7Cuser&rvlimit=1&rvdir=older View query in sandbox] | * [https://en.wikipedia.org/wiki/Special:ApiSandbox#action=query&format=json&prop=revisions&list=&meta=&titles=Panama_Papers&rvprop=ids%7Ctimestamp%7Cflags%7Ccomment%7Cuser&rvlimit=1&rvdir=older View query in sandbox] | ||
* [https://en.wikipedia.org/w/api.php?action=query&format=json&prop=revisions&list=&meta=&titles=Panama_Papers&rvprop=ids%7Ctimestamp%7Cflags%7Ccomment%7Cuser&rvlimit=1&rvdir=newer View result in browser] | * [https://en.wikipedia.org/w/api.php?action=query&format=json&prop=revisions&list=&meta=&titles=Panama_Papers&rvprop=ids%7Ctimestamp%7Cflags%7Ccomment%7Cuser&rvlimit=1&rvdir=newer View result in browser] | ||
; How many edits has the creator of Panama Papers made to Wikipedia? | ; How many edits has the creator of Panama Papers made to Wikipedia? | ||
* [https://en.wikipedia.org/wiki/Special:ApiSandbox#action=query&format=json&list=users&usprop=editcount%7Cregistration&ususers=Czar View query in sandbox] | :* [https://en.wikipedia.org/wiki/Special:ApiSandbox#action=query&format=json&list=users&usprop=editcount%7Cregistration&ususers=Czar View query in sandbox] | ||
* [https://en.wikipedia.org/w/api.php?action=query&format=json&list=users&usprop=editcount%7Cregistration&ususers=Czar View result in browser | :* [https://en.wikipedia.org/w/api.php?action=query&format=json&list=users&usprop=editcount%7Cregistration&ususers=Czar View result in browser] | ||
Line 69: | Line 61: | ||
* [https://en.wikipedia.org/wiki/Special:ApiSandbox#action=query&format=json&prop=revisions&titles=Panama+Papers&rvprop=ids%7Ctimestamp%7Ccontent&rvstart=2016-04-04T17%3A58%3A00.000Z&rvend=2016-04-04T17%3A59%3A05.000Z&rvdir=newer View query in sandbox] | * [https://en.wikipedia.org/wiki/Special:ApiSandbox#action=query&format=json&prop=revisions&titles=Panama+Papers&rvprop=ids%7Ctimestamp%7Ccontent&rvstart=2016-04-04T17%3A58%3A00.000Z&rvend=2016-04-04T17%3A59%3A05.000Z&rvdir=newer View query in sandbox] | ||
* [https://en.wikipedia.org/w/api.php?action=query&format=json&prop=revisions&titles=Panama+Papers&rvprop=ids%7Ctimestamp%7Ccontent&rvstart=2016-04-04T17%3A58%3A00.000Z&rvend=2016-04-04T17%3A59%3A05.000Z&rvdir=newer View result in browser] | * [https://en.wikipedia.org/w/api.php?action=query&format=json&prop=revisions&titles=Panama+Papers&rvprop=ids%7Ctimestamp%7Ccontent&rvstart=2016-04-04T17%3A58%3A00.000Z&rvend=2016-04-04T17%3A59%3A05.000Z&rvdir=newer View result in browser] | ||
* [https://en.wikipedia.org/w/index.php?title=Panama_Papers&oldid=713548359 revision on Wikipedia] | |||
* [https://en.wikipedia.org/w/index.php?title=Panama_Papers&oldid=713548359 | * [https://en.wikipedia.org/w/index.php?title=Panama_Papers&diff=713548359&oldid=713548357 "diff" view of revision] | ||
* [https://en.wikipedia.org/w/index.php?title=Panama_Papers&diff=713548359&oldid=713548357 | |||
Line 109: | Line 100: | ||
=== Building queries with Python <code>requests</code> === | === Building queries with Python <code>requests</code> === | ||
<!-- ''Note:'' this query returns ''data'' about each edit, but it doesn't count them for us. We need to use Python for that! --> | |||
;1. How many edits did Panama Papers receive in its first 24 hours? | ;1. How many edits did Panama Papers receive in its first 24 hours? | ||
Completing this exercise requires you to | Completing this exercise requires you to firist query the Wikipedia API to get revisions to the article that have a timestamp between 2016/4/3 17:59:05 and 2016/4/4 17:59:05, and then use Python to count the number of revisions in the resulting JSON. | ||
Line 128: | Line 113: | ||
;3. How many times was Panama Papers viewed in the first week? What proportion of those views came from mobile devices? | ;3. How many times was Panama Papers viewed in the first week? What proportion of those views came from mobile devices? | ||
Line 148: | Line 131: | ||
* [https://en.wikipedia.org/w/api.php?action=help&modules=query API documentation for the query module] | * [https://en.wikipedia.org/w/api.php?action=help&modules=query API documentation for the query module] | ||
* [https://en.wikipedia.org/wiki/Special:ApiSandbox API Sandbox] | * [https://en.wikipedia.org/wiki/Special:ApiSandbox API Sandbox] | ||
* [[Sample | * [[Sample Wikipedia API queries]] | ||
* [https://github.com/ben-zen/wikipedia-session The session lecture notes (in Markdown) and python sources.] | |||
* | |||
[[Category:DS4UX (Spring 2016)]] | [[Category:DS4UX (Spring 2016)]] |