Editing Community Data Science Workshops (Core)/Day 2 Lecture

[[Image:Place kitten-200x287.jpg|thumb|In which you learn how to use Python and web APIs to meet the likes of this guys!]]

Welcome to the Saturday lecture section of the Community Data Science Workshop Session 2! For about 140 minutes, we'll work through an introduction to web APIs using Python via both a lecture and hand-on exercises.

== Resources ==

* [https://communitydata.cc/~mako/cdsw-sp2016-lecture2-20160423.ogv Screencast/recording of the lecture from 2016] (1.1GB) — The file should be viewable in Firefox and many other browsers. If you have trouble playing it, you can download [https://www.videolan.org/vlc/index.html the VLC media player] which will be a able to play it on Windows, OSX, or GNU/Linux.

== Lecture Outline ==
;Introduction and context

* You can write some tools in Python now. Congratulations!
* Today we'll learn how to find/create data sets
* Next week we'll get into data science (asking and answering questions)

;Outline: 

* What did we learn in Session 1?
* What is an API?
* How do we use one to fetch interesting datasets?
* How do we write programs that use the internet? 
* How can we use the placekitten API to fetch kitten pictures?
* Introduction to structured data (JSON)
* How do we use APIs in general?

;What is a (web) API?

* API: a structured way for programs to talk to each other (aka an interface for programs)
* Web APIs: like a website your programs can visit (you:a website::your program:a web API)

; How do we use an API to fetch datasets?

Basic idea: your program sends a request, the API sends data back
* Where do you direct your request? The site's API endpoint.
** For example: Wikipedia's web API endpoint is http://en.wikipedia.org/w/api.php
* How do I write my request? Put together a URL; it will be different for different web APIs.
** Check the documentation, look for code samples
* How do you send a request? 
** Python has modules you can use, like <code>requests</code> (they make HTTP requests)
* What do you get back?
** Structured data (usually in the JSON format)
* How do you understand (i.e. parse) the data? 
** The requests module can do that using the <code>.json()</code> function!

; How do we write Python programs that make web requests?

To use APIs to build a dataset we will need:
* all our tools from last session: variables, etc
* the ability to open urls on the web
* the ability to create custom URLS
* the ability to save to files 
* the ability to understand (i.e., parse) JSON data that APIs usually give us

; Session 1 review

* Navigating in the terminal and using it to run programs
* Writing Python:
** using variables to manipulate data
** types of data: strings, integers, lists, dictionaries
** if statements
** for loops
** printing
** importing modules, so you can use code other people have written for you!

; New programming concepts:

* interpolate variables into a string using <code>.format</code>
* using <code>input()</code>


* [http://docs.python-requests.org/en/latest/ requests]
* open files and write to them
* parsing a string (turning the string into a data structure we can manipulate) using the json module

; How do we use an API to fetch pictures of kittens?

[http://placekitten.com/ placekitten.com]
* API that takes specially crafted URLs and gives appropriately sized picture of kittens
* Exploring placekitten in a browser:
** visit the API documentation
** kittens of different sizes
** kittens in greyscale or color
* Now we write a small program to grab an arbitrary square from placekitten by asking for the size on standard in ([http://mako.cc/teaching/2016/cdsw-spring/placekitten.py placekitten.py])

; Introduction to structured data (JSON, JavaScriptObjectNotation)

* what is json: useful for more structured data
* import json; json.loads(), or, even easier, just do it directly with requests using the <code>.json()</code> function!
* like Python (except no single quotes)
* simple lists, dictionaries
* can reflect more complicated data structures
* Example file at http://mako.cc/cdsw.json
* download it and parse it (e.g., with a program like [http://mako.cc/teaching/2015/cdsw-spring/parse_cdswjson.py parse_cdswjson.py])

; Using other APIs

* every API is different, so read the documentation!
* If the documentation isn't helpful, search online
* for popular APIs, there are python modules that help you make requests and parse json

Possible issues:
* rate limiting
* authentication
* text encoding issues

== Lecture Slides (From Fall 2014) ==

We don't use these slides anymore but they might be very useful if you're following along at home or interested in refreshing your memory since they cover a lot of the same material.

* [http://mako.cc/teaching/2014/cdsw-autumn/lecture2-web_apis.pdf Slides (PDF)] — For viewing
* [http://mako.cc/teaching/2014/cdsw-autumn/lecture2-web_apis.odp Slides (ODP Libreoffice Slides Format)] — For editing and modification

[[Category:Shared_Pages]]