DS4UX (Spring 2016)/Day 5 lecture: Difference between revisions

Revision as of 22:01, 20 April 2016

This page is a work in progress.

Lecture 1

reading/writing tsv files
reading/writing csv files

Lecture 2

In which you learn how to use Python and web APIs to meet the likes of her!

Introduction and context

You can write some tools in Python now. Congratulations!
Today we'll learn how to find/create data sets
Next week we'll get into data science (asking and answering questions)

Outline

What is an API?
How do we use one to fetch interesting datasets?
How do we write programs that use the internet?
How can we use the placekitten API to fetch kitten pictures?
Introduction to structured data (JSON)
How do we use APIs in general?

What is a (web) API?

API: a structured way for programs to talk to each other (aka an interface for programs)
Web APIs: like a website your programs can visit (you:a website::your program:a web API)

How do we use an API to fetch datasets?

Basic idea: your program sends a request, the API sends data back

Where do you direct your request? The site's API endpoint.
- For example: Wikipedia's web API endpoint is http://en.wikipedia.org/w/api.php
How do I write my request? Put together a URL; it will be different for different web APIs.
- Check the documentation, look for code samples
How do you send a request?
- Python has modules you can use, like requests (they make HTTP requests)
What do you get back?
- Structured data (usually in the JSON format)
How do you understand (i.e. parse) the data?
- There's a module for that!

How do we write Python programs that make web requests?

To use APIs to build a dataset we will need:

all our tools from last session: variables, etc
the ability to open urls on the web
the ability to create custom URLS
the ability to save to files
the ability to understand (i.e., parse) JSON data that APIs usually give us

New programming concepts

interpolate variables into a string using % and %()s
requests
open files and write to them

How do we use an API to fetch kitten pictures?

placekitten.com

API that takes specially crafted URLs and gives appropriately sized picture of kittens
Exploring placekitten in a browser:
- visit the API documentation
- kittens of different sizes
- kittens in greyscale or color
Now we write a small program to grab an arbitrary square from placekitten by asking for the size on standard in: placekitten_raw_input.py

Introduction to structured data (JSON, JavaScriptObjectNotation)

what is json: useful for more structured data
import json; json.loads()
like Python (except no single quotes)
simple lists, dictionaries
can reflect more complicated data structures
Example file at http://mako.cc/cdsw.json
You can parse data directly with .json() on a requests call

Using other APIs

every API is different, so read the documentation!
If the documentation isn't helpful, search online
for popular APIs, there are python modules that help you make requests and parse json

Possible issues:

rate limiting
authentication
text encoding issues

Other Potentially Resources

My friend Frances gave a version of this lecture last year and create slides. They are written for Python 2, so the code might not all work (remember, use print() with parentheses) but the basic ideas might be helpful:

Slides (PDF) — For viewing
Slides (ODP Libreoffice Slides Format) — For editing and modification

@@ Line 35: / Line 35: @@
 [[File:Highfivekitten.jpeg|200px|thumb|In which you learn how to use Python and web APIs to meet the likes of her!]]
-;Introduction and context
+===Introduction and context ===
 * You can write some tools in Python now. Congratulations!
@@ Line 42: / Line 42: @@
-;Outline:
+=== Outline ===
 * What is an API?
@@ Line 52: / Line 52: @@
-;What is a (web) API?
+=== What is a (web) API? ===
 * API: a structured way for programs to talk to each other (aka an interface for programs)
@@ Line 58: / Line 58: @@
-; How do we use an API to fetch datasets?
+=== How do we use an API to fetch datasets? ===
 Basic idea: your program sends a request, the API sends data back
@@ Line 73: / Line 73: @@
-; How do we write Python programs that make web requests?
+=== How do we write Python programs that make web requests? ===
 To use APIs to build a dataset we will need:
@@ Line 83: / Line 83: @@
-; New programming concepts:
+=== New programming concepts ===
 * interpolate variables into a string using % and %()s
@@ Line 90: / Line 90: @@
-; How do we use an API to fetch kitten pictures?
+=== How do we use an API to fetch kitten pictures? ===
 [http://placekitten.com/ placekitten.com]
@@ Line 101: / Line 101: @@
-; Introduction to structured data (JSON, JavaScriptObjectNotation)
+=== Introduction to structured data (JSON, JavaScriptObjectNotation) ===
 * what is json: useful for more structured data
@@ Line 111: / Line 111: @@
 * You can parse data directly with <code>.json()</code> on a <code>requests</code> call
-; Using other APIs
+=== Using other APIs ===
 * every API is different, so read the documentation!
@@ Line 131: / Line 131: @@
 <!-- FROM http://wiki.communitydata.cc/Community_Data_Science_Workshops_(Spring_2015)/Day_3_Lecture
-== Material for the lecture ==
+=== Material for the lecture ===
 For the lecture, you will need two files. Download both of these to your computer by using right or control click on the link and then using ''Save as'' or ''Save link as''. Keep track of where you put the files.