Community Data Science Workshops (Fall 2014)/Day 2 lecture: Difference between revisions
From CommunityData
Line 1: | Line 1: | ||
== Lecture Outline == | == Lecture Outline == | ||
;Introduction and context | ;Introduction and context | ||
* You can write some tools in Python now. Congratulations! | * You can write some tools in Python now. Congratulations! | ||
* Today we'll learn how to find/create data sets | * Today we'll learn how to find/create data sets | ||
* Next week we'll get into data science (asking and answering questions) | * Next week we'll get into data science (asking and answering questions) | ||
;Outline: | ;Outline: | ||
* What did we learn in Session 1? | * What did we learn in Session 1? | ||
* What is an API? | * What is an API? | ||
Line 13: | Line 16: | ||
* Introduction to structured data (JSON) | * Introduction to structured data (JSON) | ||
* How do we use APIs in general? | * How do we use APIs in general? | ||
;What is a (web) API? | ;What is a (web) API? | ||
* API: a structured way for programs to talk to each other (aka an interface for programs) | * API: a structured way for programs to talk to each other (aka an interface for programs) | ||
* Web APIs: like a website your programs can visit (you:a website::your program:a web API) | * Web APIs: like a website your programs can visit (you:a website::your program:a web API) | ||
; How do we use an API to fetch datasets? | ; How do we use an API to fetch datasets? | ||
Basic idea: your program sends a request, the API sends data back | Basic idea: your program sends a request, the API sends data back | ||
* Where do you direct your request? The site's API endpoint. | * Where do you direct your request? The site's API endpoint. | ||
Line 30: | Line 37: | ||
* How do you understand (i.e. parse) the data? | * How do you understand (i.e. parse) the data? | ||
** There's a module for that! | ** There's a module for that! | ||
; How do we write Python programs that make web requests? | ; How do we write Python programs that make web requests? | ||
Line 41: | Line 49: | ||
; Session 1 review | ; Session 1 review | ||
* Navigating in the terminal and using it to run programs | * Navigating in the terminal and using it to run programs | ||
* Writing Python: | * Writing Python: | ||
Line 52: | Line 61: | ||
; New programming concepts: | ; New programming concepts: | ||
* interpolate variables into a string using % and %()s | * interpolate variables into a string using % and %()s | ||
* requests | * requests | ||
Line 58: | Line 68: | ||
; How do we use an API to fetch kitten pictures? | ; How do we use an API to fetch kitten pictures? | ||
[http://placekitten.com/ placekitten.com] | [http://placekitten.com/ placekitten.com] | ||
* API that takes specially crafted URLs and gives appropriately sized picture of kittens | * API that takes specially crafted URLs and gives appropriately sized picture of kittens | ||
Line 67: | Line 78: | ||
; Introduction to structured data (JSON, JavaScriptObjectNotation) | ; Introduction to structured data (JSON, JavaScriptObjectNotation) | ||
* what is json: useful for more structured data | * what is json: useful for more structured data | ||
* import json; json.loads() | * import json; json.loads() | ||
Line 76: | Line 88: | ||
; Using other APIs | ; Using other APIs | ||
* every API is different, so read the documentation! | * every API is different, so read the documentation! | ||
* If the documentation isn't helpful, search online | * If the documentation isn't helpful, search online |
Revision as of 21:19, 14 November 2014
Lecture Outline
- Introduction and context
- You can write some tools in Python now. Congratulations!
- Today we'll learn how to find/create data sets
- Next week we'll get into data science (asking and answering questions)
- Outline
- What did we learn in Session 1?
- What is an API?
- How do we use one to fetch interesting datasets?
- How do we write programs that use the internet?
- How can we use the placekitten API to fetch kitten pictures?
- Introduction to structured data (JSON)
- How do we use APIs in general?
- What is a (web) API?
- API: a structured way for programs to talk to each other (aka an interface for programs)
- Web APIs: like a website your programs can visit (you:a website::your program:a web API)
- How do we use an API to fetch datasets?
Basic idea: your program sends a request, the API sends data back
- Where do you direct your request? The site's API endpoint.
- For example: Wikipedia's web API endpoint is http://en.wikipedia.org/w/api.php
- How do I write my request? Put together a URL; it will be different for different web APIs.
- Check the documentation, look for code samples
- How do you send a request?
- Python has modules you can use, like
requests
(they make HTTP requests)
- Python has modules you can use, like
- What do you get back?
- Structured data (usually in the JSON format)
- How do you understand (i.e. parse) the data?
- There's a module for that!
- How do we write Python programs that make web requests?
To use APIs to build a dataset we will need:
- all our tools from last session: variables, etc
- the ability to open urls on the web
- the ability to create custom URLS
- the ability to save to files
- the ability to understand (i.e., parse) JSON data that APIs usually give us
- Session 1 review
- Navigating in the terminal and using it to run programs
- Writing Python:
- using variables to manipulate data
- types of data: strings, integers, lists, dictionaries
- if statements
- for loops
- printing
- importing modules, so you can use code other people have written for you!
- New programming concepts
- interpolate variables into a string using % and %()s
- requests
- open files and write to them
- parsing a string (turning the string into a data structure we can manipulate)
- How do we use an API to fetch kitten pictures?
- API that takes specially crafted URLs and gives appropriately sized picture of kittens
- Exploring placekitten in a browser:
- visit the API documentation
- kittens of different sizes
- kittens in greyscale or color
- Now we write a small program to grab an arbitrary square from placekitten by asking for the size on standard in
- Introduction to structured data (JSON, JavaScriptObjectNotation)
- what is json: useful for more structured data
- import json; json.loads()
- like Python (except no single quotes)
- simple lists, dictionaries
- can reflect more complicated data structures
- Example file at http://mako.cc/cdsw.json
- download it and parse it
- Using other APIs
- every API is different, so read the documentation!
- If the documentation isn't helpful, search online
- for popular APIs, there are python modules that help you make requests and parse json
Possible issues:
- rate limiting
- authentication
- text encoding issues