Community Data Science Workshops (Core)/Day 2 Lecture: Difference between revisions
From CommunityData
No edit summary |
No edit summary |
||
(10 intermediate revisions by 5 users not shown) | |||
Line 1: | Line 1: | ||
{{Template:CDSW Header}} | |||
[[Image:Place kitten-200x287.jpg|thumb|In which you learn how to use Python and web APIs to meet the likes of this guys!]] | [[Image:Place kitten-200x287.jpg|thumb|In which you learn how to use Python and web APIs to meet the likes of this guys!]] | ||
Welcome to the Saturday lecture section of the Community Data Science Workshop Session 2! For about 140 minutes, we'll work through an introduction to web APIs using Python via both a lecture and hand-on exercises. | Welcome to the Saturday lecture section of the Community Data Science Workshop Session 2! For about 140 minutes, we'll work through an introduction to web APIs using Python via both a lecture and hand-on exercises. | ||
'''Resources:''' | |||
* [https://communitydata. | * Notebook that Tommy taught from in the morning lecutre ([https://raw.githubusercontent.com/CommunityDataScienceCollective/LectureNotes/master/Lecture_2_APIs.ipynb IPYNB file]) | ||
* Dharma's "following along" version of the notebook" ([https://communitydata.science/~mako/cdsw-wi2020-lecture2/Lecture_2_APIs_ded.html in HTML] and [https://communitydata.science/~mako/cdsw-wi2020-lecture2/Lecture_2_APIs_ded.ipynb IPYNB]) | |||
To download the IPYNB files, you need to right click and click "Save as" to download. | |||
== Lecture Outline == | == Lecture Outline == | ||
Line 66: | Line 70: | ||
; New programming concepts: | ; New programming concepts: | ||
* interpolate variables into a string using | * interpolate variables into a string using [https://docs.python.org/3/reference/lexical_analysis.html#f-strings f-strings] | ||
* using <code>input()</code> | * using <code>input()</code> | ||
Line 112: | Line 116: | ||
* [http://mako.cc/teaching/2014/cdsw-autumn/lecture2-web_apis.odp Slides (ODP Libreoffice Slides Format)] — For editing and modification | * [http://mako.cc/teaching/2014/cdsw-autumn/lecture2-web_apis.odp Slides (ODP Libreoffice Slides Format)] — For editing and modification | ||
== Videos from 2016 == | |||
* [https://communitydata.cc/~mako/cdsw-sp2016-lecture2-20160423.ogv Screencast/recording of the lecture from 2016] (1.1GB) — The file should be viewable in Firefox and many other browsers. If you have trouble playing it, you can download [https://www.videolan.org/vlc/index.html the VLC media player] which will be a able to play it on Windows, OSX, or GNU/Linux. | |||
[[Category:Shared_Pages]] | [[Category:Shared_Pages]] |
Latest revision as of 01:05, 18 February 2020
![](/upload/c/c4/Place_kitten-200x287.jpg)
Welcome to the Saturday lecture section of the Community Data Science Workshop Session 2! For about 140 minutes, we'll work through an introduction to web APIs using Python via both a lecture and hand-on exercises.
Resources:
- Notebook that Tommy taught from in the morning lecutre (IPYNB file)
- Dharma's "following along" version of the notebook" (in HTML and IPYNB)
To download the IPYNB files, you need to right click and click "Save as" to download.
Lecture Outline[edit]
- Introduction and context
- You can write some tools in Python now. Congratulations!
- Today we'll learn how to find/create data sets
- Next week we'll get into data science (asking and answering questions)
- Outline
- What did we learn in Session 1?
- What is an API?
- How do we use one to fetch interesting datasets?
- How do we write programs that use the internet?
- How can we use the placekitten API to fetch kitten pictures?
- Introduction to structured data (JSON)
- How do we use APIs in general?
- What is a (web) API?
- API: a structured way for programs to talk to each other (aka an interface for programs)
- Web APIs: like a website your programs can visit (you:a website::your program:a web API)
- How do we use an API to fetch datasets?
Basic idea: your program sends a request, the API sends data back
- Where do you direct your request? The site's API endpoint.
- For example: Wikipedia's web API endpoint is http://en.wikipedia.org/w/api.php
- How do I write my request? Put together a URL; it will be different for different web APIs.
- Check the documentation, look for code samples
- How do you send a request?
- Python has modules you can use, like
requests
(they make HTTP requests)
- Python has modules you can use, like
- What do you get back?
- Structured data (usually in the JSON format)
- How do you understand (i.e. parse) the data?
- The requests module can do that using the
.json()
function!
- The requests module can do that using the
- How do we write Python programs that make web requests?
To use APIs to build a dataset we will need:
- all our tools from last session: variables, etc
- the ability to open urls on the web
- the ability to create custom URLS
- the ability to save to files
- the ability to understand (i.e., parse) JSON data that APIs usually give us
- Session 1 review
- Navigating in the terminal and using it to run programs
- Writing Python:
- using variables to manipulate data
- types of data: strings, integers, lists, dictionaries
- if statements
- for loops
- printing
- importing modules, so you can use code other people have written for you!
- New programming concepts
- interpolate variables into a string using f-strings
- using
input()
- requests
- open files and write to them
- parsing a string (turning the string into a data structure we can manipulate) using the json module
- How do we use an API to fetch pictures of kittens?
- API that takes specially crafted URLs and gives appropriately sized picture of kittens
- Exploring placekitten in a browser:
- visit the API documentation
- kittens of different sizes
- kittens in greyscale or color
- Now we write a small program to grab an arbitrary square from placekitten by asking for the size on standard in (placekitten.py)
- Introduction to structured data (JSON, JavaScriptObjectNotation)
- what is json: useful for more structured data
- import json; json.loads(), or, even easier, just do it directly with requests using the
.json()
function! - like Python (except no single quotes)
- simple lists, dictionaries
- can reflect more complicated data structures
- Example file at http://mako.cc/cdsw.json
- download it and parse it (e.g., with a program like parse_cdswjson.py)
- Using other APIs
- every API is different, so read the documentation!
- If the documentation isn't helpful, search online
- for popular APIs, there are python modules that help you make requests and parse json
Possible issues:
- rate limiting
- authentication
- text encoding issues
Lecture Slides (From Fall 2014)[edit]
We don't use these slides anymore but they might be very useful if you're following along at home or interested in refreshing your memory since they cover a lot of the same material.
- Slides (PDF) — For viewing
- Slides (ODP Libreoffice Slides Format) — For editing and modification
Videos from 2016[edit]
- Screencast/recording of the lecture from 2016 (1.1GB) — The file should be viewable in Firefox and many other browsers. If you have trouble playing it, you can download the VLC media player which will be a able to play it on Windows, OSX, or GNU/Linux.