Community Data Science Course (Sprint 2019)/Day 7 Notes
We will be discussing this data set.
- One of the most important qualities of the Scientific Revolution was that results were broadly shared, so new results could build on top of existing knowledge.
- Repeatability is the key to science (even data science): your results are only scientific if they are repeatable by a third party.
Today's Lecture Let's go end to end on a data question: are there factors that predict injuries and fatalities in automobile accidents?
- Download data
- Explore the data: find missing values, identify categorical, numerical, ordinal data fields
- Transform (filter, project)
import requests url = 'https://data-seattlecitygis.opendata.arcgis.com/datasets/5b5c745e0f1f48e7a53acec63a0022ab_0.csv' response = requests.get(url) filehandle = open('~/Desktop/collisions.csv', 'w') filehandle.write(response.text) filehandle.close()
Opening a file is new. Note that "open" can open for reading or writing files. Be careful opening a file to write will erase that file. You can not get it back.
Explore Open the file in Excel. What columns seem to be missing sometimes?
Find a categorical, numerical, and ordinal data field.
Read and Transform
In this section, we will read and transform the data.
Code to open a file and print the first column
file_handle = open('collisions.csv', 'r') # open the csv file for line in file_handle: # loop through the file one line at a time. line_clean = line.strip() # remove the newline character at end of line line_clean_list = line_clean.split(',') # split the line into parts using split print(line_clean_list) # print the first column of data for this row.
Code to open a file, select a subset of rows and columns, and write to a new file Figure out what this code does!
(You'll want to open this window in a wide browser)
file_handle = open('collisions.csv', 'r') # open the csv file header = file_handle.readline() output_handle = open('collisions_transformed.csv', 'w') # NOTE this will overwrite for line in file_handle: # loop through the file one line at a time. line_clean = line.strip() # remove the newline character at end of line line_clean_list = line_clean.split(',') # split the line into parts using split if int(line_clean) > 0: # If the integer value in columns 17 is greater than one then... output_handle.write(line) # write that line to the output. output_handle.close() # Close the output file after the loop.
We will answer together whether accidents involving cyclists or pedestrians are more likely to result in an injury.
We will use the odds ratio to discuss the likelihood of injury.
Be sure you understand the difference between risk ratio and odds ratio!