Community Data Science Workshops (Fall 2015)/Day 3 Projects/Civic data: Difference between revisions

From CommunityData
No edit summary
 
(13 intermediate revisions by 2 users not shown)
Line 17: Line 17:
* Graphing and mapping data
* Graphing and mapping data


We will also explore an important non-technical part of data science: thinking critically about data. Thinking critically about the data you have, and what conclusions you can draw from it, is especially important when you are visualizing data, because its easy to mislead people with visualizations. We'll discuss how visualizations based on incorrect data can lead people to make false conclusions, using an example from a recent visualization of building demolitions published in the Northwest design magazine ''Arcade''.
<!-- We will also explore an important non-technical part of data science: thinking critically about data. Thinking critically about the data you have, and what conclusions you can draw from it, is especially important when you are visualizing data, because its easy to mislead people with visualizations. We'll discuss how visualizations based on incorrect data can lead people to make false conclusions, using an example from a recent visualization of building demolitions published in the Northwest design magazine ''Arcade''. -->




== Preparation ==
== Part 1: Downloads ==
''If you are confused by anything today, go back and refresh your memory with the [[Community Data Science Workshops (Fall 2015)/Day 0 setup and tutorial|Day 0 setup and tutorial]] and [[Community Data Science Workshops (Fall 2015)/Day 0 tutorial|Day 0 tutorial]]''
 
;Step 1: Download today's scripts: [http://jtmorgan.net/cdsw/nov7cdsw.zip Python scripts]
 
;Step 2: Get the dataset: Unzip the folder, navigate to it in your terminal, and run this script <pre>download_building_permit_data.py</pre>
 
Now open up the CSV output file to verify that you got all the data you asked for.


;Step 1: Download today's scripts: [http://jtmorgan.net/cdsw/nov7cdsw_scripts.zip Python scripts]
You can view the full set of permit applications at: https://data.seattle.gov/Permitting/Building-Permits-Current/mags-97de
;Step 2: Download today's datasets: [http://jtmorgan.net/cdsw/nov7cdsw_data.zip datasets]


''If you are confused by anything today, go back and refresh your memory with the [[Community Data Science Workshops (Fall 2015)/Day 0 setup and tutorial|Day 0 setup and tutorial]] and [[Community Data Science Workshops (Fall 2015)/Day 0 tutorial|Day 0 tutorial]]''
;Challenge question: how could we change our API query to download the applicant names for all COMMERCIAL building permits issued within the past year?


== Part 1: getting the building permit data ==
== Part 2: charting new construction by month ==
;Question: Has the rate of apartment construction increased since 2010?
;Question: Has the rate of residential construction increased since 2010?


Now we want to learn whether there is, in fact, a housing boom in Seattle. We'll do this by counting how many new permits are issued each month, and then plot these on a graph.


;Today's dataset: [https://data.seattle.gov/Permitting/Building-Permits-Current/mags-97de Click here to view the Seattle Building Permits database]
Run the script <pre>residential_permits_by_month.py</pre>


== Part 2: new construction by month ==
Now open up the CSV output file and check your data. Does this look right to you?


;Challenge question: how could we separate out single family homes from apartments, and count/plot them separately?


== Part 3: new construction by neighborhood ==
== Part 3: charting new construction by neighborhood ==
== Visualizing new home construction in Seattle by neighborhood ==
;Question: which Seattle neighborhoods have had the most multifamily residential construction (apartments and townhomes) since 2010?
;Question: which Seattle neighborhoods have had the most multifamily residential construction (apartments and townhomes) since 2010?


Now we want to learn where all this new construction is happening. We'll do this by sending the address for each MULTIFAMILY permit to the Google Geolocation API, which will return the neighborhood where that address is located.
Run the script <pre>multifamily_permits_by_neighborhood.py</pre>
Now open up the CSV output file and check your data. Does this look right to you?


== Mapping new home construction in Seattle ==
;Challenge question: In which Seattle neighborhood is the cost of new apartment construction projects highest, on average?
 
== Part 4: Mapping new home construction in Seattle ==
;Question: what locations have experienced the highest density of new construction since 2010?
;Question: what locations have experienced the highest density of new construction since 2010?


== Challenge questions ==
Now we will try to get an even more detailed picture of where this construction is occuring, using Google Fusion tables, a powerful visualization application that makes it easy to plot data on a map.
 
Go [https://support.google.com/fusiontables/answer/2571232?hl=en here] and follow the steps to upload your '''new_all_2010-2015.csv''' file.
 
*Create a point map and heatmap.
*Experiment with filtering and weighting the points on your map.
 
;Example map: https://www.google.com/fusiontables/data?docid=1gm0wVqnK7zQ7hgp5bsKoBIetMZ75jo-VMs5noDoJ#map:id=3
 
== Going further ==
''Try to answer these additional questions that draw on the data and methods we're learning today. Ask a mentor if you get stuck!''
''Try to answer these additional questions that draw on the data and methods we're learning today. Ask a mentor if you get stuck!''
#In which Seattle neighborhood is the cost of new construction projects highest, on average?
#Which developer has spent the most on new construction in Seattle since 2010?
#Where in Seattle are the most commercial buildings being constructed in 2015?
#How many townhouses have been constructed in Seattle since 2010?
#How does the rate of residential construction in Seattle from 2010-2015 compare to the previous 5 years?
#Where in Seattle are the most ''commercial'' buildings being constructed in 2015?
#How many townhouses have been constructed in Seattle since 2011?
#How does the rate of residential construction in Seattle from 2010-2015 compare to [https://data.seattle.gov/Permitting/Building-Permits-Older-than-5-years/47eb-r92t the previous 5 years]?




Line 55: Line 78:
== Resources ==
== Resources ==


;Press about the construction boom
*http://www.theurbanist.org/2015/10/20/fact-check-no-explosion-in-demolitions/
*http://arcadenw.org/article/changing-seattle
*http://www.seattlemag.com/article/demolitions-seattle-no-neighborhood-unaffected
*http://www.seattletimes.com/pacific-nw-magazine/seattles-building-boom-is-good-news-for-a-new-generation-of-workers/
=== Datasets ===
* Permits, last 5 years (today's dataset): https://data.seattle.gov/Permitting/Building-Permits-Current/mags-97de
* Permits older than 5 years: https://data.seattle.gov/Permitting/Building-Permits-Older-than-5-years/47eb-r92t
;Custom Socrata datasets
multifamily 2010-2015: https://data.seattle.gov/Permitting/Building-permits-new-multifamily-residential-const/ma3y-m69a
single and multifamily 2010-2015: https://data.seattle.gov/Permitting/Building-permits-new-residential-construction/kdfe-reh3
=== Sample visualizations ===
*Building permit charts: https://docs.google.com/spreadsheets/d/15DBcWnCroga4B1_ss66YjW9hlJxEk3g1UK7khFLpbQM/edit#gid=0
*Building permit Fusion table: https://www.google.com/fusiontables/DataSource?docid=1gm0wVqnK7zQ7hgp5bsKoBIetMZ75jo-VMs5noDoJ#rows:id=1
=== APIs ===
* Hurl.it API testing tool: https://www.hurl.it
;Sample API queries
* Building permit data: https://data.seattle.gov/resource/mags-97de.json?$where=issue_date%20IS%20NOT%20NULL%20&action_type=NEW&category=MULTIFAMILY&$limit=100
* Google maps location (using street address): http://maps.googleapis.com/maps/api/geocode/json?address=5601+22ND+AVE+NW+Seattle+WA
* Google maps location (using lat/long): http://maps.googleapis.com/maps/api/geocode/json?latlng=47.66979666%2C-122.38570052
=== Google tools ===
*Fusion tables: https://support.google.com/fusiontables/answer/2571232
*About Fusion table heatmaps: https://support.google.com/fusiontables/answer/1152262
*About Google maps geocoding API: https://developers.google.com/maps/documentation/geocoding/
=== Socrata open data portal===
*More about the Socrata open data portal and API: http://www.socrata.com/products/open-data-portal/
* Socrata API help resources: http://dev.socrata.com/consumers/getting-started.html
:* filtering results: http://dev.socrata.com/docs/filtering.html
:* dealing with timestamps: http://dev.socrata.com/docs/datatypes/timestamp.html
:* writing API queries: http://dev.socrata.com/docs/queries.html
;Data portal-powered apps
*https://www.seattleinprogress.com/
*http://web6.seattle.gov/mnm/
;Socrata API instructional videos
* https://data.seattle.gov/videos
* https://www.youtube.com/watch?v=YlKzXTrTLOQ
* https://www.youtube.com/watch?v=Whfp8ojMf0U
* https://www.youtube.com/watch?v=Vd6bwz3ivVA


;Some other government website that use the Socrata API
* https://data.austintexas.gov/
* https://data.cityofchicago.org/
* https://data.cityofnewyork.us/


[[Category:Fall_2015_series]]
[[Category:Fall_2015_series]]

Latest revision as of 23:48, 7 November 2015

What neighborhoods have changed the most since the beginning of the Seattle building boom?


Building and visualizing datasets using data.seattle.gov and Google apps[edit]

In this project, we will explore a few ways to analyze and visualize data from data.seattle.gov. We will examine trends in the construction of houses, townhomes, and condos in Seattle over the last five years, in order to determine the true extent of the current "construction boom".

  • We'll download data from two APIs (Socrata open data platform and Google Maps)
  • We'll analyze trends over time in our dataset
  • We'll visualize our data in graphs and heatmaps

Topics we will cover[edit]

  • Writing and testing complex API queries
  • Reading and writing CSV and JSON files
  • Filtering and aggregating data
  • Combining data from multiple APIs
  • Graphing and mapping data


Part 1: Downloads[edit]

If you are confused by anything today, go back and refresh your memory with the Day 0 setup and tutorial and Day 0 tutorial

Step 1
Download today's scripts: Python scripts
Step 2
Get the dataset: Unzip the folder, navigate to it in your terminal, and run this script
download_building_permit_data.py

Now open up the CSV output file to verify that you got all the data you asked for.

You can view the full set of permit applications at: https://data.seattle.gov/Permitting/Building-Permits-Current/mags-97de

Challenge question
how could we change our API query to download the applicant names for all COMMERCIAL building permits issued within the past year?

Part 2: charting new construction by month[edit]

Question
Has the rate of residential construction increased since 2010?

Now we want to learn whether there is, in fact, a housing boom in Seattle. We'll do this by counting how many new permits are issued each month, and then plot these on a graph.

Run the script

residential_permits_by_month.py

Now open up the CSV output file and check your data. Does this look right to you?

Challenge question
how could we separate out single family homes from apartments, and count/plot them separately?

Part 3: charting new construction by neighborhood[edit]

Question
which Seattle neighborhoods have had the most multifamily residential construction (apartments and townhomes) since 2010?

Now we want to learn where all this new construction is happening. We'll do this by sending the address for each MULTIFAMILY permit to the Google Geolocation API, which will return the neighborhood where that address is located.

Run the script

multifamily_permits_by_neighborhood.py

Now open up the CSV output file and check your data. Does this look right to you?

Challenge question
In which Seattle neighborhood is the cost of new apartment construction projects highest, on average?

Part 4: Mapping new home construction in Seattle[edit]

Question
what locations have experienced the highest density of new construction since 2010?

Now we will try to get an even more detailed picture of where this construction is occuring, using Google Fusion tables, a powerful visualization application that makes it easy to plot data on a map.

Go here and follow the steps to upload your new_all_2010-2015.csv file.

  • Create a point map and heatmap.
  • Experiment with filtering and weighting the points on your map.
Example map
https://www.google.com/fusiontables/data?docid=1gm0wVqnK7zQ7hgp5bsKoBIetMZ75jo-VMs5noDoJ#map:id=3

Going further[edit]

Try to answer these additional questions that draw on the data and methods we're learning today. Ask a mentor if you get stuck!

  1. Which developer has spent the most on new construction in Seattle since 2010?
  2. How many townhouses have been constructed in Seattle since 2010?
  3. Where in Seattle are the most commercial buildings being constructed in 2015?
  4. How does the rate of residential construction in Seattle from 2010-2015 compare to the previous 5 years?


Resources[edit]

Press about the construction boom

Datasets[edit]

Custom Socrata datasets

multifamily 2010-2015: https://data.seattle.gov/Permitting/Building-permits-new-multifamily-residential-const/ma3y-m69a single and multifamily 2010-2015: https://data.seattle.gov/Permitting/Building-permits-new-residential-construction/kdfe-reh3

Sample visualizations[edit]

APIs[edit]

Sample API queries


Google tools[edit]

Socrata open data portal[edit]

Data portal-powered apps
Socrata API instructional videos
Some other government website that use the Socrata API