Statistics and Statistical Programming (Spring 2019)/Problem Set: Week 4: Difference between revisions

From CommunityData
(Created page with "== Programming Challenges == This week we are going to do more advanced data wrangling. We'll use two datasets from [https://data.seattle.gov data.seattle.gov] related to onl...")
 
No edit summary
Line 1: Line 1:
== Programming Challenges ==
== Programming Challenges ==


This week we are going to do more advanced data wrangling. We'll use two datasets from [https://data.seattle.gov data.seattle.gov] related to online engagement. Both datasets are drawn from Google Analytics. We're going to merge them together in order to analyze the proportion of pageviews from mobile users and determine whether this proportion is going up or down over time.  
This week we are going to do more advanced data wrangling. We'll use two datasets from [https://data.seattle.gov data.seattle.gov] related to online engagement with the city of Seattle's data portal. Both datasets are drawn from Google Analytics. We're going to merge them together in order to analyze the proportion of pageviews from mobile users and determine whether this proportion is going up or down over time.  


The datasets are messy and aren't set up well to help us answer the question. Luckily, you know how to use R to solve these problems!
The datasets are messy and aren't set up well to help us answer the question. Luckily, you know how to use R to solve these problems!


:'''PC1.''' Download these two datasets from data.seattle.gov, save them into your git repository: [https://data.seattle.gov/City-Business/COS-Statistics-Top5000-Pages/yuhv-gvtm COS-Statistics-Top5000-Pages] [https://data.seattle.gov/City-Business/COS-Statistics-Mobile-Sessions/2u47-byfn COS-Statistics-Mobile Sessions]. Assume, for the purposes of this assignment that the first dataset on views to the top 5000 pages is ''all'' views made to the http://www.seattle.gov website
:'''PC1.''' The two "raw" datasets come from data.seattle.gov and are available in the data repository for the course: [https://data.seattle.gov/City-Business/COS-Statistics-Top5000-Pages/yuhv-gvtm COS-Statistics-Top5000-Pages] [https://data.seattle.gov/City-Business/COS-Statistics-Mobile-Sessions/2u47-byfn COS-Statistics-Mobile Sessions]. Assume, for the purposes of this assignment, that the first dataset on pageviews to the top 5000 pages is ''all'' pageviews made to the http://www.seattle.gov website
:'''PC2.''' Load both datasets into R as separate data frames. Explore the data to get a sense of the structure of the data. What are the columns, rows, missing data, etc? Write code to take (and then check/look at) several random subsamples of the data.
:'''PC2.''' Load both datasets into R as separate data frames. Explore the data to get a sense of the structure. What are the columns, rows, missing data, etc? Write code to take several random samples of the data (and then look at them!).
:'''PC3.''' Using the top 5000 dataset, create a new data frame where one column is each month (as described in the data) and a second column is the total number of views made to all pages in the dataset over that month.
:'''PC3.''' Using the top 5000 data, create a new data frame where one column is each month (as described in the data) and a second column is the total number of views made to all pages in the dataset over that month.
:'''PC4.''' Using the mobile dataset, create a new data frame where one column is each month described in the data and the second is a measure (estimate?) of the total number of views made by mobiles (all platforms) over each month. This will involve at least two steps since total views are included. You'll need to first use the data there to create a measure of the total views for each line in the dataset.
:'''PC4.''' Using the mobile data, create a new data frame where one column is each month described in the data and the second is a measure (estimate?) of the total number of views made by mobile devices (all platforms) over each month. This will involve at least two steps since total views are included. You'll need to first use the data there to create a measure of the total views for each line in the dataset.
:'''PC5.''' Merge your two datasets together into a new dataset with columns for each month, total views (across the top 5000 pages) and total mobile views. Are there are missing data? Can you tell why?
:'''PC5.''' Merge your two datasets together into a new dataset with columns for each month, total views (across the top 5000 pages) and total mobile views. Are there are missing data? Can you tell why?
:'''PC6.''' Create a new column in your merged dataset that describes your best estimate of the proportion (or percentage, if you really must!) of views that comes from mobile. Be able to talk about the assumptions you've made here. Make sure that date, in this final column, is a date or datetime object in R.
:'''PC6.''' Create a new column in your merged dataset that describes your best estimate of the proportion of total views that come from mobile. Be able to talk about any assumptions/decisions you've made in constructing your estimate. Make sure that date, in this final column, is a date or datetime object in R.
:'''PC7.''' Graph this over time and be ready to describe: (a) your best estimate of the proportion of views from mobiles to the Seattle City website over time and (b) an indication of whether it's going up or down.
:'''PC7.''' Graph the proportion over time and be ready to describe: (a) your best estimate of the proportion of views from mobile devices to the Seattle City website over time and (b) an indication of whether it's going up or down.


== Statistical Questions ==
== Statistical Questions ==

Revision as of 20:38, 16 April 2019

Programming Challenges

This week we are going to do more advanced data wrangling. We'll use two datasets from data.seattle.gov related to online engagement with the city of Seattle's data portal. Both datasets are drawn from Google Analytics. We're going to merge them together in order to analyze the proportion of pageviews from mobile users and determine whether this proportion is going up or down over time.

The datasets are messy and aren't set up well to help us answer the question. Luckily, you know how to use R to solve these problems!

PC1. The two "raw" datasets come from data.seattle.gov and are available in the data repository for the course: COS-Statistics-Top5000-Pages COS-Statistics-Mobile Sessions. Assume, for the purposes of this assignment, that the first dataset on pageviews to the top 5000 pages is all pageviews made to the http://www.seattle.gov website
PC2. Load both datasets into R as separate data frames. Explore the data to get a sense of the structure. What are the columns, rows, missing data, etc? Write code to take several random samples of the data (and then look at them!).
PC3. Using the top 5000 data, create a new data frame where one column is each month (as described in the data) and a second column is the total number of views made to all pages in the dataset over that month.
PC4. Using the mobile data, create a new data frame where one column is each month described in the data and the second is a measure (estimate?) of the total number of views made by mobile devices (all platforms) over each month. This will involve at least two steps since total views are included. You'll need to first use the data there to create a measure of the total views for each line in the dataset.
PC5. Merge your two datasets together into a new dataset with columns for each month, total views (across the top 5000 pages) and total mobile views. Are there are missing data? Can you tell why?
PC6. Create a new column in your merged dataset that describes your best estimate of the proportion of total views that come from mobile. Be able to talk about any assumptions/decisions you've made in constructing your estimate. Make sure that date, in this final column, is a date or datetime object in R.
PC7. Graph the proportion over time and be ready to describe: (a) your best estimate of the proportion of views from mobile devices to the Seattle City website over time and (b) an indication of whether it's going up or down.

Statistical Questions

Exercises from OpenIntro §4

SQ0. Any questions or clarifications from the OpenIntro text or lecture notes?
SQ1. Exercise 4.8 on Twitter users and news
SQ2. Exercise 4.10 which is a continuation of 4.8
SQ3. Exercise 4.19 on online communication
SQ4. Exercise 4.32 which is asking you to explain why certain statements about statistical inference are true or false

Empirical Paper

Revisit the paper we read for Week 1 of the course:

Kramer, Adam D. I., Jamie E. Guillory, and Jeffrey T. Hancock. 2014. Experimental Evidence of Massive-Scale Emotional Contagion through Social Networks. Proceedings of the National Academy of Sciences 111(24):8788–90. [Open Access]

Come to class prepared to discuss your answers to the following questions

EQ1. Describe, in your own words, the main effects estimated in this paper.
EQ2. The authors report Cohen's d] along with their regression estimates. Look up and review the formula for Cohen's d. Discuss the substantive or practical significance of the estimates given the magnitude of d reported.