Editing Human Centered Data Science (Fall 2019)/Assignments

From CommunityData

Warning: You are not logged in. Your IP address will be publicly visible if you make any edits. If you log in or create an account, your edits will be attributed to your username, along with other benefits.

The edit can be undone. Please check the comparison below to verify that this is what you want to do, and then save the changes below to finish undoing the edit.

Latest revision Your text
Line 235: Line 235:
 
''A repository with a template README file and examples of querying the ORES datastore in R and Python can be found [https://github.com/Ironholds/data-512-a2 here]''
 
''A repository with a template README file and examples of querying the ORES datastore in R and Python can be found [https://github.com/Ironholds/data-512-a2 here]''
  
Now you need to get the predicted quality scores for each article in the Wikipedia dataset. We're using a machine learning system called [https://www.mediawiki.org/wiki/ORES ORES] ("Objective Revision Evaluation Service"). ORES estimates the quality of an article (at a particular point in time), and assigns a series of probabilities that the article is in one of 6 quality categories. The options are, from best to worst:
+
Now you need to get the predicted quality scores for each article in the Wikipedia dataset. For this step, we're using a Wikimedia API endpoint for a machine learning system called [https://www.mediawiki.org/wiki/ORES ORES] ("Objective Revision Evaluation Service"). ORES estimates the quality of an article (at a particular point in time), and assigns a series of probabilities that the article is in one of 6 quality categories. The options are, from best to worst:
  
 
# FA - Featured article
 
# FA - Featured article
Line 276: Line 276:
 
==== Combining the datasets ====
 
==== Combining the datasets ====
 
   
 
   
Some processing of the data will be necessary! In particular, you'll need to - after retrieving and including the ORES data for each article - merge the wikipedia data and population data together. Both have fields containing country names for just that purpose. After merging the data, you'll invariably run into entries which ''cannot'' be merged. Either the population dataset does not have an entry for the equivalent Wikipedia country, or vis versa.  
+
Some processing of the data will be necessary! In particular, you'll need to - after retrieving and including the ORES data for each article - merge the wikipedia data and population data together. Both have fields containing country names for just that purpose. After merging the data, you'll invariably run into entries which ''cannot'' be merged. Either the population dataset does not have an entry for the equivalent Wikipedia country, or vice versa.  
  
 
Please remove any rows that do not have matching data, and output them to a CSV file called <tt>wp_wpds_countries-no_match.csv</tt>
 
Please remove any rows that do not have matching data, and output them to a CSV file called <tt>wp_wpds_countries-no_match.csv</tt>
Line 335: Line 335:
 
# Can you think of a realistic data science research situation where using these data (to train a model, perform a hypothesis-driven research, or make business decisions) might create biased or misleading results, due to the inherent gaps and limitations of the data?
 
# Can you think of a realistic data science research situation where using these data (to train a model, perform a hypothesis-driven research, or make business decisions) might create biased or misleading results, due to the inherent gaps and limitations of the data?
 
# Can you think of a realistic data science research situation where using these data (to train a model, perform a hypothesis-driven research, or make business decisions) might still be appropriate and useful, despite its inherent limitations and biases?
 
# Can you think of a realistic data science research situation where using these data (to train a model, perform a hypothesis-driven research, or make business decisions) might still be appropriate and useful, despite its inherent limitations and biases?
# How might a researcher supplement or transform this dataset to potentially ''correct for'' the limitations/biases you observed?
 
  
 
This section doesn't need to be particularly long or thorough, but we'll expect you to write at least a couple paragraphs.
 
This section doesn't need to be particularly long or thorough, but we'll expect you to write at least a couple paragraphs.
Line 365: Line 364:
 
For this assignment, you will go undercover as a member of the Amazon Mechanical Turk community. You will preview or perform Mechanical Turk tasks (called "HITs"), lurk in Turk worker discussion forums, and write an ethnographic account of your experience as a crowdworker, and how this experience changes your understanding of the phenomenon of crowdwork.
 
For this assignment, you will go undercover as a member of the Amazon Mechanical Turk community. You will preview or perform Mechanical Turk tasks (called "HITs"), lurk in Turk worker discussion forums, and write an ethnographic account of your experience as a crowdworker, and how this experience changes your understanding of the phenomenon of crowdwork.
  
The full assignment description is available [https://docs.google.com/document/d/16lZdTxkw1meUPMzA-BYl8TVtk0Jxv4Wh8mbZq_BursM/edit?usp=sharing as a Google doc].
+
The full assignment description is available [https://docs.google.com/document/d/16lZdTxkw1meUPMzA-BYl8TVtk0Jxv4Wh8mbZq_BursM/edit?usp=sharing as a Google doc] and [[:File:HCDS_Crowdwork_ethnography_instructions.pdf|as a PDF]].
  
 
=== A4: Final project proposal ===
 
=== A4: Final project proposal ===
The final project proposal is a short pitch for your final class project. It should include three basic components:
+
''to come''
* '''Motivation/problem statement:''' Why are you planning to do this analysis? Why is it potentially interesting and useful, from a scientific, practical, and/or human-centered perspective? What do you hope to learn? Note that you only need to describe your overall research goal at this stage; specific hypotheses or research questions aren’t necessary in the project proposal.
 
  
* '''Data used:''' What dataset do you plan to use, and why? Summarize what is represented in the dataset; Link to the dataset, and specify license/terms of use; Briefly justify why this dataset is relevant to your problem statement; Highlight any possible ethical considerations to using this dataset (and say why or why not).
+
=== A5: Final project plan ===
 
+
''to come''
* '''Unknowns and dependencies:''' Are there any factors outside of your control that might impact your ability to complete this project by the end of the quarter? The purpose of this section is to get you thinking, in a practical sense, about your ability to complete this project within the time allotted.
+
<!--
 +
''For examples of datasets you may want to use for your final project, see [[HCDS_(Fall_2017)/Datasets]].''
 +
-->
  
=== A5: Final project plan ===
 
 
For this assignment, you will write up a study plan for your final class project. The plan will cover a variety of details about your final project, including what data you will use, what you will do with the data (e.g. statistical analysis, train a model), what results you expect or intend, and most importantly, why your project is interesting or important (and to whom, besides yourself).
 
For this assignment, you will write up a study plan for your final class project. The plan will cover a variety of details about your final project, including what data you will use, what you will do with the data (e.g. statistical analysis, train a model), what results you expect or intend, and most importantly, why your project is interesting or important (and to whom, besides yourself).
 
The final project plan is an extension of the proposal, and should be in the same (.ipynb or .md) document in your repo. New sections to add are:
 
 
* '''Research questions and/or hypotheses:''' These describe what you hope to discover or determine in the course of your research.
 
:* Example research question: what is the impact of an MS degree on data scientist salaries over the course of their careers?
 
:* Example hypothesis: earning an MS degree is associated with an increase of x% in career data scientist salaries compared to similar data scientists who do not earn a degree
 
 
* '''Background and/or Related Work:''' What is already known about the phenomenon you are investigating? How does previous research or background info inform your decision to perform this study, the way you designed the study, or your specific research questions? Make sure to include references (endnotes and/or inline hyperlinks) to the sources of background information--whether they are websites, news articles, or peer-reviewed research.
 
 
* '''Methodology:''' Describe how you plan to investigate this phenomenon. Don't just describe what your analytical methods are (e.g. "ordinary least squares", "student's t-test", "heatmap visualization", or "recurrent neural network"), it's critical to justify why these are appropriate methods for gathering and analyzing your data, or presenting your findings. You are expected to be thorough here: please describe to the best of your ability the entire series of gathering, analysis, and presentation methods you plan to use.
 
  
 
=== A6: Final project presentation ===
 
=== A6: Final project presentation ===
For this assignment, you will give an in-class presentation of your final project. The goal of this presentation is to demonstrate that you are able to effectively communicate your research questions, methods, conclusions, and implications to a non-data-scientist audience.
+
For this assignment, you will give an in-class presentation of your final project. The goal of this assignment is to demonstrate that you are able to effectively communicate your research questions, methods, conclusions, and implications to your target audience.
 
 
The presentation will be no more than 5 minutes long. Slides are not necessary, but are probably a good idea.
 
 
 
The presentation should demonstrate the following:
 
* Your ability to give a professional research presentation.
 
* Your ability to communicate the importance of your research to a specified audience (Imagine that you are pitching your project to directors/execs at a company you work for).
 
* Your ability to communicate the nature and implications of your findings in an accurate and compelling way.
 
* Your ability to do all of the above in a very short time (Hint: please practice beforehand and time yourself)
 
  
 
=== A7: Final project report ===
 
=== A7: Final project report ===
 
For this assignment, you will publish the complete code, data, and analysis of your final research project. The goal is to demonstrate that you can incorporate all of the human-centered design considerations you learned in this course and create research artifacts that are understandable, impactful, and reproducible.
 
For this assignment, you will publish the complete code, data, and analysis of your final research project. The goal is to demonstrate that you can incorporate all of the human-centered design considerations you learned in this course and create research artifacts that are understandable, impactful, and reproducible.
 
A successful report will take the form of a well-written, well-executed research study document (a repo with a notebook + supporting data files and documentation) that includes:
 
* All your code and data, thoroughly documented and reproducible
 
* A human-centered argument for why your analysis is important
 
* Background research or related work
 
* Your research question(s)
 
* The methods, data, and approach that you used to collect and analyze the data
 
* Findings, implications, and limitations of your study
 
* A thoughtful reflection that describes the specific ways that human-centered data science principles informed your decision-making in this project—from beginning to end.
 
 
Data visualizations aren’t necessary, but are encouraged (they are often an effective way of communicating your findings!)
 
 
Your deliverables for the final project proposal and plan are part of this report: you are expected to build your report around these documents.
 
 
  
  
 
[[Category:HCDS (Fall 2019)]]
 
[[Category:HCDS (Fall 2019)]]

Please note that all contributions to CommunityData are considered to be released under the Attribution-Share Alike 3.0 Unported (see CommunityData:Copyrights for details). If you do not want your writing to be edited mercilessly and redistributed at will, then do not submit it here.
You are also promising us that you wrote this yourself, or copied it from a public domain or similar free resource. Do not submit copyrighted work without permission!

To protect the wiki against automated edit spam, we kindly ask you to solve the following CAPTCHA:

Cancel Editing help (opens in new window)