Editing Internet Research Methods (Spring 2016) (section)

=== Week 6: Wednesday May 4: Crowdsourced Data Analysis and Experimentation ===

'''Assignment:'''

* Find and complete at least 2 "hits" as a worker on [http://mturk.com Amazon Mechnical Turk]. Note that to do this you will need to create a ''worker'' account on Mturk. 
** Record (write down) details and notes about your tasks: What did you do? Who was the requester? What could you was the purpose of the task (as best you could tell)? What was the experience like? What research applications can you (not) imagine for this kind of system? 
* Design and deploy a small-scale research task on Mturk. Note that to do this, you will need to create a ''requester'' account on Mturk. Be sure to allow some time to get the task design the way you want it! Some ideas for study designs you might do:
** A small survey. 
** Classification of texts or images (e.g., label tweets, pictures, or comments from a discussion thread).
** A small experiment (e.g., you can do a survey where you insert ''different'' images and ask the same set of questions. Check out the [https://requester.mturk.com/help/getting_started.html Mturk requester getting started guide]
* Prepare to share details of your small-scale research task in class, including results (they will come fast).

''Note:'' In terms of running your task, it will cost real money and you have to put money on your Amazon account yourself. You've each got a $3 budget. Please use your credit card to put $3 on your account right away. I will pay each of you $3 in cash next week to reimburse you for the cost of running the experiment.

'''Required Readings:'''

* [https://docs.aws.amazon.com/AWSMechTurk/latest/RequesterUI/Introduction.html Amazon Mechanical Turk Requester UI Guide] ''[Free Online]''
* [https://mturkpublic.s3.amazonaws.com/docs/MTURK_BP.pdf Amazon Mechanical Turk Best Practices Guide]. ''[Free Online]''
* Weinberg, J., Freese, J., & McElhattan, D. (2014). [https://www.sociologicalscience.com/articles-vol1-19-292/ Comparing Data Characteristics and Results of an Online Factorial Survey between a Population-Based and a Crowdsource-Recruited Sample]. Sociological Science, 1, 292–310. ''[Free Online]''
* Shaw, A. (2015). [https://canvas.uw.edu/files/36419326/download?download_frd=1 Hired Hands and Dubious Guesses: Adventures in Crowdsourced Data Collection]. In E. Hargittai & C. Sandvig (Eds.), Digital Research Confidential: The Secrets of Studying Behavior Online. The MIT Press. ''[Available in Canvas]''

'''Optional Readings:'''

* Gray, M. L., Suri, S., Ali, S. S., & Kulkarni, D. (2016). [http://sidsuri.com/Publications_files/collab_paper21.pdf The Crowd is a Collaborative Network]. Proceedings of Computer-Supported Cooperative Work. ''[Free Online]''
* Kittur et al. (2013). [http://hci.stanford.edu/publications/2013/CrowdWork/futureofcrowdwork-cscw2013.pdf The Future of Crowd Work]. Proceedings of Computer-Supported Cooperative Work. ''[Free Online]''

'''Resources:'''
* [http://www.mturk-tracker.com/ Mturk Tracker]