Editing MTurk Workshop (CASBS 2019)

;When:Friday March 8 2019, 1:30-3pm
;Where:CASBS Large Conference Room
;Materials:Bring your laptop w/ 1.5h worth of power and/or a plug!

== Readings ==

Please read this paper ''before we start on Friday'':

* Shaw, Aaron. 2015. [http://aaronshaw.org/papers/Shaw-2015-hired_hands_preprint.pdf Hired hands and dubious guesses: Adventures in crowdsourced data collection], from Digital Confidential: The Secrets of Studying Behavior Online, edited by Eszter Hargittai and Christian Sandvig. Cambridge, MA: MIT Press. 

Additionally, you will want to skim these two documents. Although I have no expectation that you'll be finishing these, you'll be going back to these while we run our tasks at the session:

* [https://docs.aws.amazon.com/AWSMechTurk/latest/RequesterUI/Introduction.html Amazon Mechanical Turk Requester UI Guide] ''[Skim, but make sure you're ready to submit tasks.]''
* [https://mturkpublic.s3.amazonaws.com/docs/MTURK_BP.pdf Amazon Mechanical Turk Best Practices Guide]. ''[Skim, but make sure you're ready to submit tasks.]''

== Assignment ==

In addition to the readings, you'll want to complete the following things before we get to class:

* Find and complete at least 2 "hits" as a worker on [http://mturk.com Amazon Mechnical Turk]. Note that to do this you will need to create a ''worker'' account on MTurk. 
** Be ready to spend a few minutes talking about your experience as a worker: What did you do? Who was the requester? What could you was the purpose of the task (as best you could tell)? What was the experience like?
** ''If you are not a US citizen, skip doing this.'' Because working on MTurk involves getting paid, Amazon takes steps to ensure that US workers have legal authorization to work for payment.
* Create a "requester" account on [https://www.mturk.com/mturk/welcome MTurk]. Doing so may require up top 48 hours to be approved so please do that ''immediately'' so you have it ready to go in class.
** Put money onto your requestor account to pay workers. A $3 budget should be sufficient for our workshop/meeting. They should take any payment that Amazon does.
* ''[Optional]'' If you want to have people fill out a form, survey, etc on Qualtrics, Google Forms, or Survey Monkey, get the survey ready ahead of time. For the purposes of the workshop, have it be something minimal (e.g., just a few questions).

== In-Meeting Excercise ==

Feel free to think about this ahead of time but there's no need to start. 

When we meet, we will spend the bulk of the time doing the following things:

* Design and deploy a small-scale research task on MTurk. Note that to do this, you will need to create a ''requester'' account on MTurk. Be sure to allow some time to get the task design the way you want it! Some ideas for study designs you might do:
** A small survey. 
** Classification of texts or images (e.g., label tweets, pictures, or comments from a discussion thread).
** A small survey experiment (e.g., you can do a survey where you insert ''different'' images and ask the same set of questions. Check out the [https://requester.mturk.com/help/getting_started.html MTurk requester getting started guide]
* Prepare to share details of your small-scale research, including results (they will come fast).
@@ Line 2: / Line 2: @@
 ;Where:CASBS Large Conference Room
 ;Materials:Bring your laptop w/ 1.5h worth of power and/or a plug!
-;Notes:[[MTurk Workshop (CASBS 2019)/Notes]]
 == Readings ==
@@ Line 19: / Line 18: @@
 In addition to the readings, you'll want to complete the following things before we get to class:
-* Find and complete at least 2 "hits" as a worker on [http://mturk.com Amazon Mechanical Turk]. Note that to do this you will need to create a ''worker'' account on MTurk.
+* Find and complete at least 2 "hits" as a worker on [http://mturk.com Amazon Mechnical Turk]. Note that to do this you will need to create a ''worker'' account on MTurk.
 ** Be ready to spend a few minutes talking about your experience as a worker: What did you do? Who was the requester? What could you was the purpose of the task (as best you could tell)? What was the experience like?
 ** ''If you are not a US citizen, skip doing this.'' Because working on MTurk involves getting paid, Amazon takes steps to ensure that US workers have legal authorization to work for payment.
 * Create a "requester" account on [https://www.mturk.com/mturk/welcome MTurk]. Doing so may require up top 48 hours to be approved so please do that ''immediately'' so you have it ready to go in class.
 ** Put money onto your requestor account to pay workers. A $3 budget should be sufficient for our workshop/meeting. They should take any payment that Amazon does.
-* ''[Optional]'' If you want to have people fill out a form, survey, etc on Qualtrics, Google Forms, or SurveyMonkey, get the survey ready ahead of time. For the purposes of the workshop, have it be something minimal (e.g., just a few questions).
+* ''[Optional]'' If you want to have people fill out a form, survey, etc on Qualtrics, Google Forms, or Survey Monkey, get the survey ready ahead of time. For the purposes of the workshop, have it be something minimal (e.g., just a few questions).
-== In-Meeting Exercise ==
+== In-Meeting Excercise ==
-''Feel free to think about this ahead of time but there's no need to start since we'll do this all together.''
+Feel free to think about this ahead of time but there's no need to start.
 When we meet, we will spend the bulk of the time doing the following things:
@@ Line 37: / Line 36: @@
 ** A small survey experiment (e.g., you can do a survey where you insert ''different'' images and ask the same set of questions. Check out the [https://requester.mturk.com/help/getting_started.html MTurk requester getting started guide]
 * Prepare to share details of your small-scale research, including results (they will come fast).
-== Extra Readings ==
-Technical documentation on making requests on MTurk:
-* [https://blog.mturk.com/tutorials/home Tutorial posted by Amazon on the MTurk blog]
-* Mason, Winter, and Siddharth Suri. 2011. “Conducting Behavioral Research on Amazon’s Mechanical Turk.” Behavior Research Methods 44 (1): 1–23. https://doi.org/10.3758/s13428-011-0124-6. [''Dated but still somewhat useful.'']
-Culture and work conditions:
-* Kittur, Aniket, Jeffrey V. Nickerson, Michael Bernstein, Elizabeth Gerber, Aaron Shaw, John Zimmerman, Matt Lease, and John Horton. 2013. “The Future of Crowd Work.” In Proceedings of the 2013 Conference on Computer Supported Cooperative Work, 1301–1318. CSCW ’13. New York, NY, USA: ACM. https://doi.org/10.1145/2441776.2441923.
-* Irani, Lilly. 2015. “The Cultural Work of Microwork.” New Media & Society 17 (5): 720–39. https://doi.org/10.1177/1461444813511926.
-* Semuels, Alana. 2019. "The Internet Is Enabling a New Kind of Poorly Paid Hell." The Atlantic. January 23, 2018. https://www.theatlantic.com/business/archive/2018/01/amazon-mechanical-turk/551192/
-Systems to improve Turker experiences:
-* Irani, Lilly, and M. Six Silberman. 2013. “Turkopticon: Interrupting Worker Invisibility in Amazon Mechanical Turk.” In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 611–620. CHI ’13. New York, NY, USA: ACM. https://doi.org/10.1145/2470654.2470742.
-* Salehi, Niloufar, Lilly C. Irani, Michael S. Bernstein, Ali Alkhatib, Eva Ogbe, Kristy Milland, and Clickhappier. 2015. “We Are Dynamo: Overcoming Stalling and Friction in Collective Action for Crowd Workers.” In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, 1621–1630. CHI ’15. New York, NY, USA: ACM. https://doi.org/10.1145/2702123.2702508.