MTurk Workshop (CASBS 2019): Difference between revisions

Revision as of 03:31, 6 March 2019

When you come on Friday, please bring your laptop!

Readings

Please read this paper before we start on Friday:

Shaw, Aaron. 2015. Hired hands and dubious guesses: Adventures in crowdsourced data collection, from Digital Confidential: The Secrets of Studying Behavior Online, edited by Eszter Hargittai and Christian Sandvig. Cambridge, MA: MIT Press.

Additionally, you will want to skim these two documents. Although I have no expectation that you'll be finishing these, you'll be going back to these while we run our tasks at the session:

Amazon Mechanical Turk Requester UI Guide [Skim, but make sure you're ready to submit tasks.]
Amazon Mechanical Turk Best Practices Guide. [Skim, but make sure you're ready to submit tasks.]

Assignment

In addition to the readings, you'll want to complete the following things before we get to class:

Familiarize yourself and skim the two Amazon Mechanical Turk Guides in the readings above.
Find and complete at least 2 "hits" as a worker on Amazon Mechnical Turk. Note that to do this you will need to create a worker account on Mturk.
- For a number of reasons, I think it's important to understand the experience of the "other" side of the labor market. As a result, I want to spend a few minutes talking about our experience as a worker: What did you do? Who was the requester? What could you was the purpose of the task (as best you could tell)? What was the experience like?
- If you are not a US citizen, skip doing this. This is because working on mTurk involves getting paid and Amazon takes steps to ensure that you have authorization to work. You can try but you will likely get blocked by this.
Create a "requester" account on mTurk. Doing so may require up top 48 hours to be approved so please do that immediately so you have it ready to go in class.
- Put money onto your requestor account to pay workers. A $3 budget should be sufficient for our workshop/meeting. They should take any payment that Amazon does.
[Optional] If you want to have people fill out a form, survey, etc on Qualtrics, Google Forms, or Survey Monkey, get the survey ready ahead of time. For the purposes of the workshop, have it be something minimal (e.g., just a few questions).

In-Meeting Excercise

Feel free to think about this ahead of time but there's no need to start.

When we meet, we will spend the bulk of the time doing the following things:

Design and deploy a small-scale research task on Mturk. Note that to do this, you will need to create a requester account on Mturk. Be sure to allow some time to get the task design the way you want it! Some ideas for study designs you might do:
- A small survey.
- Classification of texts or images (e.g., label tweets, pictures, or comments from a discussion thread).
- A small survey experiment (e.g., you can do a survey where you insert different images and ask the same set of questions. Check out the Mturk requester getting started guide
Prepare to share details of your small-scale research, including results (they will come fast).

@@ Line 1: / Line 1: @@
+When you come on Friday, please bring your laptop!
 == Readings ==
-Please read this paper '''before''' you come on Friday:
+Please read this paper ''before we start on Friday'':
 * Shaw, Aaron. 2015. [http://aaronshaw.org/papers/Shaw-2015-hired_hands_preprint.pdf Hired hands and dubious guesses: Adventures in crowdsourced data collection], from Digital Confidential: The Secrets of Studying Behavior Online, edited by Eszter Hargittai and Christian Sandvig. Cambridge, MA: MIT Press.
-udying Behavior Online. The MIT Press. ''[Available in Canvas]''
-Additionally, I'll want you to skim these two documents. Although I have no expectation that you'll be finishing these, you'll be going back to these while we run our tasks at the session:
+Additionally, you will want to skim these two documents. Although I have no expectation that you'll be finishing these, you'll be going back to these while we run our tasks at the session:
 * [https://docs.aws.amazon.com/AWSMechTurk/latest/RequesterUI/Introduction.html Amazon Mechanical Turk Requester UI Guide] ''[Skim, but make sure you're ready to submit tasks.]''
@@ Line 13: / Line 14: @@
 == Assignment ==
-You'll need to complete the following things before we get to class:
+In addition to the readings, you'll want to complete the following things before we get to class:
 * Familiarize yourself and skim the two Amazon Mechanical Turk Guides in the readings above.
- * Find and complete at least 2 "hits" as a worker on [http://mturk.com Amazon Mechnical Turk]. Note that to do this you will need to create a ''worker'' account on Mturk.
+* Find and complete at least 2 "hits" as a worker on [http://mturk.com Amazon Mechnical Turk]. Note that to do this you will need to create a ''worker'' account on Mturk.
-** I want to spend a few minutes talking about our experience as a worker: What did you do? Who was the requester? What could you was the purpose of the task (as best you could tell)? What was the experience like?
+** For a number of reasons, I think it's important to understand the experience of the "other" side of the labor market. As a result, I want to spend a few minutes talking about our experience as a worker: What did you do? Who was the requester? What could you was the purpose of the task (as best you could tell)? What was the experience like?
-** ''If you are not a US citizen, just skip doing this.'' This is because working on mTurk involves getting paid and Amazon takes steps to ensure that you have authorization to work. You can try but you will likely get blocked by this.
+** ''If you are not a US citizen, skip doing this.'' This is because working on mTurk involves getting paid and Amazon takes steps to ensure that you have authorization to work. You can try but you will likely get blocked by this.
 * Create a "requester" account on [https://www.mturk.com/mturk/welcome mTurk]. Doing so may require up top 48 hours to be approved so please do that ''immediately'' so you have it ready to go in class.
 ** Put money onto your requestor account to pay workers. A $3 budget should be sufficient for our workshop/meeting. They should take any payment that Amazon does.
-* ''[Optional]'' If you want to have people fill out a form, survey, etc on Qualtrics, Google Forms, or Survey Monkey, get the survey done ahead of time. For the purposes of the workshop, have it be something minimal (e.g., just a few questions).
+* ''[Optional]'' If you want to have people fill out a form, survey, etc on Qualtrics, Google Forms, or Survey Monkey, get the survey ready ahead of time. For the purposes of the workshop, have it be something minimal (e.g., just a few questions).
-== In "class" Exercise ==
+== In-Meeting Excercise ==
-We are all going to:
+Feel free to think about this ahead of time but there's no need to start.
+When we meet, we will spend the bulk of the time doing the following things:
 * Design and deploy a small-scale research task on Mturk. Note that to do this, you will need to create a ''requester'' account on Mturk. Be sure to allow some time to get the task design the way you want it! Some ideas for study designs you might do:
@@ Line 31: / Line 34: @@
 ** Classification of texts or images (e.g., label tweets, pictures, or comments from a discussion thread).
 ** A small survey experiment (e.g., you can do a survey where you insert ''different'' images and ask the same set of questions. Check out the [https://requester.mturk.com/help/getting_started.html Mturk requester getting started guide]
-* Prepare to share details of your small-scale research task in class, including results (they will come fast).
+* Prepare to share details of your small-scale research, including results (they will come fast).
-''Note:''