Editing MTurk Workshop (CASBS 2019)/Notes
From CommunityData
The edit can be undone. Please check the comparison below to verify that this is what you want to do, and then publish the changes below to finish undoing the edit.
Latest revision | Your text | ||
Line 28: | Line 28: | ||
The MTurk platform has an enormous bias toward support of the "requestor" side of the labor market. That said, Turkers are organized: | The MTurk platform has an enormous bias toward support of the "requestor" side of the labor market. That said, Turkers are organized: | ||
* [https://www.reddit.com/r/TurkerNation/ TurkerNation] | * [https://www.reddit.com/r/TurkerNation/ TurkerNation] | ||
Line 44: | Line 45: | ||
Approaches to building these vary. | Approaches to building these vary. | ||
== Anatomy of MTurk request == | |||
<ul> | <ul> | ||
Line 50: | Line 51: | ||
<li>(Optionally) input data that you want coded, processed, etc.</li> | <li>(Optionally) input data that you want coded, processed, etc.</li> | ||
<li>Tasks/HITs come in '''Batches''' which are defined as ('''Input Data CSV''' * '''Project/Design Template''')</li> | <li>Tasks/HITs come in '''Batches''' which are defined as ('''Input Data CSV''' * '''Project/Design Template''')</li> | ||
<li>Assignments (the number of workers | <li>Assignments (the number of workers assiged to each HIT). A specific Turker can only do a given HIT/task once.</li> | ||
<li><p>[https://blog.mturk.com/managing-groups-of-hits-on-mechanical-turk-3194c045371f Groups of tasks] — If you republish multiple batches of the same task ''or'' different tasks with same name/etc, they show up as the same HIT in the worker interface!</p> | <li><p>[https://blog.mturk.com/managing-groups-of-hits-on-mechanical-turk-3194c045371f Groups of tasks] — If you republish multiple batches of the same task ''or'' different tasks with same name/etc, they show up as the same HIT in the worker interface!</p> | ||
<p>Be careful that this will mean that people can do each HIT within each batch/group (i.e., similar to doing a HIT multiple times).</p></li></ul> | <p>Be careful that this will mean that people can do each HIT within each batch/group (i.e., similar to doing a HIT multiple times).</p></li></ul> | ||
== Request Workflow == | |||
# Define the properties and design the layout of your Project. (Create a Project) | # Define the properties and design the layout of your Project. (Create a Project) | ||
Line 61: | Line 62: | ||
# View a Worker’s approval rating, award bonuses, assign qualifications, or block Workers from working on your HITs. (Managing Workers) | # View a Worker’s approval rating, award bonuses, assign qualifications, or block Workers from working on your HITs. (Managing Workers) | ||
== Creating HITs == | |||
There are two ways to do this: | There are two ways to do this: | ||
Line 70: | Line 71: | ||
Although (2) is powerful, we'll focus entirely on (1) here and on two approaches that an be accomplished within (1): (A) external surveys and material (B) created using Amazon's Crowd HTML Elements. | Although (2) is powerful, we'll focus entirely on (1) here and on two approaches that an be accomplished within (1): (A) external surveys and material (B) created using Amazon's Crowd HTML Elements. | ||
=== (A) External surveys === | |||
Sending Turkers to a survey on an external website is very popular. The challenge with involving a separate website typically relies on ensuring that the worker you hire/pay on MTurk is the same one who is doing the task on the separate website. | Sending Turkers to a survey on an external website is very popular. The challenge with involving a separate website typically relies on ensuring that the worker you hire/pay on MTurk is the same one who is doing the task on the separate website. | ||
Line 83: | Line 84: | ||
In either case, you can choose merge your data from your survey website and MTurk to verify this information before approving/paying. | In either case, you can choose merge your data from your survey website and MTurk to verify this information before approving/paying. | ||
=== (B) Crowd HTML Elements === | |||
The best way involves modifying a set of HTML templates on MTurk. Understanding at least some HTML is very helpful for doing this but not necessary. | The best way involves modifying a set of HTML templates on MTurk. Understanding at least some HTML is very helpful for doing this but not necessary. | ||
Line 91: | Line 92: | ||
Here are two options: | Here are two options: | ||
https://www.w3schools.com/html/html_intro.asp https://developer.mozilla.org/en-US/docs/Learn/HTML/Introduction_to_HTML | |||
In many cases, modifications will be extremely minimal: | In many cases, modifications will be extremely minimal: | ||
# Pick an example from the options on the website that is similar to what you want to do. | # Pick an example from the options on the website that is similar to what you want to do. | ||
# Modify it so that /incoming/ variables from your input file are marked like | # Modify it so that /incoming/ variables from your input file are marked like "${variable_name}" in the HTML. Output variables will be marked in the HTML with "name=" somewhere. | ||
I'll walk through an example on my own. | I'll walk through an example on my own. | ||
Line 103: | Line 103: | ||
'''NOTE:''' If your task involves external media (images, sounds, etc), you will need to host these yourself. MTurk does not allow you upload this material. I typically host these on my university web servers. You will need to provide these in a sheet in the form of URLs. | '''NOTE:''' If your task involves external media (images, sounds, etc), you will need to host these yourself. MTurk does not allow you upload this material. I typically host these on my university web servers. You will need to provide these in a sheet in the form of URLs. | ||
== Publishing HITs == | |||
# Publishing a batch will require providing a CSV | # Publishing a batch will require providing a CSV "input" file that works like the input to a mail merge. An example version of the CSV will be provided will be provided by the MTurk user interface. | ||
# Preview the HITS | # Preview the HITS | ||
# Publish/launch the batch! | # Publish/launch the batch! | ||
== Reviewing Workers Work == | |||
There's an interface for reviewing work. I'll walk through the example quickly. | There's an interface for reviewing work. I'll walk through the example quickly. | ||
In general, you should | In general, you should /strongly/ err on the side of paying everybody. Clean/remove data later. | ||
The damage that can come from not paying someone who did poor work on a HIT is potentially much higher than the small amoung one would save for a given HIT. | |||
== Ensuring Honesty == | |||
* Qualifications (best) | * Qualifications (best) | ||
* | * "Golden" questions (questions for which you know the answers) | ||
* Javascript web-bugs | * Javascript web-bugs to track mouse-movement, time on page, etc | ||
* Attention checks (seem to be discouraged?) | * Attention checks (seem to be discouraged?) | ||
* Assignment of 1 task to many workers (cross check results) | * Assignment of 1 task to many workers (cross check results) | ||
<blockquote>* Working with third-party labor market intermediaries w/ memory of workers across many HITs. | |||
</blockquote> | |||
= Tips and tricks = | |||
* By default there's no way to request unique works across batches or different tasks but there are tricks/scripts. Here's one example: https://uniqueturker.myleott.com/ | * By default there's no way to request unique works across batches or different tasks but there are tricks/scripts. Here's one example: https://uniqueturker.myleott.com/ | ||
* Rather than | * Rather than requring "masters," make your qualification be a 95% approval rate and 1,000 approved assignments. | ||
* Test things out in the | * Test things out in the Requester Sandbox first: https://requester.mturk.com/developer/sandbox | ||