Not logged in
Talk
Contributions
Create account
Log in
Navigation
Main page
About
People
Publications
Teaching
Resources
Research Blog
Wiki Functions
Recent changes
Help
Licensing
Page
Discussion
Edit
View history
Editing
Designing Internet Research (Winter 2020)/MTurk Notes
(section)
From CommunityData
Jump to:
navigation
,
search
Warning:
You are not logged in. Your IP address will be publicly visible if you make any edits. If you
log in
or
create an account
, your edits will be attributed to your username, along with other benefits.
Anti-spam check. Do
not
fill this in!
== The requester side of the market == There are two big classes of things Academics use MTurk for: * Surveys and/or experiments (''turkers as human subjects'') * Data collection or coding (''turkers as research assistants'') Approaches to building these vary. === Anatomy of MTurk request === <ul> <li>Project</li> <li>(Optionally) input data that you want coded, processed, etc.</li> <li>Tasks/HITs come in '''Batches''' which are defined as ('''Input Data CSV''' * '''Project/Design Template''')</li> <li>Assignments (the number of workers assigned to each HIT). A specific Turker can only do a given HIT/task once.</li> <li><p>[https://blog.mturk.com/managing-groups-of-hits-on-mechanical-turk-3194c045371f Groups of tasks] โ If you republish multiple batches of the same task ''or'' different tasks with same name/etc, they show up as the same HIT in the worker interface!</p> <p>Be careful that this will mean that people can do each HIT within each batch/group (i.e., similar to doing a HIT multiple times).</p></li></ul> === Request workflow === # Define the properties and design the layout of your Project. (Create a Project) # Publish your batch of HITs. (Publishing a Batch) # Approve or reject the work done on your HITs. You can download the results, review them offline, and republish rejected HITs. (Managing Batches) # View a Workerโs approval rating, award bonuses, assign qualifications, or block Workers from working on your HITs. (Managing Workers) === Creating HITs === There are two ways to do this: # Using the Requestor UI # Using the API or CLI to do this automatically (especially useful for creating HITs that involve working on a w/ third party website) Although (2) is powerful, we'll focus entirely on (1) here and on two approaches that an be accomplished within (1): (A) external surveys and material (B) created using Amazon's Crowd HTML Elements. ==== (A) External surveys ==== Sending Turkers to a survey on an external website is very popular. The challenge with involving a separate website typically relies on ensuring that the worker you hire/pay on MTurk is the same one who is doing the task on the separate website. There are two approaches: <ol style="list-style-type: decimal;"> <li>Ask for AMT WorkerID as part of the form (the only approach that works on Google Forms). You are ''not'' allowed to ask for emails, etc.</li> <li><p>Ask for a code generated/stored on the survey website that can be input onto MTurk. This can work on Qualtrics or Survey Monkey:</p> <p>https://blog.mturk.com/tutorial-getting-great-survey-results-from-mturk-and-qualtrics-f5366f0bd880 https://blog.mturk.com/tutorial-getting-great-survey-results-from-mturk-and-surveymonkey-f49c2891ca6f</p></li></ol> In either case, you can choose merge your data from your survey website and MTurk to verify this information before approving/paying. ==== (B) Crowd HTML Elements ==== The best way involves modifying a set of HTML templates on MTurk. Understanding at least some HTML is very helpful for doing this but not necessary. HTML is not code and you can learn enough to do this in an afternoon from any number of websites: Here are two options: * [https://www.w3schools.com/html/html_intro.asp W3C's HTML Tutorial] * [https://developer.mozilla.org/en-US/docs/Learn/HTML/Introduction_to_HTML Mozilla's Introduction to HTML] In many cases, modifications will be extremely minimal: # Pick an example from the options on the website that is similar to what you want to do. # Modify it so that /incoming/ variables from your input file are marked like ''${variable_name}'' in the HTML. Output variables will be marked in the HTML with ''name='' somewhere. I'll walk through an example on my own. '''NOTE:''' If your task involves external media (images, sounds, etc), you will need to host these yourself. MTurk does not allow you upload this material. I typically host these on my university web servers. You will need to provide these in a sheet in the form of URLs. === Publishing HITs === # Publishing a batch will require providing a CSV '''input''' file that works like the input to a mail merge. An example version of the CSV will be provided will be provided by the MTurk user interface. # Preview the HITS # Publish/launch the batch! === Reviewing Workers Work === There's an interface for reviewing work. I'll walk through the example quickly. In general, you should ''strongly'' err on the side of paying everybody. Clean/remove data later. The damage that can come from not paying someone who did poor work on a HIT is potentially much higher than the small amount one would save for a given HIT. === Ensuring quality/honesty === Approaches will very heavily depending on whether there are objective answers (i.e., Turkers as subjects or RAs) * Qualifications (best) * "Golden" questions (questions for which you know the answers) * Javascript web-bugs and similar to track mouse-movement, time on page, etc (some of this comes built-in) * Attention checks (seem to be discouraged?) * Assignment of 1 task to many workers (cross check results) * Working with third-party labor market intermediaries w/ memory of workers across many HITs.
Summary:
Please note that all contributions to CommunityData are considered to be released under the Attribution-Share Alike 3.0 Unported (see
CommunityData:Copyrights
for details). If you do not want your writing to be edited mercilessly and redistributed at will, then do not submit it here.
You are also promising us that you wrote this yourself, or copied it from a public domain or similar free resource.
Do not submit copyrighted work without permission!
To protect the wiki against automated edit spam, we kindly ask you to solve the following CAPTCHA:
Cancel
Editing help
(opens in new window)
Tools
What links here
Related changes
Special pages
Page information