MTurk Workshop (CASBS 2019)/Notes

From CommunityData

The Worker/Turker side of the market[edit]

We don't need to spend a lot of time on this but I know not everybody was able to do works as a worker so I think it might be helpful to share their experience.

In preparation for this workshop I personally:

  • Spent probably 8 minutes painstakingly drawing boxes around palm fruits for $0.90 only to have my work rejected because I took more than 5 minutes to finish the task. I submitted the work and got nothing.
  • Did a bunch of transcription of receipts for 1¢ and 2¢ which was incredibly difficult, painstaking, etc. I mostly failed to get paid. One major challenge is that I did not see/read documentation until it was too late.

Turker Experience[edit]

A few things you should know about the MTurk labor market:

  • The vast majority of work on MTurk falls into a few categories: image/video/audio transcription or transcriptions; data collection

  • Quite a lot of the work is now run in connection to AI/ML based systems. Things that a computer cannot accurately do are kicked to a person. The person often ends up training the computer.

    The vast majority of these HITs are created programmatically with an API.

  • A substantial amount of work is sketchy. "Write a positive review of X on website B" or "collect phone numbers/emails for business" are often obviously part of spamming or astroturfing operations. These are often typically against the rules but are still incredibly common.
  • There's a ton of extra work associated with finding accepting HITs that Turkers are not paid for.
  • Pay is frequently extremely shitty. Sometimes near zero. This is especially for an inexperienced worker. There is also a 20-25% (minimum 1¢) fees charged by Amazon.
  • Payment can sometimes not come because payees have the options to review and accept/reject work.
  • Things are particularly bad for new workers because the vast majority of tasks (especially better ones) on the labor market require "qualifications."
    • The most common is master qualification which is determined by Amazon using a secrete algorithm, can be incredibly difficult to gain. In general, it requires lots of high quality (i.e. approved) work across a range of types of tasks.
    • Inexperienced users have only the dregs of MTurk available to them. The first 1,000 HITs often play very little or (in some cases) nothing.

The MTurk platform has an enormous bias toward support of the "requestor" side of the labor market. That said, Turkers are organized:

The requester side of the market[edit]

There are two big classes of things Academics use MTurk for:

  • Surveys and/or experiments (turkers as human subjects)
  • Data collection or coding (turkers as research assistants)

Approaches to building these vary.

Anatomy of MTurk request[edit]

  • Project
  • (Optionally) input data that you want coded, processed, etc.
  • Tasks/HITs come in Batches which are defined as (Input Data CSV * Project/Design Template)
  • Assignments (the number of workers assigned to each HIT). A specific Turker can only do a given HIT/task once.

  • Groups of tasks — If you republish multiple batches of the same task or different tasks with same name/etc, they show up as the same HIT in the worker interface!

    Be careful that this will mean that people can do each HIT within each batch/group (i.e., similar to doing a HIT multiple times).

Request workflow[edit]

  1. Define the properties and design the layout of your Project. (Create a Project)
  2. Publish your batch of HITs. (Publishing a Batch)
  3. Approve or reject the work done on your HITs. You can download the results, review them offline, and republish rejected HITs. (Managing Batches)
  4. View a Worker’s approval rating, award bonuses, assign qualifications, or block Workers from working on your HITs. (Managing Workers)

Creating HITs[edit]

There are two ways to do this:

  1. Using the Requestor UI
  2. Using the API or CLI to do this automatically (especially useful for creating HITs that involve working on a w/ third party website)

Although (2) is powerful, we'll focus entirely on (1) here and on two approaches that an be accomplished within (1): (A) external surveys and material (B) created using Amazon's Crowd HTML Elements.

(A) External surveys[edit]

Sending Turkers to a survey on an external website is very popular. The challenge with involving a separate website typically relies on ensuring that the worker you hire/pay on MTurk is the same one who is doing the task on the separate website.

There are two approaches:

  1. Ask for AMT WorkerID as part of the form (the only approach that works on Google Forms). You are not allowed to ask for emails, etc.

  2. Ask for a code generated/stored on the survey website that can be input onto MTurk. This can work on Qualtrics or Survey Monkey:

    https://blog.mturk.com/tutorial-getting-great-survey-results-from-mturk-and-qualtrics-f5366f0bd880 https://blog.mturk.com/tutorial-getting-great-survey-results-from-mturk-and-surveymonkey-f49c2891ca6f

In either case, you can choose merge your data from your survey website and MTurk to verify this information before approving/paying.

(B) Crowd HTML Elements[edit]

The best way involves modifying a set of HTML templates on MTurk. Understanding at least some HTML is very helpful for doing this but not necessary.

HTML is not code and you can learn enough to do this in an afternoon from any number of websites:

Here are two options:

In many cases, modifications will be extremely minimal:

  1. Pick an example from the options on the website that is similar to what you want to do.
  2. Modify it so that /incoming/ variables from your input file are marked like ${variable_name} in the HTML. Output variables will be marked in the HTML with name= somewhere.

I'll walk through an example on my own.

NOTE: If your task involves external media (images, sounds, etc), you will need to host these yourself. MTurk does not allow you upload this material. I typically host these on my university web servers. You will need to provide these in a sheet in the form of URLs.

Publishing HITs[edit]

  1. Publishing a batch will require providing a CSV input file that works like the input to a mail merge. An example version of the CSV will be provided will be provided by the MTurk user interface.
  2. Preview the HITS
  3. Publish/launch the batch!

Reviewing Workers Work[edit]

There's an interface for reviewing work. I'll walk through the example quickly.

In general, you should strongly err on the side of paying everybody. Clean/remove data later.

The damage that can come from not paying someone who did poor work on a HIT is potentially much higher than the small amount one would save for a given HIT.

Ensuring quality/honesty[edit]

Approaches will very heavily depending on whether there are objective answers (i.e., Turkers as subjects or RAs)

  • Qualifications (best)
  • "Golden" questions (questions for which you know the answers)
  • Javascript web-bugs and similar to track mouse-movement, time on page, etc (some of this comes built-in)
  • Attention checks (seem to be discouraged?)
  • Assignment of 1 task to many workers (cross check results)
  • Working with third-party labor market intermediaries w/ memory of workers across many HITs.

Tips and tricks[edit]

Retrieved from "https://wiki.communitydata.science/index.php?title=MTurk_Workshop_(CASBS_2019)/Notes&oldid=130876"
Attribution-Share Alike 3.0 Unported
Powered by MediaWiki