Mechanical Turk, Tim Brady
Dr. Tim Brady
Psychology Department
University of California, San Diego
More about Dr. Brady's research
- Video of Dr. Brady's introductory talk (running time one hour) is available here.
- Video of his in-depth Tutorial (running time 3.5 hours) is available here.
Running experiments on Mechanical Turk
Mechanical Turk is a service provided Amazon that enables anybody to post short tasks and have participants from around the United States or the world complete them in exchange for payment. Workers can browse a huge set of tasks and choose which to complete. Only a small proportion of tasks on Turk are cognitive science experiments: the vast majority are short, simple tasks for business purposes (e.g., translating or transcribing speech; screening for inappropriate images, etc).
In this talk, I'll give a brief overview of using Mechanical Turk as a large-scale subject pool for research. I'll review the pros and cons of Turk as a participant sample and as a platform for experiments. I'll review what we know about the demographics of Turk users and what we know about the validity of Turk for psychology experiments. I'll then explore the technical and practical limitations of running online experiments, and give a variety of examples of Turk studies, both from my own work (e.g., Brady & Alvarez, 2011, Psych. Science; Brady & Tenenbaum, 2013, Psych. Review) and from others.