RS2: Data Collection
This problem set is only for the doctoral 6.831D version of the class.
This problem set is the second in series (RS1, RS2, RS3) about reproducing a controlled experiment from a published paper. In this problem set, you will run the experiment on some people and collect data.
Human Subjects Training (25%)
Before running users, you should take and pass the Human Subjects Training course offered by MIT's Committee on the Use of Humans as Experimental Subjects:
The course is an online course. When you register for the course, the curriculum you should choose is Social and Behavioral Research Investigators. The course material consists of about 15 web pages, each followed by a short quiz about what you've read. You must score at least 90% overall to pass, but you can repeat any module and retake its quiz if necessary. Once you have passed the course, you will get a certificate of completion.
We're requiring you to take this online course for two reasons. First, its content is relevant to this class and important to anyone doing user testing. Second, MIT requires all researchers involved in experiments with human subjects to pass this online course. So you will come out of this class already certified to do user studies in your graduate research.
If you had previously passed the training course, you don't need to take it again. Just hand in your certificate of completion or some other evidence that you passed. The training site includes a list of all people who have passed the course, so if you're not sure whether you've done it, you can scan that list for your name and date of completion.
Print your completion report by clicking Print from the CITI Main Menu page, and save it to a PDF file. Include this PDF in the zip file you submit on Stellar.
Data Collection (75%)
Conduct a within-subjects controlled experiment using the study infrastructure you built for RS1.
You should have at least 10 participants. Your 6.813/6.831 classmates (including you) will be asked to do some user testing as part of the PS5 assignment, but you can also recruit participants outside of the class.
Follow the protocol of the experiment from the paper you chose. Note that you only have two conditions (one control and one experimental condition) where the experiments in the papers had three or more, so your sessions should be shorter. You can reduce the length of your experiments so that it can be completed in roughly 15 minutes or 500 mouse clicks. Be sure to randomly assign your participants to conditions where appropriate (e.g. orderings of tasks).
The course staff will provide a mechanism for you to put your interface URL into an experimental pool that your classmates can draw from randomly for their user testing.
Collect your raw data in a clearly labeled spreadsheet in CSV format, showing independent variables, dependent variables, demographics, and conditions (values of independent variables) and results (measured dependent variables, i.e. time) Be sure to include users' subjective ratings of difficulty, satisfaction, efficiency, and frustration. You don't have to run statistical tests, generate graphs, or compute summary statistics for now -- RS3 will do that.
What to Hand In
Package your completed assignment as a zip file that contains:
- your completion report for human subjects training, as a PDF file
- your spreadsheet of data, as a CSV file (Google Spreadsheets can export CSV)
Submit your zip file on Stellar (link at the top of this handout). You must also complete the self-assessment form (link at the top of this handout) before you hand in the assignment.
This assignment will be judged on whether you did all the parts and presented them in a way that was easy to determine that you did them.