AS3: A/B Testing Analysis

You can download the data collected for the A interfaces here:

• SudokUI (Log data)
• Recurrent Calendar Event (Log data)

Analyze the data you collected for your B interface, and compare it with the data that the staff collected for the A interface. How did the efficiency differ?

You will need to write scripts to process the raw log data and compute times or rates of tasks completed, and aggregate statistics like means. Python or R are good languages for writing scripts like this.

The analysis you need to do is open-ended, and depends on your B interface design and on what events you logged and how you logged them. Your goal is to examine the data to see whether your KLM prediction that your B interface would have greater efficiency is actually supported.

It's a good idea to generate some graphs to compare the results for A and B with their KLM predictions. Graphs help you can see what happened more clearly, and also help the reader of your report. A spreadsheet (like Google Spreadsheets or Excel) can generate charts, as can Python libraries like matplotlib, ggplot, and plotly.

Write up your results in a brief report, in a PDF file called as3-report.pdf, at most 3 pages long. Your report should start with some parts from your AS1 report (which you should update if something has changed), and then add new parts:

1. Problem (from AS1)
2. Collaborators. A list of the people you discussed this assignment with, or else an explicit statement that you had no collaborators. This is an individual assignment, so be aware of the course's collaboration policy.
3. Illustration. (from AS1)
4. Experimental Hypothesis. (from AS1)
5. KLM Analysis. (from AS1)
6. Dataset. (new) An overview of the A and B data you collected, including number of participants and summary statistics about their usage (e.g. time spent, number of tasks done, etc).
7. Results. (new) Performance comparison of the A and B interfaces, and comparison with the KLM predictions. Include graphs, summary statistics (like means), and discussion of the results.

If need help with your writing, not just with AS3 but with any professional communication, the EECS Communication Lab can help. Sign up for an appointment to get one-on-one feedback from a communication lab advisor, who is a grad student or post-doc trained in communication skills. Whether you have a draft or haven't yet written down a word, they can provide guidance; the earlier in your process, the better. You can make an appointment at the EECS Communication Lab website.

Package your completed assignment as a zip file that contains:

• your as3-report.pdf report.
• a directory containing the source code of the scripts you used for your data analysis. If you did all your data analysis in a spreadsheet, provide the spreadsheet in some form, like Excel.

Submit your zip file on Gradebook (link at top of this handout). You must complete the self-assessment form (link at top of this handout) before you hand in the assignment.

This assignment will be judged on whether you did all the parts and presented them in a way that was clear, concise, and accurate. It isn't important whether interface B actually wins the A/B test.