MacKenzie, I. S., and Castellucci, S. J. (2016). Empirical research methods for human-computer interaction. Extended Abstracts of the ACM SIGCHI Conference on Computer-Human Interaction - CHI 2016. pp. 996-999. New York: ACM. doi: 10.1145/2851581.2856671. [PDF] [video]
Empirical Research Methods for Human-Computer Interaction
I. Scott MacKenzie and S. J. Castellucci
York University Toronto, ON, Canada
In this two-session course, attendees will learn how to conduct empirical research in human-computer interaction (HCI). This course delivers an A-to-Z tutorial on designing a user study and demonstrates how to write a successful CHI paper. It would benefit anyone interested in conducting a user study or writing a CHI paper. Only a general HCI knowledge is required.
Empirical research; user study; experiment design; quantitative methods; writing a CHI paper.
ACM Classification Keywords
H.5.0. Information interfaces and presentation (e.g., HCI): General.
Session One Schedule:
- What is empirical research and what is the scientific method?
- Discovering and refining topics suitable for research in HCI
- Formulating "testable" research questions
- How to design an experiment (broadly speaking) to answer research questions
- Parts of an experiment (independent variables, dependent variables, counterbalancing, etc.)
- Group participation in a real experiment
Session Two Schedule:
- Results and discussion of the experiment from Session One (this affords a strong opportunity to revisit and expand on the elements of empirical research)
- Experiment design issues ("within subjects" vs. "between subjects" factors, internal validity, external validity, counterbalancing test conditions, etc.)
- Data analyses (main effects and interaction effects, requirements to establish cause and effect relationships, etc.)
- How to organize and write a successful CHI paper (including suggestions for style and approach, as per CHI conference submissions)
Most attendees at CHI conferences will agree that an experiment (user study) is the hallmark of good research in human-computer interaction. But what constitutes an experiment? And how does one go from an experiment to a CHI paper?
This course will teach how to pose testable research questions, how to make and measure observations, and how to design an experiment. Specifically, attendees will participate in a real experiment to gain experience as both an investigator and as a participant. The second session covers the statistical tools typically used to analyze data. Most notably, attendees will learn how to organize experiment results and write a CHI paper.
The course will begin by guiding attendees on discovering research topics in HCI and formulating research questions that can be answered with a user study. Experiment design questions will be highlighted, and experiment terminology will be explained.
In addition, attendees will participate in a real experiment (lasting about 30 minutes). Working in pairs and using handouts prepared by the instructors, attendees take turns acting both as participant (i.e., perform tasks while data are collected) and as investigator (i.e., instruct the participant, administer the tasks, collect and record data). The experiment will have all the components of a real experiment: independent variables, dependent variables, counterbalancing, etc.
For this particular experiment, two different soft keyboard layouts will be compared. The primary dependent variable is task completion time, which is measured using a stopwatch. There are two independent variables: interface type (i.e., layout), and learning (i.e., trial number). The levels of interface type are administered using counterbalancing, with participants organized in groups.
Because of the time required to run the experiment, analyze the data, and present the results, this course requires two sessions. The instructors and student volunteers (SVs) will transcribe and analyze the experiment data during the break and present the results during the second session. The data will be illustrated using charts, and summarized using tables of means and standard deviations, ANOVA (analysis of variance) tests, correlation analyses, linear regression, etc. Comprehensive results from the experiment will be ready for the second session of the course.
In the second session, the instructors will present and discuss the results from the experiment of the first session. The instructors will provide step by step instructions and examples on how to generate publication-ready plots of data showing error bars, and accompanying statistics, such as linear regression equations.
Note, as well, all materials will be available to the audience through handouts and a web site prepared by the instructors. The software tools provided were used for published experiments. An ANOVA tool written by one of the instructors is one such tool. It features a full API with examples of the most common experiment designs. The tool is easy to use and bypasses the normally steep learning curve of full-featured statistics packages.
The last portion of the course will present guidelines for writing a CHI paper. This will include how to best organize and present experiment results, tips for writing style, and pitfalls to avoid.
Upon completion of this course, attendees will know how to undertake a program of empirical research on a topic in HCI relevant to their interests. Specifically, attendees will be able to:
- Discover and narrow in on topics suitable for research in HCI
- Identify the four scales of measurement and describe characteristics of each
- Formulate "testable" research questions for empirical study
- Design and conduct an experiment (aka user study) to answer research questions
- Collect and analyze empirical data from an experiment
- Write a CHI-styled research paper based on an experiment
This course caters to attendees who are motivated to learn about and use empirical research methods in HCI research. Specifically, it is for those in academia or industry who evaluate interaction techniques using quantitative methods, or those who make decisions based on usability tests.
Approximately 75 attendees is the maximum practical size for this course. If the number of registrations is large, the instructors may consider teaching the course multiple times.
No specific background is required other than a general knowledge of human-computer interaction as conveyed, for example, through an undergraduate HCI course or attendance at CHI conferences. Knowing how to enter formulae in a Microsoft Excel spreadsheet to compute means, standard deviations, etc. would be an asset. Knowledge of advanced statistics, such as the analysis of variance, is NOT required. Additionally, there is no linkage between this and any other CHI 2016 course.
The presentation will be primarily via PowerPoint slides
The use of statistical tools will be demonstrated
- Exercises and Group Discussion:
Attendees with work in pairs to both conduct and participate in an informal user study
The instructors will need a projection system for a notebook computer. Attendees will need tables (i.e., a flat writing surface) for the in-class experiment.
Two SVs are needed for this course. During the first session, attendees participate in an experiment. They record performance data and demographic information on a questionnaire and the questionnaires are collected. Then, the SVs leave the room and transcribe the data from the questionnaires into a spreadsheet, provided by the instructors. This is done as the course continues. Transcribing the data takes about 20-30 minutes with two SVs; i.e., one reads-out the data while the other inputs the data. This procedure has proved successful in previous offerings of this course.
This course was offered at CHI 2007 in San Jose, CA, at CHI 2008 in Florence, Italy, at CHI 2009 in Boston, MA, at CHI 2010 in Atlanta, GA, at CHI 2011 in Vancouver, BC, at CHI 2012 in Austin, TX, at CHI 2013 in Paris, France, and at CHI 2014 in Toronto, ON. In addition, variations on this course have been given at the University of Tampere (Tampere, Finland), the University of Central Lancashire (Preston, UK), and the IT University (Copenhagen, Denmark). Portions of the course were included in talks given in recent years for Microsoft, IBM, Nokia, Philips, etc., and at various universities in North America, Europe, and Japan.
The evaluations of the course have been very good. For CHI 2014, it garnered over 60 registrants. Attendees rated the course 6.34 out of 7 on "The course was well taught by the instructor(s)", 6.21 out of 7 on "The course material provided is helpful", and 6.27 out of 7 on "This course should be offered again next year".
A number of slight modifications are planned for CHI 2016, but these are mostly in streamlining and editing the content of the slides. No major changes are anticipated.
Advertising for this course is largely by word of mouth and its presence in the Advanced Program. Based on the course's history, this approach has been very successful. Additionally, a promotional video produced for CHI 2014 (https://youtu.be/WyVttABHWBs) could be updated and used for CHI 2016.
Scott MacKenzie's research is in HCI with an emphasis on human performance measurement and modeling, experimental methods and evaluation, interaction devices and techniques, alphanumeric entry, language modeling, and mobile computing. He has more than 160 HCI publications (including more than 40 from the SIGCHI conference and 2 HCI books) and has given numerous invited talks over the past 20 years. Since 1999, he has been Associate Professor of Electrical Engineering and Computer Science at York University, Canada.
Steven Castellucci is an Assistant Professor of Electrical Engineering and Computer Science at York University, Canada. His research interests include mobile text entry, and remote pointing techniques. In addition to having SIGCHI publications, he teaches first-year Computer Science and Engineering courses, and has lectured senior and graduate-level HCI university courses.