MacKenzie, I. S., Roig-Maimó, M. F., & Mas-Sansó, R., (2025). Empirical research methods for human-computer interaction. Extended Abstracts of the ACM SIGCHI Conference on Human Factors in Computing Systems - CHI 2025. Article No. 812. pp. 1-3. New York: ACM. doi:https:10.1145/3706599.3706640. [PDF]
Empirical Research Methods for Human-Computer Interaction
I. Scott MacKenzie, Maria Francesca Roig-Maimó, & Ramon Mas-Sansó
1Department of Electrical Engineering and Computer Science
York University Toronto, Ontario, Canada
mack@yorku.ca
2Department of Mathematics and Computer Science
University of the Balearic Islands Palma, Spain
xisca.roig@uib.es, ramon.mas@uib.es
Figure 1: Empirical Research Methods. From left, Observational, Correlational, Experimental.
Abstract
Most attendees at CHI conferences will agree that an experiment (user study) is the hallmark of good research in human-computer interaction. But what constitutes an experiment? And how does one go from an experiment to a CHI paper? This course will teach how to pose testable research questions, how to make and measure observations, and how to design and conduct an HCI experiment. Specifically, attendees will participate in a real experiment to gain experience as both an investigator and as a participant. The second session covers the statistical tools typically used to analyze data. Most notably, attendees will learn how to organize experiment results and write a CHI paper.
CCS Concepts
• Human-centered computing
Keywords
Empirical research; user study; experiment design; quantitative methods; writing a CHI paper.
1 Benefits
In this two-session course, attendees will learn how to conduct empirical research in human-computer interaction (HCI). This course delivers an A-to-Z tutorial on designing and doing a user study and demonstrates how to write a successful CHI paper. It would benefit anyone interested in conducting a user study or writing a CHI paper. Only a general HCI knowledge is required.
2 Intended Audience(s)
This course caters to attendees who are motivated to learn about, and use, empirical research methods in HCI research. Specifically, it is for those in academia or industry who evaluate interaction tech- niques using quantitative methods, or those who make decisions based on usability tests, and, in particular, user studies following an experimental methodology.
Approximately 75 attendees is the maximum practical size for this course. If the number of registrations is large, the instructors may consider teaching the course multiple times.
3 Prerequisites
No specific background is required other than a general knowledge of human-computer interaction as conveyed, for example, through an undergraduate HCI course or attendance at CHI conferences. Knowing how to enter formulae in a Microsoft Excel spreadsheet to compute means, standard deviations, etc., would be an asset. Knowledge of advanced statistics, such as the analysis of variance, is NOT required. Additionally, there is no linkage between this and any other CHI course.
4 Course History
This course has been offered numerous times over the past 20 years, with regular updating to reflect current research and practice in HCI. It was offered at CHI 2007 (San Jose), CHI 2008 (Florence), CHI 2009 (Boston), CHI 2010 (Atlanta), CHI 2011 (Vancouver), CHI 2012, (Austin), CHI 2013 (Paris), CHI 2014 (Toronto), CHI 2016 (San Jose), CHI 2017 (Denver), CHI 2018 (Montreal), CHI 2019 (Glasgow), CHI 2023 (Hamburg), and CHI 2024 (Honolulu). In addition, versions of this course have been given at the University of Tampere (Finland), the University of Central Lancashire (UK), the University of Oslo (Norway), ETH Zürich (Switzerland), the University of the Balearic Islands (Spain), the IT University (Copenhagen, Denmark), Technical University of Denmark (Lyngby, Denmark), and the University of Aalborg (Denmark).1
5 Content
This course presents selected topics from Chapter 4 (Scientific Foundations), Chapter 5 (Designing HCI Experiments), Chapter 6 (Hypothesis Testing), and Chapter 8 (Writing and Publishing a Research Paper) in Human-Computer Interaction: An Empirical Research Perspective [1].
Session 1 topics:
- What is research, what is empirical research, and what is the scientific method (see Fig. 1)?
- Formulating "testable" research questions
- How to design an experiment (broadly speaking) to answer research questions
- Parts of an experiment (independent variables, dependent variables, counterbalancing, ethics approval, etc.)
- Group participation in a real experiment
Session 2 topics:
- Results and discussion of the experiment from session 1 (this affords a strong opportunity to revisit and expand on the elements of empirical research)
- Experiment design issues (within-subjects vs. between-subjects factors, internal vs. external validity, counterbalancing test conditions, etc.)
- Data analyses (main effects and interaction effects, requirements to establish cause and effect relationships, etc.)
- How to organize and write a successful CHI paper (including suggestions for style and approach, as per CHI conference submissions)
6 Practical work
Early in session 1, participants are divided into groups of two and participate in an experiment. A hand-out is distributed for the in-class experiment. See Fig. 2.
Figure 2: Two-page handout for the in-class experiment.
Following brief instructions, the in-class experiment proceeds. During the experiment, participants take turns acting as a "participant" and as an "investigator." The participant does an experimental task – entering a text phrase five times with a non-marking stylus on the image of a soft keyboard – while the investigator measures the time to enter each phrase. This is done twice, once for keyboard layout "A" and once for keyboard layout "B". See Fig. 3. The data are entered in a log sheet. When finished, the participant and investigator switch roles and the process is repeated. This time the order of using the keyboard layouts is reversed, "B" first, then "A". This is an example of counterbalancing, as explained during the course.
Figure 3: In-class experiment from a previous version of this course.
As well as performance data, demographic information is entered on the log sheet. The in-class experiment takes about 20 minutes. Course assistants collect the hand-out sheets, leave the room, and transcribe the data from the handout sheets into a boilerplate spreadsheet, provided by the instructors. This is done as the course continues. Transcribing the data takes about 20-30 minutes with two people; i.e., one reads-out the data while the other inputs the data. This procedure has proved successful in previous offerings of this course.
During session 2, the course continues but now uses the methodology and results of the in-class user study to reinforce topics in the course. Examples of the results are shown in Fig. 4. The particular results are not important here. However, it is extremely useful from a pedagogical perspective that the results discussed are from an experiment in which the course attendees have just participated. Results of an analysis of variance are also presented.
Figure 4: Results from the in-class experiment from a previous CHI conference. See text for discussion.
7 Instructor background
Scott MacKenzie's research is in human-computer interaction with an emphasis on human performance, experimental methods and evaluation, interaction devices and techniques, etc. He has more than 200 peer-reviewed publications in the field of Human-Computer Interaction (including more than 40 from the ACM's annual SIGCHI conference). In 2015, he was elected into the ACM SIGCHI Academy. Further details: http://www.yorku.ca/mack/
Maria Francesca Roig-Maimó and Ramon Mas-Sansó have an extensive background in HCI research, including teaching undergraduate and graduate courses in HCI research methods, UI design, and so on. Their current research interests mainly focus on mobile devices and performance evaluation.
8 Resources
Attendees needn't bring any resources. Handouts will be disseminated during the course.
9 Accessibility
Attendees in need of accessibility arrangements are encouraged to contact the course organizers. Appropriate assistance will be provided in consultation with the conference organizers.
References
[1] I. Scott MacKenzie. 2024. Human-Computer Interaction: An Empirical Research Perspective (2nd ed.). Morgan Kaufmann (an imprint of Elsevier), Amsterdam. ISBN: 978-0-443-14096-9.
-----
Footnotes:
1 Please contact Scott MacKenzie, mack@yorku,ca, to discuss possibilities for your lab or institute.