Dept. of Computer Science and Engineering
York University
Toronto, Ontario, Canada M3J 1P3
mack “at” cse.yorku.ca
Last update: 18-Nov-09
The evaluation of computer
pointing devices is tricky. Although there is an abundance of published
evaluations in the disciplines of human-computer interaction and human factors,
the methodologies are often ad hoc. The experimental procedures are
inconsistent from one study to the next, and this greatly diminishes our
ability to understand the results or to undertake comparisons between studies.
As a consequence, there is a lot of literature to examine, but the reader is in
a quandary on what it all means.
Fortunately, an ISO
standard was published in 2000 that addresses this particular problem. The full
standard is ISO 9241, Ergonomic design for office work with visual display
terminals (VDTs). The standard is in seventeen parts. Part 9 of the
standard is called Requirements for non-keyboard input devices.
ISO 9241-9 describes tests
to evaluate the performance, comfort, and effort in using
computer pointing devices. This note focuses on performance evaluations.
The procedures for
conducting a performance evaluation are well laid out and, if followed, yield a
strong and valid performance evaluation of one or more pointing devices.
Between-study comparisons will be possible because of (at least) three reasons:
1.
The
methodology is consistent from one study to the next (as long as the standard
is followed).
2.
The
standard suggests including a representative device, such as a mouse, for
comparison purposes. Thus, a "base-line condition" exists, and this
will permit across-study comparisons if other studies use the same base-line
device.
3.
The
metric for comparison is Throughput, which includes both the speed and
accuracy of users' performance. Hence, Throughput has an inherent ability to
normalize for behavioural differences (i.e., speed vs. accuracy) across users
or across studies.
Obviously, within-study
comparisons are also valid, and herein lies one of the most appealing features
of the tests described in ISO9241-9. Companies in the business of developing,
manufacturing, and marketing computer pointing devices want to know how good
their product is. It might be a new pointing device, or it might be an
improvement on existing technology. The company may want the evaluation
performed for "internal" purposes (Should we bring this prototype to
market?) or for "external" purposes (Is our device as good as a
competitor's device?).
ISO testing can answer a
variety of questions, such as the following:
·
Is our
pointing device as good as a mouse?
·
Is
"Brand A" mouse as good as "Brand B" mouse?
·
Is a
touchpad as good as a pointing stick?
·
Is a
finger-controlled trackball as good as a thumb-controlled trackball?
·
Is our
trackball (or touchpad, or joystick, or whatever) as good as our competitor’s?
·
Should
buttons be placed "above" or "below" a touchpad placed in a
notebook computer?
·
Is
"lift-and-tap" as good as "buttons" for select operations
on a touchpad?
·
Is
"Device Driver A" as good as "Device Driver B" for a
certain pointing device?
The possibilities are
unlimited. You can see in the questions above that the evaluations and
comparisons are not limited to "Device A" vs. "Device B".
ISO 9241-9 can also evaluate and compare any characteristic of a device or
interaction technique that might affect user performance. For example, for a
given diameter of trackball, which of two ball masses is better? This is a
straight-forward question to answer in the context of ISO testing.
Each bullet above includes
the phrase “as good as” as the basis of comparison. Operationally, the phrase implies testing using both quantitative
and qualitative procedures. For the
quantitative tests, users are given a series of tasks and measurements are made
on their performance. These
measurements serve as the basis for comparison (This is a greatly simplified
explanation, but that's the general idea.
See any of the papers below for further details.)
For the qualitative tests,
users are given a questionnaire on their comfort and preferences. The response
items are provided in Douglas, Kirkpatrick, and MacKenzie (1999), Zhang and
MacKenzie (2007), McArthur, Castellucci, and MacKenzie (2009), and Natapov,
Castellucci, and MacKenzie (2009). See
below. As well, the experimenter maintains
a journal during the experiment and notes any anomalous behaviour that is not
captured by the experimental software (e.g., a preponderance of clutching
operations).
If you would like a copy of
ISO 9241-9, it is available for purchase through ANSI’s Electronic Standards
Store:
http://webstore.ansi.org/ansidocstore/
The author has developed an
elaborate testing tool for Task #2 – the mostly widely used performance
protocol in ISO 9241-9. It is
called FittsTaskTwo. Click here to view the API.
The tool also records and
stores trace data and performance measures associated the cursor path A separate utility facilitates viewing the
trace data. It is called FittsTaskTwoTrace. Click here
to view the API.
For more information,
please contact Scott MacKenzie at mack “at” cse.yorku.ca or visit his home page.
The following papers
present pointing device performance evaluations conforming to ISO 9241-9:
1.
Sasangohar,
F., MacKenzie, I. S., & Scott, S. D. (2009). Evaluation of mouse and touch
input for a tabletop display using Fitts’ reciprocal tapping task. Proceedings of the 53rd Annual Meeting of
the Human Factors and Ergonomics Society – HFES 2009, pp. 839-843. Santa
Monica, CA: Human Factors and Ergonomics Society.
2.
McArthur,
V., Castellucci, S. J., & MacKenzie, I. S. (2009). An empirical comparison of “Wiimote”
gun attachments for pointing tasks.
Proceedings of the ACM Symposium on Engineering Interactive Computing
Systems – EICS 2009, pp. 203-208. New York: ACM.
3.
Natapov,
D., Castellucci, S. J., & MacKenzie, I. S. (2009). ISO 9241-9 evaluation of video game
controllers. Proceedings of Graphics Interface 2009, pp. 223-230.
Toronto: Canadian Information Processing Society.
4.
Zhang,
X., & MacKenzie, I. S. (2007). Evaluating eye tracking with ISO
9241 – Part 9. Proceedings of HCI International 2007, pp. 779-788.
Heidelberg: Springer.
5.
Soukoreff,
R. W., & MacKenzie, I. S. (2004). Towards a standard for pointing
device evaluation: Perspectives on 27 years of Fitts’ law research in HCI. International
Journal of Human-Computer Studies, 61, 751-789.
6.
MacKenzie,
I. S., Jusoh, S. (2001). An evaluation of two input
devices for remote pointing. To appear in Proceedings of the Eighth IFIP
Working Conference on Engineering for Human-Computer Interaction – EHCI 2001.
Heidelberg, Germany: Springer-Verlag.
7.
Silfverberg,
M., MacKenzie, I. S., Kauppinen, T. (2001). An isometric
joystick as a pointing device for hand-held information terminals. Proceedings of Graphics Interface 2001, pp. 119-126.
Toronto, Canada: Canadian Information Processing Society.
8.
MacKenzie,
I. S., Kauppinen, T., & Silfverberg, M. (2001). Accuracy
measures for evaluating computer pointing devices. Proceedings of the
ACM Conference on Human Factors in Computing Systems – CHI 2001, pp. 9-16
New York: ACM.
9.
Douglas,
S. A, Kirkpatrick, A. E., & MacKenzie, I. S. (1999). Testing
pointing device performance and user assessment with the ISO 9241, Part 9
standard. Proceedings of the ACM Conference on Human Factors in
Computing Systems - CHI '99, pp. 215-222. New York: ACM. [PDF]
10. MacKenzie, I. S., & Oniszczak,
A. (1998). A comparison of three selection techniques for
touchpads. Proceedings of the ACM Conference on Human Factors in
Computing Systems – CHI ‘98, pp. 336-343. New York: ACM. [PDF]