Dept. of Computer Science and Engineering
Toronto, Ontario, Canada M3J 1P3
mack “at” cse.yorku.ca
Last update: 18-Nov-09
The evaluation of computer pointing devices is tricky. Although there is an abundance of published evaluations in the disciplines of human-computer interaction and human factors, the methodologies are often ad hoc. The experimental procedures are inconsistent from one study to the next, and this greatly diminishes our ability to understand the results or to undertake comparisons between studies. As a consequence, there is a lot of literature to examine, but the reader is in a quandary on what it all means.
Fortunately, an ISO standard was published in 2000 that addresses this particular problem. The full standard is ISO 9241, Ergonomic design for office work with visual display terminals (VDTs). The standard is in seventeen parts. Part 9 of the standard is called Requirements for non-keyboard input devices.
ISO 9241-9 describes tests to evaluate the performance, comfort, and effort in using computer pointing devices. This note focuses on performance evaluations.
The procedures for conducting a performance evaluation are well laid out and, if followed, yield a strong and valid performance evaluation of one or more pointing devices. Between-study comparisons will be possible because of (at least) three reasons:
1. The methodology is consistent from one study to the next (as long as the standard is followed).
2. The standard suggests including a representative device, such as a mouse, for comparison purposes. Thus, a "base-line condition" exists, and this will permit across-study comparisons if other studies use the same base-line device.
3. The metric for comparison is Throughput, which includes both the speed and accuracy of users' performance. Hence, Throughput has an inherent ability to normalize for behavioural differences (i.e., speed vs. accuracy) across users or across studies.
Obviously, within-study comparisons are also valid, and herein lies one of the most appealing features of the tests described in ISO9241-9. Companies in the business of developing, manufacturing, and marketing computer pointing devices want to know how good their product is. It might be a new pointing device, or it might be an improvement on existing technology. The company may want the evaluation performed for "internal" purposes (Should we bring this prototype to market?) or for "external" purposes (Is our device as good as a competitor's device?).
ISO testing can answer a variety of questions, such as the following:
· Is our pointing device as good as a mouse?
· Is "Brand A" mouse as good as "Brand B" mouse?
· Is a touchpad as good as a pointing stick?
· Is a finger-controlled trackball as good as a thumb-controlled trackball?
· Is our trackball (or touchpad, or joystick, or whatever) as good as our competitor’s?
· Should buttons be placed "above" or "below" a touchpad placed in a notebook computer?
· Is "lift-and-tap" as good as "buttons" for select operations on a touchpad?
· Is "Device Driver A" as good as "Device Driver B" for a certain pointing device?
The possibilities are unlimited. You can see in the questions above that the evaluations and comparisons are not limited to "Device A" vs. "Device B". ISO 9241-9 can also evaluate and compare any characteristic of a device or interaction technique that might affect user performance. For example, for a given diameter of trackball, which of two ball masses is better? This is a straight-forward question to answer in the context of ISO testing.
Each bullet above includes the phrase “as good as” as the basis of comparison. Operationally, the phrase implies testing using both quantitative and qualitative procedures. For the quantitative tests, users are given a series of tasks and measurements are made on their performance. These measurements serve as the basis for comparison (This is a greatly simplified explanation, but that's the general idea. See any of the papers below for further details.)
For the qualitative tests, users are given a questionnaire on their comfort and preferences. The response items are provided in Douglas, Kirkpatrick, and MacKenzie (1999), Zhang and MacKenzie (2007), McArthur, Castellucci, and MacKenzie (2009), and Natapov, Castellucci, and MacKenzie (2009). See below. As well, the experimenter maintains a journal during the experiment and notes any anomalous behaviour that is not captured by the experimental software (e.g., a preponderance of clutching operations).
If you would like a copy of ISO 9241-9, it is available for purchase through ANSI’s Electronic Standards Store:
The author has developed an elaborate testing tool for Task #2 – the mostly widely used performance protocol in ISO 9241-9. It is called FittsTaskTwo. Click here to view the API.
The tool also records and stores trace data and performance measures associated the cursor path A separate utility facilitates viewing the trace data. It is called FittsTaskTwoTrace. Click here to view the API.
For more information, please contact Scott MacKenzie at mack “at” cse.yorku.ca or visit his home page.
The following papers present pointing device performance evaluations conforming to ISO 9241-9:
1. Sasangohar, F., MacKenzie, I. S., & Scott, S. D. (2009). Evaluation of mouse and touch input for a tabletop display using Fitts’ reciprocal tapping task. Proceedings of the 53rd Annual Meeting of the Human Factors and Ergonomics Society – HFES 2009, pp. 839-843. Santa Monica, CA: Human Factors and Ergonomics Society.
2. McArthur, V., Castellucci, S. J., & MacKenzie, I. S. (2009). An empirical comparison of “Wiimote” gun attachments for pointing tasks. Proceedings of the ACM Symposium on Engineering Interactive Computing Systems – EICS 2009, pp. 203-208. New York: ACM.
3. Natapov, D., Castellucci, S. J., & MacKenzie, I. S. (2009). ISO 9241-9 evaluation of video game controllers. Proceedings of Graphics Interface 2009, pp. 223-230. Toronto: Canadian Information Processing Society.
4. Zhang, X., & MacKenzie, I. S. (2007). Evaluating eye tracking with ISO 9241 – Part 9. Proceedings of HCI International 2007, pp. 779-788. Heidelberg: Springer.
5. Soukoreff, R. W., & MacKenzie, I. S. (2004). Towards a standard for pointing device evaluation: Perspectives on 27 years of Fitts’ law research in HCI. International Journal of Human-Computer Studies, 61, 751-789.
6. MacKenzie, I. S., Jusoh, S. (2001). An evaluation of two input devices for remote pointing. To appear in Proceedings of the Eighth IFIP Working Conference on Engineering for Human-Computer Interaction – EHCI 2001. Heidelberg, Germany: Springer-Verlag.
7. Silfverberg, M., MacKenzie, I. S., Kauppinen, T. (2001). An isometric joystick as a pointing device for hand-held information terminals. Proceedings of Graphics Interface 2001, pp. 119-126. Toronto, Canada: Canadian Information Processing Society.
8. MacKenzie, I. S., Kauppinen, T., & Silfverberg, M. (2001). Accuracy measures for evaluating computer pointing devices. Proceedings of the ACM Conference on Human Factors in Computing Systems – CHI 2001, pp. 9-16 New York: ACM.
9. Douglas, S. A, Kirkpatrick, A. E., & MacKenzie, I. S. (1999). Testing pointing device performance and user assessment with the ISO 9241, Part 9 standard. Proceedings of the ACM Conference on Human Factors in Computing Systems - CHI '99, pp. 215-222. New York: ACM. [PDF]
10. MacKenzie, I. S., & Oniszczak, A. (1998). A comparison of three selection techniques for touchpads. Proceedings of the ACM Conference on Human Factors in Computing Systems – CHI ‘98, pp. 336-343. New York: ACM. [PDF]