Class PostHocGUI

java.lang.Object

java.awt.Component

java.awt.Container

java.awt.Window

java.awt.Frame

javax.swing.JFrame

PostHocGUI

All Implemented Interfaces:

java.awt.event.ActionListener, java.awt.event.FocusListener, java.awt.event.ItemListener, java.awt.image.ImageObserver, java.awt.MenuContainer, java.io.Serializable, java.util.EventListener, javax.accessibility.Accessible, javax.swing.RootPaneContainer, javax.swing.WindowConstants

public class PostHocGUI
extends javax.swing.JFrame
implements java.awt.event.ActionListener, java.awt.event.FocusListener, java.awt.event.ItemListener

PostHoc - GUI Version

Summary

• A utility for performing post hoc pairwise comparisons on a data set

NOTE: This API assumes the reader has studied the API for AnovaGUI and has a general understanding of the ANOVA test.

Background

In experimental research, an analysis of variance (ANOVA), or F-test, investigates the effect of an independent variable on a dependent variable. An independent variable is a condition manipulated in an experiment, such as input device (mouse vs. touchpad). A dependent variable is a human performance measurement, such as task completion time.

An independent variable is manipulated over levels. In HCI, the levels of an independent variable are often called test conditions. In statistics references, they are often called treatments. An independent variable must have at least two levels, as in the mouse vs. touchpad example above. However, an independent variable often has more than two levels. An example might be feedback mode with levels auditory, visual, tactile, none. If the independent variable has more than two levels, a significant F-test only indicates that at least one level is significantly different from one other level. It does not indicate which levels differ significantly from one another. To determine which levels (test conditions) different significantly from one another, post hoc pairwise comparisons are performed.

This utility supports four post hoc comparisons tests:

• Scheffé test
• Fisher's LSD test
• Bonferroni-Dunn test
• Tukey's HSD test

Launch PostHocGUI from GoStats as follows:

PostHocGUI accommodates data in several formats. If the data are organized in a rectangular table, just (i) select the data file, (ii) choose the desired post hoc test, and (iii) click "Analyse." The assumption in this case is that the data are organized in an n × k table, with n rows and k columns. n is the number of participants, k is the number of test conditions. This is the usual format for data in a within-subjects experiment. With k conditions there are k × (k - 1) / 2 comparisons.

Example 1

Consider an experiment where 16 participants were tested across four levels of a single within-subjects factor. The data are organized in a table with 16 rows and 4 columns. Example data are in the GoStatsExamples folder in the file posthoc-ex1.txt:

 
     11     11     21     16
     18     11     22     15
     17     10     18     13
     19     15     21     20
     13     17     23     10
     10     15     15     20
     14     14     15     13
     13     14     19     18
     19     18     16     12
     10     17     21     18
     10     19     22     13
     16     14     18     20
     10     20     17     19
     10     13     21     18
     20     17     14     18
     18     17     17     14
 

An AVOVA test (not shown) reveals a significant outcome. However, since there were four test conditions, post hoc comparisons are warranted. The pairwise comparisons are performed by PostHocGUI as follows: (click to enlarge)

Of the six possible comparisons, only two are significant. The data in columns 1 and 3 are significantly different, as are the data in columns 2 and 3. Note: Full details for this example are found in section 6.2.4 Post Hoc Comparisons in MacKenzie's Human-Computer Interaction: An Empirical Research Perspective (2nd edition).

Data Organization

If the file contains header or comment lines, select the number of lines to skip via the argument "Ignore 'n' header lines."

If counterbalancing was used and the data set includes a right-hand column for the group code, select "1" in the combobox for "Ignore 'n' right-hand columns."

For between-subjects experiments, the data are sometimes organized in a vertical column. In this case, select the option "Vertical data in left-hand column." You also need to select the number of right-hand columns to ignore, for example, 1, 2, or 3 for experiments with 1, 2, or 3 between-subjects factors. Note that the post hoc tests only apply to the groups for the first between-subjects factor. The number of groups is automatically determined, according to the codes for the first between-subjects factor (column 2).

Discussion

The example above used the Scheffé test. PostHocGUI supports four post hoc tests. If they are all selected, the outcome is as follows:

As often occurs, the results of different post hoc tests are slightly different.

The primary source for coding these tests was Chapter 21 in D. J. Sheskin's Handbook of Parametric and Nonparametric Statistical Procedures, 5th ed., CRC Press, 2011, pp. 886-999. Sheskin uses the following data set throughout the chapter:

 
     8     7     4
    10     8     8
     9     5     7
    10     8     5
     9     5     7
 

There are 3 groups of participants with 5 participants in each group. Each group was tested on a different condition, with the scores for each condition/group appearing in separate columns. There were 15 total participants.

As well as describing the calculations leading to the F-test, Sheskin demonstrates the various post hoc comparisons that often follow a statistically significant ANOVA. Sheskin's calculations are embedded in this utility. His data set is in the file sheskin-ex1.txt, slightly rearranged as per our usual organization for between-subjects experiments:

      
      8     G1
     10     G1
      9     G1
     10     G1
      9     G1
      7     G2
      8     G2
      5     G2
      8     G2
      5     G2
      4     G3
      8     G3
      7     G3
      5     G3
      7     G3
 

A post hoc analysis is performed on this data set as follows:

The results are consistent between the four post hoc tests. There is a significant difference between groups 1 and 2 and between groups 1 and 3. There is no significant difference between groups 2 and 3.

The most difficult calculation is that for the critical difference. The values above are consistent with those in the Sheskin reference. As further verification, the same test was undertaken with StatView (now JMP). The results follow:

Again, the results are consistent. There are minor deviations between some of the critical values, however. For example, the Bonferroni-Dunn critical value calculated by GoStats is 2.44. The critical value in Sheskin's handbook is 2.43 (p. 907) and by StatView 2.423 (see above). The differences may be due to rounding or to the use of embedded lookup tables in StatView versus the use of methods of the Statistics class in GoStats.

Significance is at the α = .05 level. For post hoc comparisons, the alpha level is "family-wise". This implies that the overall alpha for the set of comparisons is α = .05. To maintain the overall alpha, the individual comparisons use a more stringent alpha level. The Bonferroni-Dunn test, for example, uses .05 / n, where n is the number of comparisons. With three comparisons, α = .05 / 3 = .0167. Thus, a comparison is only deemed significant if p < .0167. See above. The other tests have different approaches to maintaining the family-wise alpha level. See Sheskin for complete details.

Note that if the above data are re-organized into the within-subjects format for StatView (compact variables), the result of the pairwise comparisons test is the same. This is not as precise since the residual mean squares are different for within-subjects data than for between-subjects data. However, the differences are minor. Sheskin provides additional details.

Author:

Scott MacKenzie, 2012-2023

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class java.awt.Window

java.awt.Window.Type

Nested classes/interfaces inherited from class java.awt.Component

java.awt.Component.BaselineResizeBehavior

Field Summary

Fields inherited from class javax.swing.JFrame

EXIT_ON_CLOSE

Fields inherited from class java.awt.Frame

CROSSHAIR_CURSOR, DEFAULT_CURSOR, E_RESIZE_CURSOR, HAND_CURSOR, ICONIFIED, MAXIMIZED_BOTH, MAXIMIZED_HORIZ, MAXIMIZED_VERT, MOVE_CURSOR, N_RESIZE_CURSOR, NE_RESIZE_CURSOR, NORMAL, NW_RESIZE_CURSOR, S_RESIZE_CURSOR, SE_RESIZE_CURSOR, SW_RESIZE_CURSOR, TEXT_CURSOR, W_RESIZE_CURSOR, WAIT_CURSOR

Fields inherited from class java.awt.Component

BOTTOM_ALIGNMENT, CENTER_ALIGNMENT, LEFT_ALIGNMENT, RIGHT_ALIGNMENT, TOP_ALIGNMENT

Fields inherited from interface javax.swing.WindowConstants

DISPOSE_ON_CLOSE, DO_NOTHING_ON_CLOSE, HIDE_ON_CLOSE

Fields inherited from interface java.awt.image.ImageObserver

ABORT, ALLBITS, ERROR, FRAMEBITS, HEIGHT, PROPERTIES, SOMEBITS, WIDTH

Method Summary

Modifier and Type	Method and Description
void	actionPerformed(java.awt.event.ActionEvent ae)
void	focusGained(java.awt.event.FocusEvent fe)
void	focusLost(java.awt.event.FocusEvent fe)
void	itemStateChanged(java.awt.event.ItemEvent ie)

Methods inherited from class javax.swing.JFrame

getAccessibleContext, getContentPane, getDefaultCloseOperation, getGlassPane, getGraphics, getJMenuBar, getLayeredPane, getRootPane, getTransferHandler, isDefaultLookAndFeelDecorated, remove, repaint, setContentPane, setDefaultCloseOperation, setDefaultLookAndFeelDecorated, setGlassPane, setIconImage, setJMenuBar, setLayeredPane, setLayout, setTransferHandler, update

Methods inherited from class java.awt.Frame

addNotify, getCursorType, getExtendedState, getFrames, getIconImage, getMaximizedBounds, getMenuBar, getState, getTitle, isResizable, isUndecorated, remove, removeNotify, setBackground, setCursor, setExtendedState, setMaximizedBounds, setMenuBar, setOpacity, setResizable, setShape, setState, setTitle, setUndecorated

Methods inherited from class java.awt.Window

addPropertyChangeListener, addPropertyChangeListener, addWindowFocusListener, addWindowListener, addWindowStateListener, applyResourceBundle, applyResourceBundle, createBufferStrategy, createBufferStrategy, dispose, getBackground, getBufferStrategy, getFocusableWindowState, getFocusCycleRootAncestor, getFocusOwner, getFocusTraversalKeys, getIconImages, getInputContext, getListeners, getLocale, getModalExclusionType, getMostRecentFocusOwner, getOpacity, getOwnedWindows, getOwner, getOwnerlessWindows, getShape, getToolkit, getType, getWarningString, getWindowFocusListeners, getWindowListeners, getWindows, getWindowStateListeners, hide, isActive, isAlwaysOnTop, isAlwaysOnTopSupported, isAutoRequestFocus, isFocusableWindow, isFocusCycleRoot, isFocused, isLocationByPlatform, isOpaque, isShowing, isValidateRoot, pack, paint, postEvent, removeWindowFocusListener, removeWindowListener, removeWindowStateListener, reshape, setAlwaysOnTop, setAutoRequestFocus, setBounds, setBounds, setCursor, setFocusableWindowState, setFocusCycleRoot, setIconImages, setLocation, setLocation, setLocationByPlatform, setLocationRelativeTo, setMinimumSize, setModalExclusionType, setSize, setSize, setType, setVisible, show, toBack, toFront

Methods inherited from class java.awt.Container

add, add, add, add, add, addContainerListener, applyComponentOrientation, areFocusTraversalKeysSet, countComponents, deliverEvent, doLayout, findComponentAt, findComponentAt, getAlignmentX, getAlignmentY, getComponent, getComponentAt, getComponentAt, getComponentCount, getComponents, getComponentZOrder, getContainerListeners, getFocusTraversalPolicy, getInsets, getLayout, getMaximumSize, getMinimumSize, getMousePosition, getPreferredSize, insets, invalidate, isAncestorOf, isFocusCycleRoot, isFocusTraversalPolicyProvider, isFocusTraversalPolicySet, layout, list, list, locate, minimumSize, paintComponents, preferredSize, print, printComponents, remove, removeAll, removeContainerListener, setComponentZOrder, setFocusTraversalKeys, setFocusTraversalPolicy, setFocusTraversalPolicyProvider, setFont, transferFocusDownCycle, validate

Methods inherited from class java.awt.Component

action, add, addComponentListener, addFocusListener, addHierarchyBoundsListener, addHierarchyListener, addInputMethodListener, addKeyListener, addMouseListener, addMouseMotionListener, addMouseWheelListener, bounds, checkImage, checkImage, contains, contains, createImage, createImage, createVolatileImage, createVolatileImage, disable, dispatchEvent, enable, enable, enableInputMethods, firePropertyChange, firePropertyChange, firePropertyChange, firePropertyChange, firePropertyChange, firePropertyChange, getBaseline, getBaselineResizeBehavior, getBounds, getBounds, getColorModel, getComponentListeners, getComponentOrientation, getCursor, getDropTarget, getFocusListeners, getFocusTraversalKeysEnabled, getFont, getFontMetrics, getForeground, getGraphicsConfiguration, getHeight, getHierarchyBoundsListeners, getHierarchyListeners, getIgnoreRepaint, getInputMethodListeners, getInputMethodRequests, getKeyListeners, getLocation, getLocation, getLocationOnScreen, getMouseListeners, getMouseMotionListeners, getMousePosition, getMouseWheelListeners, getName, getParent, getPeer, getPropertyChangeListeners, getPropertyChangeListeners, getSize, getSize, getTreeLock, getWidth, getX, getY, gotFocus, handleEvent, hasFocus, imageUpdate, inside, isBackgroundSet, isCursorSet, isDisplayable, isDoubleBuffered, isEnabled, isFocusable, isFocusOwner, isFocusTraversable, isFontSet, isForegroundSet, isLightweight, isMaximumSizeSet, isMinimumSizeSet, isPreferredSizeSet, isValid, isVisible, keyDown, keyUp, list, list, list, location, lostFocus, mouseDown, mouseDrag, mouseEnter, mouseExit, mouseMove, mouseUp, move, nextFocus, paintAll, prepareImage, prepareImage, printAll, removeComponentListener, removeFocusListener, removeHierarchyBoundsListener, removeHierarchyListener, removeInputMethodListener, removeKeyListener, removeMouseListener, removeMouseMotionListener, removeMouseWheelListener, removePropertyChangeListener, removePropertyChangeListener, repaint, repaint, repaint, requestFocus, requestFocusInWindow, resize, resize, revalidate, setComponentOrientation, setDropTarget, setEnabled, setFocusable, setFocusTraversalKeysEnabled, setForeground, setIgnoreRepaint, setLocale, setMaximumSize, setName, setPreferredSize, show, size, toString, transferFocus, transferFocusBackward, transferFocusUpCycle

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface java.awt.MenuContainer

getFont, postEvent

Method Detail

actionPerformed

public void actionPerformed(java.awt.event.ActionEvent ae)

Specified by:

actionPerformed in interface java.awt.event.ActionListener

focusGained

public void focusGained(java.awt.event.FocusEvent fe)

Specified by:

focusGained in interface java.awt.event.FocusListener

focusLost

public void focusLost(java.awt.event.FocusEvent fe)

Specified by:

focusLost in interface java.awt.event.FocusListener

itemStateChanged

public void itemStateChanged(java.awt.event.ItemEvent ie)

Specified by:

itemStateChanged in interface java.awt.event.ItemListener