public class GraffitiActivity extends android.app.Activity implements GraffitiPanel.OnStrokeListener
This Android application expands on Demo_Graffiti. For background information, see the API for Demo_Graffiti as well
as the API for the Unistroke
class where gesture recognition is performed. This API focuses on features
added to Demo_Graffiti, including saving and restoring setup parameters using the SharedPreferences
class, storing user performance measurements in data files for follow-up analyses, and auto-correcting input.
API References:
Developer's Guide:
- Storage Options (Shared Preferences)
Training:
In all, there are 10 setup parameters, organized in seven spinners and three checkboxes. The setup parameters are explained as follows.
Parameter Description Participant Code A code to identify the user being tested. In experimental research, a user is typically called a "participant".
Session Code A code to identify the session. This code could be used in an a longitudinal experiment, where participants are tested over multiple sessions. Each session might involve an hour of testing, with one session per day over multiple days. Note: There is no setup parameter for "Block code". The block code is generated automatically by the software.
Group Code A code to identify the group. If the experiment uses a within-subjects design, participants are usually assigned to groups. This code identifies the group. This procedure is known as counterbalancing, which is used to offset order effects. (See section 5.11 in the course's Suggested Readings for complete details.)
Condition Code A code to identify an external condition. This setup item is an editable text field. Tap to popup an input method editor to change the value. The condition code is useful if the experiment included an independent variable pertaining to an environmental condition or a context of use. As an example, an experiment might include "mobility" as an independent variable with levels "sitting" and "walking". The code "Sitting" might be used for the sitting condition and the code "Walking" for the walking condition. These codes appear in the output data to facilitate analyses. BTW, an example of an experiment using this independent variable is described by Fitton et al. in Exploring Tilt-Based Text Input For Mobile Devices With Teenagers.
Number of Phrases The number of phrases entered for each block of testing.
Phrases File A file containing phrases of text that are selected at random and presented to the user for input. The following options appear:
- Phrases - a collection of 500 phrases (no punctuation)
- Phrases100 - a collection of 100 phrases (with punctuation)
- Quick_Brown_Fox - the phrase "the quick brown fox jumps over the lazy dog"
- Alphabet - the alphabet ("abcdefghijklmnopqrstuvwxyz")
Note: The actual phrases are in files in
/res/values/
.
Entry mode The gesture set used for input. The following options appear:
- Graffiti - the standard Graffiti gesture set
- Unistrokes - the original Unistrokes gesture set from Goldberg and Richardson's CHI 1993 paper, Touch Typing With a Stylus
- Digits - digit gestures
- Custom - a custom gesture set
See the API for the
Unistroke
class for additional details on the gesture sets.
Show Gesture Set A checkbox item. If selected, an image showing the gesture set appears just below the input area. If not selected, the image is hidden. Showing the gesture set is useful for new users who are unfamiliar with the gestures.
Lowercase Only A checkbox item. If selected, uppercase characters in the phrases appear as lowercase
Show Recognized Text During Entry A checkbox item. If selected, the recognized characters appear in the transcribed text field after each gesture. If not selected, characters do not appear. See "Auto Correct" below for further discussion.
The setup parameters may be changed, as appropriate, for the current round of testing. They may be saved by tapping
the Save button. This feature was not used in Demo_TiltBall. Having parameters persist from one invocation to the
next is useful since it eliminates the need to change parameters each time the application is used. Retrieving and
modifying the setup parameters is possible using Android's SharedPreferences
and
SharedPreferences.Editor
classes. Objects of these classes are declared in the
GraffitiSetup
activity as follows:
SharedPreferences sp; SharedPreferences.Editor spe;In the activity's
onCreate
method, a SharedPreferences
instance is retrieved as follows:
sp = this.getPreferences(MODE_PRIVATE);Setup parameters are retrieved using get-methods. For example, retrieving the default participant code occurs as follows:
participantCode[0] = sp.getString("participantCode", participantCode[0]);Participant codes, like the other codes, are stored in arrays. The first entry in the array is the default value. It is over-written with the value associated with the key specified in the first argument of the
getString
method. This occurs only if such a key-value exists, having been saved earlier. If the value does not exist, then the
second argument is returned as a default. As seen above, this amounts leaving the first entry in the array as is.
If the Save button is tapped, the current values of the setup parameters are saved via using put-methods in
the clickSave
callback. Here's the code that saves the participant code:
spe = sp.edit(); spe.putString("participantCode", participantCode[spinParticipant.getSelectedItemPosition()]);Of course, the key in
putString
must be the same as the key in getString
. Tapping OK
bundles up the setup parameters and launches the main activity, passing the bundle. The process is much the same as
in our earlier demo program, Demo_TiltBall. When the program launches, the user is presented with a phrase of text to enter (below left). Data collection begins on the first finger touch on the input panel. Gestures are made and recognized, with the results delivered to a transcribed text field (assuming "Show recognized text during entry" is checked). The image below on the right shows the application in use.
Data Storage
As the user enters phrases of text using the selected entry mode (e.g., Graffiti), the software collects data such as the gesture recognition results, timestamps, and gesture shape. At the phrase level, additional measures are calculated such as the entry speed (in words per minute), the character-level error rate (%), and the number of gestures (aka keystrokes) entered per character of text produced (KSPC).
As noted above, the Graffiti app includes auto-correct (details below). Two error rates are calculated and
saved.
One is for the transcribed text as it was produced by the recognizer (i.e., uncorrected). The other is for the
corrected text. These data are useful to gauge the performance of the auto-correct algorithm. As an example, the
following popup dialog shows the results at the end of a phrase of entry. The error rate for the uncorrected
transcribed text was 9.7%. The error rate for the corrected text was 0.0%. (Note: "#
" is an unrecognized
gesture.)
The data are saved in files for follow-up analyses. There are three "summary data" files for each block. The file extensions and contents are as follows:
One of the first tasks in the Graffiti activity is to open and prepare data files for writing. The directory is specified using a final:
final String WORKING_DIRECTORY = "/GraffitiData/";In
onCreate
, the directory path is initialized using
File dataDirectory = new File(Environment.getExternalStorageDirectory() + WORKING_DIRECTORY);The
getExternalStorageDirectory
returns the primary external storage directory. The directory is
not external to the device. As noted in the API,
This directory can better be thought of as media/shared storage. It is a filesystem that can hold a relatively large amount of data and that is shared across all applications (does not enforce permissions). Traditionally this is an SD card, but it may also be implemented as built-in storage in a device that is distinct from the protected internal storage and can be mounted as a filesystem on a computer.
Several steps follow. Filenames for the sd1
, sd2
, and sd3
data files are
built-up using the setup parameters. "B01" is used within the filename for the block code. Including the setup
parameters in the filenames ensures the conditions tested are distinguishable in the names of the output data files.
The file system is then queried to see if the files exist. If so, new filenames are built-up using the next block
code. This is repeated until the filenames do not conflict with existing files; that is, with files created from
previous launches of the application. Then, three buffered writers are opened. With these, we're ready to go. Oh yes,
it's important that the manifest file includes the appropriate uses-permission
tag:
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" /> <uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" />As entry proceeds, data are collected. At the end of a block, data are written to the
sd1
,
sd2
, and sd3
files using the write
method of the BufferedWriter
class. This is followed by flush
and close
. Of course, all this occurs in
try-catch
statements. There is one additional small detail. If the Android device is connected to a host machine via a USB cable, then the following statement is needed to ensure the data files are viewable in the host machine's file manager (e.g., Windows Explorer):
MediaScannerConnection.scanFile(this, new String[] { f1.getAbsolutePath(), f2.getAbsolutePath(), f3.getAbsolutePath() }, null, null);Here are examples of the data files:
Graffiti-P01-S01-B01-G01-C01-Graffiti.sd1
Graffiti-P01-S01-B01-G01-C01-Graffiti.sd2
Graffiti-P01-S01-B01-G01-C01-Graffiti.sd3
The sd2 file is comma-delimited to facilitate importing into a spreadsheet or statistics application. It includes a header line that identifies the data in each column. Each row contains data for the conditions as well as for the the human performance measures gathered for that phrase of input. Here is a look at the data in the sd2 file above, as they appear in Microsoft Excel:
Auto Correct
The Graffiti
class includes an auto-correct algorithm. The motivation for this follows from a
simple observation: Users enter gestures more quickly if they do not monitor the results of recognition after
each gesture. Of course, users monitor the recognition result for a reason – to see if a gesture was correctly
recognized. Wouldn't it be nice if users didn't need to monitor their progress – if an error occurs, it gets
automatically corrected! That's a lofty goal. Achieving this goal requires an auto-correct algorithm.
Of course, if the recognition result does not appear after each gesture, there is nothing to monitor. In this case, users are likely to proceed more expeditiously. This, combined with a good auto-correct algorithm, has the potential to yield input that is both fast and accurate.
Of course, errors will occur. There are three possibilities: (a) a wrong gesture, (b) an ill-formed and misrecognized gesture, or (c) an unrecognized gesture. Auto correct is intended to work in concert with an input modality where users do not monitor their input on a gesture-by-gesture basis. This is the purpose of the "Show recognized text during entry" setup parameter (see above). The rationale and goals are further elaborated in MacKenzie and Castellucci's Reducing Visual Demand for Gestural Text Input on Touchscreen Devices.
The initial design of the auto-correct algorithm is described in Tinwala and MacKenzie's Eyes-free Text Entry with Error Correction on Touchscreen Mobile
Devices. Of course, the complete details are in the source code. Here's a high-level description of the code.
Auto correct is handled by the SpellCheck
class. In Graffiti
, a
SpellCheck
object is declared, instantiated, and initialized as follows:
SpellCheck sc; ... sc = new SpellCheck(); ... sc.setDictionary(wf);The
wf
object passed to setDictionary
is a word-frequency array – a dictionary
– for the language of input. The raw word+frequency values for the dictionary are stored in
/res/values/wordfreq.xml
. At the end of each phrase of input (in the doEndOfPhrase
method in Graffiti
), the uncorrected text phrase is passed to SpellCheck
's
correctPhrase
method. A corrected phrase is returned:
String transcribedPhraseCorrected = sc.correctPhrase(transcribedBuffer.toString());The actual correction occurs in
correctPhrase
using a look-up and compare technique applied to each word
in the phrase. If a word is in the dictionary, it is assumed correct. If a word is not in the dictionary, a
minimum-string-distance algorithm is used to compare the erroneous word with words of a similar size in the
dictionary. The closest and highest-frequency dictionary word is substituted for the erroneous word. There's a bit
more to it, of course. Consult the correctPhrase
method in SpellCheck
for the complete
details. Note: The auto-correct algorithm is only used if the "Phrases file" option in the setup dialog is set to "phrases2.txt". As well, case is ignored (e.g., "Jedi" and "jedi" are considered the same).
DEFAULT_KEYS_DIALER, DEFAULT_KEYS_DISABLE, DEFAULT_KEYS_SEARCH_GLOBAL, DEFAULT_KEYS_SEARCH_LOCAL, DEFAULT_KEYS_SHORTCUT, RESULT_CANCELED, RESULT_FIRST_USER, RESULT_OK
ACCESSIBILITY_SERVICE, ACCOUNT_SERVICE, ACTIVITY_SERVICE, ALARM_SERVICE, APP_OPS_SERVICE, APPWIDGET_SERVICE, AUDIO_SERVICE, BATTERY_SERVICE, BIND_ABOVE_CLIENT, BIND_ADJUST_WITH_ACTIVITY, BIND_ALLOW_OOM_MANAGEMENT, BIND_AUTO_CREATE, BIND_DEBUG_UNBIND, BIND_EXTERNAL_SERVICE, BIND_IMPORTANT, BIND_NOT_FOREGROUND, BIND_WAIVE_PRIORITY, BLUETOOTH_SERVICE, CAMERA_SERVICE, CAPTIONING_SERVICE, CARRIER_CONFIG_SERVICE, CLIPBOARD_SERVICE, CONNECTIVITY_SERVICE, CONSUMER_IR_SERVICE, CONTEXT_IGNORE_SECURITY, CONTEXT_INCLUDE_CODE, CONTEXT_RESTRICTED, DEVICE_POLICY_SERVICE, DISPLAY_SERVICE, DOWNLOAD_SERVICE, DROPBOX_SERVICE, FINGERPRINT_SERVICE, HARDWARE_PROPERTIES_SERVICE, INPUT_METHOD_SERVICE, INPUT_SERVICE, JOB_SCHEDULER_SERVICE, KEYGUARD_SERVICE, LAUNCHER_APPS_SERVICE, LAYOUT_INFLATER_SERVICE, LOCATION_SERVICE, MEDIA_PROJECTION_SERVICE, MEDIA_ROUTER_SERVICE, MEDIA_SESSION_SERVICE, MIDI_SERVICE, MODE_APPEND, MODE_ENABLE_WRITE_AHEAD_LOGGING, MODE_MULTI_PROCESS, MODE_NO_LOCALIZED_COLLATORS, MODE_PRIVATE, MODE_WORLD_READABLE, MODE_WORLD_WRITEABLE, NETWORK_STATS_SERVICE, NFC_SERVICE, NOTIFICATION_SERVICE, NSD_SERVICE, POWER_SERVICE, PRINT_SERVICE, RESTRICTIONS_SERVICE, SEARCH_SERVICE, SENSOR_SERVICE, SHORTCUT_SERVICE, STORAGE_SERVICE, SYSTEM_HEALTH_SERVICE, TELECOM_SERVICE, TELEPHONY_SERVICE, TELEPHONY_SUBSCRIPTION_SERVICE, TEXT_SERVICES_MANAGER_SERVICE, TV_INPUT_SERVICE, UI_MODE_SERVICE, USAGE_STATS_SERVICE, USB_SERVICE, USER_SERVICE, VIBRATOR_SERVICE, WALLPAPER_SERVICE, WIFI_P2P_SERVICE, WIFI_SERVICE, WINDOW_SERVICE
Constructor and Description |
---|
GraffitiActivity() |
Modifier and Type | Method and Description |
---|---|
void |
doEndOfPhrase() |
void |
doNewPhrase() |
int |
getDefaultDeviceOrientation() |
void |
onCreate(android.os.Bundle savedInstanceState) |
void |
onStroke(GraffitiEvent ge) |
static float |
wpm(java.lang.String text,
long msTime) |
addContentView, closeContextMenu, closeOptionsMenu, createPendingResult, dismissDialog, dismissKeyboardShortcutsHelper, dispatchGenericMotionEvent, dispatchKeyEvent, dispatchKeyShortcutEvent, dispatchPopulateAccessibilityEvent, dispatchTouchEvent, dispatchTrackballEvent, dump, enterPictureInPictureMode, findViewById, finish, finishActivity, finishActivityFromChild, finishAffinity, finishAfterTransition, finishAndRemoveTask, finishFromChild, getActionBar, getApplication, getCallingActivity, getCallingPackage, getChangingConfigurations, getComponentName, getContentScene, getContentTransitionManager, getCurrentFocus, getFragmentManager, getIntent, getLastNonConfigurationInstance, getLayoutInflater, getLoaderManager, getLocalClassName, getMediaController, getMenuInflater, getParent, getParentActivityIntent, getPreferences, getReferrer, getRequestedOrientation, getSearchEvent, getSystemService, getTaskId, getTitle, getTitleColor, getVoiceInteractor, getVolumeControlStream, getWindow, getWindowManager, hasWindowFocus, invalidateOptionsMenu, isChangingConfigurations, isChild, isDestroyed, isFinishing, isImmersive, isInMultiWindowMode, isInPictureInPictureMode, isLocalVoiceInteractionSupported, isTaskRoot, isVoiceInteraction, isVoiceInteractionRoot, managedQuery, moveTaskToBack, navigateUpTo, navigateUpToFromChild, onActionModeFinished, onActionModeStarted, onActivityReenter, onAttachedToWindow, onAttachFragment, onBackPressed, onConfigurationChanged, onContentChanged, onContextItemSelected, onContextMenuClosed, onCreate, onCreateContextMenu, onCreateDescription, onCreateNavigateUpTaskStack, onCreateOptionsMenu, onCreatePanelMenu, onCreatePanelView, onCreateThumbnail, onCreateView, onCreateView, onDetachedFromWindow, onEnterAnimationComplete, onGenericMotionEvent, onKeyDown, onKeyLongPress, onKeyMultiple, onKeyShortcut, onKeyUp, onLocalVoiceInteractionStarted, onLocalVoiceInteractionStopped, onLowMemory, onMenuItemSelected, onMenuOpened, onMultiWindowModeChanged, onNavigateUp, onNavigateUpFromChild, onOptionsItemSelected, onOptionsMenuClosed, onPanelClosed, onPictureInPictureModeChanged, onPostCreate, onPrepareNavigateUpTaskStack, onPrepareOptionsMenu, onPreparePanel, onProvideAssistContent, onProvideAssistData, onProvideKeyboardShortcuts, onProvideReferrer, onRequestPermissionsResult, onRestoreInstanceState, onRetainNonConfigurationInstance, onSaveInstanceState, onSearchRequested, onSearchRequested, onStateNotSaved, onTouchEvent, onTrackballEvent, onTrimMemory, onUserInteraction, onVisibleBehindCanceled, onWindowAttributesChanged, onWindowFocusChanged, onWindowStartingActionMode, onWindowStartingActionMode, openContextMenu, openOptionsMenu, overridePendingTransition, postponeEnterTransition, recreate, registerForContextMenu, releaseInstance, removeDialog, reportFullyDrawn, requestDragAndDropPermissions, requestPermissions, requestShowKeyboardShortcuts, requestVisibleBehind, requestWindowFeature, runOnUiThread, setActionBar, setContentTransitionManager, setContentView, setContentView, setContentView, setDefaultKeyMode, setEnterSharedElementCallback, setExitSharedElementCallback, setFeatureDrawable, setFeatureDrawableAlpha, setFeatureDrawableResource, setFeatureDrawableUri, setFinishOnTouchOutside, setImmersive, setIntent, setMediaController, setProgress, setProgressBarIndeterminate, setProgressBarIndeterminateVisibility, setProgressBarVisibility, setRequestedOrientation, setResult, setResult, setSecondaryProgress, setTaskDescription, setTheme, setTitle, setTitle, setTitleColor, setVisible, setVolumeControlStream, setVrModeEnabled, shouldShowRequestPermissionRationale, shouldUpRecreateTask, showAssist, showDialog, showDialog, showLockTaskEscapeMessage, startActionMode, startActionMode, startActivities, startActivities, startActivity, startActivity, startActivityForResult, startActivityForResult, startActivityFromChild, startActivityFromChild, startActivityFromFragment, startActivityFromFragment, startActivityIfNeeded, startActivityIfNeeded, startIntentSender, startIntentSender, startIntentSenderForResult, startIntentSenderForResult, startIntentSenderFromChild, startIntentSenderFromChild, startLocalVoiceInteraction, startLockTask, startManagingCursor, startNextMatchingActivity, startNextMatchingActivity, startPostponedEnterTransition, startSearch, stopLocalVoiceInteraction, stopLockTask, stopManagingCursor, takeKeyEvents, triggerSearch, unregisterForContextMenu
applyOverrideConfiguration, getAssets, getResources, getTheme
bindService, checkCallingOrSelfPermission, checkCallingOrSelfUriPermission, checkCallingPermission, checkCallingUriPermission, checkPermission, checkSelfPermission, checkUriPermission, checkUriPermission, clearWallpaper, createConfigurationContext, createDeviceProtectedStorageContext, createDisplayContext, createPackageContext, databaseList, deleteDatabase, deleteFile, deleteSharedPreferences, enforceCallingOrSelfPermission, enforceCallingOrSelfUriPermission, enforceCallingPermission, enforceCallingUriPermission, enforcePermission, enforceUriPermission, enforceUriPermission, fileList, getApplicationContext, getApplicationInfo, getBaseContext, getCacheDir, getClassLoader, getCodeCacheDir, getContentResolver, getDatabasePath, getDataDir, getDir, getExternalCacheDir, getExternalCacheDirs, getExternalFilesDir, getExternalFilesDirs, getExternalMediaDirs, getFilesDir, getFileStreamPath, getMainLooper, getNoBackupFilesDir, getObbDir, getObbDirs, getPackageCodePath, getPackageManager, getPackageName, getPackageResourcePath, getSharedPreferences, getSystemServiceName, getWallpaper, getWallpaperDesiredMinimumHeight, getWallpaperDesiredMinimumWidth, grantUriPermission, isDeviceProtectedStorage, isRestricted, moveDatabaseFrom, moveSharedPreferencesFrom, openFileInput, openFileOutput, openOrCreateDatabase, openOrCreateDatabase, peekWallpaper, registerReceiver, registerReceiver, removeStickyBroadcast, removeStickyBroadcastAsUser, revokeUriPermission, sendBroadcast, sendBroadcast, sendBroadcastAsUser, sendBroadcastAsUser, sendOrderedBroadcast, sendOrderedBroadcast, sendOrderedBroadcastAsUser, sendStickyBroadcast, sendStickyBroadcastAsUser, sendStickyOrderedBroadcast, sendStickyOrderedBroadcastAsUser, setWallpaper, setWallpaper, startInstrumentation, startService, stopService, unbindService, unregisterReceiver
public static float wpm(java.lang.String text, long msTime)
public void onCreate(android.os.Bundle savedInstanceState)
onCreate
in class android.app.Activity
public int getDefaultDeviceOrientation()
public void onStroke(GraffitiEvent ge)
onStroke
in interface GraffitiPanel.OnStrokeListener
public void doNewPhrase()
public void doEndOfPhrase()