Tech Report CS-04-11

Fluid Inking: Using Punctuation to Allow Modeless Combination of Marking andGesturing

Robert Zeleznik, Timothy Miller, Loring Holden, Joseph J. LaViola Jr.

July 2004

Abstract:

We present a set of principles, called Fluid Inking, for modelessly and fluidly combining gestures with free-form note-taking. The hallmark of these principles is that a gesture is a short sequence of drawn ink marks and terminal punctuation that is collectively and contextually distinct from regular notes even though individual marks and punctuation may be identical to conventional notational idioms, such as scribbling and tapping. Thus, after punctuation is drawn, the preceding ink strokes are compared to gesture templates and if they match they are interpreted as a gestural action. This approach allows gestures to be easy to perform, simple and memorable without restricting the domain of regular notes despite the modeless nature of gesturing and drawing. Further, by omitting earlier punctuation lower-level gestures can be cognitively chunked into higher-level gestures. To illustrate and evaluate our principles, we designed and implemented a full gesture set that handles the core functionality of popular note-taking applications, in addition to making available a special selection mode to support complex disjoint selections. We discuss informal user feedback about our full gesture set and a user study designed to gauge relative user preference for punctuated gestures.

(complete text in pdf or gzipped postscript)