Introduction
The central theme of our current interface research at the Brown
University Computer Graphics Lab is that gesture-based interfaces are,
in many cases, faster and more direct than traditional windows, icons,
menus and pointers (WIMP) style interaction. This theme is especially
true when dealing with interaction techniques for 3D modeling
applications. Not only are they fast and more direct, but
gesture-based interfaces allows for users to have maximum screen space
when 3D modeling because the clutter of menus and widgets such as
buttons and sliders are removed. The advantages and potential for
gesture-based interaction has led us to develop Jot, a framework that
allows Graphics Lab members to research novel gesture-based
interaction techniques for the various tasks and operations used in 3D
modeling.
The Jot framework is based on our previous work, called Sketch, a
system for creating imprecise 3D models[2]. Sketch's interface
uses only a three button mouse (no menus or 2D interface widgets are
used). Sketch interprets lines drawn by the user on the image plane
of a 3D view as operations and parameters. These operations include
primitive creation, primitive manipulation, and camera manipulation.
For example, gestures that create primitives provide enough information
to select which primitive to create, its dimensions and its place in
3D. Creating a cube, for example, requires the user to draw 3 gesture
lines one for each of the principle axes, each line meeting at a single
point. The cube is generated with it's length, width, and height
corresponding to the three gesture lines and its place in 3D based on
the intersection point. Primitives such as cylinders, cones, pyramids,
and extrusions can also be instantiated. The primitive manipulation
interface allows for automatic object constraint by gesturally
drawing a motion constraint over the object before manipulating it.
For example, to constrain an object's movement to a given axis, a
straight line is drawn indicating what axis to constrain the object to,
and when the user moves the object it will only move along that axis.
Other gestures constrain objects to move along surfaces, rotate around
a given principle axis, or scale and deform to fit a new gesture contour.
The beauty of the Jot framework is that it acts as a building block
for other related gesture-based interaction research. The framework
has been used for a wide spectrum of interface research for modeling
applications, from concept design to design of detailed feature-based
parametric parts. As members of the our graphics lab research other
venues of gesture-based interaction, the techniques and ideas are
incorporated into Jot such that it can be expanded and extended for
future use. Three of the current extensions that we are working on
include ErgoSketch, SmartSketch, and NetSketch. All three projects
are based on the Jot framework and will be described in this article.
ErgoSketch
ErgoSketch stems from the idea that although Jot makes it easy to
create 3D models, there are some tasks that mouse or pen-based
techniques do not naturally support. For example, the task of rapidly
examining objects in one's hand and annotating in three-dimensions are
better suited to higher degree-of-freedom interaction techniques. As
a result of this deficiency, we developed a system that incorporates
both the initial interaction techniques of Jot and techniques for
doing operations that require a switch of interaction metaphors[1].
ErgoSketch users sit at an ActiveDesk (a variant of the responsive
workbench developed by Input Technologies, Inc. of Toronto) and can
perform the 2D gesture-based 3D modeling operations Jot provides. In
addition, they can pick up, examine, and annotate objects with a
virtual pen. To transition between entering 2D gestures and
manipulating an object, the user puts down the pen and picks up a
tracked proxy object attached to the Active Desk. This action signals
the system to switch to stereoscopic viewing mode. The object the
user was constructing can now be rapidly examined in 3D. In addition,
while holding the virtual object with one hand, a user can use his
other hand to pick up a second tracker which acts as a virtual pen for
annotation of the held object. When the user places the initial
tracker back on the table, the system switches back to monoscopic mode
and the user can go back to creating and editing objects. It is this
seamless transition between 2D and 3D interaction that we find most
interesting.
SmartSketch
The idea behind SmartSketch is that, in many cases, users do not need
to start from such basic primitives as cubes, cones, and cylinders
when creating 3D models. For example, a user is modeling the interior
of certain rooms in a house to show customers the different furniture
arrangements possible. The model requires such primitives as tables,
chairs, desks, and the like. It would be inefficient to have to
create believable furniture objects from the basic primitive set that
Jot provides. Therefore, SmartSketch allows user to create 3D models
by using context-sensitive primitives based on the specific
application domain. These "advanced" primitives are created using
Jot's gesture-based interface metaphor but with new semantic meanings
applied to the creation of objects. In the furniture layout
application, the gesture for creating a cube is semantically changed
such that a chair, desk, window, or wall is created depending on the
context in which the gestures were drawn. For example, if the strokes
drawn take the shape of a bounding box for a chair, then a chair will
be created with those dimensions. However, if the strokes drawn take
the shape of a bounding box for a desk, then a desk is created. The
context in which the shape is drawn is used to resolve any ambiguity.
For example, one set of gestures drawn on a floor object might create
a chair, while the same gestures drawn on a desk could create a lamp.
We are currently exploring a number of other application domains such
as exterior landscaping and city planning.
NetSketch
Another important aspect of 3D modeling that we feel is interesting is
the ability to have multiple people working together in a shared
virtual space in real-time. NetSketch is a project in its early stages
that allows multiple participants to work in a shared virtual space
for 3D modeling. Since the Jot framework is built for the single user
only, we felt an important extension is interfaces for group
interaction. NetSketch will allow us to research the many issues that
come up when combining gesture-based interfaces with a collaborative
modeling application. We are currently researching how to allow a
given participant to show his camera view to others, how to let
participants interact with the same object, and visualization
techniques and communication mechanisms that allow participants to
better understand what others are doing in the virtual space.
Conclusion
Jot is an important and exciting framework for gesture-based interface
research. Its flexibility and robustness has allowed for many
extensions and new projects to develop in our lab. As Jot gets larger
and more ideas are explored, we hope that our body of interaction
research will not only help to advance gesture-based interfaces in 3D
modeling but other applications as well so interfaces are easier,
faster, and natural to use.
References
[1] Forsberg, A. S., LaViola, J. J., Markosian, L., Zeleznik, R. C. (1997)
"Seamless Interaction in Virtual Reality", IEEE Computer Graphics and
Applications, 17(6):6-9, November/December 1997.
[2] Zeleznik, R.C., Herndon, K., Hughes, J. (1996) "Sketch: An Interface
for Sketching 3D Scenes." Proceedings of SIGGRAPH'96, 163-170.