2.1 Introduction

2.2 Two-dimensional approaches for 3D manipulation

2.2.1 3D Widgets

Figure 2.1 Examples of "3D Widgets" (taken from [43] and [77])¹.

Other, more abstract, varieties of 3D widgets include the Arcball [153], Chen's Virtual Sphere [40], and the Evans & Tanner [51] rotation controllers. The Arcball and Virtual Sphere both use the metaphor of rotating an imaginary sphere; the Evans & Tanner technique, while not explicitly using this metaphor, was a predecessor of these techniques which exhibits performance similar to the Virtual Sphere [40]. These techniques will be discussed in further detail in the context of Chapter 6, "Usability Analysis of 3D Rotation Techniques." As an additional example, Nielson and Olsen's triad mouse technique allows specification of 3D points by mapping mouse motion onto the projected principle axes of an object (fig. 2.2). The resulting points can be mapped to translation, rotation, and scaling in three dimensions. This technique works well at three-quarter perspective views (as seen in figure 2.2), but does not work as well when the projection of two axes fall close to one another.

Figure 2.2 Cursor movement zones for the triad mouse technique [125].

2.2.2 SKETCH

Figure 2.3 SKETCH uses heuristics to infer 3D placement [186].

2.3 Using the hand itself for input

Figure 2.4 Glove used by Zimmerman [189].

Recently introduced gloves, called chord gloves or pinch gloves [52], detect contact between the fingertips, such as when pinching the thumb and forefinger together. Detecting contact is much more reliable than detecting gestures through finger flexion or hand movement. The gloves cost significantly less without the flexion sensors, making it more practical to use two gloves in one application, as demonstrated by the Polyshop system [1] (fig. 2.5), as well as Multigen's SmartScene application [121]. But most importantly, touching one's fingers together is much easier for users to learn than a gesture language, so fingertip contact is an effective means to provide frequently used commands.

Figure 2.5 Polyshop employs two-handed interaction with chord gloves [1].

2.3.1.1 The Virtual Wind Tunnel

Figure 2.6 Using a boom display and glove with the Virtual Wind Tunnel [25].

2.3.2 Electric field sensing

2.3.3 VIDEODESK and VIDEOPLACE

Figure 2.7 Example VIDEODESK applications using two hands [104].

Kreuger has also explored two-dimensional camera-based tracking of the entire body in the VIDEOPLACE environment [104][105], and he has linked this environment with the VIDEODESK. For example, the VIDEODESK participant can pick up the image of the VIDEOPLACE participant's body, and the VIDEOPLACE participant can then perform gymnastics on the other person's fingers [104]. Although these examples are somewhat whimsical, they demonstrate that the medium has many design possibilities.

2.3.4 Hands and voice: multimodal input

2.3.4.2 Put-That-There

The key insight of the Put-That-There system is that either modality alone is quite limited. Voice recognizers (even today) are nowhere near 100% reliable, especially when the syntax of possible commands is complex. So describing everything verbally is both tedious and error-prone. Gesture (pointing) alone allows selection and dragging, but not much else. But together, the modalities provide a rich medium. Instead of saying "Put the orange square to the right of the blue triangle," the user can speak commands such as "Put that... to the right of that." The recognizer needs to know only one object name: "that." The pronoun is disambiguated using the pointing gesture. This simplifies the syntax of voice commands, and also frees the user from having to remember the names of objects.

2.3.4.3 Two hands and voice

Weimer and Ganapathy [180] discuss voice and gesture input for a 3D free-form surface modeling system. Weimer and Ganapathy initially based the interface purely on gestural information, but added voice for three reasons: (1) people tend to use gestures to augment speech; (2) spoken vocabulary has a more standard interpretation than gestures; and (3) hand gesturing and speech complement one another. Voice is used for navigating through commands, while hand gestures provide shape information. Weimer and Ganathapy report that there was "a dramatic improvement in the interface after speech recognition was added" [180].

2.4 One-handed spatial interaction techniques

2.4.1 Schmandt's stereoscopic workspace

2.4.2 Ware's investigations of the "bat"

2.4.3 High Resolution Virtual Reality

2.4.4 JDCAD

Figure 2.9 JDCAD configuration and cone selection technique [111].

The system set-up is similar to Deering's system [46], with head tracking and a single hand-held input device used in front of a standard monitor (fig. 2.9, left). JDCAD, however, uses an object selection mechanism based on ray or cone casting. The bat is used to shoot a ray or cone into the scene, allowing the user to hold the input device in a comfortable position and rotate it to change the cone direction. The objects intersected by the cone then become candidates for selection (fig. 2.9, right).

Another innovation introduced by Liang is the ring menu. This is a menu selection technique that is designed specifically for a hand-held 3D input device, and thus eliminates the need to switch between the bat and the mouse for menu selection. The items available for selection appear in a semi-transparent belt; a gap in this belt always faces the user and indicates the selected item. Figure 2.10 shows a user selecting a cylinder geometric primitive using the ring menu. Rotation of the bat about the axis of the user's wrist causes a new item to rotate in to the gap and become selected. This technique is useful, but it does not scale well to a large number of items, or to selection of menu items which cannot be easily represented by a 3D graphics icon.

Figure 2.10 The ring menu technique for 3D menu selection [111].

Liang has evaluated JDCAD by comparing it in usability tests with commercially available 3D modelling packages [112]. Liang has not, however, performed formal experimental analysis of the factors which contributed to JDCAD's design.

2.4.5 Butterworth's 3DM (Three-Dimensional Modeler)

2.5 Two-handed spatial interaction techniques

Figure 2.11 The 3Draw computer-aided design tool [141].

2.5.2 Interactive Worlds-in-Miniature

Figure 2.12 The Worlds-In-Miniature (WIM) metaphor [164].

The WIM effectively integrates metaphors for viewing at 1:1 scale, manipulating the point-of-view, manipulation of objects which are out of physical reach or occluded from view, and navigation [133]. I will discuss some further issues raised by two-handed interaction with the WIM in the context of Chapter 4, "Design Issues in Spatial Input," as well as Chapter 8, "The Bimanual Frame-of-Reference."

2.5.3 The Virtual Workbench

The Virtual Workbench has been developed with medical applications in mind. It includes support for tasks of interest in neurosurgical visualization, including cross-sectioning a volume. Poston and Serra have not, however, performed any formal experimental evaluation of the design issues raised by the Virtual Workbench.

Figure 2.13 The Virtual Workbench [138].

2.5.4 The Reactive Workbench and ImmersaDesk

Figure 2.14 The ImmersaDesk [49].

2.5.5 Other notable systems

2.6 Theory and experiments for two hands

Guiard has proposed the Kinematic Chain as a general model of skilled asymmetric bimanual action, where a kinematic chain is a serial linkage of abstract motors. For example, the shoulder, elbow, wrist, and fingers form a kinematic chain representing the arm. For each link (e.g. the forearm), there is a proximal element (the elbow) and a distal element (the wrist). The distal wrist must organize its movement relative to the output of the proximal elbow, since the two are physically attached.

The Kinematic Chain model hypothesizes that the preferred and nonpreferred hands make up a functional kinematic chain: for right-handers, the distal right hand moves relative to the output of the proximal left hand. Based on this theory and observations of people performing manual tasks, Guiard proposes three high-order principles governing the asymmetry of human bimanual gestures, which can be summarized as follows:

(1) Motion of the preferred hand typically finds its spatial references in the results of motion of the nonpreferred hand. The preferred hand articulates its motion relative to the reference frame defined by the nonpreferred hand. For example, when writing, the nonpreferred hand controls the position and orientation of the page, while the preferred hand performs the actual writing by moving the pen relative to the nonpreferred hand (fig. 2.15). This means that the hands do not work independently and in parallel, as has often been assumed by the interface design community, but rather that the hands specify a hierarchy of reference frames, with the preferred hand moving relative to the nonpreferred hand.

(2) The preferred and nonpreferred hands are involved in asymmetric temporal-spatial scales of motion. The movements of the preferred hand are more frequent and more precise than those of the nonpreferred hand. During handwriting, for example, the movements of the nonpreferred hand adjusting the page are infrequent and coarse in comparison to the high-frequency, detailed work done by the preferred hand.

(3) The contribution of the nonpreferred hand starts earlier than that of the preferred. The nonpreferred hand precedes the preferred hand: the nonpreferred hand first positions the paper, then the preferred hand begins to write. This is obvious for handwriting, but also applies to tasks such as swinging a golf club [67].

A handwriting experiment by Guiard illustrates these principles in action (fig. 2.15). The left half of the image shows an entire sheet of paper as filled out by the subject on dictation. The right half of the image shows the impression left on a blotter which was on a desk underneath the sheet of paper. The experiment demonstrates that movement of the dominant hand occurred not with respect to the sheet of paper itself, but rather with respect to the postures defined by the non-dominant hand moving and holding the sheet of paper over time.

In a related experiment, Athenes [7], working with Guiard, had subjects repeatedly write a memorized one-line phrase at several heights on individual sheets of paper, with the nonpreferred hand excluded (no contact permitted with the sheet of paper) during half the trials. Athenes's study included 48 subjects, including a group of 16 right-handers and two groups of left-handers,² each again with 16 subjects. Athenes's results show that when the nonpreferred hand was excluded, subjects wrote between 15% and 27% slower, depending on the height of the line on the page, with an overall deficit of approximately 20%. This result clearly shows that handwriting is a two-handed behavior.

Although for consistency I have used handwriting as an example throughout this section, note that Guiard's original analysis is rich with many examples of bimanual acts, such as dealing cards, playing a guitar, swinging a golf club, hammering a nail, or unscrewing a jar cap. Guiard carefully describes how the principles which he proposes apply to a wide range of such examples.

Guiard also provides an overall taxonomy of bimanual actions. The three principles outlined above apply to the bimanual asymmetric class of manipulative actions only. The above principles do not apply to bimanual symmetric motions, such as weight lifting, climbing a rope, rowing, jogging, or swimming. In particular, during locomotion, symmetry seems to be a virtue, since asymmetric motions would result in postural instability (loss of balance). Symmetric motion is also useful to specify scale or extent, as demonstrated by Kreuger's two-handed technique for sweeping out a rectangle (fig. 2.7, right) [104]. As a final note, Guiard's principles also do not apply to the communicative gestures which accompany speech (for example, see the work of Kimura [101][102]), or those that are used for sign language. Such gestures are most likely a completely separate phenomenon from the manipulative gestures studied by Guiard.

Looking beyond the hands, one might also apply the Kinematic Chain model to reason about multiple effector systems ranging from the hands and voice (playing a piano and singing [68]), the hands and feet (operating a car's clutch and stick shift), or the multiple fingers of the hand (grasping a pen). For example, for the task of playing a flute, Guiard argues that the preferred hand organizes its action relative to the nonpreferred hand as usual, but that the mouth (breathing and tongue motion) performs the highest frequency actions relative to the hands. Similarly, one's voice can also be thought of as a link in this hierarchy. Although multimodal two handed input plus voice input will not be a subject of this dissertation, Guiard's model could offer a new perspective for studying how and when the hands and voice can work well together, and how and when they cannot. In existing studies that I am aware of, the hands and voice have generally been thought of as independent communication channels rather than as mutually dependent or hierarchically organized effectors [17][76][180]. For example, a pianist can easily sing the right-handed part of a piece of music, while continuing to perform the left-handed part, but singing the left-handed part without error is difficult or impossible to achieve, even for simple compositions [68].

2.6.2 Formal experiments

2.6.2.1 Buxton and Myers experiments

Figure 2.16 Configuration for Buxton and Myers experiments [27].

In the first experiment reported by Buxton and Myers, subjects positioned a graphical object with one hand and scaled it with the other. The task allowed subjects to adopt either a strictly serial strategy (i.e., position first, then scale) or a parallel strategy (position and scale at the same time). Buxton and Myers found that novices adopted the parallel task strategy without prompting and that task performance time was strongly correlated to the degree of parallelism employed.

In a second experiment, subjects scrolled through a document and selected a piece of text. Buxton and Myers found that both experts and novices exhibited improved performance using two hands to perform this task versus using one hand, and they also found that novices using two hands performed at the same level as experts using just one hand.

2.6.2.2 Kabbash experiments

A second experiment by Kabbash, Buxton, and Sellen [95] studied compound drawing and color selection in a "connect the dots" task (fig. 2.17). The experiment evaluated the two-handed ToolGlass technique [15]. ToolGlass consists of a semi-transparent menu which can be superimposed on a target using a trackball in the nonpreferred hand. The preferred hand can then move the mouse cursor to the target and click through the menu to apply an operation to the target. Note that this integrates the task of selecting a command (or mode) from the menu and the task of applying that command to objects being edited. In Kabbash's experiment, the ToolGlass was used to select one of four possible colors for each dot.

Figure 2.17 Second experimental task used by Kabbash [95].

Kabbash proposes that two-handed input techniques which mimic everyday tasks conforming to Guiard's bimanual asymmetric class of gestures can produce superior overall performance. Kabbash's results suggest that everyday two-handed skills can readily transfer to the operation of a computer, even in a short interval of time, and can result in superior performance to traditional one-handed techniques. This result holds true despite the benefit of years of practice subjects had with the one-handed techniques versus only a few minutes of practice with the two-handed techniques.

Kabbash also demonstrates that, if designed incorrectly, two-handed input techniques can yield worse performance than one-handed techniques [95]. In particular, the authors argue that techniques which require each hand to execute an independent sub-task can result in increased cognitive load, and hypothesize that consistency with Guiard's principles is a good initial measure of the "naturalness" of a proposed two-handed interaction.

2.6.2.3 Leganchuk's area selection experiment

For the one handed technique, Leganchuk also timed how long it took users to switch between two separate control points used to sweep out the rectangle. Even when this time was removed from the one-handed technique, the two-handed technique was still faster, especially for sweeping out larger, more difficult areas. Thus, Leganchuk argues that the difference in times cannot be attributed to the increased time-motion efficiency alone, and interprets this as evidence that the bimanual technique "reduces cognitive load."

Figure 2.18 Experimental task and apparatus used by Leganchuk [108].

2.7 Summary

My work is different and new along several dimensions. First, I have applied virtual manipulation to a develop a novel interface for volume visualization; the application to neurosurgical visualization is new. Second, much of the previous work has been driven by the technology rather than the real-world needs of an actual user. By working closely with domain experts, I have been able to focus on a real problem and I have been able to contribute not only a description of the design, but also the results of usability tests with real users. Many other efforts have been aimed at general applications without a specific focus or a specific user base.

Finally, most of the prior research has taken either a purely systems-building approach, which boils down to an account of "here is something we built," or it has taken a purely experimental approach, performing psychology experiments which may or may not be relevant to the problems faced by designers. By combining these approaches, my work offers an integrated discussion which is both timely and relevant to design.

² Athenes used two groups of left-handers because there are at least two distinct left-handed handwriting postures; these postures will be discussed further in section 7.2.1 of Chapter 7, "Issues in Bimanual Coordination."

³ Chris Shaw has performed usability studies on the THRED two-handed polygonal surface design system [150], but he has not performed any experimental analyses of the underlying behavioral issues.