Novel Multi-User Multi-Touch Interaction Concepts using TUIOFX

In this article, I want to share my experience of adding multi-user multi-touch capabilities to the JavaFX graph editor library developed by TESIS DYNAware. Shared interactive surfaces like tabletops or interactive walls are considered to improve the group collaboration and communication in office and educational settings. Still today, applications for shared interactive surfaces that support productive tasks such as document editing are rare.

multi-touch-graph_editor

 

Built on top of JavaFX, TUIOFX aims to support the development of multi-touch applications involving multiple co-located users. Therefore the goal of this project is:

  • To investigate on the applicability of TUIOFX for building multi-user multi-touch document editing/authoring applications.
  • To evaluate two novel interaction techniques that aim at reducing multi-user interaction conflicts.

As a document editing tool I chose to build a simple graph editor for UML class diagram using TUIOFX and this JavaFX graph editor library.

Contents

  1.  Features
    1. Creating New Nodes Simultaneously
    2. Single Selection: Editing Single Nodes Simultaneously via Task-Based Focus
    3. Multi-Selection: Editing Multiple Nodes Simultaneously via AdHoc-Focus-Territory
  2. Implementation
    1. Scrolling the Background
    2. Adding Support for Simultaneous Dragging of Connectors
    3. Full Press-Drag-Release Gesture
    4. Drawing Multiple Tails Simultaneously
    5. Making Single Selection Not Permanent
    6. Multi-Selection via Ad-Hoc-Focus-Territory
      1. Creating a Territory
      2. Closing a Territory
      3. Resizing Nodes
    7. Skinning Nodes
      1. UML Class Content: Name, Attributes and Methods
      2. Displaying and Hiding Connectors
    8. Supporting Simultaneous Text Entry
  3. Conclusion

Features

Most common user tasks supported by graph editors evaluated in 1 are:

  • Creating new nodes, connections and adding attributes to visual items
  • Selecting one or more visual items, (e.g. nodes, connections) to apply edit operations on them.
  • Editing attributes of selected items.
  • Deleting one or more visual items.
  • Navigating functionalities such as panning, centering the view or zooming in/out in order to view specific parts of the content.

Given these basic user tasks, the UML graph editor that I have implemented is limited in providing features to accomplish these common user tasks in multi-user settings. Now you should have a basic idea on the features of the UML graph editor I have implemented. Next, I will discuss some issues and challenges that have occurred while implementing some of these features. I will also present the solutions I used to solve these issues.

The graph node used to represent a UML class

Creating New Nodes Simultaneously

In the UML graph editor, users can create simultaneously new UML classes which are represented by a UML graph node, as depicted in Figure 1. A UML node consists of three sections separated by lines. From the top, the first section contains the class name, the second the class attributes and the last part contains methods. To allow multiple users to create multiple new nodes in parallel, users must be able to create new nodes without waiting for other users to terminate their actions. That is why adding a new node via a single global menu is inappropriate in multi-user settings because a global menu can only be used by a user at a time. Furthermore, a single global menu might raise conflicts, for instance, when a user accidentally performs an edit operation on items selected by other users.

Single Selection: Editing Single Nodes Simultaneously via Task-Based Focus

Users can select a single node by clicking on it. However, a node is marked as selected only for the period in which it is clicked. As for creating new nodes, selecting edit operations by using a global menu will hinder parallel interactions. To overcome this limitation, the UML graph editor provides a context menu with a list of appropriate edit operations for each nodes. For example, a single node can be deleted and duplicated via a context menu it which is activated by performing a tap-and-hold gesture on a node rather than just clicking on it. Furthermore, a double-tap or a double-click on the text inside the UML node will open a text input control and a soft keyboard.

Node marked as selected. Right: Selected node with its ContextMenu

The problem with both approaches is that the behaviour of the context menu and the text input control relies on the single focus model which determines that only one widget in the whole application can be selected at a time to exclusively gain the focus. For instance, with the single focus, context menus disappear immediately after they lose the focus (i.e. as soon as a touch occurs outside the PopupWindow). With multiple users interacting on a shared interactive surface, this may happen even before the user was able to make any selection. As for context menus, a text input control must first be selected to have the focus in order to interact with it. The single focus model of traditional GUI toolkits makes it hard or even impossible for multiple users to interact simultaneously using these controls. Therefore to support multi-user interaction, it is essential to make sure that users cannot accidentally close context menus of other users or prevent other users from writing into text input controls.

Fortunately, TUIOFX provides a task-based focus concept 2 to avoid accidental conflicts caused by the single user focus model. Instead of having all UI elements in the application sharing a single focus, the task-based focus gives us the ability to build multiple single focus areas (multi-foci) where each area consists of UI elements related to an individual task 3. The idea behind this concept is that we can assume that only one user will simultaneously interact with a group of UI elements related to an individual task. This group of UI elements is called a focus area. All elements inside a focus area share the same focus such that a selected element remains in focus until another element within the same focus area gets selected and gains the focus. This means that from the user perspective nothing has changed in the interaction with UI elements within a focus area since all UI elements will still behave according to the single focus model. So, what is new about this concept? What is new is that, by having a focus area, we can easily define that all elements inside a focus area will not be affected by user inputs occurring outside this focus area. This is actually what will prevent users to accidentally deactivate context menus or text input controls of other users.

Back to the UML graph editor, I defined each single node as a focus area since we can assume that only one user will simultaneously interact with a node. By doing so, user inputs outside a node (focus area) will not close the context menu or deactivate the text inputs. To close the context menu of a single node, users must either select a option of the context menu or click inside the node that owns the context menu. To allow users to edit the text fields of nodes in parallel, each nodes is attached to a soft keyboard so that double-tap or double-clicking on a text field opens a soft keyboard. Since each node is defined as a single focus area, the same soft keyboard instance is also shared between all text input controls inside the focus area.

Multi-Selection: Editing Multiple Nodes Simultaneously via AdHoc-Focus-Territory

Sometimes users want to select and group multiple items to manipulate them at once. To achieve that, users can select multiple elements by drawing a selection box  where everything inside the selection box becomes selected. Each selection box has a context menu with options for duplicating, removing, deleting all the nodes inside the selection box.

Multi-selection: Selecting multiple nodes by drawing a selection box (blue box)

Further, by clicking and moving a node, all other nodes inside the selection box will also be moved in the same direction. Resizing one node will also resize the other one in the selection box. With this, you can manipulate a group of nodes at once or interact directly with a single visual item. A selection box remains visible until an option in the associated context menu is selected or when a user clicks inside the selection box without hitting any visual element. To allow multiple users to interact with different groups of nodes in parallel, users can create multiple selection boxes simultaneously. That is, each participant can make her own selection box of multiple elements (nodes, joints, connections etc.) in order to manipulate these elements

As with the single selection, we must prevent the context menu of this selection box to disappear immediately when the next touch input occurs. Again, we can assume that the manipulation of items inside a selection box is done only by one user at a time and therefore each selection box can be defined as a focus area dynamically. Consequently, a selection box can be seen as a transient task based focus area that is created by drawing a selection rectangle. We call this concept ad-hoc focus territory since it helps the ad-hoc generation of a focus area. Being defined as a focus area, any user input outside a selection box have no influence on the UI elements inside the selection box. In the remainder of this article, I will use the terms territory and selection box interchangeably.

Implemetation

Writing an own UML graph editor from scratch would not be achievable in one semester. Therefore, I chose to build the UML graph editor on top of an existing graph editor library. Nevertheless, this is a good way to evaluate TUIOFX by adding multi-user multi-touch capabilities to a traditional document editing tool. As already mentioned, the UML graph editor was implemented by extending the graph editor library which provides functions for creating and editing graphs. The library comes with the following visual elements:

UI elements of the graph editor library

Each visual element has an abstract GSkin class. For example, all concrete connectors must extend from GConnectorSkin and all nodes from GNodeSkin. Nodes and joints can be selected either by clicking on them or by selecting many of them using the classic rubber-band selection. More details about the features provided by the graph editor can be found here.

Scrolling the Background

By default, the graph editor captures ScrollEvents to pan the view (background) regardless of where you place the mouse cursor in the application during the scrolling. This works well for mouse input devices where you typically can only scroll with the mouse wheel. However, the JavaFX specification expects touch input devices to generate TOUCH_MOVE, MOUSE_DRAGGED events besides SCROLL events when touches are moved on the input surface. Consequently, if users move a node or drag a connector then they will also equally pan the view to the same direction. This causes some strange behaviours, for example, if you tend to move a node towards another node, both nodes will never come closer.

To fix this behaviour for touch devices, I changed the setOnScroll event handler (see GraphEditorContainer class) such that it pans the view only if users’ are actually touching the view (background):

Adding Support for Simultaneous Dragging of Connectors

The graph editor library provides a ConnectorDragManager class that specifies how a user can use connectors to create connections between nodes. To establish a connection between two nodes, you must drag a connector of one node to a connector of another node. Further, when dragging a connector, the graph editor library draws a tail during the drag gesture to visualise how the connection would look like if it is created successfully. A tail is removed from the scene when the drag gesture ends.

The problem with the implementation of the ConnectorDragManager is that it is designed with a single-user in mind assuming that only one mouse cursor (or one user) will drag a connector at a time. More precisely, the ConnectorDragManager uses single data types for instance variables to store the state of parameters. These parameters determine, for instance, whether a user action should lead to the addition or removal of connections.

These single data type variables are, however, associated to one single mouse pointer dragging a connector. In a multi-user multi-touch scenarios there are multiple touch points and thus each touch point that is dragging a connector must have its own set of parameters. That is why I converted the type of these instance variables to appropriate collection types:

Full Press-Drag-Release Gesture

Normally, if you start dragging a JavaFX Node only this Node will get MouseDragEvent events. However, sometimes when you drag a Node A to a Node B then you also want Node B to receive events such that it gets notified when A is laying on top of it. To achieve that, you must activate the full press-drag-release gesture by calling the startFullDrag() method in the DRAG_DETECTED event handler of the dragged Node (i.e. Node A).

Actually the graph editor library also uses the full press-drag-release gesture to drag a connector to another connector in order to create a connection between two different nodes. The full press-drag-release gesture is activated by calling the startFullDrag() method in the DRAG_DETECTED event handler of the dragged connector (see ConnectorDragManager):

However, the DRAG_DETECTED event handler cannot be used for multi-user interaction because JavaFX’s full press-drag-release gesture implementation is designed only for one mouse pointer. Therefore, I have implemented a multi-user enabled full press-drag-release gesture recogniser for TUIOFX which was described in a previous blog post. To use TUIOFX’s press-drag-release gesture implementation in the graph editor library, I updated the DRAG_DETECTED event handler such that the TUIOFX’s press-drag-release gesture implementation is triggered whenever the MouseEvent.MOUSE_DRAGGED event is coming from a touch screen and not from a mouse input device.

Drawing Multiple Tails Simultaneously

The TailManager class of the graph editor library is responsible for creating, drawing, and removing tails. Similar to the ConnectorDragManager, the TailManager uses two instance variables with a single data type that make the multi-user concurrent interaction impossible.

The instance variable, sourcePosition, stores the source position of the tail and tailSkin stores the JavaFX node that represents the tail. With this, only one tail can be stored and drawn simultaneously. Thus, similar to the ConnectorDragManager, I replaced both variables with a HashMap (connectorTailSkins and sourcePositions):

Making Single Selection Not Permanent

As mentioned earlier, users must be able to mark a single graph element that is not inside a territory as selected by clicking on it. More importantly, the selected element needs to be immediately deselected when the click is released. To achieve this, I updated the handleNodeReleased(), handleJointReleased() to deselect a node or joint that has no territory:

Multi-Selection via Ad-Hoc-Focus-Territory

In this section I will give an overview on how I have implemented the multi-user multi-selection using the ad-hoc focus territory concept explained above.
The graph editor library allows users to select one or more items by dragging a selection box over them. The selection box (blue box in the Fig. below) is visually represented by the SelectionBox class while the SelectionCreator class is responsible for creating a selection box and marking all visual items (nodes, joints, connections etc.) inside this selection box as selected.

However, the SelectionCreator creates only one selection box instance. But as explained earlier, in order to support multi-user selection we want users to be able to create as many selection boxes as they need. To achieve this, I added a HashMap in the SelectionCreator class for storing multiple territories:

I added the class Territory that implements the ad-hoc focus territory concept. The Territory class is now responsible for updating the representation of an existing selection box. In the addDragSelectionMechanism() method of the SelectionCreator class, I replaced all mouse event handlers with touch event handlers since touch events provide more data (e.g. touch id to identify a touch) that ease the creation of multiple selection boxes. Actually, the touch id is used as the key for the territories HashMap.

Creating a Territory

The creation of a territory goes through two states: IN_PROGRESS and FINISHED. In the IN_PROGRESS state, the user is still drawing the selection box and items inside the drawn selection box are marked as selected — as long as they are not already selected. The FINISHED state indicates that the territory was created successfully with respect to the drawn selection.

IN_PROGRESS

The first state starts when the GraphEditorView(view) – which is the topmost Region that contains all visual graph items — receives a touch pressed event and the touch pressed handler (handleViewTouchPressed()) creates a new Territory object. At this stage the newly created territory is in the IN_PROGRESS state. However, if there is already a territory intersecting with the touch point that causes the touch pressed event to be sent, no Territory object will be created. This prevents the creation of a territory on top of existing one. The new Territory is stored in the territories hash map. As a key I chose the touchID of the touch point which is obtained by calling event.getTouchPoint().getId(). By doing so, territories can be identified between TOUCH_MOVED and TOUCH_RELEASED events when they are drawn.

The geometry of the selection box is defined based on the upper left point and the bottom right point. The upper left point (selectionBoxStart) is the current position of the touch pressed event while the bottom right point is the last position of the touch-point before it is removed from the surface. Further, when a touch pressed event occurs, we can only compute the upper left point (selectionBoxStart) and therefore we temporary set the bottom right point of the territory to be equal to the selectionBoxStart.

In the touch moved event handler, the size of the selection box of the territory which was previously created in the touch pressed handler is updated. The previously created territory is retrieved from the territories hash map using the touch id. If the HashMap has no mapping for the given touch id (territory == null), which is the case when an intersection was detected in the touch pressed handler, then the method terminates. If a territory has been retrieved for the current touch point, the geometry of the selection box is updated by setting the bottom right point of the selection box to current positon (selectionEnd) of the touch point.

While the user is dragging the selection box, we select all visual items that are inside the boundary of the selection box by calling territory.selectSelectableElements(). Note that connections are not selected because it is not always possible to uniquely associate a connection to a territory. Actually, a connection might belong to two different territories when it connects two nodes of different territories.
However, if the currently drawn selection box is intersecting with another selection box we do not select any visual items since this current selection box will be removed if the dragging gesture would immediately end (see TOUCH_RELEASED handler).

FINISHED

When a user stop dragging a selection box and release her finger, the touch released handler (handleViewTouchReleased) evaluates whether the previously created territory associated to her touch point should be marked as FINISHED or should the Territory be completely removed.

The creation of a territory is completed and marked as FINISHED when the boundary of the territory’s selection box contains more than two nodes. Remember, territories are only used for selecting multiple items. In order to select a single item, you should perform a single selection as explained above. If the territory has less than two nodes, then this previously created territory is removed from the scene graph and from the territories hash map. If the selection contains more than two nodes the territory.finishCreation() method is called to trigger the completion of the creation of the territory. Note, only after a territory is created successfully, all items selected during the drawing will be attached to this territory.

Closing a Territory

A Territory gets closed when a user touches the territory’s selectionBox without touching any visual element. In this case, the touch-released handler will not be able to retrieve a newly created territory for the given touch id because no territory has been created in the touch pressed handler.

Resizing Nodes

Remaining in the SelectionCreator class, I had to remove the if-condition in the handleNodePressed method in order to resize all items inside a territory relative to the item the user is actually resizing.

Skinning Nodes

In the graph editor library, each visual element (i.e. connections, tails, connectors, joints and nodes) is represented by a particular skin class. To create your own appearances for nodes, connecters etc., you can add your own custom skin as described here.

I created my own node skin, UMLNodeSkin, that extends from GNodeSkin. The visual structure of the UMLNodeSkin looks as follows:

  • ResizableBox (StackPane) is the root for all visual elements of a graph node provided by the graph editor which allows users to resize its content/children by dragging the edge or corner.
  • UMLBox extends from Region and contains three TexArea controls to represent each section of a UML class (i.e. class name, attributes and methods).
  • For the sake of simplicity, each node (UML class) has only four connectors that are placed on each side of the UMLBox.

Visual structure of the UMLNodeSkin.

UML Class Content: Name, Attributes and Methods

The content of an UML class is divided into three sections for name, list of attributes and list of methods. For each section of a UML class node, there is a TextArea where users can specify the name, attributes or methods of the current UML class node. However, each TextArea is deactivated to not receive any user input (see UMLNodeBox class). This is necessary for allowing users to move a node by dragging it from everywhere inside the node boundary. However, this would not be possible if all three TextArea controls receive any user input. Therefore I blocked them from doing so:

Consequently, to edit the content of a TextArea, users must first active the corresponding TextArea control. To do that, users must double-clicking inside one of the UML class sections (i.e. titleSelectionBox, fieldsSelectionBox, methodsSelectionBox), then the TextArea of this section gets activated and an on-screen keyboard is shown:

When the node is deselected all TextArea controls of this node are deactivated again:

Displaying and Hiding Connectors

Furthermore, each connector can have only one connection. There are three types of connectors: input, output and undefined which are define dynamically depending on the user actions.

  • An undefined connector is not connected with any other connector.
  • An output connector is a connector that has been dragged and released on top of another connector in order create a connection.
  • An input connector is a connector on which an input connector has been released.

When creating a node, four undefined connectors are created and placed on each side of the UMLBox. I increased the size of the connectors in order to improve the selection with fingers, but I feel that these connectors make the nodes not look like UML class diagrams anymore. Therefore I chose to hide at least undefined connectors and to show them only when they are needed:

  • As it is likely that users want to interact with a node immediately after its creation, the four undefined connectors are displayed after a node has been created. But the newly created node and connectors get hidden if users do not interact with them for a certain period of time (2000 ms).
  • Hidden, undefined connectors are shown again when the node containing these connectors is selected via single selection (clicked)
  • Hidden, undefined connectors are also shown when the connector of another node reaches the boundary of the node containing these hidden connectors. However, undefined connectors are hidden again when the dragged connector is moved outside the boundary (see handleMouseEntered()) and handleMouseExited() in UMLNodeSkin.

In any cases, nodes that have been deselected will hide their connectors after 2000 ms

Supporting Simultaneous Text Entry

As mentioned above, for editing the content of an UML node users must double-click the corresponding section in order to activate the text input control containing the content. However, by default, JavaFX does not support multiple users to write simultaneously into different text input controls, because only one text input can exclusively gain the focus and thus receive user inputs. Fortunately, with TUIOFX every text input control can have the focus and can also be attached to an individual soft keyboard instance such that touching a text input control opens a soft keyboard. With this it becomes possible to edit the content (name, attributes and methods) of multiple UML class nodes at a time.
However, for this UML graph editor, I wanted the same keyboard instance to be shared among multiple text-input controls within the same UML class node because I expect that most of the time only one user will edit the content on a single node at a time. To achieve this, I define each node as a focus area:

By doing so, all text area controls within the same focus area will share one soft keyboard instance. The focus area encompasses the UMLNodeBox (which is a Region) and all its children including all three TextArea controls.

Conclusion

This project shows that TUIOFX is a good candidate for evaluating novel multi-touch multi-user interaction techniques. By extending the single-user graph editor library, I feel that adding multi-user capabilities to an existing single user application might require restructuring many parts of the application. It might be easier to just rewrite most parts of the application from scratch.

  1. Gladisch, S., Kister, U., Tominski, C., Dachselt, R. and Schumann, H. Mapping Tasks to Interactions for Graph Exploration and Editing. In IEEE Conference on Information Visualization – InfoVis (Oct. 25-30, Chicago, Illinois, USA), 2016
  2. Fetter, M., Bimamisa, D. and Gross, T. Task-Based Focus and AdHoc-Focus-Territory—Novel Concepts for Shared Interactive Surfaces. In Extended Abstracts of the Conference on Human Factors in Computing Systems – CHI 2016 (May 7-12, San Jose, CA, USA). ACM Press, New York, NY, USA, 2016. pp. 1193-1200.
  3. http://tuiofx.org/blog/2016/05/03/tuiofxs-task-based-focus-concept-presented-at-chi-2016/

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *

*