Author: David Bimamisa

Novel Multi-User Multi-Touch Interaction Concepts using TUIOFX

In this article, I want to share my experience of adding multi-user multi-touch capabilities to the JavaFX graph editor library developed by TESIS DYNAware. Shared interactive surfaces like tabletops or interactive walls are considered to improve the group collaboration and communication in office and educational settings. Still today, applications for shared interactive surfaces that support productive tasks such as document editing are rare.

multi-touch-graph_editor

 

Built on top of JavaFX, TUIOFX aims to support the development of multi-touch applications involving multiple co-located users. Therefore the goal of this project is:

  • To investigate on the applicability of TUIOFX for building multi-user multi-touch document editing/authoring applications.
  • To evaluate two novel interaction techniques that aim at reducing multi-user interaction conflicts.

As a document editing tool I chose to build a simple graph editor for UML class diagram using TUIOFX and this JavaFX graph editor library.

(more…)

Full Press-Drag-Release Gesture Using TUIOFX

One major functionality of TUIOFX is the provision of touch input from TUIO-enabled 1 devices in form of  regular JavaFX mouse and touch events. For the developer these events are indistinguishable from those produced by the OS.  Recently, we added support for the full press-drag-release gesture to TUIOFX, for which we unfortunately had to sacrifice this general compatibility principle. Until now, TUIOFX was only simulating the simple press-drag-release. The difference between both mouse-based gestures is described here. Briefly, the full press-drag-release gesture is used for dragging a node to another node, for instance, “for connecting nodes by wires” etc. 2.

Normally, to enable the full press-drag-release gesture, you must call the startFullDrag method on a node or scene after receiving a MouseEvent.DRAG_DETECTED, i.e. inside of a MouseEvent.DRAG_DETECTED event handler:

The startFullDrag() method triggers the full press-drag gesture recognizer of JavaFX. The node on which the startFullDrag() method is called, is also defined as the gesture source of a dragging gesture.

JavaFX-FPDG

To enable simultaneous multi-touch multi-user interaction, TUIOFX uses its own gesture recognizers to detected touch gestures and for emulating mouse gestures like the full press-drag gesture. However–as far as we see–JavaFX does not provide a way for TUIOFX to be aware that the startFullDrag() method was called.

Therefore, we had to find a way to trigger our own full press-drag gesture recognizer implementation. Because it is not possible to reuse JavaFX’s startFullDrag() method to acknowledge our gesture recognizer, TUIOFX-Core API now provides a new static method TUIOFX.startFullDrag() which triggers TUIOFX’s full press-drag gesture recognizer.

TUIOFX-Core creates multiple gesture recognizers for each graphical element touched on by the user as shown in the following illustration.

TUIOFX-FPDG

When the user touches the Polygon in order to start a full press-drag gesture, TUIOFX-Core only creates gestures recognizers for the Polygon node 3. This means calling TUIOFX.startFullDrag(polygon) triggers the full press-drag gesture recognizer of the Polygon.

But what about using a gesture source node that is different from the actually dragged Polygon node?
Sometimes you want the gesture source to be the parent node (Pane) of the dragged node (Polygon), for instance, to ease the geometrical computation. Calling TUIOFX.startFullDrag(pane) will indeed define the Pane as the gesture source, but it will also try to trigger the press-drag gesture recognizer instance of the Pane which, however, does not exist at this point. This is because TUIOFX-Core only creates gesture recognizers for graphical elements touched by users, as already mentioned.

Therefore, it should be possible to call TUIOFX.startFullDrag(pane) to identify the pane (the parent element of the Polygon) as the gesture source while still triggering the full press-drag gesture recognizer of the Polygon.
To achieve that, the TUIOFX.startFullDrag(pane) triggers all the full press-drag gesture recognizers of the children of the given gesture source (Pane). In this way, only the child node that has actually sent the MouseEvent.DRAG_DETECTED will start a full press-drag-release gesture.

  1. TUIO
  2. MouseEvent
  3. This is an essential aspect of TUIFOX-Core that enables simultaneous multi-touch multi-user interaction.