TUIOFX full paper at EICS 2017

Last week we presented a full paper on TUIOFX at EICS 2017 – The 9th ACM SIGCHI Symposium on Engineering Interactive Computing Systems in  Lisbon, Portugal. The paper, which provides more details on the engineering and implementation of TUIOFX  can be downloaded in the ACM Digital Library following this link.

Fetter, M., Bimamisa, D. and Gross, T. TUIOFX – A JavaFX Toolkit for Shared Interactive Surfaces. Proceedings of the ACM on Human-Computer Interaction 1, 1 (June 2017). pp. 9:1-9:18. (ISSN: 2573-0142).

Novel Multi-User Multi-Touch Interaction Concepts using TUIOFX

In this article, I want to share my experience of adding multi-user multi-touch capabilities to the JavaFX graph editor library developed by TESIS DYNAware. Shared interactive surfaces like tabletops or interactive walls are considered to improve the group collaboration and communication in office and educational settings. Still today, applications for shared interactive surfaces that support productive tasks such as document editing are rare.

multi-touch-graph_editor

 

Built on top of JavaFX, TUIOFX aims to support the development of multi-touch applications involving multiple co-located users. Therefore the goal of this project is:

  • To investigate on the applicability of TUIOFX for building multi-user multi-touch document editing/authoring applications.
  • To evaluate two novel interaction techniques that aim at reducing multi-user interaction conflicts.

As a document editing tool I chose to build a simple graph editor for UML class diagram using TUIOFX and this JavaFX graph editor library.

(more…)

TUIOFX’s Task-Based Focus Concept presented at CHI 2016

We are presenting a poster on some of the novel and unique concepts behind TUIOFX at the leading HCI conference, the  CHI 2016 Conference on Human Factors in Computing Systems in San Jose, USA.

Fetter, M., Bimamisa, D. and Gross, T. Task-Based Focus and AdHoc-Focus-Territory—Novel Concepts for Shared Interactive Surfaces. In Extended Abstracts of the Conference on Human Factors in Computing Systems – CHI 2016 (May 7-12, San Jose, CA, USA). ACM Press, New York, NY, USA, 2016. pp. 1193-1200.  (Read it here!)

Mixing Console Demo

Inspired by the nice look of the gauges in Gerrit Grunwald’s blog post Harmonic Code: Building a simple Dashboard using Medusa I felt like trying out how his Medusa gauges library would look like on a Shared Interactive Surface with  TUIOFX. I started from the example in the blog, but quickly added some interactive elements in form of some sliders, to control the gauges. The design of the gauges SlimSkin and the TUIOFX-Widget styling work together quite nicely.

TUIOFX Mixer Example 2 TUIOFX Mixer Example 1

To go a little further, I put together a quick UI mockup for a mixing console. In order to have some sort of knobs as control, I remembered a blog post by Jasper Potts, where he developed a knob skin for the Slider control. While — due to some API changes in JDK 8—his code did not work out of the box, it greatly helped me to quickly come up with an own Knob control implementing custom Skin and Behaviour.

TUIOFX Mixer Example 3

AirPrintFX Demo

In order to easily share documents to an Interactive Surface we implemented an AirPrint PDF Printer in pure Java. It uses the JmDNS library to announce a PDF-capable  AirPrint-Printer to iOS- and OS X-devices in the local network via Apple’s Bonjour zeroconf-service. When a device discovers the printer and starts a print job, the incoming requests are handled by a HttpServer. The server processes the incoming IPP (Internet Printing Protocol) requests and accepts incoming print jobs. The document in the response body is passed to Apache PDFBox in form of a byte[], gets loaded into a PDDocument, and (the first page) is rendered into a BufferedImage. The document is displayed in form of an ImageView — with TUIOFX it is easily zoomable, rotatable and movable on our TUIO-enabled multi-touch table.

See this little demo video of how it works:

Full Press-Drag-Release Gesture Using TUIOFX

One major functionality of TUIOFX is the provision of touch input from TUIO-enabled 1 devices in form of  regular JavaFX mouse and touch events. For the developer these events are indistinguishable from those produced by the OS.  Recently, we added support for the full press-drag-release gesture to TUIOFX, for which we unfortunately had to sacrifice this general compatibility principle. Until now, TUIOFX was only simulating the simple press-drag-release. The difference between both mouse-based gestures is described here. Briefly, the full press-drag-release gesture is used for dragging a node to another node, for instance, “for connecting nodes by wires” etc. 2.

Normally, to enable the full press-drag-release gesture, you must call the startFullDrag method on a node or scene after receiving a MouseEvent.DRAG_DETECTED, i.e. inside of a MouseEvent.DRAG_DETECTED event handler:

The startFullDrag() method triggers the full press-drag gesture recognizer of JavaFX. The node on which the startFullDrag() method is called, is also defined as the gesture source of a dragging gesture.

JavaFX-FPDG

To enable simultaneous multi-touch multi-user interaction, TUIOFX uses its own gesture recognizers to detected touch gestures and for emulating mouse gestures like the full press-drag gesture. However–as far as we see–JavaFX does not provide a way for TUIOFX to be aware that the startFullDrag() method was called.

Therefore, we had to find a way to trigger our own full press-drag gesture recognizer implementation. Because it is not possible to reuse JavaFX’s startFullDrag() method to acknowledge our gesture recognizer, TUIOFX-Core API now provides a new static method TUIOFX.startFullDrag() which triggers TUIOFX’s full press-drag gesture recognizer.

TUIOFX-Core creates multiple gesture recognizers for each graphical element touched on by the user as shown in the following illustration.

TUIOFX-FPDG

When the user touches the Polygon in order to start a full press-drag gesture, TUIOFX-Core only creates gestures recognizers for the Polygon node 3. This means calling TUIOFX.startFullDrag(polygon) triggers the full press-drag gesture recognizer of the Polygon.

But what about using a gesture source node that is different from the actually dragged Polygon node?
Sometimes you want the gesture source to be the parent node (Pane) of the dragged node (Polygon), for instance, to ease the geometrical computation. Calling TUIOFX.startFullDrag(pane) will indeed define the Pane as the gesture source, but it will also try to trigger the press-drag gesture recognizer instance of the Pane which, however, does not exist at this point. This is because TUIOFX-Core only creates gesture recognizers for graphical elements touched by users, as already mentioned.

Therefore, it should be possible to call TUIOFX.startFullDrag(pane) to identify the pane (the parent element of the Polygon) as the gesture source while still triggering the full press-drag gesture recognizer of the Polygon.
To achieve that, the TUIOFX.startFullDrag(pane) triggers all the full press-drag gesture recognizers of the children of the given gesture source (Pane). In this way, only the child node that has actually sent the MouseEvent.DRAG_DETECTED will start a full press-drag-release gesture.

  1. TUIO
  2. MouseEvent
  3. This is an essential aspect of TUIFOX-Core that enables simultaneous multi-touch multi-user interaction.

Two TUIOFX Publications

We are happy to give a demo of TUIOFX at the INTERACT 2015 in Bamberg, Germany in September and a few days before to present a poster at the German HCI conference Mensch und Computer 2015 in Stuttgart, Germany.

  • Fetter, M. and Bimamisa, D. TUIOFX—Toolkit Support for the Development of JavaFX Applications for Interactive Tabletops. In Proceedings of the 15th IFIP TC.13 International Conference on Human-Computer Interaction – INTERACT 2015 (Sept. 14-18, Bamberg, Germany). Springer, Heidelberg, Germany, 2015. pp. 476-479.  (Read it here!)
  • Fetter, M., Bimamisa, D. and Gross, T. Plattformübergreifende Interaktive Tischanwendungen mittels TUIOFX (Platform-Independent Applications for Interactive Tables with TUIOFX; in German). In 15. Fachuebergreifende Konferenz fuer interaktive und kooperative Medien – M&C 2015 (Sept. 6-9, Stuttgart, Germany). Walter de Gruyter GmbH, Berlin/Boston, 2015. pp. 359-362.(ISBN: 987-3-11-044334-9).  (Read it here!)