Accessible text input using TUP text prediction

1. TUP

Image of Apple iPod circa 2006TUP (Transparent User guided Prediction), is a technique developed in 2006 by Morten Proschowsky, Nette Schultz, and Niels Ebbe Jacobsen. It was aimed at improving the text entry rate on touch sensitive wheels. Remember that 2006 is the year before the iPhone launched and the “in” hand-held device was the iPod with its circular thumb wheel.

The concept behind TUP was to treat touch sensitive wheels such as on the iPad as a clock-face with the letters of the alphabet located around the face of the clock at fixed positions, so that users could type content by touching the appropriate location on the wheel.

The problem with this simple technique is accuracy, especially if there is no legend on the wheel. TUP compensated for user error by implementing a stochastic based language model, predicting what the next letter would be, essentially one letter look-ahead. It did this by using a a database generated from a English corpus, so that the most popular letters were prioritized over the precise location of the user’s finger.

The problem they found with the predictive approach was that this worked great for likely letters, but made selecting less likely ones much more difficult. Their approach to resolving this was to only use the predictive algorithm when the user’s finger was moving quickly over the wheel, and dropping to a more sequential selection when the finger was moving more slowly.

To test their approach the team built a working prototype on the iPod. The iPod in those days could be hacked to install other operating systems, including Linux, and the team implemented their concept in Podzilla (a very basic GUI) on iPodLinux. The result is shown below in figure 1.

Figure 1: Image of Original TUP as implemented on the iPod
Figure1: Original TUP as implemented on the iPod

Figure 1 shows a screenshot of TUP in action editing a text note. The already input text is shown at the top of the screen, with the lower 75% of screen real estate given over to a clock-face representation of the available input characters. The character the user’s finger is currently over is highlighted (the iPod made a distinction between touch and press).

2.  TUP’s accessibility challenges

For scroll-wheel systems such as the iPod (and similar in-car touch user interfaces) TUP is clearly a huge stop forward but it has obvious accessibility concerns.

The most obvious accessibility concern is the size of the on-screen text, even users without low vision will struggle to read such tightly packed characters. The amount of real-estate taken up by the wheel simulation is also a challenge, as people with cognitive impairment may struggle to use a system where so little of the editable text is visible at any one time.

Less obvious is problems with the scroll-wheel itself. On a physical keyboard, each key is essentially separate with its own guard rail surrounding it which helps users with low vision and with fine motor impairment from pressing the wrong keys (one simple form of assistive tech is to augment that guard rail with something much higher).

There is also there  question of debounce when the user’s finger is moving around the wheel. On a static physical key, research has been done to identify effective ways of dealing with finger bounce, not so when the finger is “in flight”. For people with motor impairment e.g. poor sense of touch or hand tremors this make TUP a significant challenge.

3.  My own implementation of TUP on the iPod

Proschowsky”s paper on TUP inspired me to try installing it on my own iPod with a goal of at least investigating possible solutions to the accessibility issues within their implementation. The result is the screenshot at the top of this blog post which shows the word “hello” being entered. Below is a YouTube video showing the full input of the text. This is a screen recording rather than a video – that has been lost at some point over the last 9 years (my implementation was in early 2007).

Things to note:

  1. I dispensed with the alphabet, leaving only “A” at the top of the wheel to orient the user. I made that as large and clear as possible  and replaces the characters with a solid ring. The nominal position of the user’s finger is sill shown with a highlighted dot but the currently active letter was move to the centre of the wheel to maximize visibility
  2. The first character is still chosen by the TUP algorithm, and speed of the user’s finger still determines whether the next character is chosen by TUP or sequentially relative the the previous character. So the basics of TUP remain in place.
  3. Less obvious because this is is screen recording,  is that the user;s finger can lift from the screen. and the delays in the moving of the dot are due to me tapping the wheel gently and repeatedly. In this case debounce software kicks in  to say that the first touch is considered significant, and after a short debounce period, the finger is assumed to be moving. A minimum timeout was enforced before the same letter could be selected again (around 200ms if I remember correctly).
  4. To deal with fine motor impairment and/or poor sense of touch, wherever the finger landed on the wheel was considered to be the the original character. so landing on “e” and finishing on “g” would register as “e” on the display and moving around the wheel continued with position “g” now representing “e”. So the clock-face could move to accommodate these impairments without impacting upon default usage of TUP.
  5. A second variation was developed for users who struggle to touch a precise point on the wheel. A real-life example would be a user with MS (Multiple Sclerosis).  People with MS experience multiple impairments, two of which are a inability to sense touch, and hand tremors. They essentially guide their fingers by eye and are unsure how much force they will hit the surface with. So, if users in (4) could be said to be landing-centric, then these users are parking-centric.

4.  Reflections on TUP in 2016

My work on TUP was only months old when the iPhone was first launched, and with it the scroll-wheel lost it’s central focus as a means of user interaction. Yes, iPods continued to be produced but UI focus move to touch screen devices, And my work moved on.

What is frustrating, and what killed it dead, was Apple’s approach to users hacking their iPods: they made changes to the BIOS that prevented iPodLinux from loading. Deliberately or not, I don’t know, but the effect on iPodLinux was deadly, Other operating systems such as Rockbox continued, but iPodLinux, and with it, my experimentation with TUP was at an end.

Finally, lust for fun, here is a video from YouTube demonstrating iPodLinux in action.

Leave a Reply

Your email address will not be published. Required fields are marked *