Without doubt, the greatest challenge in accessible UX design is real-time systems where user response time is of the essence.
It’s a challenge because accessibility often involves moving or supplementing information across design spaces. There are three commonly accepted design spaces: visual, sonic, haptic (to those I would add cognitive at least). You can see these challenges clearly in computer gaming, and in the guidelines for designing accessible games. Those guidelines are not short, and excellent as they are, don’t really address rules about moving and/pr synchronizing content between design spaces, other than recommending that is done.
The easiest way to explain the problems involved is to consider the problems of an actual game, and the one I fall back on is usually Tetris, which I use because of its popularity: it needs little introduction.
1. Introducing Tetris
A screenshot of Tetris in action is shown in Figure 1 below.
Figure 1 shows a game in progress with at tile falling, some tiles already fallen, and a prediction of the next tile. Also shown are the current score and level, time elapsed, and score history.
The challenge with Tetris is the proximal content inherent in the game – rotating and guiding falling shapes to match gaps on the “floor” of the grid in a timely fashion. If we are to create an accessible version of the game, then those underlying game concepts need to be understandable in at least both the visual and sonic design spaces. Even within the “default” visual design space of the Tetris game above, it is necessary to consider users who cannot see the entire game area easily.
There are many variations on the classic Tetris game from 1985. Modern, licensed versions are required to conform to the Tetris Guidelines controlled by the copyright owner, The Tetris Company. It would appear that the guidelines are only available to licensees and not to the general public. However, a version of Tetris rules derived from studying licensed games is available.
The version of the game in Figure 1 shows the essential elements of Tetris. The game is about managing falling bricks so that they form complete lines within the playing area. Each time a line is completed, it disappears from the playing area and the player scores points; if the player manages to complete multiple lines simultaneously then the score multiplies by the number of lines.
The bricks that fall are one of seven defined shapes, each constructed within a 4×4 grid, and using exactly 4 squares of the grid. The shapes are shown in Figure 2 below.
The playing area is a visible grid of 10 columns by 20 rows, with two hidden rows at the top of the playing area in which the bricks begin to fall. Each defined shape always begins at the same horizontal offset, and as close vertically to the visible grid as possible.
As the bricks fall, the user is able to move them left and right within the playing area, and to rotate them. Rotation is constrained if the applied rotation rules would place part of the tile’s shape outside of the playing area; different versions of the game apply different rules.
Tiles continue to fall until they either reach the bottom row of the grid, or are impeded by an already fallen tile. Having reached that position, the tile is still, for a short period, movable horizontally to either edge of the playing area, or until any part of the brick’s shape is impeded by an already fallen brick. Consequently, it is possible to slide one tile under another, and potentially for the tile to begin to fall once more.
Once a tile has landed, and has timed out into the “locked” position, the next tile begins to fall. This pattern continues until the landed tiles pile up to reach the top of the visible playing area so that a newly falling brick cannot begin its descent.
The next tile shape to appear is randomly generated; although a brief reading of online user chat would suggest that the “randomness” is suspect in some games, with players relying on the poor quality of game randomness to predict the likely next several shapes.
Which shape is next, once the current tile has fallen, is always known to the user (a vertical bar in the example of figure 1). According to the Tetris Guidelines, there is also the possibility of “holding” the current falling tile. Only one tile may be held at any one time, and it is replaced with either the tile already in the holding box, or the next randomly generated tile if the hold box is empty.
All modern licensed games are required to provide a “ghost” tile. The ghost tile shows the position that the currently falling tile will reach if it dropped immediately to the bottom of the playing area. This feature goes with the user’s ability to “hard drop” tiles, if the player is happy with that predicted location; the aim being to speed up the game. Figure 1 does not contains a “ghost” tile.
Tile rotation is game version dependent, but modern licensed versions use the Super Rotation System (SRS) to describe rotation. Rotation for each tile in SRS is given in Figure 3.
Tiles are shown within a 4×4 grid. This leads to a view of each tile as a 4×4 pixel sprite with three alternative bitmaps, and with the non-populated cells of Figure 3 representing transparency. In this view, tiles move until progress of their visible grid cells is impeded by the populated cells of an existing tile/sprite i.e. the transparent areas of a sprite may overlap other sprites.
Tetris has ten levels of play, starting at level 1. The game level changes each time 10 lines of bricks have been completed. Once at level 10, the user remains there. Each step in level corresponds to an increase in the speed at which the tiles fall.
For games running on a PC, control is defined to be through the keyboard, based largely on the arrow keys, reflecting the original 1980s mini-computer origin of the game. Special rules apply to “move left” and “move right” controls; pressing and holding those controls causes the lateral movement of the tile to accelerate.
Returning to rotation, there are specific game rules related to rotation of a tile perched on the corner of another tile. This is similar to the problem of rotating tiles that are too close to the side of the playing area, but in this case the rules describe how tiles can rotate around the surface of the tile next to it whilst falling. It is generally the “T” shaped tile that gets these special rules, and they are called “T-spins” and can score additional points. Recognition of T-spins is recommended by the Tetris Guidelines, but it is not mandated.
From the Game Description, it is clear that Tetris is a deceptively simple game that places significant cognitive load on the player, especially as game speed picks up. The player is expected to:
- Recognizing seven basic tiles.
- Follow the movement of one tile as it progresses down the playing area whilst assessing its landing position at the bottom of the playing area.
- Match the outline of a moving tile to gaps in a silhouette at the bottom of a playing area.
- Optimize that match to fill horizontal lines in order to score points.
- Optimize that match to take account of the known next tile to fall.
- Optimize that match to take account of how some tiles can rotate around the corners of obstacles.
- Optimize that match to fill more than one line at the same time in order to score more highly (a multiplying factor).
- Optimizing that match by “holding” and later re-using currently unsuitable tile.
- Match not only vertical gaps in the silhouette formed by already fallen tiles, but also horizontal gaps related to the raggedness of that silhouette.
- To continue to succeed in achieving (a) to (i) whilst game speed increases.
The game as specified is almost completely visual in nature, requiring visual pattern matching skills in a time-limited environment. Sound specification is limited to requiring particular Russian folk music be played during the game. Haptically, the game is optimized to key presses, with specific keys on the PC keyboard mandated by the Tetris Guidelines.
In terms of game design, there appeared to be a number of contemporaneous elements to handle:
- The current tile and its relationship to the playing area, and the time it has dwelled at the current position (for game speed and for the “lock” timeout).
- Previous tiles still (sometimes partially) visible in the playing area i.e. the current silhouette.
- The relationship of the current tile to its ghost position relative to the silhouette.
- The shape of the next tile.
- The shape of the “held” tile, if any.
- The current game level.
- The number of lines of bricks completed.
- The current score.
- Historic high scores.
All of which need to be understandable to the user.
Clearly the importance of some elements varies during the game, with some elements rather more dynamic than others. In terms of handling the current tile, it is necessary to consider (1) through (5), whilst (6) through (9) are more relevant once a tile has “locked” in position on the playing area, and determination of line completeness has been made. That still leaves five contemporaneous information channels for the user to comprehend, some of which, such as the relationship of the tile shape and orientation relative to the landscape, are non-trivial.
2.4 Timing Considerations
There appears to be five different timeouts to consider within Tetris:
- Dwell time for a falling tile as it passes through rows of the playing area grid.
- The lock timeout when a falling tile meets an obstruction (an already locked tile, or the bottom of the playing area).
- The time before the next tile appears after the current tile has locked.
- The auto-repeat rate for “move left” and “move right”.
- The key dwell time necessary to identify a user key press.
Each of these is likely to be impacted by user capability. This is further complicated if the information expressed to the user is provided in the sonic design space where the rate of transmission of information to the user is of orders of magnitude less than for the same information presented visually.
My first observation when dealing with Tetris as a game was the lack of formally standardized metaphors to express computer games. As a developer so used to thinking in HTML for the Web, it feels almost naked to be without the familiar document metaphor of headings, paragraphs, pages, tables, divisions, and spans to support content description and rendering. Consequently, the first task in the design process would be to identify common presentation metaphors in games that have usage within Tetris, aas each would need transcoding between design spaces. My suggestions are below.
- “Cockpit”, and “Head-up display”.The version of Tetris in Figure 1 above is a cockpit, in that the playing area is surrounded by instruments that help the user operate the game. With a head-up display, some part of the instrumentation would be overlaid on the visible playing area. If Figure 1 included the “ghost brick” that helps the user know where the current brick would land if left alone, then we would have a combination of cockpit and head-up display.
- “Immersive”, and “Observational”.In an immersive game, the user would be at the centre of the action with a restricted field of view representing a “front” (or similar) view for the player; a first person shooter such as Duke Nukem (ref) would be a good example of this. In an observational game, the user is an omnipotent observer of the playing area; the user may need for practical purposes to zoom in and out, and to pan the view, but essentially the entire playing area and all the elements within it are visible to the user. Games such as Tetris, Pac Man, and Manic Miner (a game for Sinclair computers in the 1980s) are good examples of observational games. I refer to Manic Miner in that whilst the game had many levels, the whole of the playing area for the current level is visible to the user.
- “Sprite-based animation”.Sprite-based animation exists when individual elements or groups of elements are perceived by the user as appearing, moving within, or leaving the playing area. The tiles in Tetris are a good example of this. There appear to be a number of aspects of sprite-based behaviour (which are not mutually exclusive):
- Game- influenced sprites; the flying aliens in Space Invaders are one example.
- User-influenced sprites; the user-controlled gun in Spaces Invaders is one example.
- Sprite-influenced sprites; flying aliens destroyed on contact with missiles.
- Time-limited sprites; the high value flying saucers in Space Invaders.
- Morphing sprites; the up-down arms of the aliens in Space Invaders.
- Translucent sprites; the “ghost-brick” in Tetris is one example.
- Opaque sprites; the falling and fallen bricks in Tetris are examples.
- Synchronized sprites; the movement of lines of aliens in Space Invaders.
- Handshaking sprites; tessellation of tiles in Tetris.
- Pregnant sprites; exploding aliens in Space Invaders.
- Borg sprites; merging of Tetris tiles into the silhouette.
The identified aspects of behaviour are certainly not mutually exclusive: A falling tile in Tetris is: game influenced (starting position, speed, terminating position); user influenced (left/right, rotate, hold); sprite influenced (terminating position); morphing (rotate); opaque; handshaking (quality of tessellation); pregnant (separating into individual squares once “locked”); and borg (individual squares forming part of the silhouette once “locked).
Each aspect has individual properties, and potentially a life-cycle describable as a finite state model. In essence, an individual sprite may take on any aspect of behaviour as is appropriate to the life-cycle state that it has reached, and by implication also lose them. So, the sprite has a state model, and so may each aspect of its behaviour.
- “Canvas-based”, “Grid-based” 2-D playing area.A canvas-based 2-D playing area is one where the smallest movement of a sprite is the “pixel” resolution of the display; flying aliens, as the fall to earth in Space Invaders is one example of this. A grid-based 2-D playing area is one where sprites jump a noticeable distance as they move, emphasizing the grid-based nature of the playing surface; horizontal (and potentially vertical) movement in Tetris is a good example of this.
- “Canvas-based”, “Grid-based” 3-D playing area.The 3-D equivalent to (d) above. Duke Nukem is one easy example of the canvas-based 3-D playing area; whilst 3-D Tetris is an example of the grid-based model.
- “Gravity”.The idea that things fall downwards, rather than simply move towards the bottom of a screen. Typically a user moves a cursor down a screen, but potentially across a map at the same time. Tetris is another example where the tiles fall down the playing area; whilst you can play Tetris perfectly well with the grid rotated through 90 degrees, the metaphor is definitely one of falling tiles.
- “History”.The effect of the last (n) user actions are remembered and potentially can be undone. For example being able to go back a move in Solitaire. The construction of the silhouette in Tetris is another example.
- “Elapsed time”.Elapsed time is the time taken by the user since starting the game. It may be from selecting “start”, or from the user making the first move. An example of the former is Tetris, and of the latter is chess. Chess is a good example in that there are potentially multiple elapsed times to consider: for each player; and for the game as a whole.
Accessible games and their design challenges
I wanted to describe Tetris is detail, partly to show the size of the task of creating an inclusive version of the game, and partly to show ow one approaches the design of an accessible user interface. It starts with an analysis of the information conveyed between user and application, and any implied timing and metaphors that impact on design.
Having done this analysis and requirements capture, we can then go on to model and implement the game with our accessibility concerns front and centre.