Field guide 02 / recognition
Recognising intent without overreading it
A sensor can report that something moved. Recognition must decide whether that movement belongs to a command—and when it is safer not to decide.
Begin with an action sentence
“Turn the wrist to open the shield” is a better starting point than “classify wrist data.” The sentence names a physical intention and a game consequence. Add the conditions: the shield is available, the hand begins in a comfortable zone, and the player can lower it without accidentally selecting another action. These conditions define the recogniser’s job.
Now list movements that resemble the command but should not trigger it. A player may adjust a strap, reach for a drink or reposition their chair. These are not edge cases in a living room. They are part of the surrounding movement vocabulary. Recognition quality depends on rejecting ordinary activity as well as accepting carefully performed examples.
Make the state machine inspectable
A small, explicit state machine is often a useful first prototype. Distinguish unavailable, ready, candidate, committed and recovering states. Each transition needs observable evidence and a defined response. If the interface is not ready, a plausible movement should not secretly become a queued action unless that behaviour is deliberate and communicated.
Thresholds should express the action’s tolerance, not compensate blindly for a noisy signal. Hysteresis uses different entry and exit boundaries to reduce rapid switching near a threshold. A brief dwell can prevent accidental activation, but it also delays the action and may require sustained effort. A cooldown can prevent duplicate commands, yet it can also block a player who deliberately wants to repeat the action. Test these costs rather than treating stabilisation as free.
Ready is visible
The player can tell when a gesture is available and what will happen if it succeeds.
Commit is distinct
Crossing the decision boundary produces an unmistakable state change.
Cancellation is ordinary
Returning to rest or using an alternative control should end an incomplete command safely.
Use features that explain the movement
Useful features might include relative rotation, peak speed, displacement, duration or the order of movement segments. Choose them because they correspond to the action, not because they are convenient to extract. Normalisation can accommodate differences in scale, but it can also erase meaningful distinctions. If a gentle tilt and a forceful swing have different game consequences, making every trajectory the same size destroys information the design needs.
Recognition is a policy
“Uncertain” is a valid result.
A recogniser that always chooses a label can feel less trustworthy than one that pauses, asks for clearer evidence or offers a different control.
Know when learning helps
A learned model can distinguish patterns that simple rules struggle to separate, but its output is not a direct reading of intention. Training examples embody decisions about who participated, how gestures were labelled and which situations were omitted. A confidence score is not automatically a calibrated probability of player intent.
Keep evaluation participants separate from training participants where possible. Otherwise, the model may learn personal movement characteristics rather than the general action. Inspect failures by condition, including posture, hand, movement range and sensor placement. A single aggregate accuracy figure can conceal a recogniser that works well for some players and poorly for others.
Count errors in the rhythm of play
False positives and missed actions have different costs. An unintended menu highlight may be easy to reverse; an unintended item purchase or destructive action is not. Describe consequences before selecting an operating threshold. For high-consequence actions, confirmation or an alternative input may be more appropriate than trying to recognise a more elaborate gesture.
Measure accidental activations during periods when the player is not attempting a command. Also measure repeated attempts, cancellations and time to recovery. A recogniser can score well on isolated clips while becoming frustrating during continuous play. Keep a timestamped record of state transitions so an observer can connect a failure to what the game believed, not only what the player visibly did.
Respect the whole interaction vocabulary
Nintendo Wii gestures, Valve SteamVR input mappings and contemporary hand-tracking interfaces demonstrate different relationships between movement and game action. They do not establish one universal gesture language. A familiar-looking movement can still acquire a new meaning in a different game. Teach the local contract, and avoid assuming that theatrical resemblance makes a gesture self-explanatory.
Paul Fitts’s work on aimed movement is a useful reminder that target size and distance affect performance, but it is not a complete model for recognising expressive gestures. Use established interaction research to sharpen questions, not to bypass testing. The practical goal is a vocabulary players can enter, repeat and leave without continual negotiation with the classifier.
Should recognition adapt during play?
Possibly, but adaptation should be bounded and understandable. Avoid changing action thresholds invisibly after every attempt. Preserve a stable fallback and let the player reset or recalibrate.
For runtime context, consult the SteamVR Input documentation. Then continue from interpretation to the response a player actually perceives.
Next: make recognition readable →