Affective Pseudo-Haptics in Hand-Tracked XR

Hand-tracking enables controller-free XR interaction, but it does not provide the tactile feedback of physical controllers. Rather than treating this only as a missing-sensation problem, this work explores whether pseudo-haptic cues on an embodied virtual hand can act as tactile or affective substitutes that shape how interactions feel.

We developed a mixed-reality prototype that keeps the contacted surface visually neutral and renders cues on the hand through motion modulation, texture, color glow, and movement-coupled sound. In a within-subjects study with 12 participants, participants experienced 12 conditions across four effects and three modalities: audio, visual, and combined audio-visual feedback.

The results suggest that pseudo-haptic cues can influence affect without recreating sustained tactile or thermal sensations. Rough-hot cues lowered valence and increased arousal, while smooth-cold cues produced calmer and more pleasant states.

Our poster was presented at the CHI 2026 conference. The open access version can be found here.

More details can be found in the ACM Digital Library.