Haptic Feedback: Feeling the Dance You Cannot See

Written by Gabrielle Di Raddo

A foot, planted firmly on the ground, slides backward. The knees bend and bend and push the feet against gravity and the stage floor. One foot releases and the toes curl and draw up the leg to the knee. The other, balancing on the toes. Arms, held tight in a circle, help the rest of the body rotate and rotate until gravity finds its way back. Both heels find the floor and the knees bend in release.

Dance relies on the audience’s sense of sight to experience a performance. However, dancing itself is a very tactile experience. A dancer’s precise movements depend on a heightened internal awareness of the body – how joints are stacked and what muscles are activated. They must also maintain an awareness of external forces – how one limb moves in relation to the other, how their skin feels sliding against the floor, and where they sense other dancers moving with them in space.

The experience of the audience is vastly different from that of the dancer. Is there a way to bridge this divide and communicate a multi-sensory experience to the audience, and allow those who cannot see to feel? Tapping into this sense of touch might help audiences experience dance in a new way.

The word haptic derives from the Greek word “touch”. Simply put, haptics is the coordination of touch, perception, and action. Together, our skin, musculoskeletal system, and nervous system communicate information to our brains about the outside world we experience. These systems work together to make up a human’s haptic sense. Although one could classify any communication received via a tactile sensation as haptic feedback, it is most often used today regarding haptic technology that simulates the human touch experience.

Haptic feedback is becoming more common in everyday technology as a communication tool between the device and the end-user. Apple is certainly a fan, with Macbook force touch trackpads providing haptic communication for various degrees of force, or the Apple Watch sending communication via haptic “bumps” to user's wrists to tell them the time. Video game controllers are famous for their Dual Shock haptic feedback, rumbling the entire device during an intense boss battle. In newer, more sophisticated controllers, haptic feedback creates the sensation of a joystick beneath the player’s thumb by using precise vibrations.

How Haptic Feedback Works

Haptic feedback relies on two systems to engage the haptic sense – one on the human side and one on the machine side. Each requires an input, a translation, and an output.

On the human side, the skin cell of a finger make contact with a haptic feedback device and receives an input. This alerts the finger’s tactile sensor nerves which sends whatever information it just received from the cells to the brain. The brain translates this information and neuron synapses fire a call to action to the body. This output might be a sudden jerk if the haptic feedback is programmed to feel sharp, or its denseness may make for curious fingers to explore the larger shape.

A similar process happens on the machine side. Haptic feedback sensors receive an input, either data that has been programmed or some external force, and send it to a computer. The computer then translates this data as commands to actuators, the machine’s device that produces motion. Based on the data it receives, it outputs specific vibration frequencies, that are too high pitched for the human ear, to exert different levels of force, motion, and sensation.

Figure 1: A diagram showing how the process of haptic feedback. Source: Sreelakshmi and Subash.

Through this, haptic feedback can simulate touch, tricking the brain into firing neuron synapses that make us think we are experiencing a surface different than how it appears to exist in the real world. Human haptic sense is incredibly sensitive, which can detect and perceive thousands of surface textures, as well as an object's directional motion, velocity, and temperature. This sense is so delicate that one spike from a single touch receptor neuron is enough to elicit conscious recognition in the brain. This allows for precise applications of haptic feedback to simulate texture and 3D objects.

Figure 2: A video demonstration of how haptic technology works. Source: YouTube.

The tactile experience is profound and felt deeply throughout the human ecosystem. Touch also often elicits an emotional response, allowing a carefully crafted gesture to have a visceral impact. Haptic feedback is uniquely suited to serve as a reinterpretation of dance. Both occur in a three-dimensional space, depending on variations in force and motion to communicate with an audience. Using haptic feedback to translate dance enables choreographers to communicate nonverbally to multiple senses and gives those who cannot experience the visual performance an alternative interpretation that can still be moving and engaging.

Haptic feedback technology spans a breadth of uses, its dance applications are primarily focused around surface haptics and vibrotactile haptics. Unlike surface haptics, like the Macbook’s trackpad, vibrotactile haptics do not require a precise input from the user through a surface to elicit feedback. Instead, data, like recorded choreography or live dance, can be fed to the haptic technology’s computer to determine the haptic feedback without requiring direct contact with the end-user. This allows haptic feedback technology to remain stationary in a space or sit fixed on the body like a wearable. 

Choreography and Hapitc Feedback

The Choreo-haptic project uses surface haptics, having audience members rest their palms on haptic pads in order to experience dance. A Microsoft Kinect captures movement input in real-time, tracking dance’s movement throughout a space. This information is sent to an Arduino processor in the haptic pad, translating data to commands and activating a 5x6 motor board. Vibrotactile haptic feedback communicates different movement qualities and patterns as they occur.

The Microsoft Kinect captures an abstraction of the movement being performed. To communicate the quality of the movement, an algorithm breaks down the dynamism of a body and translates it into an intensity value which is signaled back to the haptic device. The algorithm, developed to translate movement to haptic feedback, takes note of motion patterns, spatial movements, the body’s depth in space, where the center of mass is placed, and the position of particular limbs. It has the ability to capture which part of the body is moving, the tempo and timing, and a movement’s effect on the body posture.

The more a movement affects body posture, the more intense the haptic feedback output will be. Each of these qualities is assigned different intensities, motions, and rhythms to replicate the most noticeable aspects of a live-performed dance.

Since the setup is relatively inexpensive and easily programmable, it is best used as a choreographic medium to understand what movements are best experienced through haptic feedback for visually impaired audiences. While this device gives dancers a greater range of motion and freedom, it is at the expense of a more detailed haptic experience for the audience. Abstractions in choreography allow only the large, grand movements to be communicated to the visually impaired audience member, while the small movements, which are often the most vital in dance expression, are lost.

Haptic Cushion

The Dance Haptics project at Deakin Motion.Lab sought to create a multi-sensory experience for dance audiences, especially those that are visually impaired, by translating dance movements to haptic feedback. To make this accessible in theatre spaces, the team designed the haptic device as a seat back cushion to provide haptic feedback to the audience member’s back in a way that can be easily integrated into existing performance spaces and would not encumber the audience.

The haptic cushion is composed of 64 vibrotactile actuators arranged on an 8x8 grid. An Arduino board inside the cushion serves as the device’s computer to translate input data to the actuators. A dance performance choreographed for a proportionally sized 8x8 grid provides input for the haptic feedback.

To translate dance to haptics, a dance is recorded via a motion capture system overhead. While the top-down information communicates floor patterns across the audience’s back, vertical cameras capture movement qualities on the vertical plane. Using the audio track from the performance, along with the overhead floor plan, the haptic cushion can translate more detailed physical action to physical sensation.

Figure 3: A flowchart showing how a haptic cushion is used to translate movement from a dancer to a sensory experience for visually impaired audiences. Source: McCormick et al.

The haptic cushion can be used with a prerecorded “haptic track” while a live performance occurs, or it can be fed movement information during a live performance.  With a bigger surface area than most haptic feedback devices used in the field, the haptic cushion provides the most visceral sensory experience and is the most accurate for feeling a dance performed throughout the body of the sensory receiver. The haptic cushion’s engaging interpretation of dance translation embedded in a theater-friendly device could radically change an audience member’s experience.

This technology is limited by the necessity to choreograph for the device, both in terms of movement qualities that translate to haptic feedback best and special constraints of the 8x8 grid. The vast majority of works today would need to be adapted in some way to be experienced through the haptic cushion. This application is best suited for dance on film projects that could be widely distributed as a standalone piece or be played in tandem with a haptic track in a theater space.

Conclusion

No technology will be able to identically replicate human touch or exactly recreate the tactile experience of dancing. There is potential, however, to communicate dance in new ways that are more accessible and immersive for audiences. The demand for multi-sensory experiences is ever-increasing, given the rise of immersive art and theatre in our post-pandemic world. Audiences have a desire to experience more deeply, which often means with more than one sense. Multiple interpretations of a nonverbal art form give choreographers the opportunity to communicate artistic expression in an entirely new medium that is engaging to a wider population. For visually impaired audiences, including haptic feedback in dance performance is a critical next step in accessibility for the art form.

+ Resources

Biswas, Shantonu, and Yon Visell. “Emerging Material Technologies for Haptics.” Advance Materials Technology, March 13, 2019. https://doi.org/10.1002/admt.201900042.

Blenkinsopp, Robert. “What Is Haptics? Definition of Haptic Feedback and Technology | Ultraleap.” Ultraleap, June 10, 2019. https://www.ultraleap.com/company/news/blog/what-is-haptics/.

De Boeck, Joan, Lode Vanacken, and Karin Coninx. “Target Acquisition with Force Feedback: The Effect of Different Forces on the User’s Performance.” In 4th International Conference, 176. Dresden, Germany, 2009. https://link.springer.com/chapter/10.1007/978-3-642-04076-4_1.

Gomes, Norma, Sandrine Cochet, and Alice Guyon. “Dance and Embodiment: Therapeutic Benefits on Body-Mind Health.” Journal of Interdisciplinary Methodologies and Issues in Science 9 (August 2021). https://doi.org/10.18713/JIMIS-02072021-9-4.

Graham-Rowe, Duncan. “The Cutting Edge of Haptics.” MIT Technology Review, August 25, 2006. https://www.technologyreview.com/2006/08/25/100586/the-cutting-edge-of-haptics/.

Hall, Brian. “Taptic, Haptics, and the Body Fantastic: The Real Apple Watch Revolution.” Macworld (blog), October 2, 2014. https://www.macworld.com/article/224270/taptic-haptics-and-the-body-fantastic-the-real-apple-watch-revolution.html.

Hossny, M., Saeid Nahavandi, M. Fielding, J. Mullins, S. Mohamed, Doug Creighton, J. McCormick, K. Vincs, J. Vincent, and S. Hutchison. “Haptically-Enabled Dance Visualisation Framework for Deafblind-Folded Audience and Artists.” In 2015 IEEE International Conference on Systems, Man, and Cybernetics, 446–50, 2015. https://doi.org/10.1109/SMC.2015.89.

Hutson, Matthew. “Here’s What the Future of Haptic Technology Looks (Or Rather, Feels) Like.” Smithsonian Magazine, December 18, 2018. https://www.smithsonianmag.com/innovation/heres-what-future-haptic-technology-looks-or-rather-feels-180971097/.

Lycouris, Sophia, Wendy Timmons, Mark Wright, John Ravenscroft, and Stathis Vafeias. “(Choreo)-Haptic Experiments: Kinaesthetic Empathy and Non-Sighted Dance Audiences.” University of Chichester, 2012. https://www.pure.ed.ac.uk/ws/portalfiles/portal/8127316/documentation_of_haptic_project.pdf.

McCormick, John, Mohammed Hossny, Michael Fielding, James Mullins, Jordan Beth Vincent, Mostafa Hossny, Kim Vincs, et al. “Feels Like Dancing: Motion Capture-Driven Haptic Interface as an Added Sensory Experience for Dance Viewing.” Leonardo 53, no. 1 (February 2020): 45–49. https://doi.org/10.1162/leon_a_01689.

Novasentis, Ralph Russo and Christophe Ramstein. “Wearables and the ‘Neo-Sensory’ Age.” Wired, August 8, 2014. https://www.wired.com/insights/2014/08/wearables-neo-sensory-age/.

Profis, Sharon, and Matt Elliott. “How the MacBook’s Force Touch Trackpad Works.” CNET, October 27, 2016. https://www.cnet.com/tech/computing/what-is-force-touch-macbook/.

Shaghaghi, Navid, Yu Yang Chee, Jesse Mayer, and Alissa LaFerriere. “SwingBeats: An IoT Haptic Feedback Ankle Bracelet (HFAB) for Dance Education.” In Intelligent Technologies for Interactive Entertainment, edited by Navid Shaghaghi, Fabrizio Lamberti, Brian Beams, Reza Shariatmadari, and Ahmed Amer, 82–101. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering. Cham: Springer International Publishing, 2021. https://doi.org/10.1007/978-3-030-76426-5_6.

Shibasaki, Mina, Youichi Kamiyama, and Kouta Minamizawa. “Designing a Haptic Feedback System for Hearing-Impaired to Experience Tap Dance.” In Proceedings of the 29th Annual Symposium on User Interface Software and Technology, 97–99. Tokyo Japan: ACM, 2016. https://doi.org/10.1145/2984751.2985716.

Sreelakshmi, M, and T.D. Subash. “Haptic Technology: A Comprehensive Review on Its Applications and Future Prospects | Elsevier Enhanced Reader.” Materials Today 4, no. 2, Part B (2017): 4182–87. https://doi.org/10.1016/j.matpr.2017.02.120.

Steamworks. “Steam Controller (Steamworks Documentation).” Accessed March 4, 2022. https://partner.steamgames.com/doc/features/steam_controller/device/steam_controller.

The Royal Institution. Haptic Technology Demonstration - with Danielle George, 2017. https://www.youtube.com/watch?v=uPnHzQ7qJ2Y.

Yon Visell. Haptics: Science and Engineering for the Sense of Touch. University of California Santa Barbara, 2018. https://www.youtube.com/watch?v=bdgmGrESe14.