In diesem Artikel sollen folgende Fragen untersucht werden:
Was ist eine audioreaktive Animation?
Welche Funktionen bieten Blender und Cinema 4D?
Welche Vorteile bietet diese Technik für ein Videomapping gegenüber einer herkömmlichen Animation?
Was ist eine audioreaktive Animation?
Eine audioreaktive Animation beschreibt den Prozess, bei dem ein Audiosignal von der Animationssoftware erkannt und als Information weiterverarbeitet wird, um bestimmte Parameter eines Objekts zu verändern. Ein einfaches Beispiel: Wird eine Kickdrum im 4/4-Takt mit 90 BPM (Beats Per Minute) erkannt, kann der entsprechende Frequenzbereich ausgewählt werden, um beispielsweise die Größe einer Kugel zu steuern. Die Skalierung der Kugel würde sich dann synchron zum Beat der Kickdrum verändern.
Die Herausforderung besteht darin, gezielt Einfluss auf die gewünschten Parameter wie Skalierung, Position oder Farbe zu nehmen und gleichzeitig dynamische Anschlags- und Ausklangszeiten zu definieren, um eine flüssige, ansprechende Animation zu erzeugen, die harmonisch mit der Musik interagiert.
Blender und Cinema 4D bieten hierfür unterschiedliche Ansätze und Werkzeuge, um audioreaktive Animationen umzusetzen.
Audioreaktive Animation in Blender mit Geometry und Simulation Nodes
In Blender ermöglichenGeometry Nodes, 3D-Objekte prozedural zu erstellen, zu verändern und zu animieren, indem visuelle Knoten (Nodes) miteinander verbunden werden.
Ein häufiger Ansatz ist es, einen Keyframe auf einen Value Node zu setzen und diesen mit einer Audiodatei zu verknüpfen. Je nach ausgewähltem Frequenzbereich wird der Wert des Nodes dynamisch beeinflusst. So lassen sich die Eigenschaften eines geometrischen Objekts, wie Skalierung, Position oder Rotation, direkt durch das Audiosignal steuern.
Besonders interessant sind die Simulation Nodes, die es ermöglichen, komplexe Bewegungen und Interaktionen zu generieren. Dies umfasst Loop-basierte Simulationen, die über mehrere Frames hinweg berechnet und präzise gesteuert werden können. Dadurch lassen sich flüssige Bewegungen und dynamische Ease-In- und Ease-Out-Animationen erzeugen, die nahtlos auf Audiosignale reagieren.
Audioreaktive Animation in Cinema 4D mit Soundfields und Mograph
Cinema 4D bietet mit seinen Soundfields und Mograph-Tools eine benutzerfreundliche Möglichkeit, audioreaktive Animationen zu erstellen.
In meinem Beispiel wird ein Cube mehrfach geklont und über einen Schritt- und Formeleffektor, die beide ein Soundfield enthalten, animiert. Die Klone werden dann als Simulationsobjekte definiert, um eine Partikel-Replikation des ursprünglichen Turms zu erzeugen.
Das Soundfield ermöglicht es, Audiosignale direkt auf Parameter wie Position, Skalierung oder Rotation zu übertragen, was besonders für rhythmische oder abstrakte Animationen nützlich ist.
Vorteile für ein Videomapping gegenüber herkömmlichen Animationen
Im Kontext des Videomappings bieten audioreaktive Animationen erhebliche Vorteile, insbesondere bei der Gestaltung von längeren Visualisierungen, da sie zum Großteil ohne Keyframes auskommen und die Musik den Rhythmus vorgibt.
Ein besonderer Vorteil zeigt sich, wenn das Sounddesign selbst entwickelt wird. In diesem Fall entsteht eine einzigartige Symbiose zwischen Bild und Ton. Durch gezieltes Komponieren und das Einsetzen bestimmter Sounds und Frequenzen übernimmt der Sounddesigner gewissermaßen auch die Rolle des Animators.
Warum diese Technik besonders für Videomapping geeignet ist:
Echtzeit-Reaktion: Visuelle Effekte können live auf Musik reagieren.
Effizienz: Keine manuelle Anpassung von Keyframes erforderlich, die Animation passt sich dynamisch an.
Kohärentes Erlebnis: Durch die direkte Verbindung von Ton und Bild entsteht ein intensiveres visuelles Erlebnis.
Flexibilität: Änderungen im Audiosignal erfordern keine zeitaufwendige Anpassung der Animation.
Blender featuring C4D – Ein möglicher hybrider Workflow für die Zukunft?
Eine vielversprechende Methode, die Stärken beider Programme zu kombinieren, wäre der Einsatz von FBX, OBJ oder Alembic-Dateien, um Meshes und Animationen, die in Blender mit Simulation Nodes erstellt wurden, in den Workflow von Cinema 4D zu integrieren, um dort weiterverarbeitet zu werden.
So lassen sich die prozeduralen und simulationsbasierten Möglichkeiten von Blender mit den intuitiven Tools und der Benutzerfreundlichkeit von Cinema 4D verbinden, um ein noch leistungsfähigeres Setup für audioreaktive Animationen zu schaffen.
🤖🧠Disclaimer zur Nutzung von Künstlicher Intelligenz (KI):
Dieser Blogbeitrag wurde unter Zuhilfenahme von Künstlicher Intelligenz (ChatGPT) erstellt. Die KI wurde zur Recherche, zur Korrektur von Texten, zur Inspiration und zur Einholung von Verbesserungsvorschlägen verwendet. Alle Inhalte wurden anschließend eigenständig ausgewertet, überarbeitet und in den hier präsentierten Beitrag integriert.
Embodied Resonance is an experimental audio performance that investigates the interplay between trauma, physiological responses, and immersive sound. By integrating biofeedback sensors with spatial sound, this project translates the body’s real-time emotional states into an evolving sonic landscape. Through this process, Embodied Resonance aims to create an intimate and immersive experience that bridges personal narrative with universal themes of emotional resilience and healing.
Reference Works
Inspiration for this project draws heavily from groundbreaking works in biofeedback art. For instance, Tobias Grewenig’s Emotion’s Defibrillator (2005) inspired me to explore how visual imagery can serve as emotional triggers, sparking physiological responses that drive sound. Grewenig’s project combines sensory input with dynamic visual feedback, using breathing, pulse, and skin sensors to create a powerful interactive experience. His exploration of binaural beats and synchronized visuals provided a foundation for my use of AR imagery and biofeedback systems.
Another profound influence is the project BODY ECHOES, which integrates EMG sensors, breathing monitors, and sound design to capture inner bodily movements and translate them into a spatialized audio experience. This project highlights how subtle physiological states, such as changes in muscle tension or breathing rhythms, can form the basis of a compelling sonic narrative. It has inspired my approach to using EMG and respiratory sensors as key components for translating physical states into sound.
How Does It Work?
The performance involves the use of biofeedback sensors to capture physiological data such as:
Electromyography (EMG) to measure muscle tension
Electrodermal Activity (EDA/GSR) to track stress levels via skin conductivity
Heart Rate (ECG/PPG) to monitor pulse fluctuations and emotional arousal
Respiratory Sensors to analyze breath patterns
This real-time data is processed using software like Max/MSP and Ableton Live, which maps physiological changes to dynamic sound elements. Emotional triggers, such as augmented reality (AR) images chosen by the audience, influence the performer’s physiological responses, which in turn shape the sonic environment.
Core Components of the Project
Emotional Triggers and Biofeedback: The audience plays an active role by selecting AR-displayed imagery, which elicits emotional and physiological responses from the performer.
Sound Mapping and Generation: Physiological changes dynamically alter elements of the soundscape.
Spatial Audio and Immersion: An Ambisonic sound system enhances the experience, surrounding the audience in a three-dimensional sonic space.
Interactive Performance Structure: The performer’s emotional and physical state directly influences the performance, creating a unique, real-time interaction between artist and audience.
Why is This Project Important?
Embodied Resonance is an innovative approach to understanding how trauma manifests in the body and how it can be externalized through sound. This project:
Explores the intersection of biofeedback technology, music, and performance art
Provides a new medium for emotional processing and healing through immersive sound
Pushes the boundaries of interactive performance, inviting the audience into a participatory experience
Challenges conventional notions of musical composition by integrating the human body as an instrument
Why Do I Want to Work on It?
As a sound producer, performer, and music editor, I have always been fascinated by the connections between sound, emotion, and the body. My personal journey with trauma and healing has shaped my artistic explorations, driving me to create a performance that not only expresses these experiences but also fosters a shared space for reflection and empathy. By combining my technical skills with deep personal storytelling, I aim to push the boundaries of sonic expression.
How Will I Realize This Project?
Methods & Techniques
Research: Studying trauma, somatic therapy, and the physiological markers of emotional states.
Technology: Utilizing biofeedback sensors and signal processing tools to create real-time sound mapping.
Performance Development: Experimenting with gesture analysis and embodied interaction.
Audience Engagement: Exploring ways to integrate audience input via AR-triggered imagery.
Necessary Skills & Resources
Sound Design & Synthesis: Proficiency in Ableton Live, Max/MSP, and Envelop for Live.
Sensor Technology: Understanding EMG, ECG, and GSR sensor integration.
Spatial Audio Engineering: Knowledge of Ambisonic techniques for immersive soundscapes.
Programming: Implementing interactive elements using coding languages and software.
Theoretical Research: Studying literature on biofeedback art, music therapy, and embodied cognition.
Challenges and Anticipated Difficulties
Spatial Audio Optimization: Achieving an immersive sound experience that maintains clarity and emotional depth.
Technical Complexity: Ensuring seamless integration of biofeedback data into real-time sound processing requires rigorous calibration and testing.
Emotional Vulnerability: The deeply personal nature of the performance may present emotional challenges, requiring careful preparation.
Audience Interaction: Designing a system that effectively incorporates audience input without disrupting the emotional flow.
The project’s core goal is to create an embodied, immersive experience where the performer’s movements and physiological signals interact with dynamic soundscapes, reflecting states of stress, panic, and resolution. This endeavor seeks to explore the intersection of the body, trauma, and sound as a medium of expression and understanding.
Tasks Fulfilled by the Project:
Expressive Performance: Convey the visceral experience of stress and trauma through movement and sound.
Interactive Soundscapes: Use real-time biofeedback to dynamically alter sound parameters, enhancing the audience’s sensory engagement.
Therapeutic Exploration: Demonstrate the potential of somatic expression and sound for trauma exploration and healing.
Main Goals:
Develop a cohesive interaction between biofeedback, sound design, and movement.
Design an immersive auditory space using ambisonics.
Create an emotionally impactful narrative through choreography and sound dynamics.
Steps for Project Implementation
Identifying Subtasks:
Movement and Choreography Exploration:
Research and refine body movements that mirror states of stress and release.
Develop movement scores aligned with sound triggers.
Biofeedback and Technology Integration:
Select and test wearable sensors for movement and physiological signals (e.g., heart rate monitors, EMG sensors).
Map sensor data to sound parameters using tools like Max/MSP or Pure Data.
Sound Design and Ambisonics:
Create a palette of sound textures representing emotional states.
Test and refine 3D spatial audio setups.
Rehearsal and Iteration:
Practice interaction between movement and sound.
Adjust mappings and refine performance flow.
Determining the Sequence:
Begin with movement research and initial choreography.
Set up and test biofeedback systems.
Integrate sound design with real-time data mappings.
Conduct iterative rehearsals and refine dynamics.
Description of Subtasks
Required Information and Conditions:
Knowledge of movement techniques representing trauma.
Understanding biofeedback sensors and data processing.
Familiarity with ambisonic sound design principles.
Methods:
Employ somatic techniques and physical theater practices for movement.
Use biofeedback-driven sound generation software for real-time interaction.
Apply iterative testing and rehearsal methods for refinement.
Existing Knowledge and Skills:
Dance and performance experience.
Basic knowledge of sensor technologies and sound design tools.
Understanding of trauma’s physical manifestations through literature.
Additional Resources:
Sensors and biofeedback devices.
Ambisonic Toolkit and spatial audio software.
Research materials on trauma and biofeedback in art.
Timeline Overview
Current Semester – “Explore” Phase:
Research movement responses to stress and trauma.
Test sensors and sound mapping tools.
Document all findings to create the exposé and prepare for the oral presentation.
Second Semester – “Experiment” Phase:
Prototype interactions between movement, biofeedback, and sound.
Evaluate the feasibility and emotional resonance of the prototypes.
Incorporate feedback and iterate designs.
Third Semester – “Product” Phase:
Combine prototypes into a cohesive performance.
Optimize the interplay between sound and movement.
Conclude with final documentation and a presentation of the complete performance.
Questions for Exploration
What additional biofeedback sensors and sound techniques can enhance the performance?
How can movement scores effectively translate the emotional states into physical expressions?
What feedback mechanisms will refine the audience’s immersive experience?
A live performance where the body’s movement and physiological responses interact with real-time, 3D soundscapes, creating an auditory and sensory experience that embodies the physical and emotional states associated with trauma, stress, or panic.
Core Elements
Live Movement and Performance:
Physical Expression: Expressive body movements are used to convey states of stress, panic, and tension. Movements could be choreographed or improvised, incorporating controlled gestures, sudden shifts, and spasmodic motions that mirror the body’s natural reactions to trauma.
Sensor Integration: The performer will be equipped with wearable sensors (e.g., accelerometers, heart rate monitors, muscle tension sensors) to capture real-time data that triggers sound changes.
Sound Design and Biofeedback:
Real-time Data to Sound Mapping: The data from the sensors can be mapped to sound parameters such as volume, pitch, and spatial positioning.
Spatial Audio (Ambisonics): the 3D sound environment where the sound moves with the performer, simulating the feeling of being surrounded by or caught in an experience of panic.
Sound Layers and Textures: Layer sounds that range from chaotic, dissonant clusters to more open, calming tones, symbolizing shifts between heightened panic and brief moments of relief.
Interactive Performance Dynamics:
Feedback Loops: The performer’s movements could influence sound parameters, and changes in sound could, in turn, affect how the performer responds (e.g., sudden loud or abrupt sounds causing physical shifts).
Immersive Auditory Space: Spatial audio setup will immerse the audience, making them feel as though they are within the performance’s sonic realm or inside the performer’s body.
Choreography and Movement Techniques:
Imitating Panic and Stress:
Breath Control: Rapid, shallow breathing or uneven breathing patterns to simulate panic.
Body Tension and Release: Show how different areas of the body can tense up and release in response to imagined threats.
Sudden, Erratic Movements: Imitate fight-or-flight reactions through jerky, uncoordinated gestures.
Movement Scores: Create a set of movement phrases that can be triggered by specific sound cues, with each phase representing a different level of intensity or emotional state.
Implementation Steps:
Initial Research and Movement Exploration:
Spend time exploring how the body naturally responds to stress through dance or physical theatre techniques.
Record and analyze your body’s response to various stimuli to understand how to replicate these in a performance context.
Tech Setup and Testing:
Choose sensors capable of tracking movement and vital signs, such as wearable accelerometers and heart rate monitors.
Connect the sensors to real-time audio processing software (e.g., Max/MSP, Pure Data) to create dynamic sound generation based on data input.
Experiment with one biofeedback sensor (e.g., heartbeat or EMG) and connect it to sound manipulation software.
Test simple ambisonic setups to understand spatial audio placement.
Sound Design:
Use ambisonics to experiment with how sounds can be positioned and moved in 3D space.
Create a palette of sound elements that represent different stress levels, such as soft background noise, mechanical sounds, distorted human voices, and deep bass thuds.
Rehearsals and Iteration:
Conduct rehearsals where you practice the movement and sound interaction, making adjustments to the data-to-sound mappings to achieve the desired response.
Test with different inputs to refine the sonic representation of the body’s signals.
Refine the performance flow by timing the intensity of movements and sound shifts to ensure coherence and emotional impact.
Resources
Body and Trauma
The Body Keeps the Score by Bessel van der Kolk
Waking the Tiger: Healing Trauma by Peter Levine
Sound Design and Technology
Sound Design: The Expressive Power of Music, Voice and Sound Effects in Cinema by David Sonnenschein
Immersive Sound: The Art and Science of Binaural and Multi-Channel Audio edited by Agnieszka Roginska and Paul Geluso
Tools and Tutorials
Ambisonic Toolkit (ATK)
Cycling ’74 Max/MSP Tutorials
Artistic and Conceptual References
Janet Cardiff – Known for immersive sound installations, especially her 40-Part Motet.
Meredith Monk – Combines movement and sound to explore human experience.
Christine Sun Kim – Explores sound and silence through the lens of the body and perception.
Academic Research in Sound and Perception
Music, Cognition, and Computerized Sound: An Introduction to Psychoacoustics by Perry Cook
My journey into sound and technology started with my experiments in movement-based sound design. One of my first projects used ultrasonic sensors and Arduino technology to transform body movement into music. I was fascinated by the idea of turning motion into sound, mapping gestures into an interactive sonic experience. This led me to explore other ways of integrating physical action with sound manipulation, such as using MIDI controllers and custom-built sensors.
I see sound as more than just music—it’s a form of expression, communication, and interaction. My interest in sound design is rooted in its ability to create immersive experiences, whether through spatial sound, interactivity, or emotional storytelling. I love experimenting with unconventional ways of generating and manipulating sound, pushing beyond traditional composition to explore new territories.
Right now, I’m particularly interested in how sound connects to the body. How can movement or internal processes be used as an instrument? How do physical states influence the way we experience sound? These are the questions that drive my current explorations.
Idea Draft for a Future Project
At first, I was focused on transforming movement into sound. My early idea was to explore sensors that could read touch, direction, and motion, allowing me to control different sound layers by moving my body. I imagined a 3D sound composition where gestures could manipulate textures, rhythms, and effects in real-time. Maybe even integrating voice elements, allowing me to shape effects with both movement and singing.
Over time, my focus shifted. Instead of external movement, I started thinking about internal body processes—breath, heartbeat, muscle tension. What if sound could react to what happens inside the body rather than just external gestures? This led to the idea of biofeedback-driven sound, where physiological data becomes a source of real-time sonic transformation.
The concept is still in development, but the main idea remains the same: exploring the relationship between the body and sound in a way that is immersive, interactive, and emotionally driven. Whether through movement or internal signals, I want to create a performance where sound is a direct extension of the body’s state, turning invisible experiences into something that can be heard and felt.
Moving Forward
This project is still evolving. It might become a performance, an installation, or something entirely different. Right now, I’m in the phase of exploring what’s possible. Sound and the body are deeply connected, and I want to keep pushing that connection in new and unexpected ways.
Shift of intention of the project due to time plan:
By narrowing down the topic to ensure the feasibility of this project the focus or main purpose of the project will be the artistic approach. The tool will still combine the use of direct image to audio translation and the translation via sonification into a more abstract form. The main use cases will be generating unique audio samples for creative applications, such as sound design for interactive installations, brand audio identities, or matching image soundscapes and the possibility to be a versatile instrument for experimental media artists and display tool for image information.
By further research on different possibilities of sonification of image data and development of the sonification language itself the translation and display purpose is going to get more clear within the following weeks.
Testing of Google Gemini API for AI Object and Image Recognition:
The first testing of the Google Gemini Api started well. There are different models for dedicated object recognition and image recognition itself which can be combined to analyze pictures in terms of objects and partly scenery. These models (SSD, EfficientNET,…) create similar results but not always the same. It might be an option to make it selectable for the user (so that in a failure case a different model can be tried and may give better results). The scenery recognition itself tends to be a problem. It may be a possibility to try out different apis.
The data we get from this AI model is a tag for the recognized objects or image content and a percentage of the probability.
The next steps for the direct translation of it into realistic sound representations will be to test the possibility of using the api of freesound.org to search directly and automated for the recognized object tags and load matching audio files. These search calls also need to filter by copyright type of the sounds and a choosing rule / algorithm needs to be created.
object recognition: efficient float 16 model (Photo by Jason Oh on unsplash)object recognition: image splice test – recognition fail (Photo by Jason Oh on unsplash)object recognition: accurate but low score (Photo: https://lernen.zoner.de/)object recognition (photo: zdf.de)
Research on sonification of images / video material and different approaches:
The world of image sonification is rich with diverse techniques, each offering unique ways to transform visual data into auditory experiences. The world of image sonification is rich with diverse techniques, each offering unique ways to map visual data into auditory experiences. One of the most straightforward methods is raster scanning, introduced by Yeo and Berger. This technique maps the brightness values of grayscale image pixels directly to audio samples, creating a one-to-one correspondence between visual and auditory data. By scanning an image line by line, from top to bottom, the system generates a sound that reflects the texture and patterns of the image. For example, a smooth gradient might produce a steady tone, while a highly textured image could result in a more complex, evolving soundscape. The process is fully reversible, allowing for both image sonification and sound visualization, making it a versatile tool for artists and researchers alike. This method is particularly effective for sonifying image textures and exploring the auditory representation of visual filters, such as “patchwork” or “grain” effects.(Yeo and Berger, 2006)
Principle raster scanning (Yeo and Berger, 2006)
In contrast, Audible Panorama (Huang et al. 2019) automates sound mapping for 360° panorama images used in virtual reality (VR). It detects objects using computer vision, estimates their depth, and assigns spatialized audio from a database. For example, a car might trigger engine sounds, while a person generates footsteps, creating an immersive auditory experience that enhances VR realism. A user study confirmed that spatial audio significantly improves the sense of presence. It contains a interesting concept regarding to choosing a random audio file from a sound library to avoid producing similar or same results. Also it mentions the aspect of postprocessing the audios which also would be a relevant aspect for the image extender project.
principle audible panorama (Huang et al. 2019)
Another approach, HindSight (Schoop, Smith, and Hartmann 2018), focuses on real-time object detection and sonification in 360° video. Using a head-mounted camera and neural networks, it detects objects like cars and pedestrians, then sonifies their position and danger level through bone conduction headphones. Beeps increase in tempo and pan to indicate proximity and direction, providing real-time safety alerts for cyclists.
Finally, Sonic Panoramas (Kabisch, Kuester, and Penny 2005) takes an interactive approach, allowing users to navigate landscape images while generating sound based on their position. Edge detection extracts features like mountains or forests, mapping them to dynamic soundscapes. For instance, a mountain ridge might produce a resonant tone, while a forest creates layered, chaotic sounds, blending visual and auditory art. It also mentions different approaches for sonification itself. For example the idea of using micro (timbre, pitch and melody) and macro level (rhythm and form) mapping.
principle sonic panoramas (Kabisch, Kuester, and Penny 2005)
Each of these methods—raster scanning, Audible Panorama, HindSight, and Sonic Panoramas—demonstrates the versatility of sonification as a tool for transforming visual data into sound and lead keeping these different approaches in mind for developing my own sonification language or mapping method. It also leads to further research by checking some useful references they used in their work for a deeper understanding of sonification and extending the possibilities.
References
Huang, Haikun, Michael Solah, Dingzeyu Li, and Lap-Fai Yu. 2019. “Audible Panorama: Automatic Spatial Audio Generation for Panorama Imagery.” In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 1–11. Glasgow, Scotland: ACM. https://doi.org/10.1145/3290605.3300851.
Kabisch, Eric, Falko Kuester, and Simon Penny. 2005. “Sonic Panoramas: Experiments with Interactive Landscape Image Sonification.” In Proceedings of the 2005 International Conference on Artificial Reality and Telexistence (ICAT), 156–163. Christchurch, New Zealand: HIT Lab NZ.
Schoop, Eldon, James Smith, and Bjoern Hartmann. 2018. “HindSight: Enhancing Spatial Awareness by Sonifying Detected Objects in Real-Time 360-Degree Video.” In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, 1–12. Montreal, QC, Canada: ACM. https://doi.org/10.1145/3173574.3173717.
Yeo, Woon Seung, and Jonathan Berger. 2006. “Application of Raster Scanning Method to Image Sonification, Sound Visualization, Sound Analysis and Synthesis.” In Proceedings of the 9th International Conference on Digital Audio Effects (DAFx-06), 311–316. Montreal, Canada: DAFx.
This project reimagines the surfboard as a data-driven tool, integrating advanced sensors to measure wave interaction, surfer dynamics, and board performance. By merging this scientific data with creative visualization, it opens new dimensions for surfboard shaping, surfer training, and interactive art. Using innovative tools like the x-IMU3 sensor, Pure Data, and TouchDesigner, this project seeks to translate surfing’s raw energy into visuals and soundscapes. The outcome will not only enhance understanding for surfers and shapers but also inspire broader cultural and artistic engagement with the sport. By transforming raw surfing data into emotionally resonant soundscapes and visuals, this project creates a new artistic medium for experiencing the sport, pushing the boundaries of what’s possible in surfing, technology, and art.
The project is divided into four main phases: research and preparation, prototype development and field testing, data processing and visualization, and finalization and presentation. Each phase is designed to ensure the project’s success, from the initial selection of sensors to the final interactive installation. The project also lays the foundation for a future master thesis exploring real-time applications of this technology, with potential commercial applications such as an app or software for surfers and shapers.
02. Introduction and Background
Surfing is a deeply technical sport, where the relationship between the surfer, the board, and the wave is essential. However, much of this interaction remains intuitive, with limited data-driven insights available to inform board design or surfer performance. Current technologies like TRACE and Surflogic GPS focus on external metrics such as speed and location, leaving critical factors—such as board flex, wave impact, and surfer positioning—unexplored.
Building on the 2015 TorFlex project by Cabianca Surfboards, this research uses the latest sensor technology to achieve levels of accuracy and data detail that were not done before. Collaborating with professional shapers and surfers, this project will integrate sensors into surfboards, transforming them into tools for analysis, visualization, and artistic expression. While existing technologies focus on performance metrics, this project goes beyond by exploring the artistic potential of surfing. By translating motion, speed, and vibrations into sound and visuals, we aim to create a new way to experience and appreciate the sport.
The project also draws inspiration from other fields, such as computational fluid dynamics (CFD) and sports technology, to ensure a robust scientific foundation. By combining these elements, the project aims to create a surfboard that not only performs well but also provides valuable data for surfers and shapers, while also serving as a medium for artistic expression.
03. Research Question
How can embedded sensors on a surfboard capture environmental and performance data to create auditory and visual representations of surfing?
Sub-Questions:
Can sensor data be used to create emotionally resonant sound and visuals that enhance the surfing experience?
How can a sensor-embedded surfboard improve performance without compromising the traditional surfing experience?
What are the most effective methods for visualizing and sonifying complex surfing data in real-time?
How can the data collected from sensor-embedded surfboards inform future surfboard design and surfer training?
Research on Existing Projects and Technological Advancements
To develop a sensor-integrated surfboard that captures and translates surfing data into artistic visualizations and soundscapes, it is crucial to understand past and ongoing research in this field. Several projects have laid the groundwork for data-driven surfboard innovation, yet technological advancements in machine learning and sensor accuracy now enable deeper exploration and improved results.
One of the most notable initiatives is the SurfSens Project, a collaboration between Pukas Surf and Tecnalia, which equipped surfboards with pressure sensors, flex sensors, GPS, and accelerometers. The data was recorded via an embedded computer and later analyzed through the Robot Operating System (ROS). While this project provided valuable insights into surfer technique and board performance, it was conducted several years ago, meaning that modern sensors and data analysis tools can now achieve even greater precision and applicability. (https://www.ros.org/news/2011/02/robots-using-ros-surfsens-high-performance-surfboard-with-integrated-sensors.html) – video: https://vimeo.com/20197603
Another compelling approach was explored in the Data-Generated Surfboards Project, which utilized onboard sensors to analyze movement and pressure data. This data informed the creation of CNC-shaped surfboards customized for individual surfers. Though promising, this project remained relatively small in scope and did not fully explore the integration of artistic visualization or real-time data processing. (https://hackaday.io/project/166977-data-generated-surfboards)
The Smartfin Project took a different approach, focusing on environmental data collection. By embedding sensors into a surfboard fin, it recorded ocean parameters like temperature and wave characteristics, transmitting data over cellular networks. While this project contributed to oceanographic research, it did not directly address the dynamics of board performance or surfer technique. (https://blog.scistarter.org/2021/09/with-smartfin-surfers-collect-ocean-data-while-they-hang-ten/)
Insights from Industry and Academic Collaborations
Beyond analyzing existing projects, I have actively engaged with industry professionals to gain deeper insights. I connected with Jonny from Cabianca Surfboards, who previously conducted extensive research into surfboard flex through the TorFlex Project. This system allowed shapers to measure flex, torsion, and vibration in boards to refine their performance. However, Jonny mentioned that the project was halted due to high costs and the complexity of testing different board designs. With today’s more accessible and advanced sensor technology, alongside machine learning applications, I believe these challenges can be overcome, allowing for a more streamlined and scalable approach.
Additionally, I am in discussions with Pukas Surf regarding their past research and potential collaboration. I have applied for an internship with them, which would allow me to gain firsthand knowledge of their findings and integrate their expertise into my project. I am also considering working with Cabianca Surfboards to build and test my prototype surfboard.
Leveraging Modern Technology for a New Approach
While these previous projects laid a strong foundation, I aim to push the boundaries further by integrating:
Machine Learning for Data Analysis: Unlike past projects, I will apply AI models to recognize movement patterns, board flex characteristics, and wave interactions, providing deeper insights into surfer performance and board design.
Real-Time Data Visualization and Sonification: Using Pure Data and TouchDesigner, I will transform surfboard motion and environmental data into an immersive, artistic experience, making the project not just a scientific tool but also an expressive medium.
Advanced Sensor Integration: With support from my university, I will have access to cutting-edge sensors and funding, allowing me to integrate high-precision IMUs (like the x-IMU3), pressure sensors, and hydrophonesinto the surfboard for detailed data collection.
Collaboration with Experts: I will work closely with professors specializing in sensor integration and data visualization, ensuring a rigorous research approach.
Conclusion
By combining elements from previous research projects with modern advancements in machine learning, real-time data processing, and artistic representation, my project will not only provide insights into surfboard performance but also transform raw surfing data into a unique audiovisual experience. Given the rapid evolution of sensor technology and data science, this project has the potential to set a new standard in surfboard innovation, offering both scientific and artistic contributions to the field.
04. Objectives
Primary Objective: To develop a surfboard prototype equipped with sensors that collects performance and environmental data, which is then translated into immersive visual and auditory experiences.
Specific Goals:
Data Collection:
Capture motion, wave interaction, and board dynamics using sensors like the x-IMU3.
Explore additional measurements, including flex, pressure distribution, and surfer positioning.
Visualization and Sound Design:
Use tools like TouchDesigner and Pure Data to transform collected data into compelling visuals and soundscapes.
Ensure the artistic output reflects surf culture, board shaping processes, and wave dynamics.
Collaboration and Practical Application:
Work closely with professional shapers to design boards informed by collected data.
Test the prototype in real surfing conditions with professional surfers.
Documentation:
Document the entire process to establish a foundation for a master thesis and future research, potentially including a commercial application like an app or software for surfers and shapers.
Artistic Expression:
Create an immersive artpiece that allows audiences to experience the rhythm and beauty of surfing through sound and visuals.
05. Methodology
Phase 1: Research and Preparation (Summer Semester 2024)
Sensor Exploration:
Initial trials with x-IMU3 for motion tracking and gyroscopic data.
Investigate additional sensors for pressure mapping and flex analysis.
Collaborations:
Partner with Cabianca Surfboards and other professional shapers for guidance on sensor placement.
Consult with professors specializing in sensor integration and data visualization.
Skateboard Simulations:
Attach sensors to skateboards for controlled land-based testing.
Phase 2: Prototype Development and Field Testing (July–August 2024)
Sensor Integration:
Embed sensors into a surfboard during the shaping process.
Ensure waterproofing and durability for real-world testing.
Maybe also putting sensors on the surfer (shoulders) to capture spesific movement patterns
Field Testing:
Conduct trials in various surf conditions (e.g., small waves, large swells) to gather comprehensive data.
Interview surfers to evaluate the board’s performance and usability.
Phase 3: Data Processing and Visualization (Winter Semester 2024/25)
Data Analysis:
Process collected data to identify patterns in motion, wave dynamics, and surfer-board interaction.
Use tools like Grafana and Kafka for in-depth analysis
Using AI and machine learing tools to define clear patterns and give concret numbers which are important for the surfboard shaper and surfers
Sound Design and Visualization:
Map motion and wave data to sound parameters (e.g., speed → pitch, pressure → amplitude).
Create real-time visual representations inspired by ocean waves and board dynamics using TouchDesigner.
Phase 4: Finalization and Presentation (Early 2026)
Refine the prototype and integrate feedback from field tests.
Create an artpiece and a film showcasing the project.
Prepare final documentation and a pitch for academic and industry presentations.
06. Risk Analysis
Technical Risks:
Sensor failure due to water exposure or impact during surfing.
Data loss or corruption during transmission from the surfboard to the processing unit.
Mitigation Strategies:
Use waterproof and shock-resistant sensors.
Implement redundant data storage and backup systems.
Conduct rigorous testing in controlled environments before field deployment.
07. Stakeholder Engagement Plan
Surfers: Conduct interviews and surveys to understand their needs and preferences. Involve them in field testing to gather feedback on the prototype’s performance.
Shapers: Collaborate with professional shapers like Cabianca Surfboards to ensure the sensors do not compromise the board’s design or performance.
Artists and Technologists: Conduct independent creative research to explore innovative ways to visualize and sonify the data. Immerse myself deeply in the surf culture, studying wave dynamics, board designs, and the aesthetics of surfing to create visually and thematically fitting designs for the project. Experiment with artistic techniques and technologies to develop unique visual and auditory representations that resonate with the essence of surfing.
08. Ethical Considerations
Data Privacy: Ensure that any personal data collected from surfers (e.g., performance metrics) is anonymized and stored securely.
Environmental Impact: Use eco-friendly materials for the surfboard and sensors to minimize environmental harm.
Consider the long-term sustainability of the technology, especially if it were to be commercialized.
09. Broader Impact Statement
Cultural Impact: The project could inspire new forms of artistic expression by merging sports data with creative visualization.
Educational Impact: The technology could be used in educational settings to teach students about data science, oceanography, and sports technology.
Economic Impact: If commercialized, the technology could create new opportunities in the surfing industry, such as data-driven surfboard design or interactive art installations. Helps shapers have a better understanding of their products and supports surfers and professional athlets to improve their interaction with the board.
10. Potential Future Applications
Commercialization: Develop a consumer-friendly app that allows surfers to track their performance and visualize their data in real-time.
Expansion to Other Sports: Adapt the technology for use in other board sports, such as snowboarding or skateboarding.
Scientific Research: Use the data collected to contribute to oceanographic research, such as studying wave patterns or the impact of climate change on surfing conditions.
11. Detailed Evaluation Metrics
Technical Metrics: Accuracy of sensor data (e.g., motion tracking, pressure mapping). Reliability of the system in various surf conditions.
User Experience Metrics: Feedback from surfers on the board’s performance and usability. Audience engagement during the interactive installation.
Artistic Metrics: Emotional impact of the soundscapes and visuals on the audience. Creativity and innovation in the artistic representation of surfing data.
12. Collaboration with Academic and Industry Partners
Academic Collaborations: Work with professors specializing in sensor integration, data visualization, and computational fluid dynamics.
Industry Partnerships: Partner with companies like Cabianca Surfboards for surfboard design and sensor integration. Collaborate with software companies specializing in real-time data processing and visualization tools.
Professional Organizations: Engage with surfing associations to promote the project and gather feedback from professional surfers.
13. Detailed Timeline with Milestones
Phase
Tasks
Timeline
Research & Prototyping
Sensor selection, skateboard testing, collaboration with shapers, and preparation for field testing.
Summer 2024
Field Testing
Integration of sensors into surfboards, data collection, and surfer feedback.
July–August 2024
Data Processing
Analysis of data, sound and visual mapping, and adjustments based on findings.
Winter 2024/25
Final Presentation
Prototype refinement, surf film creation, and academic/public presentations.
Early 2026
14. Budget Justification
Item
Cost Estimate (EUR)
Sensors (x-IMU3, pressure, etc.)
2,500
Surfboard materials
1,500
Software and hardware
1,000
Travel costs
2,000
Miscellaneous
1,000
Total
8,000
15. Conclusion
This project is a pioneering step in merging surfing, technology, and art. By providing real-time data on the interplay between surfers, boards, and waves, it offers transformative possibilities for surfboard design, athletic performance, and cultural expression. The strong technical foundation, combined with artistic innovation, ensures this project’s relevance to both scientific and creative communities. With its potential applications in sports analytics, art, and education, this project is poised to leave a lasting impact on the surfing world and beyond. The Sonic Wave is a project that pushes the boundaries of what’s possible in surfing, technology, and art. By transforming data into sound and visuals, we create a new way to experience and appreciate the sport. The project has the potential to inspire new ways of thinking about the intersection of sports, technology, and art, and I’m excited to see where it takes us.
16. Bibliography
Grand View Research. Surfing equipment market size, share & trends analysis report by product (apparel & accessories, surfing boards), by distribution channel (online, offline), by region (APAC, North America), and segment forecasts, 2021–2028 (2022, accessed 30 Sep 2022). Link.
Elshahomi, A. et al. Computational fluid dynamics performance evaluation of grooved fins for surfboards. MRS Adv. DOI (2022).
Shormann, D. E. & in het Panhuis, M. Performance evaluation of humpback whale-inspired shortboard surfing fins based on ocean wave fieldwork. PLoS ONE 15(4), e0232035. DOI (2020).
Gately, R. D. et al. Additive manufacturing, modeling and performance evaluation of 3D printed fins for surfboards. MRS Adv. 2, 913–920. DOI (2017).
Gudimetla, P., Kelson, N. & El-Atm, B. Analysis of the hydrodynamic performance of three- and four-fin surfboards using computational fluid dynamics. Aust. J. Mech. Eng. 7(1), 61–67. DOI (2009).
Falk, S. et al. Computational hydrodynamics of a typical 3-fin surfboard setup. J. Fluids Struct. 90, 297–314. DOI (2019).
Falk, S. et al. Numerical investigation of the hydrodynamics of changing fin positions within a 4-fin surfboard configuration. Appl. Sci. 10(3), 816. DOI (2020).
Romanin, A. et al. Surfing equipment and design: A scoping review. Sports Eng. 24, 1–13. DOI(2021).
Roberts, J. R., Jones, R., Mansfield, N. J. & Rothberg, S. J. Evaluation of vibrotactile sensations in the feel of a golf shot. J. Sound Vibr. 285, 303–319. DOI (2004).
Fisher, C. et al. What static and dynamic properties should slalom skis possess? Judgements by advanced and expert skiers. J. Sports Sci. 25(14), 1567–1576. DOI (2007).
Title: “Looking Ahead: Preparing for the Next Steps”
This week was focused on planning and setting the stage for the next phases of the project. While no physical progress was made, the time spent organizing and reaching out to potential collaborators was essential for moving forward.
INTERNSHIP UPDATE:
I reached out to the owner of Cabianca Surfboards to discuss the possibility of an internship this summer. While I haven’t received a definitive answer yet, the initial response was encouraging. If confirmed, this internship would provide invaluable hands-on experience and access to professional surfboard builders, as well as potential connections to the WSL (World Surf League).
RESEARCH AND DEVELOPMENT TIMELINE:
Based on my current progress and future plans, I’ve adjusted the timeline for the project:
Until June 2024: Focus on research, sensor selection, and software exploration.
July-August 2024: Internship at Cabianca Surfboards (if confirmed). During this time, I’ll work on integrating sensors into a surfboard and conducting initial tests.
September 2024 – Spring 2025: Develop the software for data visualization and sound synthesis. Conduct interviews with surfers and experts to refine the project.
Summer 2025: Finalize the prototype and prepare for the final presentation in autumn 2025.
CHALLENGES:
The internship is not yet confirmed, which adds some uncertainty to the timeline.
Balancing research with practical work will be crucial as the project progresses.
The timeline is ambitious, and there’s a lot to accomplish in the next year and a half.
NEXT STEPS:
Follow up with Cabianca Surfboards to confirm the internship.
Continue researching sensors and software tools.
Begin planning for interviews and how they’ll inform the project.
This week was about setting the stage for the next phases. While there’s still much to be determined, the planning process has provided clarity and direction for the road ahead.
Title: “Exploring Data Visualization Platforms: A Week of Discovery”
This week shifted the focus from hardware to software, as I explored various platforms for visualizing the data that the sensors will eventually collect. While no physical progress was made, the exploration of these tools was a necessary step in shaping the project’s creative direction.
PLATFORMS EXPLORED:
Grafana: A robust tool for creating dashboards and visualizing time-series data. Its customization options make it a strong contender for displaying real-time sensor data.
TouchDesigner: A visual programming language ideal for creating interactive visuals. I’m considering using it to design dynamic, wave-inspired visuals that respond to the surfboard’s motion.
Pure Data: An open-source platform for audio synthesis. It could be used to map sensor data to sound, creating an immersive auditory experience.
CHALLENGES:
Each platform has its own learning curve, and I need to determine which one aligns best with the project’s goals.
Syncing the sensor data with these platforms in real-time will require further exploration.
The challenge lies in creating visuals and sounds that are not only technically sound but also emotionally resonant.
NEXT STEPS:
Dive deeper into TouchDesigner and Pure Data to assess their suitability for the project.
Experiment with sample data sets to understand how they can be transformed into sound and visuals.
Continue researching other visualization tools that might offer a better fit.
This week was about laying the groundwork for the creative aspects of the project. While no concrete outcomes were achieved, the exploration of these tools has opened up exciting possibilities.
Title: “Exploring Sensor Options: A Week of Research and Consultation”
This week was dedicated to deepening my understanding of the technical aspects of the project. While no physical tests were conducted, the focus on research and consultation with professors proved to be incredibly valuable. The goal was to identify the most suitable sensors for capturing the surfboard’s motion, and the discussions opened up new possibilities.
SENSOR RESEARCH:
One of the highlights of the week was being introduced to the x-IMU3 by x-io Technologies during a consultation with a professor. This sensor combines an accelerometer, gyroscope, and magnetometer, offering a comprehensive solution for tracking motion. Its advanced capabilities make it a strong candidate for the project, though its cost and integration requirements will need careful consideration.
CHALLENGES:
The x-IMU3 is more sophisticated than the MPU-6050 I initially considered, but its higher price point could impact the project budget.
Integrating the sensor into the surfboard without compromising performance remains a key concern.
Powering the sensor during extended surfing sessions is another hurdle that needs addressing.
NEXT STEPS:
Continue researching the x-IMU3 and compare it with other sensor options.
Reach out to x-io Technologies for technical specifications and potential support.
Begin planning for the internship and how it can facilitate sensor integration.
This week reinforced the importance of thorough research. By exploring the available tools and consulting with experts, I’m better equipped to make informed decisions as the project progresses.
