My journey into sound and technology started with my experiments in movement-based sound design. One of my first projects used ultrasonic sensors and Arduino technology to transform body movement into music. I was fascinated by the idea of turning motion into sound, mapping gestures into an interactive sonic experience. This led me to explore other ways of integrating physical action with sound manipulation, such as using MIDI controllers and custom-built sensors.
I see sound as more than just music—it’s a form of expression, communication, and interaction. My interest in sound design is rooted in its ability to create immersive experiences, whether through spatial sound, interactivity, or emotional storytelling. I love experimenting with unconventional ways of generating and manipulating sound, pushing beyond traditional composition to explore new territories.
Right now, I’m particularly interested in how sound connects to the body. How can movement or internal processes be used as an instrument? How do physical states influence the way we experience sound? These are the questions that drive my current explorations.
Idea Draft for a Future Project
At first, I was focused on transforming movement into sound. My early idea was to explore sensors that could read touch, direction, and motion, allowing me to control different sound layers by moving my body. I imagined a 3D sound composition where gestures could manipulate textures, rhythms, and effects in real-time. Maybe even integrating voice elements, allowing me to shape effects with both movement and singing.
Over time, my focus shifted. Instead of external movement, I started thinking about internal body processes—breath, heartbeat, muscle tension. What if sound could react to what happens inside the body rather than just external gestures? This led to the idea of biofeedback-driven sound, where physiological data becomes a source of real-time sonic transformation.
The concept is still in development, but the main idea remains the same: exploring the relationship between the body and sound in a way that is immersive, interactive, and emotionally driven. Whether through movement or internal signals, I want to create a performance where sound is a direct extension of the body’s state, turning invisible experiences into something that can be heard and felt.
Moving Forward
This project is still evolving. It might become a performance, an installation, or something entirely different. Right now, I’m in the phase of exploring what’s possible. Sound and the body are deeply connected, and I want to keep pushing that connection in new and unexpected ways.
Shift of intention of the project due to time plan:
By narrowing down the topic to ensure the feasibility of this project the focus or main purpose of the project will be the artistic approach. The tool will still combine the use of direct image to audio translation and the translation via sonification into a more abstract form. The main use cases will be generating unique audio samples for creative applications, such as sound design for interactive installations, brand audio identities, or matching image soundscapes and the possibility to be a versatile instrument for experimental media artists and display tool for image information.
By further research on different possibilities of sonification of image data and development of the sonification language itself the translation and display purpose is going to get more clear within the following weeks.
Testing of Google Gemini API for AI Object and Image Recognition:
The first testing of the Google Gemini Api started well. There are different models for dedicated object recognition and image recognition itself which can be combined to analyze pictures in terms of objects and partly scenery. These models (SSD, EfficientNET,…) create similar results but not always the same. It might be an option to make it selectable for the user (so that in a failure case a different model can be tried and may give better results). The scenery recognition itself tends to be a problem. It may be a possibility to try out different apis.
The data we get from this AI model is a tag for the recognized objects or image content and a percentage of the probability.
The next steps for the direct translation of it into realistic sound representations will be to test the possibility of using the api of freesound.org to search directly and automated for the recognized object tags and load matching audio files. These search calls also need to filter by copyright type of the sounds and a choosing rule / algorithm needs to be created.
object recognition: efficient float 16 model (Photo by Jason Oh on unsplash)object recognition: image splice test – recognition fail (Photo by Jason Oh on unsplash)object recognition: accurate but low score (Photo: https://lernen.zoner.de/)object recognition (photo: zdf.de)
Research on sonification of images / video material and different approaches:
The world of image sonification is rich with diverse techniques, each offering unique ways to transform visual data into auditory experiences. The world of image sonification is rich with diverse techniques, each offering unique ways to map visual data into auditory experiences. One of the most straightforward methods is raster scanning, introduced by Yeo and Berger. This technique maps the brightness values of grayscale image pixels directly to audio samples, creating a one-to-one correspondence between visual and auditory data. By scanning an image line by line, from top to bottom, the system generates a sound that reflects the texture and patterns of the image. For example, a smooth gradient might produce a steady tone, while a highly textured image could result in a more complex, evolving soundscape. The process is fully reversible, allowing for both image sonification and sound visualization, making it a versatile tool for artists and researchers alike. This method is particularly effective for sonifying image textures and exploring the auditory representation of visual filters, such as “patchwork” or “grain” effects.(Yeo and Berger, 2006)
Principle raster scanning (Yeo and Berger, 2006)
In contrast, Audible Panorama (Huang et al. 2019) automates sound mapping for 360° panorama images used in virtual reality (VR). It detects objects using computer vision, estimates their depth, and assigns spatialized audio from a database. For example, a car might trigger engine sounds, while a person generates footsteps, creating an immersive auditory experience that enhances VR realism. A user study confirmed that spatial audio significantly improves the sense of presence. It contains a interesting concept regarding to choosing a random audio file from a sound library to avoid producing similar or same results. Also it mentions the aspect of postprocessing the audios which also would be a relevant aspect for the image extender project.
principle audible panorama (Huang et al. 2019)
Another approach, HindSight (Schoop, Smith, and Hartmann 2018), focuses on real-time object detection and sonification in 360° video. Using a head-mounted camera and neural networks, it detects objects like cars and pedestrians, then sonifies their position and danger level through bone conduction headphones. Beeps increase in tempo and pan to indicate proximity and direction, providing real-time safety alerts for cyclists.
Finally, Sonic Panoramas (Kabisch, Kuester, and Penny 2005) takes an interactive approach, allowing users to navigate landscape images while generating sound based on their position. Edge detection extracts features like mountains or forests, mapping them to dynamic soundscapes. For instance, a mountain ridge might produce a resonant tone, while a forest creates layered, chaotic sounds, blending visual and auditory art. It also mentions different approaches for sonification itself. For example the idea of using micro (timbre, pitch and melody) and macro level (rhythm and form) mapping.
principle sonic panoramas (Kabisch, Kuester, and Penny 2005)
Each of these methods—raster scanning, Audible Panorama, HindSight, and Sonic Panoramas—demonstrates the versatility of sonification as a tool for transforming visual data into sound and lead keeping these different approaches in mind for developing my own sonification language or mapping method. It also leads to further research by checking some useful references they used in their work for a deeper understanding of sonification and extending the possibilities.
References
Huang, Haikun, Michael Solah, Dingzeyu Li, and Lap-Fai Yu. 2019. “Audible Panorama: Automatic Spatial Audio Generation for Panorama Imagery.” In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 1–11. Glasgow, Scotland: ACM. https://doi.org/10.1145/3290605.3300851.
Kabisch, Eric, Falko Kuester, and Simon Penny. 2005. “Sonic Panoramas: Experiments with Interactive Landscape Image Sonification.” In Proceedings of the 2005 International Conference on Artificial Reality and Telexistence (ICAT), 156–163. Christchurch, New Zealand: HIT Lab NZ.
Schoop, Eldon, James Smith, and Bjoern Hartmann. 2018. “HindSight: Enhancing Spatial Awareness by Sonifying Detected Objects in Real-Time 360-Degree Video.” In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, 1–12. Montreal, QC, Canada: ACM. https://doi.org/10.1145/3173574.3173717.
Yeo, Woon Seung, and Jonathan Berger. 2006. “Application of Raster Scanning Method to Image Sonification, Sound Visualization, Sound Analysis and Synthesis.” In Proceedings of the 9th International Conference on Digital Audio Effects (DAFx-06), 311–316. Montreal, Canada: DAFx.
This project reimagines the surfboard as a data-driven tool, integrating advanced sensors to measure wave interaction, surfer dynamics, and board performance. By merging this scientific data with creative visualization, it opens new dimensions for surfboard shaping, surfer training, and interactive art. Using innovative tools like the x-IMU3 sensor, Pure Data, and TouchDesigner, this project seeks to translate surfing’s raw energy into visuals and soundscapes. The outcome will not only enhance understanding for surfers and shapers but also inspire broader cultural and artistic engagement with the sport. By transforming raw surfing data into emotionally resonant soundscapes and visuals, this project creates a new artistic medium for experiencing the sport, pushing the boundaries of what’s possible in surfing, technology, and art.
The project is divided into four main phases: research and preparation, prototype development and field testing, data processing and visualization, and finalization and presentation. Each phase is designed to ensure the project’s success, from the initial selection of sensors to the final interactive installation. The project also lays the foundation for a future master thesis exploring real-time applications of this technology, with potential commercial applications such as an app or software for surfers and shapers.
02. Introduction and Background
Surfing is a deeply technical sport, where the relationship between the surfer, the board, and the wave is essential. However, much of this interaction remains intuitive, with limited data-driven insights available to inform board design or surfer performance. Current technologies like TRACE and Surflogic GPS focus on external metrics such as speed and location, leaving critical factors—such as board flex, wave impact, and surfer positioning—unexplored.
Building on the 2015 TorFlex project by Cabianca Surfboards, this research uses the latest sensor technology to achieve levels of accuracy and data detail that were not done before. Collaborating with professional shapers and surfers, this project will integrate sensors into surfboards, transforming them into tools for analysis, visualization, and artistic expression. While existing technologies focus on performance metrics, this project goes beyond by exploring the artistic potential of surfing. By translating motion, speed, and vibrations into sound and visuals, we aim to create a new way to experience and appreciate the sport.
The project also draws inspiration from other fields, such as computational fluid dynamics (CFD) and sports technology, to ensure a robust scientific foundation. By combining these elements, the project aims to create a surfboard that not only performs well but also provides valuable data for surfers and shapers, while also serving as a medium for artistic expression.
03. Research Question
How can embedded sensors on a surfboard capture environmental and performance data to create auditory and visual representations of surfing?
Sub-Questions:
Can sensor data be used to create emotionally resonant sound and visuals that enhance the surfing experience?
How can a sensor-embedded surfboard improve performance without compromising the traditional surfing experience?
What are the most effective methods for visualizing and sonifying complex surfing data in real-time?
How can the data collected from sensor-embedded surfboards inform future surfboard design and surfer training?
Research on Existing Projects and Technological Advancements
To develop a sensor-integrated surfboard that captures and translates surfing data into artistic visualizations and soundscapes, it is crucial to understand past and ongoing research in this field. Several projects have laid the groundwork for data-driven surfboard innovation, yet technological advancements in machine learning and sensor accuracy now enable deeper exploration and improved results.
One of the most notable initiatives is the SurfSens Project, a collaboration between Pukas Surf and Tecnalia, which equipped surfboards with pressure sensors, flex sensors, GPS, and accelerometers. The data was recorded via an embedded computer and later analyzed through the Robot Operating System (ROS). While this project provided valuable insights into surfer technique and board performance, it was conducted several years ago, meaning that modern sensors and data analysis tools can now achieve even greater precision and applicability. (https://www.ros.org/news/2011/02/robots-using-ros-surfsens-high-performance-surfboard-with-integrated-sensors.html) – video: https://vimeo.com/20197603
Another compelling approach was explored in the Data-Generated Surfboards Project, which utilized onboard sensors to analyze movement and pressure data. This data informed the creation of CNC-shaped surfboards customized for individual surfers. Though promising, this project remained relatively small in scope and did not fully explore the integration of artistic visualization or real-time data processing. (https://hackaday.io/project/166977-data-generated-surfboards)
The Smartfin Project took a different approach, focusing on environmental data collection. By embedding sensors into a surfboard fin, it recorded ocean parameters like temperature and wave characteristics, transmitting data over cellular networks. While this project contributed to oceanographic research, it did not directly address the dynamics of board performance or surfer technique. (https://blog.scistarter.org/2021/09/with-smartfin-surfers-collect-ocean-data-while-they-hang-ten/)
Insights from Industry and Academic Collaborations
Beyond analyzing existing projects, I have actively engaged with industry professionals to gain deeper insights. I connected with Jonny from Cabianca Surfboards, who previously conducted extensive research into surfboard flex through the TorFlex Project. This system allowed shapers to measure flex, torsion, and vibration in boards to refine their performance. However, Jonny mentioned that the project was halted due to high costs and the complexity of testing different board designs. With today’s more accessible and advanced sensor technology, alongside machine learning applications, I believe these challenges can be overcome, allowing for a more streamlined and scalable approach.
Additionally, I am in discussions with Pukas Surf regarding their past research and potential collaboration. I have applied for an internship with them, which would allow me to gain firsthand knowledge of their findings and integrate their expertise into my project. I am also considering working with Cabianca Surfboards to build and test my prototype surfboard.
Leveraging Modern Technology for a New Approach
While these previous projects laid a strong foundation, I aim to push the boundaries further by integrating:
Machine Learning for Data Analysis: Unlike past projects, I will apply AI models to recognize movement patterns, board flex characteristics, and wave interactions, providing deeper insights into surfer performance and board design.
Real-Time Data Visualization and Sonification: Using Pure Data and TouchDesigner, I will transform surfboard motion and environmental data into an immersive, artistic experience, making the project not just a scientific tool but also an expressive medium.
Advanced Sensor Integration: With support from my university, I will have access to cutting-edge sensors and funding, allowing me to integrate high-precision IMUs (like the x-IMU3), pressure sensors, and hydrophonesinto the surfboard for detailed data collection.
Collaboration with Experts: I will work closely with professors specializing in sensor integration and data visualization, ensuring a rigorous research approach.
Conclusion
By combining elements from previous research projects with modern advancements in machine learning, real-time data processing, and artistic representation, my project will not only provide insights into surfboard performance but also transform raw surfing data into a unique audiovisual experience. Given the rapid evolution of sensor technology and data science, this project has the potential to set a new standard in surfboard innovation, offering both scientific and artistic contributions to the field.
04. Objectives
Primary Objective: To develop a surfboard prototype equipped with sensors that collects performance and environmental data, which is then translated into immersive visual and auditory experiences.
Specific Goals:
Data Collection:
Capture motion, wave interaction, and board dynamics using sensors like the x-IMU3.
Explore additional measurements, including flex, pressure distribution, and surfer positioning.
Visualization and Sound Design:
Use tools like TouchDesigner and Pure Data to transform collected data into compelling visuals and soundscapes.
Ensure the artistic output reflects surf culture, board shaping processes, and wave dynamics.
Collaboration and Practical Application:
Work closely with professional shapers to design boards informed by collected data.
Test the prototype in real surfing conditions with professional surfers.
Documentation:
Document the entire process to establish a foundation for a master thesis and future research, potentially including a commercial application like an app or software for surfers and shapers.
Artistic Expression:
Create an immersive artpiece that allows audiences to experience the rhythm and beauty of surfing through sound and visuals.
05. Methodology
Phase 1: Research and Preparation (Summer Semester 2024)
Sensor Exploration:
Initial trials with x-IMU3 for motion tracking and gyroscopic data.
Investigate additional sensors for pressure mapping and flex analysis.
Collaborations:
Partner with Cabianca Surfboards and other professional shapers for guidance on sensor placement.
Consult with professors specializing in sensor integration and data visualization.
Skateboard Simulations:
Attach sensors to skateboards for controlled land-based testing.
Phase 2: Prototype Development and Field Testing (July–August 2024)
Sensor Integration:
Embed sensors into a surfboard during the shaping process.
Ensure waterproofing and durability for real-world testing.
Maybe also putting sensors on the surfer (shoulders) to capture spesific movement patterns
Field Testing:
Conduct trials in various surf conditions (e.g., small waves, large swells) to gather comprehensive data.
Interview surfers to evaluate the board’s performance and usability.
Phase 3: Data Processing and Visualization (Winter Semester 2024/25)
Data Analysis:
Process collected data to identify patterns in motion, wave dynamics, and surfer-board interaction.
Use tools like Grafana and Kafka for in-depth analysis
Using AI and machine learing tools to define clear patterns and give concret numbers which are important for the surfboard shaper and surfers
Sound Design and Visualization:
Map motion and wave data to sound parameters (e.g., speed → pitch, pressure → amplitude).
Create real-time visual representations inspired by ocean waves and board dynamics using TouchDesigner.
Phase 4: Finalization and Presentation (Early 2026)
Refine the prototype and integrate feedback from field tests.
Create an artpiece and a film showcasing the project.
Prepare final documentation and a pitch for academic and industry presentations.
06. Risk Analysis
Technical Risks:
Sensor failure due to water exposure or impact during surfing.
Data loss or corruption during transmission from the surfboard to the processing unit.
Mitigation Strategies:
Use waterproof and shock-resistant sensors.
Implement redundant data storage and backup systems.
Conduct rigorous testing in controlled environments before field deployment.
07. Stakeholder Engagement Plan
Surfers: Conduct interviews and surveys to understand their needs and preferences. Involve them in field testing to gather feedback on the prototype’s performance.
Shapers: Collaborate with professional shapers like Cabianca Surfboards to ensure the sensors do not compromise the board’s design or performance.
Artists and Technologists: Conduct independent creative research to explore innovative ways to visualize and sonify the data. Immerse myself deeply in the surf culture, studying wave dynamics, board designs, and the aesthetics of surfing to create visually and thematically fitting designs for the project. Experiment with artistic techniques and technologies to develop unique visual and auditory representations that resonate with the essence of surfing.
08. Ethical Considerations
Data Privacy: Ensure that any personal data collected from surfers (e.g., performance metrics) is anonymized and stored securely.
Environmental Impact: Use eco-friendly materials for the surfboard and sensors to minimize environmental harm.
Consider the long-term sustainability of the technology, especially if it were to be commercialized.
09. Broader Impact Statement
Cultural Impact: The project could inspire new forms of artistic expression by merging sports data with creative visualization.
Educational Impact: The technology could be used in educational settings to teach students about data science, oceanography, and sports technology.
Economic Impact: If commercialized, the technology could create new opportunities in the surfing industry, such as data-driven surfboard design or interactive art installations. Helps shapers have a better understanding of their products and supports surfers and professional athlets to improve their interaction with the board.
10. Potential Future Applications
Commercialization: Develop a consumer-friendly app that allows surfers to track their performance and visualize their data in real-time.
Expansion to Other Sports: Adapt the technology for use in other board sports, such as snowboarding or skateboarding.
Scientific Research: Use the data collected to contribute to oceanographic research, such as studying wave patterns or the impact of climate change on surfing conditions.
11. Detailed Evaluation Metrics
Technical Metrics: Accuracy of sensor data (e.g., motion tracking, pressure mapping). Reliability of the system in various surf conditions.
User Experience Metrics: Feedback from surfers on the board’s performance and usability. Audience engagement during the interactive installation.
Artistic Metrics: Emotional impact of the soundscapes and visuals on the audience. Creativity and innovation in the artistic representation of surfing data.
12. Collaboration with Academic and Industry Partners
Academic Collaborations: Work with professors specializing in sensor integration, data visualization, and computational fluid dynamics.
Industry Partnerships: Partner with companies like Cabianca Surfboards for surfboard design and sensor integration. Collaborate with software companies specializing in real-time data processing and visualization tools.
Professional Organizations: Engage with surfing associations to promote the project and gather feedback from professional surfers.
13. Detailed Timeline with Milestones
Phase
Tasks
Timeline
Research & Prototyping
Sensor selection, skateboard testing, collaboration with shapers, and preparation for field testing.
Summer 2024
Field Testing
Integration of sensors into surfboards, data collection, and surfer feedback.
July–August 2024
Data Processing
Analysis of data, sound and visual mapping, and adjustments based on findings.
Winter 2024/25
Final Presentation
Prototype refinement, surf film creation, and academic/public presentations.
Early 2026
14. Budget Justification
Item
Cost Estimate (EUR)
Sensors (x-IMU3, pressure, etc.)
2,500
Surfboard materials
1,500
Software and hardware
1,000
Travel costs
2,000
Miscellaneous
1,000
Total
8,000
15. Conclusion
This project is a pioneering step in merging surfing, technology, and art. By providing real-time data on the interplay between surfers, boards, and waves, it offers transformative possibilities for surfboard design, athletic performance, and cultural expression. The strong technical foundation, combined with artistic innovation, ensures this project’s relevance to both scientific and creative communities. With its potential applications in sports analytics, art, and education, this project is poised to leave a lasting impact on the surfing world and beyond. The Sonic Wave is a project that pushes the boundaries of what’s possible in surfing, technology, and art. By transforming data into sound and visuals, we create a new way to experience and appreciate the sport. The project has the potential to inspire new ways of thinking about the intersection of sports, technology, and art, and I’m excited to see where it takes us.
16. Bibliography
Grand View Research. Surfing equipment market size, share & trends analysis report by product (apparel & accessories, surfing boards), by distribution channel (online, offline), by region (APAC, North America), and segment forecasts, 2021–2028 (2022, accessed 30 Sep 2022). Link.
Elshahomi, A. et al. Computational fluid dynamics performance evaluation of grooved fins for surfboards. MRS Adv. DOI (2022).
Shormann, D. E. & in het Panhuis, M. Performance evaluation of humpback whale-inspired shortboard surfing fins based on ocean wave fieldwork. PLoS ONE 15(4), e0232035. DOI (2020).
Gately, R. D. et al. Additive manufacturing, modeling and performance evaluation of 3D printed fins for surfboards. MRS Adv. 2, 913–920. DOI (2017).
Gudimetla, P., Kelson, N. & El-Atm, B. Analysis of the hydrodynamic performance of three- and four-fin surfboards using computational fluid dynamics. Aust. J. Mech. Eng. 7(1), 61–67. DOI (2009).
Falk, S. et al. Computational hydrodynamics of a typical 3-fin surfboard setup. J. Fluids Struct. 90, 297–314. DOI (2019).
Falk, S. et al. Numerical investigation of the hydrodynamics of changing fin positions within a 4-fin surfboard configuration. Appl. Sci. 10(3), 816. DOI (2020).
Romanin, A. et al. Surfing equipment and design: A scoping review. Sports Eng. 24, 1–13. DOI(2021).
Roberts, J. R., Jones, R., Mansfield, N. J. & Rothberg, S. J. Evaluation of vibrotactile sensations in the feel of a golf shot. J. Sound Vibr. 285, 303–319. DOI (2004).
Fisher, C. et al. What static and dynamic properties should slalom skis possess? Judgements by advanced and expert skiers. J. Sports Sci. 25(14), 1567–1576. DOI (2007).
Title: “Looking Ahead: Preparing for the Next Steps”
This week was focused on planning and setting the stage for the next phases of the project. While no physical progress was made, the time spent organizing and reaching out to potential collaborators was essential for moving forward.
INTERNSHIP UPDATE:
I reached out to the owner of Cabianca Surfboards to discuss the possibility of an internship this summer. While I haven’t received a definitive answer yet, the initial response was encouraging. If confirmed, this internship would provide invaluable hands-on experience and access to professional surfboard builders, as well as potential connections to the WSL (World Surf League).
RESEARCH AND DEVELOPMENT TIMELINE:
Based on my current progress and future plans, I’ve adjusted the timeline for the project:
Until June 2024: Focus on research, sensor selection, and software exploration.
July-August 2024: Internship at Cabianca Surfboards (if confirmed). During this time, I’ll work on integrating sensors into a surfboard and conducting initial tests.
September 2024 – Spring 2025: Develop the software for data visualization and sound synthesis. Conduct interviews with surfers and experts to refine the project.
Summer 2025: Finalize the prototype and prepare for the final presentation in autumn 2025.
CHALLENGES:
The internship is not yet confirmed, which adds some uncertainty to the timeline.
Balancing research with practical work will be crucial as the project progresses.
The timeline is ambitious, and there’s a lot to accomplish in the next year and a half.
NEXT STEPS:
Follow up with Cabianca Surfboards to confirm the internship.
Continue researching sensors and software tools.
Begin planning for interviews and how they’ll inform the project.
This week was about setting the stage for the next phases. While there’s still much to be determined, the planning process has provided clarity and direction for the road ahead.
Title: “Exploring Data Visualization Platforms: A Week of Discovery”
This week shifted the focus from hardware to software, as I explored various platforms for visualizing the data that the sensors will eventually collect. While no physical progress was made, the exploration of these tools was a necessary step in shaping the project’s creative direction.
PLATFORMS EXPLORED:
Grafana: A robust tool for creating dashboards and visualizing time-series data. Its customization options make it a strong contender for displaying real-time sensor data.
TouchDesigner: A visual programming language ideal for creating interactive visuals. I’m considering using it to design dynamic, wave-inspired visuals that respond to the surfboard’s motion.
Pure Data: An open-source platform for audio synthesis. It could be used to map sensor data to sound, creating an immersive auditory experience.
CHALLENGES:
Each platform has its own learning curve, and I need to determine which one aligns best with the project’s goals.
Syncing the sensor data with these platforms in real-time will require further exploration.
The challenge lies in creating visuals and sounds that are not only technically sound but also emotionally resonant.
NEXT STEPS:
Dive deeper into TouchDesigner and Pure Data to assess their suitability for the project.
Experiment with sample data sets to understand how they can be transformed into sound and visuals.
Continue researching other visualization tools that might offer a better fit.
This week was about laying the groundwork for the creative aspects of the project. While no concrete outcomes were achieved, the exploration of these tools has opened up exciting possibilities.
Title: “Exploring Sensor Options: A Week of Research and Consultation”
This week was dedicated to deepening my understanding of the technical aspects of the project. While no physical tests were conducted, the focus on research and consultation with professors proved to be incredibly valuable. The goal was to identify the most suitable sensors for capturing the surfboard’s motion, and the discussions opened up new possibilities.
SENSOR RESEARCH:
One of the highlights of the week was being introduced to the x-IMU3 by x-io Technologies during a consultation with a professor. This sensor combines an accelerometer, gyroscope, and magnetometer, offering a comprehensive solution for tracking motion. Its advanced capabilities make it a strong candidate for the project, though its cost and integration requirements will need careful consideration.
CHALLENGES:
The x-IMU3 is more sophisticated than the MPU-6050 I initially considered, but its higher price point could impact the project budget.
Integrating the sensor into the surfboard without compromising performance remains a key concern.
Powering the sensor during extended surfing sessions is another hurdle that needs addressing.
NEXT STEPS:
Continue researching the x-IMU3 and compare it with other sensor options.
Reach out to x-io Technologies for technical specifications and potential support.
Begin planning for the internship and how it can facilitate sensor integration.
This week reinforced the importance of thorough research. By exploring the available tools and consulting with experts, I’m better equipped to make informed decisions as the project progresses.
Title: “From Concept to Reality: The First Steps Toward a Sensor-Embedded Surfboard”
This week was all about laying the groundwork for the project. I spent hours researching sensors, microcontrollers, and data visualization tools. The goal is to create a surfboard that not only performs well in the water but also captures the essence of the surfing experience through sound and visuals.
SENSOR SELECTION:
After some research, I’ve decided to start with the MPU-6050 accelerometer and gyroscope. These sensors are affordable, widely available, and perfect for capturing motion data. I’m also considering adding a waterproof microphone (hydrophone) to capture the sounds of the water as the board moves through it.
PROTOTYPING:
I’ve started planning to attach sensors to a skateboard to simulate the motion of a surfboard. This will allow me to test the sensors in a controlled environment before moving to the water. I will be using an Arduino to collect and transmit the data, which will later be processed using Pure Data for sound synthesis and TouchDesigner for visual effects.
COLLABORATIONS:
I reached out to a local surfboard shaper to discuss the possibility of embedding sensors into a board. He was intrigued by the idea and offered to help me with the design. This collaboration will be crucial in ensuring that the board remains functional and aesthetically pleasing.
CHALLENGES:
Waterproofing the sensors and electronics is a major concern.
I need to figure out how to power the system while it’s in the water.
The data needs to be transmitted in real-time, which requires a reliable wireless connection.
NEXT STEPS:
Finalizing the idea of the sensor setup and start collecting data from the skateboard.
Begin experimenting with Pure Data and TouchDesigner to map the data to sound and visuals.
Continue discussions with the surfboard
This week was all about turning the concept into something tangible. It’s exciting to see the project taking shape, but there’s still a long way to go.
Title: “The Sonic Wave: Where Surfing Meets Sound and Technology”
This week marked the beginning of an exciting journey. The idea of merging surfing, sound, and technology has been on my mind for some time, and now it’s time to bring it to life. The concept is simple yet profound: embed sensors into a surfboard to capture the motion, speed, and vibrations of the board as it rides the waves. Then, transform this data into sound and visuals, creating an immersive experience that highlights the rhythm and beauty of surfing.
RESEARCH AND INSPIRATION:
I started by exploring existing projects that combine sports and technology. The Surflogic GPS Tracker and TRACE were particularly inspiring. These tools track surfers’ performance metrics like speed and wave count, but they don’t delve into the artistic side of things. I want to go beyond performance tracking and explore how surfing can be experienced as a multisensory art form.
CHALLENGES AND QUESTIONS:
How do I integrate sensors into a surfboard without affecting its performance?
What kind of sensors will give me the most accurate and meaningful data?
How can I translate raw data into something that resonates emotionally with an audience?
NEXT STEPS:
Research sensor technology (accelerometers, gyroscopes, hydrophones).
Reach out to surfboard shapers and tech experts for advice.
Start sketching out a prototype design.
This project feels like a perfect blend of my passions—surfing, technology, and art. I’m eager to see where this journey takes me.
The Image Extender project bridges accessibility and creativity, offering an innovative way to perceive visual data through sound. With its dual-purpose approach, the tool has the potential to redefine auditory experiences for diverse audiences, pushing the boundaries of technology and human perception.
The project is designed as a dual-purpose tool for immersive perception and creative sound design. By leveraging AI-based image recognition and sonification algorithms, the tool will transform visual data into auditory experiences. This innovative approach is intended for:
1. Visually Impaired Individuals 2. Artists and Designers
The tool will focus on translating colors, textures, shapes, and spatial arrangements into structured soundscapes, ensuring clarity and creativity for diverse users.
Core Functionality: Translating image data into sound using sonification frameworks and AI algorithms.
Target Audiences: Visually impaired users and creative professionals.
Platforms: Initially desktop applications with planned mobile deployment for on-the-go accessibility.
User Experience: A customizable interface to balance complexity, accessibility, and creativity.
Working Hypotheses and Requirements
Hypotheses:
Cross-modal sonification enhances understanding and creativity in visual-to-auditory transformations.
Intuitive soundscapes improve accessibility for visually impaired users compared to traditional methods.
Requirements:
Develop an intuitive sonification framework adaptable to various images.
Integrate customizable settings to prevent sensory overload.
Ensure compatibility across platforms (desktop and mobile).
Subtasks
1. Project Planning & Structure
Define Scope and Goals: Clarify key deliverables and objectives for both visually impaired users and artists/designers.
Research Methods: Identify research approaches (e.g., user interviews, surveys, literature review).
Project Timeline and Milestones: Establish a phased timeline including prototyping, testing, and final implementation.
Identify Dependencies: List libraries, frameworks, and tools needed (Python, Pure Data, Max/MSP, OSC, etc.).
2. Research & Data Collection
Sonification Techniques: Research existing sonification methods and metaphors for cross-modal (sight-to-sound) mapping and research different other approaches that can also blend in the overall sonification strategy.
Psychoacoustics & Perceptual Mapping: Review how different sound frequencies, intensities, and spatialization affect perception.
Existing Tools & References: Study tools like Melobytes, VOSIS, and BeMyEyes to understand features, limitations, and user feedback.
object detection from python yolo library
3. Concept Development & Prototyping
Develop Sonification Mapping Framework: Define rules for mapping visual elements (color, shape, texture) to sound parameters (pitch, timbre, rhythm).
Simple Prototype: Create a basic prototype that integrates:
AI content recognition (Python + image processing libraries).
Sound generation (Pure Data or Max/MSP).
Communication via OSC (e.g., using Wekinator).
Create or collect Sample Soundscapes: Generate initial soundscapes for different types of images (e.g., landscapes, portraits, abstract visuals).
example of puredata with rem library (image to sound in pure data by Artiom Constantinov)
4. User Experience Design
UI/UX Design for Desktop:
Design intuitive interface for uploading images and adjusting sonification parameters.
Mock up controls for adjusting sound complexity, intensity, and spatialization.
Accessibility Features:
Ensure screen reader compatibility.
Develop customizable presets for different levels of user experience (basic vs. advanced).
Mobile Optimization Plan:
Plan for responsive design and functionality for smartphones.
5. Testing & Feedback Collection
Create Testing Scenarios:
Develop a set of diverse images (varying in content, color, and complexity).
Usability Testing with Visually Impaired Users:
Gather feedback on the clarity, intuitiveness, and sensory experience of the sonifications.
Identify areas of overstimulation or confusion.
Feedback from Artists/Designers:
Assess the creative flexibility and utility of the tool for sound design.
Iterate Based on Feedback:
Refine sonification mappings and interface based on user input.
6. Implementation of Standalone Application
Develop Core Application:
Integrate image recognition with sonification engine.
Implement adjustable parameters for sound generation.
Error Handling & Performance Optimization:
Ensure efficient processing for high-resolution images.
Handle edge cases for unexpected or low-quality inputs.
Cross-Platform Compatibility:
Ensure compatibility with Windows, macOS, and plan for future mobile deployment.
7. Finalization & Deployment
Finalize Feature Set:
Balance between accessibility and creative flexibility.
Ensure the sonification language is both consistent and adaptable.
Documentation & Tutorials:
Create user guides for visually impaired users and artists.
Provide tutorials for customizing sonification settings.
Deployment:
Package as a standalone desktop application.
Plan for mobile release (potentially a future phase).
Technological Basis Subtasks:
Programming: Develop core image recognition and processing modules in Python.
Sonification Engine: Create audio synthesis patches in Pure Data/Max/MSP.
Integration: Implement OSC communication between Python and the sound engine.
UI Development: Design and code the user interface for accessibility and usability.
Testing Automation: Create scripts for automating image-sonification tests.
Possible academic foundations for further research and work:
Chatterjee, Oindrila, and Shantanu Chakrabartty. “Using Growth Transform Dynamical Systems for Spatio-Temporal Data Sonification.” arXiv preprint, 2021.
Chion, Michel. Audio-Vision. New York: Columbia University Press, 1994.
Ziemer, Tim. Psychoacoustic Music Sound Field Synthesis. Cham: Springer International Publishing, 2020.
Ziemer, Tim, Nuttawut Nuchprayoon, and Holger Schultheis. “Psychoacoustic Sonification as User Interface for Human-Machine Interaction.” International Journal of Informatics Society, 2020.
Ziemer, Tim, and Holger Schultheis. “Three Orthogonal Dimensions for Psychoacoustic Sonification.” Acta Acustica United with Acustica, 2020.
The project would be a program that uses either AI-content recognition or a specific sonification algorithm by using equivalent of the perception of sight (cross-model metaphors).
examples of cross modal metaphors (Görne, 2017, S.53)
This approach could serve two main audiences:
1. Visually Impaired Individuals: The tool would provide an alternative to traditional audio descriptions, aiming instead to deliver a sonic experience that evokes the ambiance, spatial depth, or mood of an image. Instead of giving direct descriptive feedback, it would use non-verbal soundscapes to create an “impression” of the scene, engaging the listener’s perception intuitively. Therefore, the aspect of a strict sonification language might be a good approach. Maybe even better than just displaying the sounds of the images. Or maybe a mixture of both.
2. Artists and Designers: The tool could generate unique audio samples for creative applications, such as sound design for interactive installations, brand audio identities, or cinematic soundscapes. By enabling the synthesis of sound based on visual data, the tool could become a versatile instrument for experimental media artists.
Purpose
The core purpose would be the mixture of both purposes before, a tool that supports and helps creating in the same suite.
The dual purpose of accessibility and creativity is central to the project’s design philosophy, but balancing these objectives poses a challenge. While the tool should serve as a robust aid for visually impaired users, it also needs to function as a practical and flexible sound design instrument.
The final product can then be used by people who benefit from the added perception they get of images and screens and for artists or designers as a tool.
Primary Goal
A primary goal is to establish a sonification language that is intuitive, consistent, and adaptable to a variety of images and scenes. This “language” would ideally be flexible enough for creative expression yet structured enough to provide clarity for visually impaired users. Using a dynamic, adaptable set of rules tied to image data, the tool would be able to translate colors, textures, shapes, and contrasts into specific sounds.
To make the tool accessible and enjoyable, careful attention needs to be paid to the balance of sound complexity. Testing with visually impaired individuals will be essential for calibrating the audio to avoid overwhelming or confusing sensory experiences. Adjustable parameters could allow users to tailor sound intensity, frequency, and spatialization, giving them control while preserving the underlying sonification framework. It’s important to focus on realistic an achievable goal first.
planning on the methods (structure)
research and data collection
simple prototyping of key concept
testing phases
implementation in an standalone application
ui design and mobile optimization
The prototype will evolve in stages, with usability testing playing a key role in refining functionality. Early feedback from visually impaired testers will be invaluable in shaping how soundscapes are structured and controlled. Incorporating adjustable settings will likely be necessary to allow users to customize their experience and avoid potential overstimulation. However, this customization could complicate the design if the aim is to develop a consistent sonification language. Testing will help to balance these needs
Initial development will target desktop environments, with plans to expand to smartphones. A mobile-friendly interface would allow users to access sonification on the go, making it easier to engage with images and scenes from any device.
In general, it could lead to a different perception of sound in connection with images or visuals.
Needed components
Technological Basis:
Programming Language & IDE: The primary development of the image recognition could be done in Python, which offers strong libraries for image processing, machine learning, and integration with sound engines. Also wekinator could be a good start for the communication via OSC for example.
Sonification Tools: Pure Data or Max/MSP are ideal choices for creating the audio processing and synthesis framework, as they enable fine-tuned audio manipulation. These platforms can map visual data inputs (like color or shape) to sound parameters (such as pitch, timbre, or rhythm).
Testing Resources: A set of test images and videos will be required to refine the tool’s translations across various visual scenarios.
Existing Inspirations and References:
– Melobytes: Software that converts images to music, highlighting the potential for creative auditory representations of visuals.
– VOSIS: A synthesizer that filters visual data based on grayscale values, demonstrating how sound synthesis can be based on visual texture.
– image-sonification.vercel.app: A platform that creates audio loops from RGB values, showing how color data can be translated into sound.
– BeMyEyes: An app that provides auditory descriptions for visually impaired users, emphasizing the importance of accessibility in technology design.
Academic Foundations:
Literature on sonification, psychoacoustics, and synthesis will support the development of the program. These fields will help inform how sound can effectively communicate complex information without overwhelming the listener.
