Author: alice.zanutti
15. Building the First Decay Prototype
Sculpting the Core
In Blender. I modeled a central, organic symbol, a 3d flower.
I moved the flower model into KeyShot, for its Texture, Light, and Animation where I focused on giving it a simple machine like vibe, animated the flower slowly turning, anexported the frames only, which gave me full control later in video editing.
Once rendered, the frames were stitched together in Premiere Pro to produce a looping animation, lightweight, editable.
Resolume + FaceOSC + Max/MSP
Since the flower has to respond to people, I used FaceOSC to track facial movements, interpreting gestures and expressions. A Max 9 patch to process this incoming data and map it to decay parameters. UDPsend to push the control signals directly to Resolume Arena, where the flower video lives. Resolume became the decay environment. As a viewer approaches or moves, the flower visuals begin to glitch, shake and distort.
What’s Next
Now that the pipeline works, from face input to visual output the next steps would be working on:
-More advanced facial tracking
-Sound integration: the next phase will let the flower sound different based on mood
-Visual regeneration zones: can stillness heal the decay?
Final Thoughts
Prototypes are weird. You spend a lot of time worrying about how close they should be to the final piece, how polished, how functional, how perfect. But in the end, what really mattered to me was the process.Just figuring out how to get even a little closer to the feeling, the behavior I imagined was enough.
14. Prototyping an Exhibition with Intelligent Decay
Define the Core Experience
Before any code or visuals, I need to be clear about the emotional arc of the installation. The users would be free to feel peace, tension, or any emotion, which would make the space restorative (biophilic), provocative (biophobic).
Build the Decay Engine
This is the technical heart of the project. I’m designing a system that remembers and changes based on how it’s treated through inputs like the time spent in front of the work; motion, sound, or proximity and speed, frequency, or intensity of interaction.
The outputs would resolve into blur, color shift, fragmentation, erosion of digital textures and the sound could be detuning, fading, glitching.
Tools I’m considering for a final product are TouchDesigner, Max/MSP with OSC (for generative sound) and Sensors (motion, mic, or camera input)
Micro-Prototypes
Rather than build the full system at once, I’ll start with small sketches like a digital painting that becomes more glitchy if you interact with it.
13. Intelligent Decay in Visual and Sonic Environments
We live in a world of infinite loops. Screensavers cycle endlessly. Playlists repeat. But humans aren’t like that, we age, we erode, we carry the marks of memory and emotion. What if our interactive systems did the same?
This is what Intelligent Decay is. A design principle where sound and visuals change over time, not randomly, but in response to emotional intensity, frequency of use, or even neglect.
Visual and Sonic Systems That Learn to Forget
Traditional installations loop. They restart fresh each morning. But in an intelligent-decay system a color palette may fade with overexposure, an image blurs or pixelate based on emotional tension in the space. Textures could disintegrate, mimicking time or even emotional burnout.
The system could “remember” every touch, every sound, every moment of silence. And it reflects that back through both sight and sound.
Biophilia Meets Digital Erosion
The concept finds deep resonance in biophilia the concept we introduced in the previous post. Natural systems evolve, they grow, decay, and change with the seasons. There are no perfect loops in the forest. So in my installation, visual and auditory decay could mirror this. A space that becomes a kind of living organism, responding to presence, absence, intensity, and stillness.
When Art Remembers You
Most digital works track physical time, but few track emotional or cognitive load. What if this interaction wore down the system? If you rush, the visuals become frantic. If you overuse it, parts may “burn out,” leaving visual scars or silent zones and so on.
This makes the piece personal, temporal, and irreproducible so that no one else experiences the exact same artwork.
Emotion-Responsive Decay
While some games and artworks use entropy or erosion as aesthetic motifs, few explore intelligent decay as a behavioral mirror especially in public exhibit contexts.This opens a space for a multi-sensory system that ages with the viewer, a feedback loop between emotion, memory, and environment. And probably most of all a poetic take on how we affect the world, visibly and invisibly
By designing with intelligent decay we reconnect technology with something deeply human: the ability to change, to carry meaning over time, and to never be quite the same again.
12. Our Deep Connection to Nature: The Biophilic Power of Sound
Sound moves us, heals us, and shapes how we think and feel. But what if part of its power lies in something even more primal, our connection to the natural world?
Here lies an intersection of sound, neuroscience, and biophilia our innate tendency to seek harmony with nature.
Music as a Cognitive Enhancer
Let’s start with what we already know: ssound activates multiple areas of the brain. It can help us focus; enhance creativity and Problem-Solving; but also in theprocess of rehabilitation and memory.
Biophilic Soundscapes
Here’s where biophilia enters the conversation.
Coined by biologist E.O. Wilson, biophilia refers to the innate human attraction to nature. We are wired to find comfort in the sound of a forest, the rhythm of waves, or the chirping of birds. Interestingly, music can mimic these elements, creating biophilic soundscapes that soothe the nervous system and enhance mental health. Think of ambient tracks that use rain, wind, or rustling leaves or classical compositions that follow natural cadences or imitate birdsong.
These sounds create a kind of sonic shelter, reconnecting us to environments where we feel most at ease, even when we’re stuck indoors or stressed out.
The Other Side
On the flip side, there’s biophobia, a stress response to environments (or sounds) that feel unnatural, threatening, or chaotic. Harsh alarms, dissonant chords, or industrial noise can trigger anxiety and discomfort. And while some music uses this intentionally it’s important to recognize the psychological impact.
Why This Matters
Understanding music through the lens of biophilic and biophobic systems adds a powerful dimension to how we use it.
Our brains evolved alongside birdsong, rustling trees, and the pulse of the earth. So next time you press play, ask yourself: does this music bring me closer to nature—or push me away from it?
11. A 1-on-1 Experiment in Physical Storytelling
In class, we did a rapid-fire round of 1-on-1 prototype testing. Each of us had about three minutes to present our prototype to a classmate, who would try to figure out how it works and what it represents, without much explanation.
My prototype was a small-scale physical room, just 10x10cm, constructed from paper. It had two vertical walls and a floor, with a “0” drawn on each of these surfaces to mark the starting point. From there, the interaction unfolded in layers.
The next components were two small paper strips, 1x3cm each. Folded in half and placed on top of the walls like little blades, these represented projectors. Then came two larger 10x10cm sheets, each marked with a “1” in the top left corner and colored pink. One of these was placed on the floor, the other on one of the walls. The pink floor piece had footprints drawn on it as signal of movement.
Finally, two more sheets were added this time green, labeled with a “2” and layered over the pink ones. On the green floor piece, small ripples of water replaced the original marks. Meanwhile the wall had raindrops in the same axis of the water ripples.
As classmates explored the prototype, the general response was encouraging. Many found it intuitive, the layering, the footprints, the shift from step to splash. However, the most consistent point of confusion was the tiny projector pieces. People weren’t sure what they were for, and in the fast-paced 3-minute window, that uncertainty took up valuable time.
This exercise reminded me how even small unclear elements can disrupt an otherwise understandable experience. But it also showed how much can be communicated through tactile storytelling. Overall, it was nice to see my idea come to life in others’ hands.
WebExpo Conference: Designing Beyond the Happy Path
In this talk the speaker made one thing clear: we’re all designing for users who never face loading screens, never make mistakes, and always have perfect internet. Meanwhile, actual humans struggle with our products in ways we did’t take in account
Something to keep in mind: if my design only works when everything goes right, it doesn’t really work at all.
The Loading Screen
We imagine users navigatin through our interfaces, but truth is that sometimes the pages take time to load. The speaker showed how even a half-second delay needs to be considered.
It is better to add proper communication about what actually went wrong and what the user can realistically do about it.
When Content Misbehaves
Designers design tidy cards that look good with short text like “Pizza.” Then real users come along with titles like “Martha’s Gluten-Free Artisanal Sourdough Breakfast Flatbread” and in that moment the layout is not working out anymore.
The solution? Design components that can stretch, shrink. Good design is about handling whatever users throw at it.
The Myth of the Mouse User
Ofter designer design for people with fast internet and desktop computers. But many users navigate using only keyboards, rely on screen readers or use touchscreens where hover effects don’t work
Real design considers all these ways people interact, not just the one we’re used to. It’s about making sure everyone can actually use what we build
Adapt or Die
Modern designs can now adjust to how people want to use them. Dark mode respects night owls eyes. Websites can remember content for when your signal drops.
Good design doesn’t force one experience, it adapts to each user’s needs.
The Hard Pill to Swallow
In short, the speaker’s closing argument is that if you’re not designing these states, you’re not doing your job.
After this talk I can’t look at a design without seeing all the invisible failure points. Because in the end, what separates good design from great design it’s how it gets thorugh the chaos of real world use.
WebExpo Conference: Creating meaningful gamified experiences
At a recent conference, I attended a fascinating talk on the use of game design elements in non-game contexts and how it can boost user engagement and satisfaction. The speaker opened with a compelling case: gamification isn’t just a trend, it’s a tool that taps into deep-seated cognitive biases to motivate behavior and enhance user experience.
The speaker opened with the example of one of the early brands in this field. Nike used gamification strategies to encourage people to run more. Their apps allowed users to track progress, set personal goals, and even compete with friends. This created a sense of achievement and motivation.
Supporting this, a study from the University of Colorado found that using gamification in educational or workplace settings can lead to a 48% increase in engagement and satisfaction. That’s a significant jump, and one that shows the potential of integrating these strategies into various systems.
A particularly interesting example mentioned in the talk was Bluepoint, a kind of virtual currency system where users receive points from their colleagues for being helpful or kind. This builds positive behavior but also a sense of community and recognition in the workplace.
And here’s where I have to be honest: some of it really made me raise an eyebrow. For example, the idea that getting a digital badge on an app can create genuine emotional attachment? That sounds ridiculous, a bit like rewarding yourself with a gold star for remembering to do the laundry. Even more unsettling is the concept of colleagues rating each other for kindness. I mean, really in a Black Mirror episode?
But surprisingly, psychology seems to back it up. The speaker explained that rewards and progress indicators are powerful tools. When users see that they’re getting closer to completing a task, it becomes psychologically harder to stop, this phenomenon is called the Zeigarnik effect.
Another key concept was the Endowment Effect, the idea that people place more value on things simply because they feel ownership over them. Apparently, even small things like digital badges can trigger this sense of emotional investment. Still sounds like a stretch to me, but I guess the science says otherwise.
Then there’s the IKEA effect, which suggests that when people build or customize something themselves, they tend to value it more. In gamification, this translates to letting users set daily goals or personalize their experience. When users feel like they’ve helped create something, they’re more likely to stick with it.
So while some parts of gamification still sound like nonsense to me and maybe even a little dystopian I can’t ignore the results. When used thoughtfully, these strategies clearly tap into human psychology in ways that drive engagement and motivation. Whether I like it or not its worth paying attention to it.
NIME: Design and Exhibition of Loudspeaker-Based, Environmentally-Reactive, Soundscape Augmentation Artifacts in Outdoor Natural Environments
Legatus is a soundscape augmentation artifact designed for outdoor environments, providing a way of engaging with and enhancing isonic environments. It is a lightweight, self-powered device that can listen to and interact with its environment using sensors, and then generate audio responses based on the captured sounds. It leverages audio playback, synthesis, and real-time effects, creating a interaction between the natural world and its sonic representation. The device has been tested in various outdoor installations, with the goal of encouraging environmental listening and fostering a deeper connection between visitors and their surroundings.
The hardware design of Legatus includes several key components: a microcontroller, environmental sensors, an audio codec, a digital signal processor, and an amplifier, all housed within an acrylic body. The device is powered by rechargeable batteries and is designed for easy transport and setup, with modular feet and simple controls.
Legatus operates through a series of interactive installation scenarios, where it listens to its surroundings, records sounds, and either replays or synthesizes new sounds in response to environmental conditions. These scenarios include spatially relocated soundscapes, temporally relocated soundscapes, pitch-based synthesis, and feedback chamber modes.
WHAT ARE IT’S STRENGHTS?
Legatus is a promising tool for sound installations and environmental art projects. Its key strength lies in its ability to engage the natural environment through audio responses. The integration of sensors that detect ambient light, temperature, humidity, and sound levels makes it highly reactive to its surroundings. This reactivity allows for real-time interaction between the artifact and its environment.
Another strength is its portability. Legatus is lightweight and compact, making it easy to transport and set up in various outdoor locations.
Moreover, Legatus’s durability in outdoor conditions is not to take for granted. The device is resistant to water splashes and dust, making sure that it can work proprrly in various weather conditions without significant risk of damage.
Additionally, Legatus promotes non-cochlear visual feedback through the use of RGB LEDs, directing attention away from the artifact’s vocalizations and towards the ambient soundscape. This feedback loop helps maintain the visitor’s engagement with the surrounding environment.
WHAT ARE IT’S WEAKNESSES?
While Legatus is without any doubt a promising product it does have several weaknesses that limit its scope, particularly in larger or more complex installations.
One of the primary issues is the volume and audio clarity. The current speaker design, while functional, is limited in terms of sound output. In larger spaces or environments with high background noise, the audio may not be loud or clear enough to be heard effectively.
Another significant weakness is the lack of networking capabilities. In installations with multiple Legatus units, there is no communication between the devices, meaning they cannot work together to create a more cohesive or immersive soundscape.
Additionally, the environmental mappings that drive Legatus’s behavior can sometimes feel too abstract for visitors to understand. The relationship between sensor readings and the artifact’s responses is not always intuitive, making it difficult for audiences to connect the device’s actions with the environmental conditions they represent, and making necessary a prior explanation or guidance.
The interaction between the artifact and the audience is also quite passive, they have no direct control over it’s actions. This lack of interactivity may reduce engagement and limit the potential for more personalized experiences.
IMPROVEMENTS
Audio Output and Durability: Upgrading the speaker to a more robust, weatherproof model could prevent damage and improve sound quality.
Networking Capabilities: Integrating Wi-Fi or Bluetooth networking would enable Legatus units to communicate with each other. This would allow for synchronized sound playback and the creation of multi-channel audio experiences in installations with multiple devices.
Environmental Mappings: To make the mappings between environmental conditions and artifact behavior more intuitive, clearer visual cues, such as a simple mobile app interface, could be introduced. Visitors could see and hear more directly how environmental changes influence the device’s behavior, improving understanding.
Interactivity: Adding user-controlled elements, such as touch sensors, proximity sensors, or a mobile app for controlling sound parameters, could make the experience more engaging. By allowing visitors to influence the artifact’s behavior there would be space to create a more personalized and immersive installation.
CONCLUSION
Legatus offers an innovative approach to blending audio, sensors, and real-time interaction, encouraging visitors to engage with and reflect on the sonic environment around them. While there are notable weaknesses, such as limited audio output, lack of networking, and passive interaction, the device’s strengths, including its portability, adaptability, and environmental responsiveness, make it a powerful tool for artistic expression. With some improvements, particularly in terms of interactivity, audio capabilities, and networking, Legatus could become an even more versatile and engaging platform for sound art installations.
10. Reflections and Next Steps
LEARNINGS SO FAR
Through this research into audiovisual interactivity and its impact on immersive environments, I have found foundational insights that are shaping the trajectory of my topic. These learnings span theoretical frameworks, practical applications, and the challenges of designing user-centered, multisensory experiences.
When properly integrated, audiovisual elements can enhance emotional responses, direct attention, and foster deeper immersion. Examples such as teamLab Borderless and the MoMA SoundScape installation illustrate how carefully synchronized soundscapes and visuals create environments where users feel emotionally and physically connected.
Drawing on Gibson’s and Norman’s theories, I’ve realized the importance of perceived affordances in designing interactive environments. Users need clear cueS, whether visual, auditory, or tactile, to understand how to interact with installations. Misaligned or ambiguous affordances risk creating frustration and disengagement, underscoring the importance of intuitive design.
From Your Brain on Music to the principles of ecological perception, I’ve learned how sensory stimuli influence emotional states and cognitive processes. For instance, the cerebellum’s role in movement and rhythm underscores how bodily interaction with immersive spaces can evoke joy or excitement, while the amygdala’s role in emotional memory highlights the potential of audiovisual design to create lasting impressions.
Different cultures interpret visual and auditory stimuli in distinct ways. Similarly, individuals’ personal experiences shape their responses to interactive environments. Designing for inclusivity and flexibility is essential to ensure that installations resonate universally while allowing space for personal meaning.
NEXT STEPS IN MY RESEARCH
The insights gained so far set the stage for deeper exploration into how audiovisual interactivity shapes user experiences in immersive environments. My next steps will focus on refining these learnings and addressing gaps to create actionable frameworks for designers.
I aim to design and test small-scale prototypes that integrate sound and visual elements with interactive affordances. These prototypes will allow me to study how users respond to specific audiovisual cues and identify design principles for creating intuitive and engaging experiences.
To validate my findings, I will conduct user studies in controlled settings. By observing how participants interact with audiovisual environments, I can gather data on emotional and cognitive responses, usability challenges, and cultural variations. This will provide valuable insights for refining design strategies.
Incorporating AI and real-time feedback systems is a priority. AI can analyze user behavior and dynamically adapt audiovisual elements to enhance engagement and personalization. This approach will also explore how real-time interactivity can deepen user immersion and foster a sense of agency.
Given the interdisciplinary nature of this field, I plan to collaborate with experts in interaction design, psychology, and cultural studies. These collaborations will provide diverse perspectives and strengthen the theoretical and practical dimensions of my research.
Ultimately, my goal is to create a comprehensive framework for designing audiovisual interactivity in immersive environments. This framework will provide guidelines for balancing sound and visuals, optimizing affordances, and tailoring experiences to diverse audiences.
CLOSING THOUGHTS
he journey so far has revealed the immense potential of audiovisual design to transform interactive environments into spaces of emotional and cognitive resonance. It has also highlighted the challenges, from managing sensory overload to designing for cultural inclusivity. As I move forward, I am excited to deepen this exploration, contribute meaningful insights to the field, and pave the way for innovative, user-centered experiences.