Author: alice.zanutti

APPLICATIONS IN INTERACTIVE DESIGN

Affordances are a cornerstone of user experience (UX) and interaction design, as they guide users on how to interact with objects, interfaces, and environments. The goal is to make these interactions intuitive by aligning design elements with users’ expectations and capabilities.

Visual Affordances: Visual cues, such as arrows, color contrasts, or animations, can suggest how to interact with an interface or environment.
In an interactive video mapping installation, glowing pathways might afford movement, while pulsating shapes suggest touch or interaction.
Tactile Affordances: Haptic feedback can reinforce perceived affordances.
A vibrating surface in an interactive installation can signal that it affords touch or proximity.
Auditory Affordances: Sound can provide additional clues about action possibilities.
A rising tone accompanying a visual projection can suggest that a user should raise their hand or move upward.
Dynamic Affordances: In interactive environments, affordances can change in response to user actions.
A visual pattern that spreads outward when touched signals that further interaction is possible.

APPLICATION IN IMMERSIVE EXPERIENCES

In immersive environments like video mapping, interactive exhibitions, or installations, affordances help bridge the gap between users and the system by making interactions natural and engaging. By designing elements with clear affordances, users are encouraged to explore the space and interact. For instance, a visual element that brightens as users approach affords proximity and engagement.
Dynamic affordances are often used in projection mapping to guide user behavior. For example, footsteps that trigger ripple effects in projections afford walking, exploration, and movement. Affordances can guide users through a narrative. For example, a glowing doorway in a visual environment might afford passage and signify the transition to the next part of the story.

“Volume” is an interactive cube of responsive mirrors that redirect light and sound to spatialize and reflect the excitement of surrounding festival goers.

“Dream Machine” piece was a “multi-sensory scent organ” connected to a network of bicycle pumps that spray scents into the air. Festival-goers can experience a range of human emotions by playing the organ, which activates smells, sound and light to communicate feelings such as contentment, sadness, fear, anger, disgust and shame

CHALLANGES WITH AFFORDANCES

It’s important to keep in mind what the challenges of affordance can be, too many in one environment can overwhelm users, making it unclear where to focus attention or how to interact. Think of a multi-sensory installation, conflicting visual, auditory, and tactile affordances might create confusion. Affordances can be misinterpreted based on cultural contexts, for instance gestures or symbols that afford interaction in one culture might not be understood in another. Beware of misleading users with false affordances, this can result in frustration and disengagement.

Affordances, a concept originally introduced by James J. Gibson and expanded upon by Don Norman, are central to understanding how we interact with objects, environments, and systems. Diving deeper into affordances involves exploring their origins, types, applications, and implications, particularly in the context of interaction design, audiovisual environments, and immersive experiences.

THE CONCEPT OF AFFORDANCES

In Gibson’s “The Ecological Approach to Visual Perception“, affordances refer to the action possibilities that an environment or object offers to an organism that perceives and behave, that is to say, animals. As an example: A chair affords sitting because its surface is flat and elevated and stairs affords climbing because of its shape and spacing.

Gibson emphasized that affordances exist independently of perception—they are inherent in the object or environment. Whether or not an individual recognizes an affordance depends on their ability to perceive and act upon it.

Norman, in “The Design of Everyday Things“, narrowed this concept to perceived affordances, focusing on how design can make these action possibilities obvious and intuitive. A door with a pull handle affords pulling because of the way the handle is shaped and positioned.

TYPES OF AFFORDANCES

Physical Affordances: These are based on the physical properties of an object or environment.
A button affords pressing because it protrudes and can be pushed.
Perceived Affordances: These rely on visual, auditory, or tactile cues that signal how to interact with a object or system.
A glowing touch screen affords interaction by signaling its active state.
Hidden Affordances: These are affordances that exist but are not immediately perceivable.
A touch-sensitive area on a wall projection may afford interaction, but without visual or auditory cues, users might not recognize it.
False Affordances: These occur when an object suggests an affordance that it does not actually provide.
A decorative handle on a door that does not open but looks like it should be pulled.
Cultural Affordances: These are shaped by cultural norms and user expectations.
Red often signals danger or stop in Western cultures, while it may have different meanings in other cultural contexts.

Immersive experiences that combine interaction design, video mapping, and visual design have the power to shape collective experiences that are not only visually stunning but also socially and culturally resonant. When these fields intersect, they can evoke emotions and foster connections in a way that mirrors the roles music has played in human history, such as group bonding, cultural expression, and social healing. Here’s how these visual technologies serve similar functions in social and cultural contexts:

GROUP BONDING AND SURVIVAL

In immersive environments, visuals have the potential to act as a tool for fostering group cohesion, much like how music once played a role in aligning individuals within a group for survival. Through interactive visual experiences, people can engage together, share moments of awe, and feel united in a collective experience. For instance, large-scale projection mapping installations, such as those used during public festivals or social events, often transform shared spaces—like the sides of buildings or urban streets—into visually captivating environments. When these projections are designed to respond to the movement and actions of the group, they encourage communal participation, creating a sense of belonging and collective identity. This shared interaction can foster cooperation and unity, with participants feeling that they are part of something larger than themselves.

Crown Fountain in Millennium Park, Chicago. It incorporates multimedia technology into the fountain,but also “invites” passers-by to join the shallow pool, and water interaction.

CULTURAL IDENTITY AND COMMUNICATION

Immersive visuals also play a key role in reflecting and communicating cultural identities. Video mapping and interactive installations allow for the exploration of cultural symbols, historical narratives, and visual traditions, helping communities preserve and express their heritage in an engaging, modern context. This can take the form of interactive exhibitions that use projections to showcase traditional art or storytelling techniques, blending old and new forms of expression. These installations can serve as a platform for communities to communicate their stories, histories, and values in a way that resonates with both the present and the future. Visual design, through its ability to engage and captivate, creates a powerful space where cultural identity can be shared, celebrated, and explored by both local and global audiences.

MUSIC AND SOCIAL HEALING

Although this point primarily concerns sound, immersive visual environments also play an essential role in social healing and emotional restoration. Visuals have been used in therapeutic settings to calm the mind, reduce anxiety, and provide an outlet for emotional expression. For instance, in interactive art therapy or healing installations, participants might engage with immersive projections that change according to their movements or emotions, creating a dynamic, responsive environment that encourages introspection and emotional release. By interacting with visuals in these spaces, individuals can process their feelings, find calm, and experience a sense of healing. This type of visual interaction can also be used in public installations where the aim is to heal divisions within a community, creating spaces of reconciliation and connection.

Through projections, light, and motion design, immersive experiences can act as tools for emotional regulation and group healing, providing a shared space for personal and collective recovery. For example, projections of nature scenes, calming landscapes, or abstract visuals have been used in hospital rooms, therapy centers, or community spaces to help people relax, feel connected, and improve their mental well-being.

Levitin explores music’s foundational role in shaping and maintaining human social structures, emphasizing its power as a unifying force across cultures and communities.

GROUP BONDING AND SURVIVAL

Music historically served as a tool for fostering cooperation and cohesion within groups.
In early human societies, communal music-making, such as drumming or chanting, likely strengthened social bonds and promoted group survival. This is because coordinated musical activities helped align group members physically and emotionally, enhancing trust and reducing conflict.
Moreover, singing together has been shown to increase oxytocin levels, a hormone linked to bonding and empathy. This chemical connection explains why group singing in choirs, religious rituals, or protests fosters a sense of unity.

CULTURAL IDENTITY AND COMMUNICATION

Music is a universal language but also reflects the unique values, norms, and histories of different cultures, it acts as a repository for cultural memory, preserving traditions, values, and histories through lyrics, melodies, and performance styles. For example, folk songs often encapsulate a community’s shared experiences and struggles. Music transcends spoken language, allowing individuals from different linguistic backgrounds to communicate emotions and ideas. For instance, a mournful melody universally conveys sadness, regardless of cultural context. We also have some examples from history like the role of spirituals during the American Civil Rights Movement or the use of national anthems to foster patriotism. These examples illustrate music’s ability to inspire collective action and express shared identity.

MUSIC AND SOCIAL HEALING

Music is also an effective tool for conflict resolution and emotional healing in social contexts such as some programs like musical collaborations between conflicting cultural groups (e.g., Israeli and Palestinian youth orchestras) have shown how music can build bridges by focusing on shared creative experiences. But also music therapy which can help individuals process trauma, foster empathy, and rebuild relationships after crises or conflicts.

CEREBELLUM: MUSIC AND MOVEMENT

The cerebellum, primarily known for its role in coordinating movement, plays a crucial part in our rhythmic and physical responses to music. This interaction enhances the emotional experience of music through embodied engagement.

Synchronization with Rhythm: The cerebellum helps align our body’s movements with the rhythm of music. This is why we naturally tap our feet, sway, or dance to a beat. It connects auditory inputs with motor outputs, creating a physical embodiment of musical patterns.
Emotional Amplification: The cerebellum interacts with the brain’s emotional centers, amplifying feelings associated with music. Rhythmic patterns and tempo changes influence the intensity of these emotional responses. For instance, a fast, driving rhythm might energize us, while a slow, steady rhythm can calm us.
Evolutionary Perspective: Levitin suggests that the cerebellum’s role in rhythm and movement likely evolved to foster social cohesion. Group activities like singing and dancing strengthened bonds and facilitated communication in early human societies.

HIPPOCAMPUS: MUSIC AND MEMORY

The hippocampus, essential for memory formation and retrieval, plays a central role in how music connects to our personal histories and emotions, making it a powerful tool for nostalgia and therapeutic interventions.

Music as a Memory Trigger: Music activates the hippocampus, often unlocking vivid memories tied to specific songs. This is why hearing a song from childhood can transport us back in time, complete with emotional context.
Emotional Encoding: Music’s emotional impact strengthens memory encoding. Songs experienced during emotionally significant moments (e.g., a wedding or breakup) are more deeply imprinted in the brain, making them easier to recall later.
Neurological Basis for Music Therapy: Levitin points to studies showing how music engages the hippocampus even in individuals with memory disorders like Alzheimer’s. Familiar songs can evoke memories and emotions that seem otherwise inaccessible, providing a therapeutic effect.

AMYGDALA: MUSIC AND EMOTION

The amygdala, the brain’s emotional processing center, helps explain why music has such a profound effect on our mood and why it can be used to evoke specific emotional states.

Emotional Resonance: The amygdala assesses the emotional content of music, influencing feelings like joy, sadness, or fear. For example, dissonant harmonies can activate the amygdala’s fear response, while consonant melodies may evoke feelings of peace or happiness.
Fight-or-Flight Response to Sound: The amygdala’s evolutionary role in processing auditory stimuli is tied to survival. Sudden, loud sounds might trigger a fight-or-flight response, while gentle, soothing sounds signal safety and relaxation. Music capitalizes on this mechanism to create tension and release, often driving emotional highs.
Interplay with Memory: The amygdala works closely with the hippocampus, reinforcing the emotional weight of memories associated with music. This partnership explains why emotionally charged songs often leave a lasting impression.

EMOTIONALLY DIRVEN ROOMS

Emotionally driven rooms in interactive design use music and visuals to create specific emotional atmospheres that immerse participants. By pairing the right kind of music with complementary visuals, these rooms evoke feelings such as calmness, excitement, or awe. For instance, a “Calm Room” could feature slow-tempo, minor-key music alongside visuals of flowing water and cool colors like blues and purples, providing a serene atmosphere. In contrast, an “Excitement Room” might use fast-paced, syncopated rhythms paired with sharp, dynamic visuals in warm tones like reds and oranges, generating energy and intensity. To further enhance the experience, environmental factors such as scent or temperature can be incorporated—cool air and the scent of lavender could accompany the calm setting, while the excitement room could feel warmer with energetic vibrations in the floor. Visitors can also influence the tone of the room through touchscreens or voice commands, adjusting the mood and music to create a more personalized emotional response.

Real-world examples of emotionally driven rooms include the “Museum of Ice Cream”, which creates various themed environments using color and music to evoke joy and nostalgia, and “TeamLab Borderless in Tokyo“, where interactive, immersive rooms shift based on visitors’ movements, evoking a range of emotions.

INTERACTIVE MUSIC-VISUAL FEEDBACK SISTEMS

Interactive music-visual feedback systems allow visitors to control both sound and visuals in real-time, giving them agency in shaping their own emotional experiences. Gesture-based systems, for instance, use motion sensors to detect users’ movements, letting them “conduct” music or trigger visual effects through their actions. As visitors move their hands upward, they might increase the pitch of the music, while corresponding changes in lighting or visual elements could brighten or shift in color. Wearable technology, like gloves or bracelets, could be used to control sound parameters such as pitch or tempo, creating a seamless blend of touch and sound that is directly reflected in the visuals. Collaborative interaction is also a possibility—several participants can engage with the system simultaneously, creating a shared musical and visual experience that blends their input into a unique, communal result.

Real-life examples of interactive systems include the “Dandelion Dome at Expo 2020 Dubai”, where visitors could blow on virtual dandelions to trigger sound and visuals, and Google’s AI Experiments, such as the “Chrome Music Lab”, which allows users to manipulate sound and visuals together in a digital format. Another example is the “Wavefield in Montreal“, where users swing on illuminated swings that trigger musical notes and synchronized lighting changes.

Wavefield in Montreal

EMOTIONALLY RESPONSIVE INSTALLATIONS

Emotionally responsive installations use AI and biometric data to dynamically adjust music and visuals based on participants’ emotional states, creating a deeply personalized experience. These systems can monitor facial expressions, heart rate, or skin responses to gauge how a person is feeling. For example, an AI system might detect signs of tension and shift the music to a calming melody while changing the visuals to soothing, fluid patterns. This creates a responsive environment that adapts to the user’s emotions. Additionally, layered responses can allow users to interact with the system, letting them customize aspects such as tempo or color in reaction to the emotional cues detected by the AI. This dynamic interplay between AI and human interaction guides visitors through emotional transitions, helping them move from a state of anxiety or stress to relaxation or excitement.

A real-world example of emotionally responsive technology includes “The Alchemist’s Garden in Prague“, where AI analyzes participants’ emotional states and adjusts the music and visual patterns accordingly. IBM’s Mood Mixer, used in collaboration with the Grammys, also offers a similar experience, where users’ responses help generate a personalized music playlist with visuals designed to match their emotional profile. Refik Anadol’s art installations, such as “Melting Memories,” utilize biometric data to generate dynamic, mesmerizing visual patterns synchronized with ambient soundscapes, further demonstrating the power of AI in creating emotionally responsive art.

Refik Anadol’s art installation “Melting Memories,”

ENHANCEMENTS TO CONSIDER

To enhance the emotional depth of these exhibitions, designers can incorporate immersive technologies like 360-degree projection mapping, spatial audio, or virtual reality (VR). These tools fully envelop the visitor in a multi-sensory environment, heightening emotional engagement. Additionally, exhibitions can serve an educational purpose by showing how music and visuals work together to influence emotion. Visitors might see how specific changes in tempo or key can alter the tone of an environment, or how visual elements like lighting or color shift alongside the emotional trajectory of the music. For a broader impact, cross-cultural elements can also be integrated, showcasing how different cultures use music and visuals to evoke emotions, allowing visitors to explore diverse emotional languages and experiences. Combining these enhancements with sound and visual integration can transform any exhibition into a deeply personal and emotionally engaging journey for each visitor.

Music and visuals have immense power to evoke emotions and immerse participants in interactive experiences. By combining the insights from Chapter 1 of This Is Your Brain on Music with design principles, we can create emotionally engaging designs. Below is a framework for how music, visuals, and interactivity can work together, with examples and practical applications.

EMOTIONAL TRIGGERS AND THEIR INTERACTIVE USES

1. Pattern Recognition and Expectation

An interactive design application could incorporate dynamic lighting that reacts to musical patterns, such as intensifying lights during a crescendo and dimming or shifting colors during resolution. To encourage participation, users could control musical sequences through touch panels or motion sensors, creating harmonious results that satisfy their pattern-seeking instincts and evoke joy. For example, a touch-sensitive wall could allow participants to tap and create melodies, triggering synchronized visual effects like ripples or sparks that align with melodic expectations.

2. Tension and Release

Emotional arcs in music, such as the buildup and resolution of tension, mirror life experiences, making them deeply relatable. Designers can enhance these arcs by pairing music with immersive visuals that reflect emotional shifts. For instance, tense musical passages could be accompanied by jagged, dark visuals, which then transform into soft, flowing imagery as the music resolves. Interactive exhibits can further engage participants by giving them control over tension and release, such as pressing a button to trigger a visual explosion synced with a musical climax. An example could be a VR experience where users navigate an environment that becomes brighter and more colorful as the music resolves, evoking feelings of triumph or relief.

3. Visual Analogies (Colors and Shapes)

The brain naturally links musical qualities with visuals, such as warm colors for happy music or smooth shapes for calm music. Aligning music with complementary visuals amplifies its emotional impact. Designers can map musical elements to visual stimuli, like using higher pitches to trigger vibrant colors (reds, yellows) and lower pitches to evoke cool tones (blues, purples). Rhythmic changes could influence shapes, with sharp angles for fast tempos and flowing curves for slower ones. Interactive experiences can further engage users by allowing them to mix visual effects or adjust color schemes in sync with musical features, fostering a sense of agency over the emotional tone. For example, an interactive light installation could let participants control both the soundtrack and visuals, where selecting a major key generates bright, colorful imagery, while a minor key creates darker, subdued visuals.

“FLORA” light installation, 2015/2016, Philipp Artus

DESIGN CONSIDERATIONS

Designing for maximum emotional impact in interactive exhibitions requires a seamless integration of sound and visuals, leveraging their ability to evoke and amplify emotions. Synchronizing music with visual transitions ensures an immersive and cohesive experience, while personalization fosters a sense of agency, making the interaction deeply engaging and memorable. By incorporating contrast and surprise, designers can mimic music’s tension-and-release mechanism to create emotional highs and lows. Finally, considering cultural contexts ensures the experience resonates universally, honoring diverse interpretations of sound and imagery. Together, these principles can transform exhibitions into powerful, multisensory journeys that connect deeply with audiences.

In the opening chapter of This Is Your Brain on Music, Daniel J. Levitin lays the foundation for understanding music as a structured collection of sounds that appeal to our brain’s innate ability to recognize patterns. He breaks down the core components of music-pitch (how high or low a note is), rhythm (the timing of sounds), and harmony (how different notes work together)-and explores their role in eliciting emotional responses.

HOW MUSIC TRIGGERS EMOTIONS IN OUT BRAIN

Levitin emphasizes that music’s emotional power lies in its ability to play with our brain’s expectations. Our brains are wired to predict patterns in sensory information, and music leverages this predictive mechanism to create moments of satisfaction, surprise, or tension. The brain processes music through several interconnected systems:

Pattern Recognition: The auditory cortex is responsible for identifying musical patterns. When a melody resolves in a way our brain predicts, it brings feelings of comfort or pleasure. Conversely, when a pattern is disrupted bye a surprising chord or rhythm, it can evoke excitement, curiosity, or even anxiety.
Tension and Release: Music creates emotional arcs by building tension and then resolving it. For example, a suspended chord creates a sense of unease that resolves when it transitions to the expected harmonic endpoint. This mirrors emotional experiences in life, such as anticipating and resolving conflict, which makes the emotional impact of music relatable.
Dynamic Contrasts: Changes in loudness and tempo can influence the intensity of emotional responses. A gradual crescendo builds excitement and anticipation, while a sudden decrescendo might evoke calmness or introspection.
Cultural and Personal Context: Our emotional reaction to music is shaped by prior experiences and cultural conditioning. For instance, certain scales (like the major scale) are culturally associated with happiness, while others (like the minor scale) are linked to sadness.

EXAMPLES OF EMOTIONAL TRIGGERS

Pitch and Emotion: High-pitched melodies are often perceived as bright or happy, while low-pitched tones convey seriousness or sadness. For instance, a slow cello piece in a minor key might evoke the image of a dark, moody landscape, resonating with feelings of sadness or introspection.
Rhythm and Excitement: Fast, syncopated rhythms (like those in dance music) stimulate the brain’s motor areas, often triggering excitement and a physical urge to move.
Timbre and Texture: The unique “color” of an instrument’s sound can evoke specific associations. For instance, the sharp, brassy timbre of a trumpet might evoke energy or heroism, while the soft timbre of a flute might suggest tranquility or nostalgia.

INSIGHTS

Levitin’s analysis of music in Chapter 1 reveals that the emotions triggered by music stem from the brain’s inherent drive to find order, combined with the sensory and cultural associations we attribute to sound. The visual and color analogies he uses underscore the multi-sensory nature of our emotional experiences with music, showing how sound transcends auditory processing to evoke vivid, almost tangible feelings and images.