by David Adlberger -

Product XI: Image Extender

From Notebook Prototype to Local, Exhibitable Software

This iteration was less about adding new conceptual capabilities and more about solidifying the system as an actual, deployable artifact. The core task was migrating the image extender from its experimental form into a standalone local application. What sounds like a technical refactor turned out to be a decisive shift in how the system is meant to exist, be used, and be encountered.

Until now, the notebook environment functioned as a kind of protected laboratory. It encouraged rapid iteration, verbose configuration, and exploratory branching. Moving out of that space meant confronting a different question: what does this system look like when it stops being a research sketch and starts behaving like software?

The transition from Colab-style execution to a locally running script forced a re-evaluation of assumptions that notebooks quietly hide:

  • Implicit state becomes explicit
  • Execution order must be deterministic
  • Errors can no longer be “scrolled past”
  • Configuration must be intentional, not convenient

Porting the logic meant flattening the notebook’s narrative structure into a single, readable execution flow. Cells that once assumed context had to be restructured into functions, initialization stages, and clearly defined entry points. This wasn’t just cleanup, it was an architectural clarification.

In the notebook, ambiguity is tolerated. In running software, it accumulates as friction.

Reduction as Design: Cutting Options to Increase Clarity

One of the more deliberate changes during this phase was a reduction in exposed settings. The notebook version allowed extensive tweaking, model switches, resolution variants, prompt behaviors, fallback paths, all useful during development, but overwhelming in a public-facing context.

For the exhibition version, optionality became noise.

Instead of presenting the system as a configurable toolkit, I reframed it as a guided instrument. Core behaviors remain intact, but the number of visible parameters was intentionally constrained. This aligns with a recurring principle in the project: flexibility should live inside the system, not on its surface.

Adapting for Exhibition: Y2K as Interface Language

Alongside the structural changes, the interface was visually adapted to match the exhibition context. The decision to lean into a Y2K-inspired color palette wasn’t purely aesthetic; it functioned as a form of contextual grounding.

The visual layer needed to communicate that this is not a neutral utility, but a situated artifact. The Y2K styling introduced:

  • High-contrast synthetic colors
  • Clear visual hierarchy
  • A subtle nod to early digital optimism and machinic playfulness

Rather than competing with the system’s conceptual weight, the styling makes its artificiality explicit.

Stability Over Novelty

Another quiet but important shift was prioritizing stability over feature expansion. The migration process exposed several edge cases that were easy to ignore in a notebook but unacceptable in a live context: silent failures, unclear loading states, brittle dependencies.

Addressing these didn’t add visible functionality, but they fundamentally changed how trustworthy the system feels. In an exhibition setting, reliability is part of the experience. A system that hesitates or crashes invites interpretation for the wrong reasons.

Here, robustness became a form of authorship.

Reframing the System’s Status

By the end of this iteration, the most significant change wasn’t technical, it was ontological. The system is no longer best described as “a notebook that does something interesting.” It is now a runnable, bounded piece of software, designed to be encountered without explanation.

This transition marks a subtle but important moment in the project’s lifecycle:

  • From private exploration to public behavior
  • From configurable experiment to opinionated instrument
  • From development environment to exhibited system

The constraints introduced in this phase don’t limit future growth, they define a stable core from which growth can happen meaningfully.

If earlier updates were about expanding the system’s conceptual reach, this one was about giving it a body.

by David Adlberger -

Product X: Image Extender

Extending the System from Image Interpretation to Image Synthesis

This update marked a conceptual shift in the system’s scope: until now, images functioned purely as inputs, sources of visual information to be analyzed, interpreted, and mapped onto sound. With this iteration, I expanded the system to also support image generation, enabling users not only to upload visual material but to synthesize it directly within the same creative loop.

The goal was not to bolt on image generation as a novelty feature, but to integrate it in a way that respects the system’s broader design philosophy: user intent first, semantic coherence second, and automation as a supportive, not dominant, layer.

Architectural Separation: Reasoning vs. Rendering

A key early decision was to separate prompt reasoning from image rendering. Rather than sending raw user input directly to the image model, I introduced a two-stage pipeline:

  1. Prompt Interpretation & Enrichment (GPT-4.1)
    Responsible for understanding vague or underspecified user prompts and rewriting them into a semantically complete, realistic scene description.
  2. Image Synthesis (gpt-image-1 → DALL-E 2/3)
    Dedicated purely to rendering the final image from the enriched prompt. Through implementation, I discovered that while the original spec referenced gpt-image-1, OpenAI’s actual models are DALL-E 2 (60% cheaper, faster, but less detailed) and DALL-E 3 (higher quality but more expensive).

This separation mirrors the system’s audio architecture, where semantic interpretation and signal processing are deliberately decoupled. GPT-4.1 acts as a semantic mediator, while the image model remains a deterministic renderer.

The Response Format Learning Curve

During implementation, I encountered a subtle but important API nuance that forced a deeper understanding of the system’s data flow: DALL-E models return URLs by default, not base64 data. The initial implementation failed with a confusing “NoneType” error because I was trying to decode a base64 field that didn’t exist.

The fix was elegantly simple, adding response_format=”b64_json” to the API call—but the debugging process revealed something more fundamental about API design: different services have different default behaviors, and understanding those defaults is crucial for robust system integration.

This also led to implementing proper fallback logic: if base64 isn’t available, the system gracefully falls back to downloading from the image URL, ensuring reliability across different OpenAI model versions and configurations.

Interactive Workflow Integration with Toggle Architecture

To maintain consistency with the existing interactive toolset while adding flexibility, I implemented a mode-toggle architecture:

  • Upload Mode: Traditional file upload with drag-and-drop support
  • Generate Mode: Text-to-image synthesis with prompt enrichment
  • State Preservation: The system maintains a single IMAGE_FILE variable that can be overwritten by either mode, ensuring seamless transitions between workflows

The interface exposes this through clean toggle buttons, showing only the relevant UI for each mode. This reduces cognitive load while preserving full functionality, a principle I’ve maintained throughout the system’s evolution.

Cost-Aware Design with Caching and Model Selection

Image synthesis presents unique cost challenges compared to text generation or audio processing. I implemented several cost-mitigation strategies learned through experimentation:

  1. Resolution Control: Defaulting to 1024×1024  or 512×512 (for DALL-E 2)
  2. Quality Parameter Awareness: Only DALL-E 3 supports quality=”standard” vs “hd”—using the wrong parameter with DALL-E 2 causes API errors

The cost considerations weren’t just about saving money—they were about enabling iteration. When artists can generate dozens of variations without financial anxiety, they explore more freely. The system defaults to the cheapest viable path, with quality controls available but not forced.

Prompt Realism as a Soft Constraint

Rather than enforcing hard validation rules (e.g., predefined lists of places or objects), I chose to treat realism as a soft constraint enforced by language, not logic.

User prompts are passed through a prompt-enrichment step where GPT-4.1 is instructed to:

  • Reframe the input as a photographic scene
  • Ensure the presence of spatial context (location, environment)
  • Ground the description in physical objects and lighting
  • Explicitly avoid illustrated, cartoon, or painterly styles

This approach preserves creative freedom while ensuring that the downstream image generation remains visually coherent and photo-realistic. Importantly, the system does not reject user input—it interprets it.

Design Philosophy: Generation as a First-Class Input

What this update ultimately enabled is a shift in how the system can be used:

  • Images are no longer just analyzed artifacts
  • They can now be constructed, refined, and immediately fed into downstream processes (visual analysis, audio mapping, spatial inference)

This closes a loop that previously required external tools. The system now supports a full cycle: imagine → generate → interpret → sonify.

Crucially, the same principle that guided earlier updates still applies: automation should amplify intent, not replace it. Image generation here is not about producing spectacle, but about giving users a controlled, semantically grounded way to define the visual worlds their soundscapes respond to.

The implementation journeyfrom API quirks to cost optimization to user experience design, reinforced that even “simple” features require deep consideration when integrating into a complex creative system. Each new capability should feel like it was always there, waiting to be discovered.

by David Adlberger -

Product IX: Image Extender

Moving Beyond Dry Audio to Spatially Intelligent Soundscapes

My primary objective for this update was to bridge a critical perceptual gap in the system: while the previous iterations successfully mapped visual information to sonic elements with precise panning and temporal placement, the resulting audio mix remained perceptually “dry” and disconnected from the image’s implied acoustic environment. This update introduces adaptive reverberation, not as a cosmetic effect, but as a semantically grounded spatialization layer that transforms discrete sound objects into a coherent, immersive acoustic scene.

System Architecture

The existing interactive DAW interface, with its per-track volume controls, sound replacement engine, and user feedback mechanisms, was extended with a comprehensive spatial audio processing module. This module interprets the reverb parameters derived from image analysis (room detection, size estimation, material damping, and spatial width) and provides interactive control over their application.

Global Parameter State & Data Flow Integration

A crucial architectural challenge was maintaining separation between the raw audio mix (user-adjustable volume levels) and the reverb-processed version. I implemented a dual-state system with:

  • current_mix_raw: The continuously updated sum of all audio tracks with current volume slider adjustments.
  • current_mix_with_reverb: A cached, processed version with reverberation applied, recalculated only when reverb parameters change or volume sliders are adjusted with reverb enabled.

This separation preserves processing efficiency while maintaining real-time responsiveness. The system automatically pulls reverb parameters (room_size, damping, wet_level, width) from the image analysis block when available, providing image-informed defaults while allowing full manual override.

Pedalboard-Based Reverb Engine

I integrated the pedalboard audio processing library to implement professional-grade reverberation. The engine operates through a transparent conversion chain:

  1. Format Conversion: AudioSegment objects (from pydub) are converted to NumPy arrays normalized to the [-1, 1] range
  2. Pedalboard Processing: A Reverb effect instance applies parameters with real-time adjustable controls
  3. Format Restoration: Processed audio is converted back to AudioSegment while preserving sample rate and channel configuration

The implementation supports both mono and stereo processing chains, maintaining compatibility with the existing panning system.

Interactive Reverb Control Interface

A dedicated control panel was added to the DAW interface, featuring:

  • Parameter Sliders: Four continuous controls for room size, damping, wet/dry mix, and stereo width, pre-populated with image-derived values when available
  • Toggle System: Three distinct interaction modes:
    1. “🔄 Apply Reverb”: Manual application with current settings
    2. “🔇 Remove Reverb”: Return to dry mix
    3. “Reverb ON/OFF Toggle”: Single-click switching between states
  • Contextual Feedback: Display of image-based room detection status (indoor/outdoor)

Seamless Playback Integration

The playback system was redesigned to dynamically switch between dry and wet mixes:

  • Intelligent Routing: The play_mix() function automatically selects current_mix_with_reverb or current_mix_raw based on the reverb_enabled flag
  • State-Aware Processing: When volume sliders are adjusted with reverb enabled, the system automatically reapplies reverberation to the updated mix, maintaining perceptual consistency
  • Export Differentiation: Final mixes are exported with _with_reverb or _raw suffixes, providing clear version control

Design Philosophy: Transparency Over Automation

This phase reinforced a critical design principle: spatial effects should enhance rather than obscure the user’s creative decisions. Several automation approaches were considered and rejected:

  • Automatic Reverb Application: While the system could automatically apply image-derived reverb, I preserved manual activation to maintain user agency
  • Dynamic Parameter Adjustment: Real-time modification of reverb parameters during playback was technically feasible but introduced perceptual confusion
  • Per-Track Reverb: Individual reverberation for each sound object would create acoustic chaos rather than coherent space

The decision was made to implement reverb as a master bus effect, applied consistently to the entire mix after individual track processing. This approach creates a unified acoustic space that respects the visual scene’s implied environment while preserving the clarity of individual sound elements.

Technical Challenges & Solutions

State Synchronization

The most significant challenge was maintaining synchronization between the constantly updating volume-adjusted mix and the computationally expensive reverb processing. The solution was a conditional caching system: reverb is only recalculated when parameters change or when volume adjustments occur with reverb active.

Format Compatibility

Bridging the pydub-based mixing system with pedalboard‘s NumPy-based processing required careful attention to sample format conversion, channel configuration, and normalization. The implementation maintains bit-perfect round-trip conversion.

by David Adlberger -

Product VIII: Image Extender

Iterative Workflow and Feedback Mechanism

The primary objective for this update was to architect a paradigm shift from a linear generative pipeline to a nonlinear, interactive sound design environment

System Architecture & Implementation of Interactive Components

The existing pipeline, comprising image analysis (object detection, semantic tagging), importance-weighted sound search, audio processing (equalization, normalization, panoramic distribution based on visual coordinates), and temporal randomization was extended with a state-preserving session layer and an interactive control interface, implemented within the collab notebook ecosystem.

Data Structure & State Management
A critical prerequisite for interactivity was the preservation of all intermediate audio objects and their associated metadata. The system was refactored to maintain a global, mutable data structure, a list of processed_track objects. Each object encapsulates:

  • The raw audio waveform (as a NumPy array).
  • Semantic source tag (e.g., “car,” “rain”).
  • Track type (ambience base or foreground object).
  • Temporal onset and duration within the mix.
  • Panning coefficient (derived from image x-coordinate).
  • Initial target loudness (LUFS, derived from object importance scaling).

Dynamic Mixing Console Interface
A GUI panel was generated post-sonification, featuring the following interactive widgets for each processed_track:

  • Per-Track Gain Sliders: Linear potentiometers (range 0.0 to 2.0) controlling amplitude multiplication. Adjustment triggers an immediate recalculation of the output sum via a create_current_mix() function, which performs a weighted summation of all tracks based on the current slider states.
  • Play/Stop Controls: Buttons invoking a non-blocking, threaded audio playback engine (using IPython.display.Audio and threading), allowing for real-time auditioning without interface latency.

On-Demand Sound Replacement Engine
The most significant functional addition is the per-track “Search & Replace” capability. Each track’s GUI includes a dedicated search button (🔍). Its event handler executes the following algorithm:

  1. Tag Identification: Retrieves the original semantic tag from the target processed_track.
  2. Targeted Audio Retrieval: Calls a modified search_new_sound_for_tag(tag, exclude_id_list) function. This function re-executes the original search logic, including query formulation, Freesound API calls, descriptor validation (e.g., excluding excessively long or short files), and fallback strategies—while maintaining a session-specific exclusion list to avoid re-selecting previously used sounds.
  3. Consistent Processing: The newly retrieved audio file undergoes an identical processing chain as in the initial pipeline: target loudness normalization (to the original track’s LUFS target), application of the same panning coefficient, and insertion at the identical temporal position.
  4. State Update & Mix Regeneration: The new audio data replaces the old waveform in the processed_track object. The create_current_mix() function is invoked, seamlessly integrating the new sonic element while preserving all other user adjustments (e.g., volume levels of other tracks).

Integrated Feedback & Evaluation Module
To formalize user evaluation and gather data for continuous system improvement, a structured feedback panel was integrated adjacent to the mixing controls. This panel captures:

  • A subjective 5-point Likert scale rating.
  • Unstructured textual feedback.
  • Automated attachment of complete session metadata (input image description, derived tags, importance values, processing parameters, and the final processed_track list).
    This design explicitly closes the feedback loop, treating each user interaction as a potential training or validation datum for future algorithmic refinements.
  • Automated sending of the feedback via email
by David Adlberger -

Product VII: Image Extender

Room-Aware Mixing – From Image Analysis to Coherent Acoustic Spaces

Instead of attempting to recover exact physical properties, the system derives normalized, perceptual room parameters from visual cues such as geometry, materials, furnishing density, and openness. These parameters are intentionally abstracted to work with algorithmic reverbs.

The introduced parameters are:

  • room_detected (bool)
    Indicates whether the image depicts a closed indoor space or an outdoor/open environment.
  • room_size (0.0–1.0)
    Represents the perceived acoustic size of the room (small rooms → short decay, large spaces → long decay).
  • damping (0.0–1.0)
    Estimates high-frequency absorption based on visible materials (soft furnishings, carpets, curtains vs. glass and hard walls).
  • wet_level (0.0–1.0)
    Describes how reverberant the space naturally feels.
  • width (0.0–1.0)
    Estimates perceived stereo width derived from room proportions and openness.

All parameters are stored flat within the same dictionary as objects, panning, and importance values, forming a single coherent scene representation.

Dereverberation: Explored, Then Intentionally Abandoned

As part of this phase, automatic analysis of existing reverberation (RT60, DRR estimation) and dereverberation was evaluated.

The outcome:

  • Computationally expensive, especially in Google Colab
  • Inconsistent and often unsatisfactory audio results
  • High complexity with limited practical benefit

Decision:
Dereverberation is not pursued further in this project. Instead, the system relies on:

  • Consistent room estimation
  • Controlled, unified reverb application
  • Preventive design rather than corrective processing

The next step will be to focus on the analysis of the sounds (especially rt60 and drr values) to make the reverb (if its a closed room) less on the specific sound.

by David Adlberger -

Product VI: Image Extender

Intelligent Balancing – progress of automated mixing

This development phase introduces a sophisticated dual-layer audio processing system that addresses both proactive and reactive sound masking, creating mixes that are not only visually faithful but also acoustically optimal. Where previous systems focused on semantic accuracy and visual hierarchy, we now ensure perceptual clarity and natural soundscape balance through scientific audio principles.

The Challenge: High-Energy Sounds Dominating the Mix

During testing, we identified a critical issue: certain sounds with naturally high spectral energy (motorcycles, engines, impacts) would dominate the audio mix despite appropriate importance-based volume scaling. Even with our masking analysis and EQ correction, these sounds created an unbalanced listening experience where the mix felt “crowded” by certain elements.

Dual-Layer Solution Architecture

Layer 1: Proactive Energy-Based Gain Reduction

This new function analyzes each sound’s spectral energy across Bark bands (psychoacoustic frequency scale) and applies additional gain reduction to naturally loud sounds. The system:

  1. Measures average and peak energy across 24 Bark bands
  2. Calculates perceived loudness based on spectral distribution
  3. Applies up to -6dB additional reduction to high-energy sounds
  4. Modulates reduction based on visual importance (high importance = less reduction)

Example Application:

  • Motorcycle sound: -4.5dB additional reduction (high energy in 1-4kHz range)
  • Bird chirp: -1.5dB additional reduction (lower overall energy)
  • Both with same visual importance, but motorcycle receives more gain reduction

Layer 2: Reactive Masking EQ (Enhanced)

Improved Feature: Time-domain masking analysis now works with consistent positioning

We fixed a critical bug where sound positions were being randomized twice, causing:

  • Overlap analysis using different positions than final placement
  • EQ corrections applied to wrong temporal segments
  • Inconsistent final mix compared to analysis predictions

Solution: Position consistency through saved_positions system:

  • Initial random placement saved after calculation
  • Same positions used for both masking analysis and final timeline
  • Transparent debugging output showing exact positions used

Key Advancements

  1. Proactive Problem Prevention: Energy analysis occurs before mixing, preventing issues rather than fixing them
  2. Preserved Sound Quality: Moderate gain reduction + moderate EQ = better than extreme EQ alone
  3. Phase Relationship Protection: Gain reduction doesn’t affect phase like large EQ cuts do
  4. Mono Compatibility: Less aggressive processing improves mono downmix results
  5. Transparent Debugging: Complete logging shows every decision from energy analysis to final placement

Integration with Existing System

The new energy-based system integrates seamlessly with our established pipeline:

text

Sound Download → Energy Analysis → Gain Reduction → Importance Normalization

→ Timeline Placement → Masking EQ (if needed) → Final Mix

This represents an evolution from reactive correction to intelligent anticipation, creating audio mixes that are both visually faithful and acoustically balanced. The system now understands not just what sounds should be present, but how they should coexist in the acoustic space, resulting in professional-quality soundscapes that feel natural and well-balanced to the human ear.

by David Adlberger -

Product V: Image Extender

Dynamic Audio Balancing Through Visual Importance Mapping

This development phase introduces sophisticated volume control based on visual importance analysis, creating audio mixes that dynamically reflect the compositional hierarchy of the original image. Where previous systems ensured semantic accuracy, we now ensure proportional acoustic representation.

The core advancement lies in importance-based volume scaling. Each detected object’s importance value (0-1 scale from visual analysis) now directly determines its loudness level within a configurable range (-30 dBFS to -20 dBFS). Visually dominant elements receive higher volume placement, while background objects maintain subtle presence.

Key enhancements include:

– Linear importance-to-volume mapping creating natural acoustic hierarchies

– Fixed atmo sound levels (-30 dBFS) ensuring consistent background presence

– Image context integration in sound validation for improved semantic matching

– Transparent decision logging showing importance values and calculated loudness targets

The system now distinguishes between foreground emphasis and background ambiance, producing mixes where a visually central “car” (importance 0.9) sounds appropriately prominent compared to a distant “tree” (importance 0.2), while “urban street atmo” provides unwavering environmental foundation.

This represents a significant evolution from flat audio layering to dynamically balanced soundscapes that respect visual composition through intelligent volume distribution.

by David Adlberger -

Product IV: Image Extender

Semantic Sound Validation & Ensuring Acoustic Relevance Through AI-Powered Verification

Building upon the intelligent fallback systems developed in Phase III, this week’s development addressed a more subtle yet critical challenge in audio generation: ensuring that retrieved sounds semantically match their visual counterparts. While the fallback system successfully handled missing sounds, I discovered that even when sounds were technically available, they didn’t always represent the intended objects accurately. This phase introduces a sophisticated description verification layer and flexible filtering system that transforms sound retrieval from a mechanical matching process to a semantically intelligent selection.

The newly implemented description verification system addresses this through OpenAI-powered semantic analysis. Each retrieved sound’s description is now evaluated against the original visual tag to determine if it represents the actual object or just references it contextually. This ensures that when Image Extender layers “car” sounds into a mix, they’re authentic engine recordings rather than musical tributes.

Intelligent Filter Architecture: Balancing Precision and Flexibility

Recognizing that overly restrictive filtering could eliminate viable sounds, we redesigned the filtering system with adaptive “any” options across all parameters. The Bit-Depth filter got removed because it resulted in search errors which is also mentioned in the documentation of the freesound.org api.

Scene-Aware Audio Composition: Atmo Sounds as Acoustic Foundation

A significant architectural improvement involves intelligent base track selection. The system now distinguishes between foreground objects and background atmosphere:

  • Scene & Location Analysis: Object detection extracts environmental context (e.g., “forest atmo,” “urban street,” “beach waves”)
  • Atmo-First Composition: Background sounds are prioritized as the foundational layer
  • Stereo Preservation: Atmo/ambience sounds retain their stereo imaging for immersive soundscapes
  • Object Layering: Foreground sounds are positioned spatially based on visual detection coordinates

This creates mixes where environmental sounds form a coherent base while individual objects occupy their proper spatial positions, resulting in professionally layered audio compositions.

Dual-Mode Object Detection with Scene Understanding

OpenAI GPT-4.1 Vision: Provides comprehensive scene analysis including:

  • Object identification with spatial positioning
  • Environmental context extraction
  • Mood and atmosphere assessment
  • Structured semantic output for precise sound matching

MediaPipe EfficientDet: Offers lightweight, real-time object detection:

  • Fast local processing without API dependencies
  • Basic object recognition with positional data
  • Fallback when cloud services are unavailable

Wildcard-Enhanced Semantic Search: Beyond Exact Matching

Multi-Stage Fallback with Verification Limits

The fallback system evolved into a sophisticated multi-stage process:

  1. Atmo Sound Prioritization: Scene_and_location tags are searched first as base layer
  2. Object Search: query with user-configured filters
  3. Description Verification: AI-powered semantic validation of each result
  4. Quality Tiering: Progressive relaxation of rating and download thresholds
  5. Pagination Support: Multiple result pages when initial matches fail verification
  6. Controlled Fallback: Limited OpenAI tag regeneration with automatic timeout

This structured approach prevents infinite loops while maximizing the chances of finding appropriate sounds. The system now intelligently gives up after reasonable attempts, preventing computational waste while maintaining output quality.

Toward Contextually Intelligent Audio Generation

This week’s enhancements represent a significant leap from simple sound retrieval to contextually intelligent audio selection. The combination of semantic verification, adaptive filtering and scene-aware composition creates a system that doesn’t just find sounds, it finds the right sounds and arranges them intelligently.

by David Adlberger -

Product III: Image Extender

Intelligent Sound Fallback Systems – Enhancing Audio Generation with AI-Powered Semantic Recovery

After refining Image Extender’s sound layering and spectral processing engine, this week’s development shifted focus to one of the system’s most practical yet creatively crucial challenges: ensuring that the generation process never fails silently. In previous iterations, when a detected visual object had no directly corresponding sound file in the Freesound database, the result was often an incomplete or muted soundscape. The goal of this phase was to build an intelligent fallback architecture—one capable of preserving meaning and continuity even in the absence of perfect data.

Closing the Gap Between Visual Recognition and Audio Availability

During testing, it became clear that visual recognition is often more detailed and specific than what current sound libraries can support. Object detection models might identify entities like “Golden Retriever,” “Ceramic Cup,” or “Lighthouse,” but audio datasets tend to contain more general or differently labeled entries. This mismatch created a semantic gap between what the system understands and what it can express acoustically.

The newly introduced fallback framework bridges this gap, allowing Image Extender to adapt gracefully. Instead of stopping when a sound is missing, the system now follows a set of intelligent recovery paths that preserve the intent and tone of the visual analysis while maintaining creative consistency. The result is a more resilient, contextually aware sonic generation process—one that doesn’t just survive missing data, but thrives within it.

Dual Strategy: Structured Hierarchies and AI-Powered Adaptation

Two complementary fallback strategies were introduced this week: one grounded in structured logic, and another driven by semantic intelligence.

The CSV-based fallback system builds on the ontology work from the previous phase. Using the tag_hierarchy.csv file, each sound tag is part of a parent–child chain, creating predictable fallback paths. For example, if “tiger” fails, the system ascends to “jungle,” and then “nature.” This rule-based approach guarantees reliability and zero additional computational cost, making it ideal for large-scale batch operations or offline workflows.

In contrast, the AI-powered semantic fallback uses GPT-based reasoning to dynamically generate alternative tags. When the CSV offers no viable route, the model proposes conceptually similar or thematically related categories. A specific bird species might lead to the broader concept of “bird sounds,” or an abstract object like “smartphone” could redirect to “digital notification” or “button click.” This layer of intelligence brings flexibility to unfamiliar or novel recognition results, extending the system’s creative reach beyond its predefined hierarchies.

User-Controlled Adaptation

Recognizing that different projects require different balances between cost, control, and creativity, the fallback mode is now user-configurable. Through a simple dropdown menu, users can switch between CSV Mode and AI Mode.

  • CSV Mode favors consistency, predictability, and cost-efficiency—perfect for common, well-defined categories.
  • AI Mode prioritizes adaptability and creative expansion, ideal for complex visual inputs or unique scenes.

This configurability not only empowers users but also represents a deeper design philosophy: that AI systems should be tools for choice, not fixed solutions.

Toward Adaptive and Resilient Multimodal Systems

This week’s progress marks a pivotal evolution from static, database-bound sound generation to a hybrid model that merges structured logic with adaptive intelligence. The dual fallback system doesn’t just fill gaps, it embodies the philosophy of resilient multimodal AI, where structure and adaptability coexist in balance.

The CSV hierarchy ensures reliability, grounding the system in defined categories, while the AI layer provides flexibility and creativity, ensuring the output remains expressive even when the data isn’t. Together, they form a powerful, future-proof foundation for Image Extender’s ongoing mission: transforming visual perception into sound not as a mechanical translation, but as a living, interpretive process.

by David Adlberger -

Product II: Image Extender

Dual-Model Vision Interface – OpenAI × Gemini Integration for Adaptive Image Understanding

Following the foundational phase of last week, where the OpenAI API Image Analyzer established a structured evaluation framework for multimodal image analysis, the project has now reached a significant new milestone. The second release integrates both OpenAI’s GPT-4.1-based vision models and Google’s Gemini (MediaPipe) inference pipeline into a unified, adaptive system inside the Image Extender environment.

Unified Recognition Interface

In The current version, the recognition logic has been completely refactored to support runtime model switching.
A dropdown-based control in Google Colab enables instant selection between:

  • Gemini (MediaPipe) – for efficient, on-device object detection and panning estimation
  • OpenAI (GPT-4.1 / GPT-4.1-mini) – for high-level semantic and compositional interpretation

Non-relevant parameters such as score threshold or delegate type dynamically hide when OpenAI mode is active, keeping the interface clean and focused. Switching back to Gemini restores all MediaPipe-related controls.
This creates a smooth dual-inference workflow where both engines can operate independently yet share the same image context and visualization logic.

Architecture Overview

The system is divided into two self-contained modules:

  1. Image Upload Block – handles external image input and maintains a global IMAGE_FILE reference for both inference paths.
  2. Recognition Block – manages model selection, executes inference, parses structured outputs, and handles visualization.

This modular split keeps the code reusable, reduces side effects between branches, and simplifies later expansion toward GUI-based or cloud-integrated applications.

OpenAI Integration

The OpenAI branch extends directly from Last week but now operates within the full environment.
It converts uploaded images into Base64 and sends a multimodal request to gpt-4.1 or gpt-4.1-mini.
The model returns a structured Python dictionary, typically using the following schema:

{

    “objects”: […],

    “scene_and_location”: […],

    “mood_and_composition”: […],

    “panning”: […]

}

A multi-stage parser (AST → JSON → fallback) ensures robustness even when GPT responses contain formatting artifacts.

Prompt Refinement

During development, testing revealed that the English prompt version initially returned empty dictionaries.
Investigation showed that overly strict phrasing (“exclusively as a Python dictionary”) caused the model to suppress uncertain outputs.
By softening this instruction to allow “reasonable guesses” and explicitly forbidding empty fields, the API responses became consistent and semantically rich.

Debugging the Visualization

A subtle logic bug was discovered in the visualization layer:
The post-processing code still referenced German dictionary keys (“objekte”, “szenerie_und_ort”, “stimmung_und_komposition”) from Last week.
Since the new English prompt returned English keys (“objects”, “scene_and_location”, etc.), these lookups produced empty lists, which in turn broke the overlay rendering loop.
After harmonizing key references to support both language variants, the visualization resumed normal operation.

Cross-Model Visualization and Validation

A unified visualization layer now overlays results from either model directly onto the source image.
In OpenAI mode, the “panning” values from GPT’s response are projected as vertical lines with object labels.
This provides immediate visual confirmation that the model’s spatial reasoning aligns with the actual object layout, an important diagnostic step for evaluating AI-based perception accuracy.

Outcome and Next Steps

The project now represents a dual-model visual intelligence system, capable of using symbolic AI interpretation (OpenAI) and local pixel-based detection (Gemini).

Next steps

The upcoming development cycle will focus on connecting the openAI API layer directly with the Image Extender’s audio search and fallback system.