The experimental row called SINGING ROCKS is an attempt to visualize sound with known representations, we can find in nature. Acoustic sounds are converted in timecoded fft spectrum graphics. These graphics are used as depth information for a proper image diffusion process.

recorded voice saying „on the rocks“ + depthMap guided rendering

recorded voice saying „rock wall rocking wall“ + free mouse direction + depthMap guided rendering

audio to depthmap

https://editor.p5js.org/brucexxxbanner/sketches/ZtmcwFaem

SYNESTHETISCHE SCHLEIFEN

Unter Synesthesie verstehen wir die Verschmelzung von getrennten Sinnesmodalitäten zu einer hybriden Wahrnehmungsform. Etwa 4 % der Menschen besitzen synesthetische Eigenschaften. Mit generativer KI und entsprechenden Werkzeugen können wir solche Schnittstellen synthetisch erzeugen. Zum Beispiel können wir aus dem Klangbild einer Stimme ein charakteristisches Bild einer Felswand erstellen. Dieses Bild visualisiert die Audiodaten direkt, kann jedoch auch in andere visuelle Richtungen weiterentwickelt werden. Aber das ist noch nicht alles: Dieses Bild kann von LLMs beschrieben werden, um Songtexte und Anweisungen zu erzeugen, die wiederum neue Musikstücke hervorbringen. So schließt sich der mediale Kreis.

raw FFT output of a voice singing „rocky road“ 3x

rendered depth2Img

Florence-2 based image description: „The image is a black and white close-up of a textured surface. The surface appears to be made up of small, irregularly shaped blocks or blocks of different sizes and shapes. The blocks are arranged in a grid-like pattern, with some blocks overlapping each other. The texture of the surface is rough and uneven, with deep grooves and crevices running through them. The image is taken from a top-down perspective, looking down on the surface.“

genre description chatGPT 3.5: Ambient, Post-Rock, Minimalism, Electronic, Classical, Atmospheric, Subtle, Ethereal, Dreamlike, Reverberant, Textured, Cinematic, Melancholic, Soft-spoken, Reflective, Haunting, Gentle, Intimate, Whispered, Enigmatic, Layered

Song lyrics created by chatGPT 3.5:
„The Tapestry of Shadows

In monochrome whispers, the surface unfolds, A mosaic of shadows in whispers of old. Blocks of wonder, in sizes unbound, Arrayed in a dance, where secrets are found.

Irregular fragments in a grid’s embrace, Overlap and intertwine, in a textured lace. Rough and uneven, the landscape reveals, Deep grooves and crevices where silence conceals.

From a top-down view, the canvas reveals, A rugged terrain where the heart gently heals. Each line, each shadow, a tale softly told, In the tapestry of shadows, both timid and bold.“