ViVo: by Diemo Schwarz and Matéo Fayet

ViVo Video analysis for concatenative audiovisual corpus synthesis

  • Research
  • Seminars

Matéo Fayet and Diemo Schwarz, from the Interaction Son Musique Mouvement (ISMM) team at the STMS laboratory (Ircam - CNRS - Sorbonne University - Ministry of Culture), invite you to a presentation of ViVo, a video analysis system for concatenative audiovisual synthesis by corpus, the subject of their work in collaboration with the PRISM laboratory (CNRS - Aix-Marseille University, Ministry of Culture).

Abstract

Audiovisual concatenative synthesis by corpus extends the principle of sound concatenative synthesis to the visual domain, where in addition to the sound corpus (a collection of recorded sound segments accompanied by a perceptive description of their characteristics), the artist uses a corpus of static images with their perceptive visual characteristics (colour, texture, detail, luminosity, entropy, movement). He thus aims to create a musical audiovisual performance by navigating in real time through these spaces of descriptors: through a collection of sound grains in a space of perceptual audio descriptors, and through a space of visual descriptors, selecting images from a visual corpus for rendering. As a result, navigate in parallel through both corpora interactively using tactile gesture control. We extend this principle here to the analysis of videos to form the visual corpus, with the addition of a few specific descriptors. Two arts/sciences questions are explored, in the case of an artistic creation: which visual descriptors are suitable for multimodal interaction? How can they be integrated, from real-time video data analysis, into a corpus-based concatenative sound synthesis system, to create an embodied multimodal audiovisual experience? This is what is possible with Max/Jitter/MuBu in the ViVo toolbox, briefly presented in this seminar.

Speakers

Diemo Schwarz is a researcher at Ircam (STMS laboratory), electronic music composer and improvising musician. His scientific research focuses on the interaction between musician and machine, on the exploitation of large masses of sound for real-time and interactive sound synthesis, either to give musicians immediate and expressive access to electronic sound worlds through gesture control, or for consumer installations with intuitive tangible interfaces, such as DIRTI (Dirty Tangible Interfaces). In 2017, he was a visiting professor at the Technical University of Berlin as part of the DAAD's Edgar-Varèse programme, and in 2022, a resident at the IMéRA Institute for Advanced Research, Aix-Marseille University, as part of the arts, sciences, societies programme.

Matéo Fayet graduated from the Master's programme in Acoustics and Musicology (Aix-Marseille University) in 2023. His Master's thesis focused on the development of ViVo and the creation of an immersive audiovisual work in the ISMM team and the PRISM laboratory. He is currently a student on a Research Year in Research-Creation (ENS Paris-Saclay), doing his placement with the ISMM team (DOTS project) and the Computer Music Production team at Ircam.

VIVO

En poursuivant votre navigation sur ce site, vous acceptez l'utilisation de cookies pour nous permettre de mesurer l'audience, et pour vous permettre de partager du contenu via les boutons de partage de réseaux sociaux. En savoir plus.