The interfaces have changed, but a question connects them: what becomes possible when listeners can act on the parts and perspectives inside a recording?
Around 2005 · Historical Flash interface
MusicDelta: a stage you could change
Our work in this area began with a collaboration with the Norwegian company MusicDelta around 2005. Its Flash-based music-history platform combined articles, recordings and video with an invitation to change what learners heard.
Rather than only reading about an ensemble or watching it perform, a learner could move its animated musicians. Moving a musician forward or backward changed the volume of that musician’s audio stem; moving left or right changed its stereo placement. The stage made a multitrack recording something to explore through familiar physical actions.
Grieg Music Education, founded in 2005, created Music Delta. OIID launched in 2015, extending the exploration of interactive music experiences.
Animated musicians in MusicDelta’s historical ensemble interface.
10 February 2016 · Public event and archival prototype
Step Inside Mahler: listening through the archives
The MusicDelta experiments helped inspire a later collaboration involving OIID and the New York Philharmonic archives. Step Inside Mahler brought the invitation to “step inside” music into an orchestral and archival setting.
The MusEDLab team led by Alex Ruthmann developed the prototype with the New York Philharmonic and Norwegian developer MusIT. Built with MusIT’s OIID technology, the prototype brought together four historical performances of Mahler’s Sixth Symphony and annotated scores from the Philharmonic’s Leon Levy Digital Archives.
Listeners could compare recordings and scores, while a spatial-listening interface offered instrumental perspectives. The February 10 event at Lincoln Center’s David Rubenstein Atrium included interactive iPad stations. Barbara Haws, Lawrence Rock, Erik Ryding and Alex Ruthmann took part in the event.
“Mahler Grooves”: annotated scores and instrumental perspectives.
Continuing explorations
More ways to listen, move and play
Later MusEDLab projects pursue related questions through different interfaces and collaborations. The Aeolus recordings invite listening from within an ensemble. Virtual Piano Duet adds a way to perform alongside a recorded musician. SpatialAudioNav makes listening position part of exploring Boulez’s music. Spatial Music Lab represents individual sounds as movable objects, while Air Traffic exposes the balance of a recording through a mixing console.
Across these collaborations, performers, designers and developers have created different ways to explore musical relationships through listening and action.
Improvise with recorded pianist Steve Wogaman of the Chamber Music Society of Detroit and see your melodies take shape in a Melody Notebook. V2 automatically captures your played notes, timing and chords as piano rolls. Name and save phrases in your browser, replay them alone or with the duet, and export or import compatible melody JSON files. These are note recordings, not microphone audio; saved phrases stay in this browser rather than syncing to an account.
Step inside the ensemble through immersive recordings developed with the Aeolus String Quartet, Steve Wogaman and the Chamber Music Society of Detroit. Spatial recording and mixing offer new ways to experience chamber music through headphones and web platforms.
MusEDLab and IRCAM developed an interactive exploration of Boulez’s spatial electroacoustic work, combining recordings, score materials and spatial audio visualization. Alex Zhang, James Chang and Zehao Wang collaborated on the platform.
Preview 06 offers Listen, Explore and Mix views for Transition I–II, with shared speaker-volume indicators and room navigation. This is a development preview; musical alignment, asset credits and cross-device listening checks remain under review.
Preview 06: the Transition I–II listening interface.
Explore: Chinese traditional music
In collaboration with Yiting Liu, MusEDLab explores traditional instrumental performances through immersive multichannel recording, spatial listening and interactive media. The project connects musical performance and cultural heritage with creative learning and research.
Spatial Music Lab opens with Chinese Instrumental, featuring music by Justin Scholar 玉刻. Move individual sounds to explore the spatial mix.
Building on Yiting Liu’s completed prototype, MusEDLab has further developed the current version with a green interface and a Movement Notebook for capturing, saving and replaying movement takes.
See the interfaces and the ideas behind them in lab demonstrations. Alex Ruthmann’s TEDx talk, Lowering the Barriers to Creative Musical Expression, presents the approach in a public setting. The talk includes a live demonstration of interactive musical exploration.
MusEDLab App Demos — lab demonstrations
Loading connects to the external provider. Open directly
From the lab
See the tools in action
Watch MusEDLab’s demonstrations of musical exploration, then try an experience yourself.
Historical Flash interface. Explore the illustrated history above. This page does not offer a live Flash player.
Step Inside Mahler
Historical prototype and event documentation. Read about its recordings, scores and collaborators in the historical project record. The original interactive tools are offline.
MusicDelta image courtesy of Alex Ruthmann. The Mahler prototype was a collaboration between NYU MusEDLab, the New York Philharmonic and MusIT using OIID technology; the Philharmonic’s Leon Levy Digital Archives supplied archival material.
Later project credits include the Aeolus String Quartet, Steve Wogaman and the Chamber Music Society of Detroit; IRCAM and developers Alex Zhang, James Chang and Zehao Wang; Yiting Liu and musician Justin Scholar 玉刻; and Clara Berry & Wooldog. Follow the individual project records and source repositories for further credits.
Music rights remain with the credited artists and publishers.
Privacy preferences
You can browse without optional Google Analytics. Videos and other embeds load only when you click them.
