On Feature Design, Overcoming Data Scarcity, and Multimodal Feature Learning—A Music Informatics Journey
Speaker
Juan Pablo Bello
New York University (NYU)
Host
Justin Solomon
CSAIL MIT
MIT Departments of Music & Theater Arts and the Schwarzman College of Computing Special Music Technology Seminars.
Abstract: The automatic symbolization of recorded music has long been a driver of music information retrieval (MIR) research, with the choice of symbols such as notes, chords, beats, sections, instrumental and genre labels, having great influence in how we define tasks, design systems, label data and deploy applications. Yet, the means by which the field tries to achieve symbolization have evolved over the past two decades, a story that I will try to recreate in three parts using examples from my own work. First, I will illustrate how, in early MIR work, the challenge of symbolization was primarily the design and parametrization of features, resulting in highly specialized and heterogeneous methods. Second, and as the field adopted supervised deep learning approaches with minimal to no specialization, I will show how the challenge of symbolization shifted to data curation and labeling, with system design now focusing on overcoming scarcity and bias in labeled data. Third, I will discuss how in the last few years, machine listening research including MIR, has used self-supervised learning, especially multimodal approaches, to reduce the need for labeled data and increase the generalizability of feature representations. I will finalize by arguing that these methods also open the door to a new space of multimodal MIR, less focused on music recordings and notation, and more on gestures, technique and the physical experience of performing music.
Short Bio: Juan Pablo Bello is a Professor of Music Technology, Computer Science & Engineering, Electrical & Computer Engineering, and Urban Science at New York University. In 1998 he received a BEng in Electronics from the Universidad Simón Bolívar in Caracas, Venezuela, and in 2003 he earned a doctorate in Electronic Engineering at Queen Mary, University of London. Juan’s expertise is in digital signal processing, machine listening and music information retrieval, topics that he teaches and in which he has published more than 150 papers and articles in books, journals and conference proceedings. Since 2016, he is the director of the Music and Audio Research Lab (MARL), a multidisciplinary research center at the intersection of science, technology, music and sound. Between 2019-2022 He was also the director of the NYU Center for Urban Science and Progress (CUSP). A fellow of the IEEE and a Fulbright scholar, his work has been supported by public and private institutions in Venezuela, the UK, and the US, including Frontier and CAREER awards from the National Science Foundation.
Abstract: The automatic symbolization of recorded music has long been a driver of music information retrieval (MIR) research, with the choice of symbols such as notes, chords, beats, sections, instrumental and genre labels, having great influence in how we define tasks, design systems, label data and deploy applications. Yet, the means by which the field tries to achieve symbolization have evolved over the past two decades, a story that I will try to recreate in three parts using examples from my own work. First, I will illustrate how, in early MIR work, the challenge of symbolization was primarily the design and parametrization of features, resulting in highly specialized and heterogeneous methods. Second, and as the field adopted supervised deep learning approaches with minimal to no specialization, I will show how the challenge of symbolization shifted to data curation and labeling, with system design now focusing on overcoming scarcity and bias in labeled data. Third, I will discuss how in the last few years, machine listening research including MIR, has used self-supervised learning, especially multimodal approaches, to reduce the need for labeled data and increase the generalizability of feature representations. I will finalize by arguing that these methods also open the door to a new space of multimodal MIR, less focused on music recordings and notation, and more on gestures, technique and the physical experience of performing music.
Short Bio: Juan Pablo Bello is a Professor of Music Technology, Computer Science & Engineering, Electrical & Computer Engineering, and Urban Science at New York University. In 1998 he received a BEng in Electronics from the Universidad Simón Bolívar in Caracas, Venezuela, and in 2003 he earned a doctorate in Electronic Engineering at Queen Mary, University of London. Juan’s expertise is in digital signal processing, machine listening and music information retrieval, topics that he teaches and in which he has published more than 150 papers and articles in books, journals and conference proceedings. Since 2016, he is the director of the Music and Audio Research Lab (MARL), a multidisciplinary research center at the intersection of science, technology, music and sound. Between 2019-2022 He was also the director of the NYU Center for Urban Science and Progress (CUSP). A fellow of the IEEE and a Fulbright scholar, his work has been supported by public and private institutions in Venezuela, the UK, and the US, including Frontier and CAREER awards from the National Science Foundation.