Analysis of Phoneme-based Audio to MIDI Translation for Live Music Performance using Deep Learning

Project Abstract

This study presents a novel deep learning framework for phoneme-based audio to MIDI translation tailored to live music performance, addressing the challenge of real-time articulation, timbre, and expressive control in automated music transcription. We propose a multi-stream neural architecture that jointly models linguistic content (phonemes) and prosodic cues (intonation, rhythm, dynamics) extracted from vocal and instrument sounds, enabling high-fidelity conversion of expressive performance into MIDI sequences suitable for synthesis. The core model integrates a phoneme-aware encoder with a temporal alignment mechanism and a MIDI decoder that forecasts note events, velocities, and control changes with low latency. To handle ambiguities in polyphonic textures and speaker/instrument variability, we introduce a probabilistic decoding layer that estimates uncertainty and enables robust performance across genres. The dataset comprises extensive annotated recordings across voice and multi-instrument ensembles, augmented with synthetic phoneme labels using a hybrid alignment strategy that fuses forced alignment and phoneme-aware self-supervision. We implement a real-time inference pipeline optimized for low CPU/GPU overhead, leveraging quantization and model pruning without sacrificing musical nuance. Our evaluation framework combines objective metrics—note-onset F1, pitch accuracy, onset-to-velocity correlation, and MIDI control fidelity—with perceptual listening tests conducted by professional musicians to assess expressiveness, timing precision, and timbral appropriateness. Results demonstrate significant improvements over traditional audio-to-MIDI baselines, particularly in capturing expressive timing (rubato), dynamic shading, and vibrato/ornamentation as explicit MIDI parameters. The model shows robustness to acoustic noise and room reverberation, aided by data augmentation strategies that simulate live performance environments. We conduct ablation studies to quantify the contribution of phoneme conditioning, temporal alignment, and probabilistic decoding to overall musical quality. A key contribution is the introduction of a versatile loss function that balances transcription accuracy with musical plausibility, including a perceptual loss component aligned with human judgments of musicality. The framework supports end-to-end training and can be extended to language-agnostic phoneme sets to accommodate multilingual vocal performances. The study also explores integration with stage-ready synthesizers and live-looping systems, enabling performers to generate MIDI-driven accompaniments in real time while maintaining natural phrasing and articulation. Potential applications span live performance augmentation, music education, and accessibility tools for hearing-impaired musicians. Limitations include handling highly complex polyphony with rapid note changes and the reliance on high-quality phoneme annotations; future work will seek unsupervised or weakly supervised alternatives and cross-domain transfer to non-vocal instrumental timbres. Overall, the research advances the feasibility of phoneme-informed MIDI translation for expressive live music, offering a scalable, real-time solution that bridges linguistic cues and musical syntax.

Project Overview

What This Project Is About

---

This project explores how spoken phonemes from a voice or singing input can be translated into musical notes and timing (MIDI) so that live performances can be recreated or assisted by computer models. The aim is to map how sounds relate to musical structure and convert that into a playable digital score in real time or near real time.

---

---

The Problem It Addresses

---

Current music transcription often requires ear-trained humans or complicated workflows. This project seeks a scalable AI-based approach that can interpret natural speech or singing and generate accurate MIDI data, enabling faster prototyping for composers, performers, and assistive music tech. It addresses gaps in real-time adaptability, portability, and accessibility for non-experts.

---

---

Objectives of the Project

---

1. Understand how phonemes relate to musical notes, rhythm, and timing.
2. Build a simple model that converts phoneme sequences into MIDI data.
3. Evaluate the accuracy of the generated MIDI against ground truth.
4. Test in a live-performance scenario to assess latency and reliability.
5. Provide a practical toolkit or pipeline for further development.

---

---

What You Will Do Step by Step

---

1) Review basic concepts of phonemes, MIDI, and audio-to-score ideas. 2) Collect a small dataset of spoken/sung phoneme sequences paired with MIDI outputs. 3) Design a straightforward neural model that predicts notes and timing from phoneme input. 4) Train and validate the model, adjust for latency. 5) Implement a simple live-demo where spoken input becomes MIDI playback. 6) Analyze results, identify limitations, and suggest improvements.

---

---

Expected Outcome

A functional demonstration showing phoneme-to-MIDI translation with measurable accuracy and latency metrics, plus a basic, shareable toolkit for similar experiments.