Adaptive Accompaniment System for Solo Instrumentation Using Real-Time Music Emotion Recognition

Project Abstract

This study presents the design, implementation, and evaluation of an adaptive accompaniment system for solo instrumental performance leveraging real-time music emotion recognition (MER) to dynamically tailor accompaniment styles, timing, and dynamics to the performer’s expressive state. The system integrates signal processing, machine learning, and music theory to infer affective dimensions such as valence, arousal, and mood from live audio input and contextual cues, enabling immediate modulation of harmonic progressions, rhythmic feel, articulation, and dynamic contour. We introduce a multimodal MER framework that fuses spectral features, tempo and meter estimation, articulation cues, and learned representations from a curated dataset of expressive performances across genres. A lightweight real-time inference engine, deployed on a consumer-grade workstation with low-latency audio I/O, ensures response times suitable for live performance, preserving musical coherence and interpretive intent. Methodologically, the project comprises (i) data collection and annotation of expressive solo performances with aligned accompaniment styles, (ii) feature extraction pipelines for real-time MER including mel-frequency cepstral coefficients, chroma features, onset detection, and prosodic cues, (iii) development of a supervised learning model capable of predicting continuous emotion dimensions and discrete mood states, (iv) design of a flexible accompaniment generator that maps emotion predictions to adaptive musical parameters such as tonality, progression borrowing, harmonic rhythm, dynamics, and articulation, (v) design of a low-latency interface that synchronizes accompaniment with solo timing and expressive rubato, and (vi) a perceptual evaluation protocol involving professional musicians to assess musicality, synchronization, and perceived expressiveness. Results from extensive simulations and live audition experiments demonstrate that the adaptive system improves perceived synchronization and emotional congruence between solo performance and accompaniment by a statistically significant margin compared to fixed or rule-based accompaniment baselines. The MER component achieves competitive accuracy in estimating arousal and valence in real-time, with robust performance under diverse instrumentations and acoustic environments. The accompaniment generator maintains musical coherence while adapting harmonic pace, voice-leading, and dynamics in response to detected emotional states, thereby enabling nuanced collaborative performance without imposing rigid scripts on the performer. User studies reveal enhanced interpretive flexibility, reduced cognitive load for performers, and positive reception of the system’s ability to reflect expressive intent. Limitations include dependency on high-quality audio capture, potential drift in emotion estimation during rapid tempo changes or extreme dynamics, and the necessity for domain-specific tuning to accommodate varied performance styles. Future work will explore unsupervised and few-shot learning to broaden generalization across genres, integration with stage-level latency compensation techniques, and extended evaluation in ensemble settings. The research contributes a practical blueprint for real-time affect-aware accompaniment systems, offering a pathway to more expressive and responsive human–computer musical collaboration.

Project Overview

What This Project Is About

---

A simple system that helps a solo musician by listening to what they play and automatically adding accompaniment that fits the mood and tempo in real time. The project explores how music emotion (feelings like happy, sad, or tense) can be detected from audio and how an intelligent system can respond with appropriate chords, rhythms, and melodies.

---

---

The Problem It Addresses

---

Musicians practicing alone often lack immediate, responsive accompaniment to practice with. Existing tools may be static or require complex setups. This project aims to create a responsive, easy-to-use assistant that enhances solo performance and practice by providing adaptive backing that aligns with the performer’s expressive intent.

---

---

Objectives of the Project

---

1. Identify musical emotion from live audio in real time.
2. Generate suitable accompaniment that matches tempo, key, and mood.
3. Keep latency low so the accompaniment feels natural.
4. Provide a simple user interface for setup and control.
5. Evaluate the system with real performers and collect feedback.

---

---

What You Will Do Step by Step

---

1. Study basics of audio processing and emotion cues in music.
2. Collect sample performances and label them by mood.
3. Develop a lightweight mood-detection model using features like tempo, dynamics, and harmony.
4. Design a rule-based or small neural system to choose accompaniment patterns.
5. Implement a real-time audio pipeline that streams input and outputs backing tracks.
6. Test with musicians, refine latency and musical fit, and gather feedback.

---

---

---

Expected Outcome

An accessible prototype that can accompany a solo instrument in real time, with convincing mood-aligned backing and practical guidance for further development or classroom use.