Automated Mood-Responsive Composition System Using Deep Learning and Spectral Features for Live Performance

Project Abstract

This research presents an automated system that composes mood-responsive music in real time for live performance by leveraging deep learning and spectral feature analysis. The core objective is to translate perceived emotional states of a performer or audience into musically coherent and expressive compositions that adapt dynamically to the evolving context of a performance. We integrate a multimodal pipeline consisting of perceptual emotion estimation, feature extraction from audio signals, and a generative core that ensures musicality, structure, and responsiveness. The emotion estimation module fuses acoustic cues (tonal center, tempo, timbre, loudness, spectral flux, and rhythm patterns) with contextual indicators (scene metadata, performer gestures, and audience engagement proxies) to infer discrete or dimensional mood representations (valence, arousal, and possible arousal gradients). These mood signals feed the spectral-informed generative model, which utilizes a hybrid architecture combining neural networks for high-level musical planning with rule-based constraints that preserve stylistic coherence with the chosen genre and live performance constraints. Spectral features such as chroma, mel-frequency cepstral coefficients (MFCCs), spectral centroid, bandwidth, and defeat-free spectral contrast are employed to capture timbral and harmonic progressions that characteristically encode emotional content. The generative component comprises a variational autoencoder conditioned on mood vectors and a recurrent decoder that languages musical phrases across scales, rhythms, and textures. A dynamic orchestration layer maps generated material to a virtual ensemble, enabling instrument-specific articulations, dynamic expressions, and real-time tempo adjustments to align with the performer’s tempo fluctuations. The system emphasizes low-latency inference to maintain real-time interaction and uses an adaptive planning horizon to balance musical coherence with spontaneity, ensuring that emergent motifs are developed while avoiding jarring transitions. We evaluate the system through a mixed-methods study involving expert musicians and audiences in live settings, complemented by objective metrics for musicality (tonal stability, harmonic progression plausibility, rhythmic regularity) and responsiveness (latency, fit to mood, and adaptability to sudden mood shifts). A controlled corpus featuring staged mood trajectories across genres provides a baseline for quantitative assessment, while qualitative feedback from performers informs interface usability, perceived expressiveness, and the perceived alignment between the mood input and the generated music. Results indicate that the mood-conditioned generation approach enhances interpretive depth and audience immersion, with statistically significant improvements in perceived coherence, emotional congruence, and performative spontaneity compared to non-mood-aware baselines. The study also identifies constraints related to dataset diversity, real-time conditioning bandwidth, and the need for genre-specific tuning to preserve stylistic integrity. The research contributes a scalable framework for live, mood-responsive composition that integrates perceptual emotion modeling, spectral feature-driven analysis, and conditioned generative synthesis. It lays groundwork for future exploration of adaptive accompaniment systems, broadcast-ready improvisation engines, and interactive performance interfaces that bridge human expression with machine-assisted creativity.

Project Overview

What This Project Is About

---

A beginner-friendly overview of creating an automatic system that helps compose music based on mood. The project explores how to read music-like signals from audio, decide the mood, and then generate or adapt melodies and harmonies to match that mood using simple computer techniques.

---

---

The Problem It Addresses

---

Musicians and venues often need music that fits a moment or atmosphere. Creating mood-appropriate pieces manually can be time-consuming and requires skill. This project looks for a practical way to assist performers by automatically shaping music to a chosen mood in real time.

---

---

Objectives of the Project

---

1. Understand how audio features relate to mood in music.
2. Build a simple model that classifies mood from audio features.
3. Create a basic system that uses mood to guide melody and harmony choices.

---

---

What You Will Do Step by Step

---

1) Learn fundamental audio features (like loudness and tempo).

2) Collect or simulate short music samples labeled by mood.

3) Train a basic mood classifier using these features.

4) Design a simple music generator that adjusts notes and chords based on mood outputs.

5) Test the system with live or prerecorded audio and refine the results.

---

---

Expected Outcome

The project should deliver a working prototype that can listen to audio, predict a mood, and produce or adapt music in that mood, along with a short guide on how to use it in practice.