Adaptive Real-Time Music Genre Classification Using Transformer-Based Auditory Perception Models

Project Abstract

Adaptive Real-Time Music Genre Classification Using Transformer-Based Auditory Perception Models investigates a robust framework for instantaneous genre identification from live audio streams. This work addresses the challenge of real-time performance where latency, accuracy, and generalization must be balanced across diverse musical contexts, including cross-genre blends, streaming constraints, and noisy environments. We propose a hierarchical transformer architecture inspired by auditory perception principles, combining a learned mel-spectrogram frontend with selective attention mechanisms to simulate human perceptual focus on salient timbral and rhythmic cues. The model leverages a multi-scale temporal encoding strategy to capture modulations from sub-second rhythms to sustained tonal textures, enabling resilient genre discrimination even under tempo and key variations. Key contributions include the design of an online inference pipeline that orchestrates streaming feature extraction, windowed self-attention, and incremental classification with bounded latency. We introduce a novel dynamic frame-skipping policy that adapts to computational load while maintaining reliability of genre predictions, and a cross-domain adaptation module trained with self-supervised objectives to enhance performance on unseen datasets and in-the-wild recordings. The system integrates uncertainty-aware decision fusion to mitigate misclassifications in noisy segments, producing probabilistic genre estimates with calibrated confidence scores suitable for downstream applications such as adaptive playlisting, live DJ systems, and real-time music recommendation engines. To evaluate effectiveness, we curate a benchmark that encompasses multiple datasets with varying recording conditions, instrumentations, and production qualities, including studio tracks, live performances, and user-generated content. We benchmark against state-of-the-art baselines on metrics capturing accuracy, latency, and robustness to noise and domain shift. Experimental results demonstrate a substantial improvement in real-time genre accuracy while maintaining sub-200 ms end-to-end latency on standard hardware, with pronounced gains in complex mixtures and overlapping genres. Ablation studies reveal the critical roles of multi-scale temporal encoding, spectral-aware attention, and uncertainty fusion in achieving stable performance across challenging scenarios. Additionally, we examine deployment considerations such as model compression via knowledge distillation and quantization, ensuring compatibility with edge devices and streaming platforms. A qualitative analysis of error cases provides insights into genre boundary ambiguity and the influence of production techniques on perceptual cues, informing avenues for future enhancements in perceptual modeling and dataset curation. Overall, the work advances real-time music genre classification by integrating transformer-based perceptual abstractions with practical streaming constraints, delivering accurate, low-latency, and robust genre inference suitable for dynamic, real-world audio environments.

Project Overview

What This Project Is About

---

A straightforward exploration of how computer models can listen to music and guess its genre in real time. The project combines how we represent sound with a type of machine learning model that can focus on important parts of the sound as it streams.

---

---

The Problem It Addresses

---

Many music systems struggle to identify genres quickly and accurately when the song is playing live or with varying audio quality. There is a need for models that can adapt to new songs and still classify genres as the music plays, without long delays.

---

---

Objectives of the Project

---

1. Understand how musical features relate to genres in streaming data.
2. Implement a lightweight transformer-based model suitable for real-time processing.
3. Evaluate accuracy and latency across different genres and audio qualities.
4. Explore how to adapt the model to new or evolving music styles.
5. Provide practical guidelines for deploying the system in apps or services.

---

---

What You Will Do Step by Step

---

1. Study basic audio concepts and genre labeling basics.
2. Collect or build a streaming-friendly dataset with labeled genres.
3. Convert audio into simple, informative features suitable for real-time use.
4. Build and train a transformer-based model optimized for speed.
5. Test the model on live or simulated streaming data to measure latency and accuracy.
6. Analyze errors to understand which genres are confused and why.
7. Experiment with lightweight optimizations for faster inference.
8. Prepare a short guide on how to deploy the system in simple apps.

---

---

Expected Outcome

A functioning, real-time music genre classifier that can infer genres with low delay and reasonable accuracy, along with a practical deployment plan and insights into its limitations and future improvements.