3D Mapping of Facial Muscle Activation during Expression Recognition using Ultrasonography and Machine Learning

Project Abstract

This study presents a multidisciplinary approach to decoding facial expressions by mapping 3D muscle activation patterns using ultrasonography integrated with machine learning algorithms. The core aim is to establish a robust, non-invasive framework that links dynamic myoelastic activity of facial muscles to recognizable affective and cognitive states, enabling real-time interpretation of expressions with high spatial-temporal fidelity. Ultrasonography provides millimeter-scale resolution of superficial and deep facial muscles in vivo, capturing subtle temporal changes that accompany micro-expressions, social interactions, and affective states. The collected data comprise synchronized 2D ultrasonographic image sequences and high-frame-rate video of facial expressions elicited through standardized affective stimuli, posed tasks, and spontaneous interactions. To translate rich imaging data into meaningful activation maps, a multi-stage pipeline is developed (i) preprocessing including motion compensation, speckle noise reduction, and normalization across subjects; (ii) segmentation and tracking of target muscles using semi-supervised deep learning models calibrated with a curated anatomical atlas; (iii) reconstruction of 3D muscle geometry and deformation fields through a fusion of ultrasonography with optical tracking, enabling voxel-wise quantification of muscle strain and activation likelihood over time; (iv) feature extraction that combines kinematic descriptors, texture-based cues, and muscle activation signatures; (v) machine learning classification and regression models—ranging from convolutional neural networks to temporal sequence models like LSTMs and transformers—that map activation patterns to a predefined set of basic emotions and compound expressions. The study also investigates the stability of the mappings across individuals with diverse ages, genders, and facial morphology, employing transfer learning and domain adaptation to mitigate inter-subject variability. A crucial component is the development of explainable AI techniques to interpret model decisions, including saliency maps aligned with anatomical muscle groups and perturbation analyses that assess the causal contribution of specific muscles to predicted expressions. Validation occurs through cross-subject and time-sliced experiments, benchmarking against established facial expression corpora and electromyography (EMG) where available, while also exploring potential cross-modal synergy with rgb-depth imaging and voice prosody in multimodal emotion recognition. The anticipated outcomes include a publicly accessible annotated dataset of synchronized ultrasonography and expression labels, a suite of open-source tools for 3D muscle activation reconstruction, and validated models capable of real-time inference on standard hardware. This research advances our understanding of the biomechanical underpinnings of facial communication and paves the way for applications in clinical assessment of neuromuscular disorders, human-computer interaction, affective computing, and enhanced nonverbal communication analytics in fields such as psychology, cognitive neuroscience, and social robotics. Limitations and ethical considerations are addressed, including subject comfort during ultrasonography, data privacy, and potential misuse of affective surveillance technologies, with recommendations for governance and responsible deployment.

Project Overview

What This Project Is About

---

A plain-language overview of the topic and what the project investigates.

---

---

The Problem It Addresses

---

What problem or gap this project tackles and why it matters to the field or society.

---

---

Objectives of the Project

---

1. Understand how facial muscles move during expressions
2. Use ultrasound to visualize underlying muscle activity
3. Apply machine learning to recognize patterns linked to expressions
4. Develop a 3D map that links muscle signals to visible facial changes
5. Evaluate the accuracy and reliability of the mapping approach

---

---

What You Will Do Step by Step

---

1. Learn basic anatomy of facial muscles and how ultrasound works
2. Set up a simple ultrasound recording session with volunteers
3. Acquire motion data while participants perform expressions
4. Process images to extract muscle movement signals
5. Train a basic machine learning model to classify expressions
6. Combine signals into a 3D activation map
7. Validate maps against known reference movements
8. Document methods, results, and potential limitations

---

---

Expected Outcome

A clear, user-friendly 3D map showing which facial muscles activate for common expressions, along with notes on accuracy and potential educational or clinical uses.