Investigating the Effect of Multimodal Eye-Tracking on Real-Time L2 Pragmatic Competence in Telepresence Communication

Project Abstract

This study investigates how multimodal eye-tracking data can be utilized to enhance real-time L2 pragmatic competence within telepresence communication environments. Leveraging a mixed-methods design, we examine whether integrating gaze patterns, pupil dilation, and fixation sequences into real-time feedback mechanisms improves learners’ ability to perform contextually appropriate speech acts, manage discourse structure, and interpret interlocutor intent across telepresence scenarios. The research deploys a controlled experiment with intermediate-to-advanced L2 learners interacting in simulated telepresence meetings, incorporating eye-tracking-enabled prompts and adaptive feedback that respond to learners’ multimodal cues. Data collection combines quantitative metrics—accuracy and speed of pragmatic realizations, incidence of floor-taking and turn-taking errors, error rates in illocutionary force, and nonverbal coordination measures—with qualitative insights from stimulated recall interviews and discourse analysis of interaction transcripts. The study also assesses sociolinguistic variables such as register appropriateness, politeness strategies, and tactful refusals, examining how learners adjust to interlocutor feedback under varying communicative pressures (e.g., task complexity, time constraints, and cultural distance). A key aim is to determine whether real-time, multimodal feedback grounded in gaze and physiological indicators accelerates the acquisition of pragmatic norms beyond gains achieved by text-based or audio-only feedback. We explore the reliability and validity of eye-tracking signals as proxies for cognitive load, attention allocation, and communicative intent, and we examine potential individual differences, including prior exposure to telepresence technologies, working memory capacity, and motivation. The theoretical framework integrates pragmatics, second language acquisition, and information-processing accounts of gaze behavior, drawing on Conversation Analysis for interactional structure and Dynamic Assessment for learning potential. The intervention comprises three conditions (i) baseline telepresence communication with standard feedback; (ii) eye-tracking-informed feedback with gaze-driven prompts; and (iii) enhanced multimodal feedback combining gaze data with affective indicators and prosodic cues. We hypothesize that multimodal feedback will yield statistically significant improvements in pragmatic appropriateness, faster repair of miscommunications, and more fluent negotiation of meaning in real-time interactions. Additionally, we investigate transfer effects to offline tasks such as written and spoken pragmatics, and assess learner perceptions regarding the usefulness, intrusiveness, and trust in eye-tracking-guided feedback. The study contributes to the literature on computer-mediated pragmatics, L2 telepresence, and educational data mining by offering a principled account of how multimodal physiological signals can be translated into actionable instructional feedback. Findings are expected to inform the design of adaptive CALL systems and teleconferencing platforms that support pragmatic competence, with implications for language teacher training, assessment practices, and inclusive telepresence curricula. Policy implications regarding ethical use of biometric data and data privacy in educational technologies are addressed to guide responsible deployment in classroom and remote learning contexts.

Project Overview

What This Project Is About

---

A straightforward look at how people use eye movements and other cues while speaking a second language in real-time conversations. The project asks whether tracking eye gaze and other signals can reveal patterns that help learners understand and respond appropriately in practical conversations (like asking for directions, negotiating, or clarifying meaning) when communicating through video or telepresence tools.

---

---

The Problem It Addresses

---

Many second-language learners struggle with real-time pragmatics—knowing how to say things at the right time and tone. Traditional teaching focuses on grammar and vocabulary but glosses over how eye contact, facial cues, and timing affect meaning. This project explores whether measuring where learners look and how they react can improve real-time speaking skills and overall communication effectiveness.

---

---

Objectives of the Project

---

1. Identify key eye-tracking patterns linked to successful L2 pragmatic use in telepresence talks.
2. Explain how multimodal cues influence real-time language choices.
3. Develop practical guidelines or exercises to improve L2 pragmatics for online interactions.
4. Evaluate whether feedback based on gaze and other signals helps learners adjust strategies.

---

---

What You Will Do Step by Step

---

1. Review basic concepts of eye-tracking and telepresence in language learning. 2. Design simple tasks that require real-time L2 responses in video chats. 3. Collect data on learners’ gaze patterns and responses during tasks. 4. Analyze how gaze correlates with successful pragmatic choices. 5. Compare performance before and after targeted practice. 6. Interpret findings and suggest practical activities. 7. Write up results with clear implications for teaching. 8. Reflect on limitations and future work.

---

---

Expected Outcome

Anticipated outcomes include a better understanding of how eye movements and multimodal cues relate to pragmatic success in L2 conversation, and practical teaching tips that can be used in classrooms or online tutoring to help learners communicate more effectively.