Smart City Traffic Anomaly Detection Using Real-Time Multi-Modal Data Fusion and Explainable AI

Project Abstract

This study presents a novel framework for detecting traffic anomalies in urban environments by fusing real-time multi-modal data streams and leveraging explainable artificial intelligence to provide transparent, actionable insights for traffic management. The proposed system integrates heterogeneous data sources, including CCTV video streams, vehicle GPS traces, road-side sensor data (inductive loop detectors, acoustic sensors, weather stations), and social-computing signals from crowdsourced traffic apps, to capture both macroscopic flow patterns and microscopic vehicular behaviors. A unified data architecture is developed to address asynchronous sampling, missing data, and varying data quality across modalities, employing a streaming data pipeline with real-time preprocessing, feature extraction, and alignment using time-warped cross-modal embeddings. At the core, a modular anomaly detection engine combines supervised, semi-supervised, and unsupervised learning components to identify deviations from normative traffic patterns under diverse conditions such as incidents, adverse weather, and infrastructural faults. The detection module utilizes a multi-task attention-based neural network that fuses spatial-temporal features from video analytics (object detection, trajectory reconstruction), trajectory clustering, and sensor-derived indicators (speed, occupancy, flow, anomaly scores) to produce granular anomaly scores at road segment and intersection levels. To ensure robustness, the system incorporates probabilistic forecasting and scenario-based modelling to distinguish transient fluctuations from persistent anomalies, enabling early warning and adaptive traffic control interventions. Explainability is embedded through model-agnostic interpretability tools and intrinsic attention maps that highlight which modalities and geographic zones contributed most to a detected anomaly, along with counterfactual explanations suggesting potential mitigation actions (e.g., rerouting, signal timing adjustments). The research investigates the impact of data fusion strategies, temporal granularity, and sensor reliability on anomaly detection performance, conducting extensive experiments on a large-scale urban dataset synthesized from real-world city deployments and simulated incident scenarios. Evaluation metrics include precision, recall, F1-score, detection latency, false alarm rate, and interpretability scores based on user feedback from traffic operators. A comparative analysis against baseline approaches—single-modality detectors, late fusion models, and traditional statistical methods—demonstrates that multi-modal fusion with explainability substantially improves detection accuracy and reduces operator cognitive load. The work also explores deployment considerations such as edge computing deployment for latency-critical tasks, data privacy and security, and scalability to city-wide networks. Findings indicate that real-time multi-modal fusion enhances situational awareness, enabling proactive traffic management and rapid incident response, while maintainable explanations foster trust and adoption by non-technical stakeholders. The research contributes a generalized blueprint for building explainable anomaly detection systems in smart cities, including data schemas, processing pipelines, model architectures, evaluation methodologies, and a set of guidelines for integrating such systems with existing traffic management centers. The developed framework is extensible to other urban sensing domains, offering a pathway toward more resilient and intelligent transportation ecosystems.

Project Overview

What This Project Is About

---

A simple, hands-on exploration of detecting unusual traffic patterns in a city using different data sources. The project investigates how live data from cameras, road sensors, and GPS-enabled devices can be combined to spot anomalies like sudden congestion, unusual speeds, or unexpected road closures, and explain why these events happen in plain terms.

---

---

The Problem It Addresses

---

Cities rely on timely detection of traffic problems, but data is scattered and noisy. Relying on one source can miss events or misinterpret normal variations. This project aims to create a unified approach that fuses multiple data types to improve detection accuracy and provide clear explanations for why an anomaly was flagged.

---

---

Objectives of the Project

---

1. Identify the main data sources used in modern traffic monitoring (video, sensors, GPS).
2. Design a simple pipeline to combine these data sources into a common view.
3. Develop a lightweight anomaly detection method suitable for real-time use.
4. Provide human-friendly explanations for detected anomalies.
5. Evaluate the approach on a small city dataset and compare with single-source methods.

---

---

What You Will Do Step by Step

---

1) Gather publicly available traffic data from cameras, sensors, and GPS traces. 2) Preprocess data to a common format. 3) Create a fusion model that combines signals from all sources. 4) Implement a simple anomaly detector that runs in real time. 5) Generate explanations that describe why an alert was raised in plain language. 6) Test the system on sample scenarios and refine thresholds. 7) Document limitations and potential improvements.

---

---

Expected Outcome

An easy-to-use prototype that detects traffic anomalies using multiple data sources and explains alerts clearly. The project should demonstrate improved detection over single-source methods and provide insights into how different data streams contribute to each alert.