AI-based Traffic Anomaly Analysis

We developed a system to detect, reason about, and understand traffic anomalies in video. It is powered by multiple large multimodal language models (MLLMs), including OpenAI's ChatGPT, Google's Gemini, and our self-developed Qwen-VL-based model. These models can reason about both visual and textual information to identify anomalies and explain what happened.

The system has two variants for different deployment priorities. The accuracy-focused framework uses multiple agents to handle different parts of a question, decomposing complex traffic-analysis tasks into several steps before producing an evidence-based answer. The speed-focused framework prioritises inference efficiency and can run real-time streaming monitoring on NVIDIA Jetson Orin edge devices.

The framework supports applications such as continuous traffic anomaly monitoring and automated incident reporting. For example, when a collision is detected, it can send a structured report to a traffic control dashboard with an explanation of what happened and the evidence supporting it. The accuracy-focused system won the runner-up award in Track 3 of the NVIDIA AI City Challenge.

During the project, we collaborated with the City of Carmel, Indiana, to collect traffic anomaly data and create Roundabout-TAU. We open-sourced this dataset to support future research into traffic anomaly detection, reasoning, and understanding.