HydraView.

Improving Viewpoint-Invariance and Temporal Consistency for Action Detection

Yannick Porto1,2, Renato Martins1, Thomas Chalumeau2, Cédric Demonceaux1,
1Université Bourgogne Europe, 2TEB Group
Code
Overview of our method.

Overview of our temporal action detection method. For each view, an untrimmed sequence is passed to our spatio-temporal encoder to generate features for each window of time. These features are then refined by our multi-view and multi-scale temporal encoder before being projected to find the beginning and end of each action over time.

Abstract

Temporal Action Detection (TAD) necessitates the precise recognition and localization of actions within untrimmed videos. Current approaches predominantly focus on single-view systems, which are constrained by a single perspective during both training and inference. Furthermore, existing skeleton-based methods typically rely on local temporal windows, often failing to capture long-range dependencies between these windows. To address these limitations, this paper introduces a novel multi-view framework designed to leverage complementary perspectives for more accurate action boundary detection. Our method employs a specialized encoder to extract motion features from localized temporal windows. These features are then integrated by HydraView, a multi-view and multi-scale temporal encoder that aggregates information across different perspectives to perform frame-level action detection. To mitigate the high computational overhead associated with managing long sequences in multi-view systems, we build HydraView upon the recent Mamba architecture, ensuring linear scaling and reduced inference time. Experimental results demonstrate that our approach outperforms several state-of-the-art TAD models on the BABEL and PKU-MMD datasets. Our code and pre-trained models will be made publicly available.

Method

Window feature encoding.

SWGCN. Motion features are learned by small time windows for each viewpoint. The encoder model (SWGCN) is then frozen to extract window features with improved temporal consistency.

Temporal Encoding with HydraView.

Hydraview. Multi-view Multi-Scale Learning for Temporal Encoding with HydraView. HydraView is composed of multiple ViewMamba blocks each responsible of a specific scale in views and time. The ViewMamba block is a 2D convolution followed by a dilation module scaling the input for SS2D. The representation from each ViewMamba block is then processed by a Multi-Scale Fuser to provide the action predictions.

Scanning strategies.

ViewScan. Designed bidirectional scanning strategies along both view and temporal dimensions. This scheme shows actions seen from two viewpoints, and final empty cells mean a missing skeleton detection.

Results

Results on PKUMMD.

PKUMMD. Results on two sequences from PKUMMDv1 (cross-subject split). Our model effectively scales to long-form sequences (up to 4 minutes) involving a large action vocabulary (51 classes). For visual clarity, we aggregate ``others'' instances into a single category, though they retain their individual class labels for evaluation purposes. The results demonstrate the model's ability to maintain temporal consistency across extended durations.

Results on BABEL.

BABEL. Results on two sequences from BABEL (split 1). The predictions indicate the classification for each temporal window. Each action instance is correctly labeled. While some Detections exhibit minor temporal shifts at the boundaries, they maintain an Intersection over Union (IoU) exceeding 50%.