G-TAD: Sub-Graph Localization for Temporal Action Detection

Temporal action detection is a fundamental yet challenging task in video understanding. Video context is a critical cue to effectively detect actions, but current works mainly focus on temporal context, while neglecting semantic context as well as other important context properties. In this work, we propose a graph convolutional network (GCN) model to adaptively incorporate multi-level semantic context into video features and cast temporal action detection as a sub-graph localization problem. Specifically, we formulate video snippets as graph nodes, snippet-snippet correlations as edges, and actions associated with context as target sub-graphs. With graph convolution as the basic operation, we design a GCN block called GCNeXt, which learns the features of each node by aggregating its context and dynamically updates the edges in the graph. To localize each sub-graph, we also design an SGAlign layer to embed each sub-graph into the Euclidean space. Extensive experiments show that G-TAD is capable of finding effective video context without extra supervision and achieves state-of-the-art performance on two detection benchmarks. On ActivityNet-1.3, it obtains an average mAP of 34.09%; on THUMOS14, it reaches 51.6% at IoU@0.5 when combined with a proposal processing method. G-TAD code is publicly available at https://github.com/frostinassiky/gtad.

PDF Abstract CVPR 2020 PDF CVPR 2020 Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Temporal Action Localization ActivityNet-1.3 G-TAD mAP IOU@0.5 50.36 # 23
mAP 34.09 # 26
mAP IOU@0.75 34.60 # 20
mAP IOU@0.95 9.02 # 8
Temporal Action Localization EPIC-KITCHENS-100 G-TAD (verb) Avg mAP (0.1-0.5) 9.4 # 5
mAP IOU@0.1 12.1 # 5
mAP IOU@0.2 11.0 # 5
mAP IOU@0.3 9.4 # 5
mAP IOU@0.4 8.1 # 5
mAP IOU@0.5 6.5 # 5
Temporal Action Localization FineAction G-TAD (i3d feature) mAP 9.06 # 6
mAP IOU@0.5 13.74 # 4
mAP IOU@0.75 8.83 # 4
mAP IOU@0.95 3.06 # 4
Temporal Action Localization THUMOS’14 G-TAD mAP IOU@0.5 40.2 # 28

Methods