Distribution Shift
UCF101-DS [Distribution Shift]
The UCF101-DS dataset focuses on real-world distribution shifts in user-generated videos without synthetic augmentation. It addresses the challenge of model robustness by exhibiting 63 distribution shifts categorized into 15 types, such as changes in actor, behavior, or scenery.
| No. of classes | No. of Videos | Distribution Shifts |
|---|---|---|
| 47 | 4708 | 63 |
A Large-scale Robustness Analysis of Video Action Recognition Models
Madeline Chantry, Naman Biyani, Prudvi Kamtam, Shruti Vyas, Hamid Palangi, Vibhav Vineet, and Yogesh Singh Rawat
CVPR 2023
@misc{schiappa2023largescalerobustnessanalysisvideo, title={Large-scale Robustness Analysis of Video Action Recognition Models}, author={Madeline Chantry Schiappa and Naman Biyani and Prudvi Kamtam and Shruti Vyas and Hamid Palangi and Vibhav Vineet and Yogesh Rawat}, year={2023}, eprint={2207.01398}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2207.01398}, }
Perturbation
HMDB51-P
The dataset introduces 90 different perturbations to the original HMDB51 dataset, including changes in video quality, motion, and appearance, to evaluate how well models can maintain performance under such conditions.
| No. of classes | No. of Videos | Perturbations |
|---|---|---|
| 51 | 137,610 | 90 |
A Large-scale Robustness Analysis of Video Action Recognition Models
Madeline Chantry, Naman Biyani, Prudvi Kamtam, Shruti Vyas, Hamid Palangi, Vibhav Vineet, and Yogesh Singh Rawat
CVPR 2023
@misc{schiappa2023largescalerobustnessanalysisvideo, title={Large-scale Robustness Analysis of Video Action Recognition Models}, author={Madeline Chantry Schiappa and Naman Biyani and Prudvi Kamtam and Shruti Vyas and Hamid Palangi and Vibhav Vineet and Yogesh Rawat}, year={2023}, eprint={2207.01398}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2207.01398}, }
UCF101-P
The dataset introduces 90 different perturbations to the original UCF101 dataset, including changes in video quality, motion, and appearance, to evaluate how well models can maintain performance under such conditions.
| No. of classes | No. of Videos | Perturbations |
|---|---|---|
| 101 | 340,380 | 90 |
A Large-scale Robustness Analysis of Video Action Recognition Models
Madeline Chantry, Naman Biyani, Prudvi Kamtam, Shruti Vyas, Hamid Palangi, Vibhav Vineet, and Yogesh Singh Rawat
CVPR 2023
@misc{schiappa2023largescalerobustnessanalysisvideo, title={Large-scale Robustness Analysis of Video Action Recognition Models}, author={Madeline Chantry Schiappa and Naman Biyani and Prudvi Kamtam and Shruti Vyas and Hamid Palangi and Vibhav Vineet and Yogesh Rawat}, year={2023}, eprint={2207.01398}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2207.01398}, }
Kinetics400-P
The dataset introduces 90 different perturbations to the original Kinetics400 dataset, including changes in video quality, motion, and appearance, to evaluate how well models can maintain performance under such conditions.
| No. of classes | No. of Videos | Perturbations |
|---|---|---|
| 400 | 1,616,670 | 90 |
A Large-scale Robustness Analysis of Video Action Recognition Models
Madeline Chantry, Naman Biyani, Prudvi Kamtam, Shruti Vyas, Hamid Palangi, Vibhav Vineet, and Yogesh Singh Rawat
CVPR 2023
@misc{schiappa2023largescalerobustnessanalysisvideo, title={Large-scale Robustness Analysis of Video Action Recognition Models}, author={Madeline Chantry Schiappa and Naman Biyani and Prudvi Kamtam and Shruti Vyas and Hamid Palangi and Vibhav Vineet and Yogesh Rawat}, year={2023}, eprint={2207.01398}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2207.01398}, }
SSv2-P
The dataset introduces 90 different perturbations to the original SSv2 dataset, including changes in video quality, motion, and appearance, to evaluate how well models can maintain performance under such conditions.
| No. of classes | No. of Videos | Perturbations |
|---|---|---|
| 174 | 2,229,930 | 90 |
A Large-scale Robustness Analysis of Video Action Recognition Models
Madeline Chantry, Naman Biyani, Prudvi Kamtam, Shruti Vyas, Hamid Palangi, Vibhav Vineet, and Yogesh Singh Rawat
CVPR 2023
@misc{schiappa2023largescalerobustnessanalysisvideo, title={Large-scale Robustness Analysis of Video Action Recognition Models}, author={Madeline Chantry Schiappa and Naman Biyani and Prudvi Kamtam and Shruti Vyas and Hamid Palangi and Vibhav Vineet and Yogesh Rawat}, year={2023}, eprint={2207.01398}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2207.01398}, }
MSRVTT-P
The dataset is designed to evaluate the robustness of video-language models against real-world perturbations. It introduces 90 visual and 35 textual perturbations to the original MSRVTT dataset to assess how existing models perform under various distribution shifts.
| Videos | Captions | Video-Text Pairs |
|---|---|---|
| 90K | 35K | 2,766K |
Robustness Analysis of Video-Language Models Against Visual and Language Perturbations
Madeline C. Schiappa, Shruti Vyas, Hamid Palangi, Yogesh S. Rawat, and Vibhav Vineet
NeurIPS 2022
@misc{schiappa2023robustnessanalysisvideolanguagemodels, title={Robustness Analysis of Video-Language Models Against Visual and Language Perturbations}, author={Madeline C. Schiappa and Shruti Vyas and Hamid Palangi and Yogesh S. Rawat and Vibhav Vineet}, year={2023}, eprint={2207.02159}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2207.02159}, }
You2Cook-P
The dataset is designed to evaluate the robustness of video-language models against real-world perturbations. It introduces 90 visual and 35 textual perturbations to the original You2Cook dataset to assess how existing models perform under various distribution shifts.
| Videos | Captions | Video-Text Pairs |
|---|---|---|
| 41K | 103K | 9,266K |
Robustness Analysis of Video-Language Models Against Visual and Language Perturbations
Madeline C. Schiappa, Shruti Vyas, Hamid Palangi, Yogesh S. Rawat, and Vibhav Vineet
NeurIPS 2022
@misc{schiappa2023robustnessanalysisvideolanguagemodels, title={Robustness Analysis of Video-Language Models Against Visual and Language Perturbations}, author={Madeline C. Schiappa and Shruti Vyas and Hamid Palangi and Yogesh S. Rawat and Vibhav Vineet}, year={2023}, eprint={2207.02159}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2207.02159}, }
Occlusion
O-UCF
The dataset is an occluded version of the original UCF24 test set, featuring both static and dynamic occlusions at varying severity levels. It is designed to evaluate the robustness of action detection models under different occlusion scenarios.
| No. of Videos | No. of Classes | No. of Instances |
|---|---|---|
| 20306 | 24 | 2284 |
On Occlusions in Video Action Detection: Benchmark Datasets And Training Recipes
Rajat Modi, Vibhav Vineet, and Yogesh Singh Rawat
NeurIPS 2023
@inproceedings{NEURIPS2023_b3640c2d, author = {Modi, Rajat and Vineet, Vibhav and Rawat, Yogesh}, booktitle = {Advances in Neural Information Processing Systems}, editor = {A. Oh and T. Naumann and A. Globerson and K. Saenko and M. Hardt and S. Levine}, pages = {57306–57335}, publisher = {Curran Associates, Inc.}, title = {On Occlusions in Video Action Detection: Benchmark Datasets And Training Recipes}, url = {https://proceedings.neurips.cc/paper_files/paper/2023/file/b3640c2d3e58f716c67066046318db0f-Paper-Datasets_and_Benchmarks.pdf}, volume = {36}, year = {2023} }
O-JHMDB
The dataset is an occluded version of the original JHMDB21 test set, featuring both static and dynamic occlusions at varying severity levels. It is designed to evaluate the robustness of action detection models under different occlusion scenarios.
| No. of Videos | No. of Classes | No. of Instances |
|---|---|---|
| 20306 | 24 | 2284 |
On Occlusions in Video Action Detection: Benchmark Datasets And Training Recipes
Rajat Modi, Vibhav Vineet, and Yogesh Singh Rawat
NeurIPS 2023
@inproceedings{NEURIPS2023_b3640c2d, author = {Modi, Rajat and Vineet, Vibhav and Rawat, Yogesh}, booktitle = {Advances in Neural Information Processing Systems}, editor = {A. Oh and T. Naumann and A. Globerson and K. Saenko and M. Hardt and S. Levine}, pages = {57306–57335}, publisher = {Curran Associates, Inc.}, title = {On Occlusions in Video Action Detection: Benchmark Datasets And Training Recipes}, url = {https://proceedings.neurips.cc/paper_files/paper/2023/file/b3640c2d3e58f716c67066046318db0f-Paper-Datasets_and_Benchmarks.pdf}, volume = {36}, year = {2023} }
OVIS-UCF
The dataset incorporates realistic occlusions from the OVIS dataset, superimposed onto the UCF-24 test set. This design simulates authentic occluder movements, providing a benchmark to evaluate the performance of action detection models under real-world occlusion scenarios.
| No. of Videos | No. of Classes | No. of Instances |
|---|---|---|
| 5896 | 21 | 928 |
On Occlusions in Video Action Detection: Benchmark Datasets And Training Recipes
Rajat Modi, Vibhav Vineet, and Yogesh Singh Rawat
NeurIPS 2023
@inproceedings{NEURIPS2023_b3640c2d, author = {Modi, Rajat and Vineet, Vibhav and Rawat, Yogesh}, booktitle = {Advances in Neural Information Processing Systems}, editor = {A. Oh and T. Naumann and A. Globerson and K. Saenko and M. Hardt and S. Levine}, pages = {57306–57335}, publisher = {Curran Associates, Inc.}, title = {On Occlusions in Video Action Detection: Benchmark Datasets And Training Recipes}, url = {https://proceedings.neurips.cc/paper_files/paper/2023/file/b3640c2d3e58f716c67066046318db0f-Paper-Datasets_and_Benchmarks.pdf}, volume = {36}, year = {2023} }
OVIS-JHMDB
The dataset incorporates realistic occlusions from the OVIS dataset, superimposed onto the JHMDB21 test set. This design simulates authentic occluder movements, providing a benchmark to evaluate the performance of action detection models under real-world occlusion scenarios.
| No. of Videos | No. of Classes | No. of Instances |
|---|---|---|
| 5896 | 21 | 928 |
On Occlusions in Video Action Detection: Benchmark Datasets And Training Recipes
Rajat Modi, Vibhav Vineet, and Yogesh Singh Rawat
NeurIPS 2023
@inproceedings{NEURIPS2023_b3640c2d, author = {Modi, Rajat and Vineet, Vibhav and Rawat, Yogesh}, booktitle = {Advances in Neural Information Processing Systems}, editor = {A. Oh and T. Naumann and A. Globerson and K. Saenko and M. Hardt and S. Levine}, pages = {57306–57335}, publisher = {Curran Associates, Inc.}, title = {On Occlusions in Video Action Detection: Benchmark Datasets And Training Recipes}, url = {https://proceedings.neurips.cc/paper_files/paper/2023/file/b3640c2d3e58f716c67066046318db0f-Paper-Datasets_and_Benchmarks.pdf}, volume = {36}, year = {2023} }
Real-OUCF
The dataset addresses the limitations of the UCF-24 dataset, which predominantly features single-actor scenarios. By incorporating videos of tandem events from the Olympics, where multiple actors perform synchronized movements and occlude each other, Real-OUCF provides a more challenging benchmark for evaluating action detection models under realistic occlusion conditions.
| No. of Videos | No. of Classes | No. of Instances |
|---|---|---|
| 1743 | 24 | 6920 |
On Occlusions in Video Action Detection: Benchmark Datasets And Training Recipes
Rajat Modi, Vibhav Vineet, and Yogesh Singh Rawat
NeurIPS 2023
@inproceedings{NEURIPS2023_b3640c2d, author = {Modi, Rajat and Vineet, Vibhav and Rawat, Yogesh}, booktitle = {Advances in Neural Information Processing Systems}, editor = {A. Oh and T. Naumann and A. Globerson and K. Saenko and M. Hardt and S. Levine}, pages = {57306–57335}, publisher = {Curran Associates, Inc.}, title = {On Occlusions in Video Action Detection: Benchmark Datasets And Training Recipes}, url = {https://proceedings.neurips.cc/paper_files/paper/2023/file/b3640c2d3e58f716c67066046318db0f-Paper-Datasets_and_Benchmarks.pdf}, volume = {36}, year = {2023} }
UCF-101-O
The dataset is constructed by synthetically introducing occlusions into the original UCF101 videos using objects from the Pascal VOC dataset. This approach allows for controlled experiments to assess the impact of occlusions on action recognition models. By varying occlusion properties such as size, position, and movement, researchers can systematically evaluate model robustness under different occlusion scenarios.
| No. of Videos | No. of Classes | Occlusion Size |
|---|---|---|
| 3783 | 101 | 40-60% |
Revealing the unseen: Benchmarking video action recognition under occlusion
Shresth Grover, Vibhav Vineet, and Yogesh S Rawat
NeurIPS 2023
@inproceedings{10.5555/3666122.3668987, author = {Grover, Shresth and Vineet, Vibhav and Rawat, Yogesh S}, title = {Revealing the unseen: benchmarking video action recognition under occlusion}, year = {2024}, publisher = {Curran Associates Inc.}, address = {Red Hook, NY, USA}, booktitle = {Proceedings of the 37th International Conference on Neural Information Processing Systems}, articleno = {2865}, numpages = {23}, location = {New Orleans, LA, USA}, series = {NIPS ’23} }
K400-O
The dataset is constructed by synthetically introducing occlusions into the original Kinetics400 videos using objects from the Pascal VOC dataset. This approach allows for controlled experiments to assess the impact of occlusions on action recognition models. By varying occlusion properties such as size, position, and movement, researchers can systematically evaluate model robustness under different occlusion scenarios.
| No. of Videos | No. of Classes | Occlusion Size |
|---|---|---|
| 40,000 | 400 | 40-60% |
Revealing the unseen: Benchmarking video action recognition under occlusion
Shresth Grover, Vibhav Vineet, and Yogesh S Rawat
NeurIPS 2023
@inproceedings{10.5555/3666122.3668987, author = {Grover, Shresth and Vineet, Vibhav and Rawat, Yogesh S}, title = {Revealing the unseen: benchmarking video action recognition under occlusion}, year = {2024}, publisher = {Curran Associates Inc.}, address = {Red Hook, NY, USA}, booktitle = {Proceedings of the 37th International Conference on Neural Information Processing Systems}, articleno = {2865}, numpages = {23}, location = {New Orleans, LA, USA}, series = {NIPS ’23} }
UCF-19-Y-OCC
The dataset consists of real-world videos that naturally exhibit occlusions, providing a benchmark to evaluate the performance of action recognition models under such conditions. This dataset aids in understanding how models handle occlusions in practical scenarios.
| No. of Videos | No. of Classes | Duration |
|---|---|---|
| 570 | 19 | 5s |
Revealing the unseen: Benchmarking video action recognition under occlusion
Shresth Grover, Vibhav Vineet, and Yogesh S Rawat
NeurIPS 2023
@inproceedings{10.5555/3666122.3668987, author = {Grover, Shresth and Vineet, Vibhav and Rawat, Yogesh S}, title = {Revealing the unseen: benchmarking video action recognition under occlusion}, year = {2024}, publisher = {Curran Associates Inc.}, address = {Red Hook, NY, USA}, booktitle = {Proceedings of the 37th International Conference on Neural Information Processing Systems}, articleno = {2865}, numpages = {23}, location = {New Orleans, LA, USA}, series = {NIPS ’23} }
Low Resolution
TinyVIRAT: Low-resolution Video Action Recognition
The dataset comprises realistic, low-resolution video samples extracted from surveillance footage, specifically designed to evaluate action recognition models under challenging visual conditions.
| Resolution | No. of Videos | No. of Classes | Label Type |
|---|---|---|---|
| 10×10 – 128×128 | 12,829 | 26 | Multi-Label |
TinyVIRAT: Low-resolution Video Action Recognition
Ugur Demir, Yogesh S Rawat, and Mubarak Shah
CVPR 2021 Workshop
@misc{demir2020tinyviratlowresolutionvideoaction, title={TinyVIRAT: Low-resolution Video Action Recognition}, author={Ugur Demir and Yogesh S Rawat and Mubarak Shah}, year={2020}, eprint={2007.07355}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2007.07355}, }
Action Detection
UCF-MAMA [Multi Actor Multi Action]
The dataset is designed to address limitations in existing video action detection datasets by providing complex, real-world scenarios where multiple actors perform simultaneous actions, often occluding each other and exhibiting mirrored movements. The dataset includes high-resolution videos, enabling the evaluation of models under conditions that closely resemble real-world applications.
| Resolution | No. of Videos | No. of Classes | Label Type |
|---|---|---|---|
| 1920×1080 | 32,726 | 35 | Multi-Label |
Video Action Detection: Analysing Limitations and Challenges
Rajat Modi, Aayush Jung Rana, Akash Kumar, Praveen Tirupattur, Shruti Vyas, Yogesh Singh Rawat, Mubarak Shah
CVPR Workshop 2022
@misc{modi2022videoactiondetectionanalysing, title={Video Action Detection: Analysing Limitations and Challenges}, author={Rajat Modi and Aayush Jung Rana and Akash Kumar and Praveen Tirupattur and Shruti Vyas and Yogesh Singh Rawat and Mubarak Shah}, year={2022}, eprint={2204.07892}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2204.07892}}
Perception
Probe-RCB: Relational, Compositional and Contextual Understanding
The datasets derived from Visual Genome evaluate visual-language (V+L) models on relational (probe R), compositional (probe C), and contextual (probe B) understanding. Probe R tests relational understanding by presenting images with predicate swaps in textual descriptions. Probe C examines compositional understanding through prompts altering object compositions or attributes. Probe B evaluates contextual understanding by modifying or removing image backgrounds to assess reliance on context. Together, these probes comprehensively evaluate V+L models’ ability to interpret relationships, compositions, and contextual cues.
| Dataset | # Images | # Objects | Other |
|---|---|---|---|
| Probe R | 99,960 | 6006 | 2456 Relations |
| Probe C | 99,886 | 2462 | 114 Compositions |
| Probe B | 33,229 | 80 | 4 Fillers |
Probing Conceptual Understanding of Large Visual-Language Models
Madeline Schiappa, Raiyaan Abdullah, Shehreen Azad, Jared Claypoole, Michael Cogswell, Ajay Divakaran, Yogesh Rawat
CVPR Workshop 2024
@misc{schiappa2024probingconceptualunderstandinglarge, title={Probing Conceptual Understanding of Large Visual-Language Models}, author={Madeline Schiappa and Raiyaan Abdullah and Shehreen Azad and Jared Claypoole and Michael Cogswell and Ajay Divakaran and Yogesh Rawat}, year={2024}, eprint={2304.03659}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2304.03659}, }
GeoMeter
This benchmark probes the depth and height perception capability of large visual language models. Our key finding is that all models struggle with depth and height perception.
| No. of Image-Text Pairs |
|---|
| 11k |
GeoMeter: Probing Depth and Height Perception of Large Visual-Language Models
Shehreen Azad, Yash Jain, Rishit Garg, Yogesh S Rawat, Vibhav Vineet
@misc{azad2024geometerprobingdepthheight, title={GeoMeter: Probing Depth and Height Perception of Large Visual-Language Models}, author={Shehreen Azad and Yash Jain and Rishit Garg and Yogesh S Rawat and Vibhav Vineet}, year={2024}, eprint={2408.11748}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2408.11748}, }
Action Recognition
NoisyActions2M
This is a large-scale dataset comprising approximately 2 million web-sourced videos, each accompanied by user-generated annotations and metadata. It is designed to facilitate video understanding tasks and to benchmark techniques for learning from noisy labels. By utilizing freely available user-generated labels, this dataset addresses the challenge of training deep learning models without the need for costly manual annotations.
| No. of Videos | No. of Classes | Label Type | Avf. Duration |
|---|---|---|---|
| 2M | 7000 | Multi-Label | 123s |
NoisyActions2M: A Multimedia Dataset for Video Understanding from Noisy Labels
Mohit Sharma, Raj Patra, Harshal Desai, Shruti Vyas, Yogesh Rawat, Rajiv Ratn Shah
ACM MM Asia 2021
@misc{sharma2021noisyactions2mmultimediadatasetvideo, title={NoisyActions2M: A Multimedia Dataset for Video Understanding from Noisy Labels}, author={Mohit Sharma and Raj Patra and Harshal Desai and Shruti Vyas and Yogesh Rawat and Rajiv Ratn Shah}, year={2021}, eprint={2110.06827}, archivePrefix={arXiv}, primaryClass={cs.MM}, url={https://arxiv.org/abs/2110.06827}, }
Segmentation
MS COCO-P
This dataset is designed to evaluate the robustness of segmentation models against real-world distribution shifts. It introduces various perturbations, such as blur and compression artifacts, to the original MS COCO images, enabling a comprehensive assessment of model performance under challenging conditions. Experiments show that both multimodal and unimodal visual foundation models exhibit vulnerabilities to these perturbations.
| Things Categories | Stuff Categories | Total Images |
|---|---|---|
| 80 | 53 | 425,000 |
Robustness Analysis on Foundational Segmentation Models
Madeline Chantry Schiappa, Shehreen Azad, Sachidanand VS, Yunhao Ge, Ondrej Miksik, Yogesh S Rawat, and Vibhav Vineet
CVPR Workshop 2024
@misc{schiappa2024robustnessanalysisfoundationalsegmentation, title={Robustness Analysis on Foundational Segmentation Models}, author={Madeline Chantry Schiappa and Shehreen Azad and Sachidanand VS and Yunhao Ge and Ondrej Miksik and Yogesh S. Rawat and Vibhav Vineet}, year={2024}, eprint={2306.09278}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2306.09278}, }
ADE20K-P
This dataset is designed to evaluate the robustness of segmentation models against real-world distribution shifts. It introduces various perturbations, such as blur and compression artifacts, to the original ADE20K images, enabling a comprehensive assessment of model performance under challenging conditions. Experiments show that both multimodal and unimodal visual foundation models exhibit vulnerabilities to these perturbations.
| Things Categories | Stuff Categories | Total Images |
|---|---|---|
| 100 | 50 | 170,000 |
Robustness Analysis on Foundational Segmentation Models
Madeline Chantry Schiappa, Shehreen Azad, Sachidanand VS, Yunhao Ge, Ondrej Miksik, Yogesh S Rawat, and Vibhav Vineet
CVPR Workshop 2024
@misc{schiappa2024robustnessanalysisfoundationalsegmentation, title={Robustness Analysis on Foundational Segmentation Models}, author={Madeline Chantry Schiappa and Shehreen Azad and Sachidanand VS and Yunhao Ge and Ondrej Miksik and Yogesh S. Rawat and Vibhav Vineet}, year={2024}, eprint={2306.09278}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2306.09278}, }