Big Data

This material is based upon work supported by the National Science Foundation under Grant No. 1741431.

Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

Award Number:  1741431

Duration (expected):  3 years

Award Amount:  $662,431.00

Award Title:  BIGDATA: IA: Distributed Semi-Supervised Training of Deep Models and Its Applications in Video Understanding

PI:  Mubarak Shah

Co-PI:  Liqiang Wang

Student(s):  (Supported) Ehsan Kazemi Foroushani, Siavash Khodadadeh, Jyoti Kini, Yandong Li, Zixia Liu, Amir Mazaheri, Dongdong Wang, Yang Zhang; (Co-Authors) Pooya Abolghasemi, Deliang Fan, Yunhui Guo, Mahdi M. Kalayeh, Nasim Souly, Xiang Wei

Project Goals

We propose to design new algorithms to explore distributed Learning with Less Labels (LWL) including unsupervised, semi-supervised, active learning and meta learning training strategies for the deep models as well as its applications to computer vision and machine learning problems including semantic segmentation, video retrieval, image and video synthesis and video human action recognition/localization/segmentation, visuomotor object manipulation etc. We also conduct fundamental research on unsupervised learning techniques such as structure-preserving data selection and multi-norm batch normalization.
To leverage the labeled and unlabeled data at scale, we are designing novel hybrid synchronized and asynchronous frameworks for efficient distributed training. We emphasize that whereas we expect to make novel discoveries in above mentioned domains, our integrated solution to the distributed semi-supervised training of deep models could be applied to other scientific domains. The proposed research will enrich the family of deep learning methods by modeling the inherent characteristics of data to enable effective learning from the distributed, and heterogeneous image and video data. To further facilitate faster and scalable training of the deep models, we are also developing a novel parallel computing framework tailored for the deep neural networks. The carefully chosen application domains enable us to additionally explore the temporal structures and models.

Research Challenges

The idiosyncrasies of the deep models and the distributed unlabeled data prevent us from directly applying existing semi-supervised algorithms. First of all, the existing methods mostly rely on fixed data representations (e.g., handcrafted features of images and acoustic signals), while a key distinction of deep models from the other machine learning methods is that they automatically learn the data representations from raw data (e.g., images). Besides, the computational challenge becomes more profound in the semi-supervised learning: The unlabeled data are often unstructured and distributed, making it infeasible to transmit or store the data to a single machine. How shall we design the architectures of the deep models to facilitate distributed training? How to use unlabeled data to facilitate semi-supervised learning? How to analyze video effectively? Our team endeavours to work on these challenging research questions.

Current/Final Results (summary)

[1] Fast and Accurate Deep Learning using Data Selection from Nonlinear Manifolds (CVPR 2020)

Finding a small subset of data whose linear combination spans other data points, also called column subset selection problem (CSSP), is an important open problem in computer science with many applications in computer vision and deep learning. There are some studies that solve CSSP in a polynomial time complexity w.r.t. the size of the original dataset. A simple and efficient selection algorithm with a linear complexity order, referred to as spectrum pursuit (SP), is proposed that pursuits spectral components of the dataset using available sample points. The proposed non-greedy algorithm aims to iteratively find K data samples whose span is close to that of the first K spectral components of entire data. SP has no parameter to be fine-tuned and this desirable property makes it problem-independent. The simplicity of SP enables us to extend the underlying linear model to more complex models such as nonlinear manifolds and graph-based models. The non-linear extension of SP is introduced as kernel-SP (KSP). The superiority of the proposed algorithms is demonstrated in a wide range of applications.

Result: To evaluate the performance of our proposed selection algorithms we considered several applications and conducted extensive experiments. Some applications of our method are (i) fast GAN training using reduced dataset; (ii) semi-supervised learning on graph-based datasets; (iii) large graph summarization; (iv) few-shot learning; and (v) openset identification. For example, in data selection for GAN training, SP and KSP outperform other methods. Also, KSP performs better than SP due to the selection from a nonlinear manifold. Note that SP needs less memory requirement and its computational complexity is less than its peers.

[2] Norm-Preservation using Procrustes Residual Networks (PAMI 2020)

Augmenting deep neural networks with skip connections, as introduced in the so-called ResNet architecture, surprised the community by enabling the training of networks of more than 1000 layers with significant performance gains. It has been shown that identity skip connections eliminate singularities and improve the optimization landscape of the network. Our research deciphers ResNet by analyzing the effect of skip connections in the backward path and sets forth new theoretical results on the advantages of identity skip connections in deep neural networks. We prove that the skip connections in the residual blocks facilitate preserving the norm of the gradient and lead to well-behaved and stable back-propagation, which is a desirable feature from an optimization perspective. We also show that, perhaps surprisingly, as more residual blocks are stacked, the network becomes more norm-preserving. Traditionally, norm-preservation is enforced on the network only at beginning of the training, by using initialization techniques. However, we show that identity skip connection retains norm-preservation during the training procedure.

Result: Our theoretical arguments are supported by extensive empirical evidence. Our results and the introduced modification for ResNet, referred to as Procrustes ResNets, can be used as a guide for studying more complex architectures such as DenseNet, training deeper networks, and inspiring new architectures.

[3] Subspace Capsule Network (AAAI 2020)

In this paper we propose the general subspace capsule network that can be successfully applied in generative as well as discriminative tasks. Subspace capsule network models the possible variations in the appearance of entities trough a group of learned capsule subspaces. Then the capsules are created by projecting the input feature vector onto these learned capsule subspaces using leaned transformations.

Result: We evaluated the effectiveness and generalizability of SCN through a comprehensive set of experiments in three applications namely, high resolution image generation, semi-supervised image classification using the GAN framework and supervised image classification. We demonstrate the scalability of SCN to large datasets and model architectures by applying it on supervised classification of ImageNet dataset.

[4] Unsupervised Meta-Learning For Few-Shot Image and Video Classification (NeurIPS 2019)

Few-shot classification refers to classify N different concepts based on just a few examples of them. Few-shot learning refers to methods or techniques which enables deep neural networks to learn a few-shot classification task by just few samples. Few-shot or one-shot learning of classifiers requires a significant inductive bias towards the type of task to be learned. One way to acquire this is by meta-learning on tasks similar to the target task. In order to achieve this, meta-learning requires access to many different few-shot learning tasks and aims to learn how to learn those tasks from scratch with just a few samples for each. When we face a new classification task (target task), we hope to perform it by the network which is meta-learned on how to learn classification tasks with just a few samples. Note that the target classification task does not share any of its classes with meta-learning tasks. The subjects should not have any intersection with the tasks which we learned how to classify.

Result: In this paper, we propose UMTRA, an algorithm that performs unsupervised, model-agnostic meta-learning for classification tasks. UMTRA does not require label information during meta-learning. In other words, UMTRA trades off some classification accuracy for a reduction in the required labels of several orders of magnitude.

[5] Improving Object Detection with Selective Self-Supervised Self-Training (ECCV 2020)

We study how to leverage Web images to augment human-curated object detection datasets. We propose a selective net to rectify the supervision signals in Web images. It not only identifies positive bounding boxes but also creates a safe zone for mining hard negative boxes. We report state-of-the-art results on detecting backpacks and chairs from everyday scenes, along with other challenging object classes. We propose a novel approach to improving object detection with massive unlabeled Web images. We collect the Web images with image-to-image search, leading to smaller domain mismatch between the retrieved Web images and the curated dataset than text-to-image search does. Besides, we incorporate a principled selective net into self-training rather than threshold confidence scores as a simple heuristic for selecting bounding boxes. Moreover, we impose a self-supervised training loss over the high-quality boxes chosen by the selective net to make better use of the Web images. The improvement in detecting challenging objects is significant over the competing methods, Our approach works consistently well on not only the Web-augmented object detection but also the traditional semi-supervised object detection.

Result: To the best of our knowledge, this work is the first to explore unlabeled, out-of-domain Web images to augment curated object detection datasets. We report state-of-the-art results for detecting backpacks and chairs, along with other challenging objects.

[6] Neural Networks Are More Productive Teachers Than Human Raters: Active Mixup for Data-Efficient Knowledge Distillation from a Blackbox Model (CVPR 2020)

We study how to train a student deep neural network for visual recognition by distilling knowledge from a blackbox teacher model in a data-efficient manner. Progress on this problem can significantly reduce the dependence on large-scale datasets for learning high-performing visual recognition models. We propose an approach that blends mixup and active learning. The former effectively augments the few unlabeled images by a big pool of synthetic images sampled from the convex hull of the original images, and the latter actively chooses from the pool hard examples for the student neural network and query their labels from the teacher model. We validate our approach with extensive experiments.

Result: We propose an approach combining mixup and active learning. Mixup allows the student model to probe diverse regions of the input space. We then employ active learning to reduce the query transactions to the teacher model. To the best of our knowledge, we are the first to distill knowledge from a blackbox teacher model while underscoring the need for data-efficiency and query-efficiency. Experiments show that, despite the blackbox teacher in our work, our approach performs on par or better than the competing methods that learn from whitebox teachers.

[7] Rethinking Class-Balanced Methods for Long-Tailed Visual Recognition from a Domain Adaptation Perspective (CVPR 2020)

Object frequency in the real world often follows a power law, leading to a mismatch between datasets with long-tailed class distributions seen by a machine learning model and our expectation of the model to perform well on all classes. We analyze this mismatch from a domain adaptation point of view. First of all, we connect existing class-balanced methods for long-tailed classification to target shift, a well-studied scenario in domain adaptation. The connection reveals that these methods implicitly assume that the training data and test data share the same class-conditioned distribution, which does not hold in general and especially for the tail classes. We propose to augment the classic class-balanced learning by explicitly estimating the differences between the class-conditioned distributions with a meta-learning approach. We validate our approach with six benchmark datasets and three loss functions.

Result: We propose to investigate long-tailed visual recognition from a domain adaptation point of view. We employ ResNet-32 and ResNet-152 for the experiments on Places-LT. We arrive at 62.90% top-1 error (39.86/29.87% top-3/5 error) on PlacesLT, which is on par with 64.1% by OLTR or 63.3% by cRT.

[8] Iterative Projection and Matching: Finding Structure-preserving Representatives (CVPR 2019)

We have developed a new algorithm for finding structure-perserving representative and show its superiority of the proposed algorithm on active learning for video action recognition dataset on UCF-101; learning using representatives on ImageNet; training a generative adversarial network (GAN) to generate multi-view images from a single-view input on CMU Multi-PIE dataset; and video summarization on UTE Egocentric dataset.

Result: Our Iterative Projection Method (IPM ) achieves 30% on using only 50 samples, compared to 48% employing 1.2 million samples.

[9] Training Faster by Separating Modes of Variation in Batch-normalized Models (IEEE PAMI)

We have devleoped new algorithm for traning deep networks by separatign modes of vatiation in Batch Normalized Models. Through extensive set of experiments on CIFAR-10 and CIFAR-100, using both a 5-layers deep CNN and modern Inception-V3 architecture, we show that mixture normalization reduces required number of gradient updates to reach the maximum test accuracy of the batch normalized model by ∼31%-47% across a variety of training scenarios.

Result: Our mixture normalization algorithm reduces required number of gradient updates to reach the maximum test accuracy of the batch normalized model by ∼31%-47% across a variety of training scenarios.

[10] An Efficient 3D CNN for Action/Object Segmentation in Video (BMVC 2019)

We have developed a new algorithm for the video object segmentation in the unsupervised setting. We propose an end-to-end encoder-decoder style 3D CNN based method to solve the video object segmentation problem efficiently. We evaluate our method on video object segmentation and action segmentation benchmarks and demonstrate state-of-the-art performance. Extensive experiments on several video datasets demonstrate the superior performance of the proposed approach for action and object segmentation compared to the state-of-the-art.

Result: Our Efficient 3D CNN model for video object segmentation achieves accuracy of 77.4% with 6 Billion parameters and 11 MB employing separable convolution compared to Standard 3D Conv which achieves 77.1% accuracy with 136 Billion parameters and 255 MB.

[11] Robustifying Visuomotor Policy by Task Focused Visual Attention (CVPR 2019)

We have developed an approach for augmenting a deep visuomotor policy trained through demonstrations with Task Focused visual Attention (TFA). The manipulation task is specified with a natural language text such as “move the red bowl to the left”. Averaged over all the objects, the recovery rate is only 5% for the baseline policy in pickup and push tasks, while it is 57% for the policy with the TFA.

Result: Our Task Focused visual Attention (TFA) visuomotor policy achieves recovery rate 57% for pickup and push tasks compared to 5% for the baseline policy.

[12] Photography and Exploration of Tourist Locations Based on Optimal Foraging Theory (IEEE CSVT 2019)

We have developed a new algorithm for location recommendation to explore a particular tourist attraction from the photography perspective. The recommendation is learned from YFCC100M dataset which has around 98M user-generated images with around 300K users. We make use of social media images and associated meta-data to understand the past tourist behavior and the location environment.

Result: Our Foraging Theory based method utilizes big multimedia data shared on social media platforms and can generate effective personalized tour recommendation in real-time with 42% improvement in similarity performance.

[13] How Local is the Local Diversity? Reinforcing Sequential Determinantal Point Processes with Dynamic Ground Sets for Supervised Video Summarization. (ECCV 2018)

In this project, we propose a novel probabilistic model, built upon SeqDPP, to dynamically control the time span of a video segment upon which the local diversity is imposed. In particular, we enable SeqDPP to learn to automatically infer how local the local diversity is supposed to be from the input video. The resulting model is extremely involved to train by the hallmark maximum likelihood estimation (MLE), which further suffers from the exposure bias and non-differentiable evaluation metrics. To tackle these problems, we instead devise a reinforcement learning algorithm for training the proposed model. Extensive experiments verify the advantages of our model and the new learning algorithm over MLE-based methods.

Result: Our video summarization algorithms utilize local diversity as a guideline to tackle the dynamic diverse selection problem, where our both SeqDPP and DySeqDPP algorithms significantly outperform the state-of-the-art methods and DySeqDPP ranks to the first by a big margin (around 4%) on the benchmark SumMe.

[14] Depthwise Convolution is All You Need for Learning Multiple Visual Domains (AAAI 2019)

In this project, we propose a multi-domain learning architecture based on depthwise separable convolution. The proposed approach is based on the assumption that images from different domains share cross-channel correlations but have domain-specific spatial correlations. The proposed model is compact and has minimal overhead when being applied to new domains. Additionally, we introduce a gating mechanism to promote soft sharing between different domains.

Result: In the project of investigating reusable structure in models, our depth-wise separable convolution based architecture shows the highest score while only requiring 50% of the parameters compared with the state-of-the-art approaches, according to an experiment on Visual Decathlon Challenge.

[15] NATTACK: Learning the Distributions of Adversarial Examples for an Improved Black-Box Attack on Deep Neural Networks (ICML 2019)

In this project, we propose a black-box adversarial attack algorithm that can defeat both vanilla DNNs and those generated by various defense techniques developed recently. Instead of searching for an “optimal” adversarial example for a benign input to a targeted DNN, our algorithm finds a probability density distribution over a small region centered around the input, such that a sample drawn from this distribution is likely an adversarial example, without the need of accessing the DNN’s internal layers or weights. Our approach is universal as it can successfully attack different neural networks by a single algorithm. It is also strong; according to the testing against 2 vanilla DNNs and 13 defended ones, it outperforms state-of-the-art black-box or white-box attack methods for most test cases.

Result: In the black-box attaching project, our algorithm defeats 13 defended DNNs, better than or on par with state-of-the-art white-box attack methods.

Listed below are the major accomplishments:

• IPM was found to be an efficient and robust method for structure-preserving representative selection which can be generalized to a wide range of problem domains.
• A training cost optimization based on separating mode variation was observed to be robust across networks with varying size.
• The use of separable convolutions for video object segmentation is effective and leads to drastic reduction in memory consumption.
• The text based visual attention was found to be effective for visuomotor policy learning and robust in the presence of visual and physical disturbances.
• The integration of behavior science from foraging theory was found to be useful in providing effective personalized recommendation which considers both content as well as user context.
• In the black-box attaching project, we investigate the transferability of adversarial examples across the defended DNNs, interestingly, we observe that unlike the high transferability across vanilla DNNs, it is difficult to transfer the attacks on the defended DNNs.
• In the project of distributed machine learning, we observe that asynchronous proximal algorithms can be highly efficient when being used to solve large scale non-convex non-smooth problems.

Publications

[1] Joneidi, Mohsen, Saeed Vahidian, Ashkan Esmaeili, Weijia Wang, Nazanin Rahnavard, Bill Lin, and Mubarak Shah. Select to better learn: Fast and accurate deep learning using data selection from nonlinear manifolds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7819-7829 (CVPR 2020).

[2] Zaeemzadeh, Alireza, Nazanin Rahnavard, and Mubarak Shah. Norm-Preservation: Why Residual Networks Can Become Extremely Deep?. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI 2020).

[3] Marzieh Edraki, Nazanin Rahnavard and Mubarak Shah. Subspace Capsule Network. 34th Conference on Association for the Advancement of Artificial Intelligence (AAAI 2020), New York, USA. [Project Page] [Code]

[4] Khodadadeh, Siavash, Ladislau Bölöni, and Mubarak Shah. Unsupervised Meta-Learning For Few-Shot Image and Video Classification. 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada. [Project Page] [Code]

[5] Li, Yandong, Di Huang, Danfeng Qin, Liqiang Wang, and Boqing Gong. Improving Object Detection with Selective Self-supervised Self-training. In European Conference on Computer Vision, pp. 589-607. Springer, Cham (ECCV 2020).

[6] Wang, Dongdong, Yandong Li, Liqiang Wang, and Boqing Gong. Neural Networks Are More Productive Teachers Than Human Raters: Active Mixup for Data-Efficient Knowledge Distillation from a Blackbox Model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1498-1507 (CVPR 2020).

[7] Jamal, Muhammad Abdullah, Matthew Brown, Ming-Hsuan Yang, Liqiang Wang, and Boqing Gong. Rethinking Class-Balanced Methods for Long-Tailed Visual Recognition from a Domain Adaptation Perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7610-7619 (CVPR 2020).

[1] Zaeemzadeh, Alireza, Joneidi, Mohsen, Rahnavard, Nazanin, Shah, Mubarak: Iterative Projection and Matching: Finding Structure-preserving Representatives and Its Application to Computer Vision. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2019), Long Beach, CA, June 16-20, 2019. [Project Page] [Code]

[2] M. M. Kalayeh and M. Shah, “Training Faster by Separating Modes of Variation in Batch-normalized Models,” in IEEE Transactions on Pattern Analysis and Machine Intelligence. doi: 10.1109/TPAMI.2019.2895781

[3] Rui Hou, Chen Chen, Rahul Sukthankar, Mubarak Shah, An Efficient 3D CNN for Action/Object Segmentation in Video, British Machine Vision Conference (BMVC 2019), UK, Sep 9-10, 2019.

[4] Abolghasemi, Pooya, Amir Mazaheri, Mubarak Shah, and Ladislau Boloni, Pay attention!-Robustifying a Deep Visuomotor Policy through Task-Focused Attention, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2019), Long Beach, CA, June 16-20, 2019. [Project Page] [Code]

[5] Rawat, Yogesh Singh, Mubarak Shah, and Mohan S. Kankanhalli. Photography and Exploration of Tourist Locations Based on Optimal Foraging Theory. IEEE Transactions on Circuits and Systems for Video Technology (2019). [Code]

[6] Yandong Li, Liqiang Wang, Tianbao Yang, and Boqing Gong. How Local is the Local Diversity? Reinforcing Sequential Determinantal Point Processes with Dynamic Ground Sets for Supervised Video Summarization. In the European Conference on Computer Vision (ECCV) 2018. Munich, Germany. Sept. 8-14, 2018. [Code]

[7] Yandong Li#, Yunhui Guo#, Liqiang Wang, and Tajana Rosing. Depthwise Convolution is All You Need for Learning Multiple Visual Domains. In The Thirty-Third AAAI Conference on Artificial Intelligence (AAAI). Honolulu, Hawaii, USA. 2019. (# Equal contribution.)

[8] Ehsan Kazemi* and Liqiang Wang. Asynchronous Delay-Aware Accelerated Proximal Coordinate Descent for Nonconvex Nonsmooth Problems. In The Thirty-Third AAAI Conference on Artificial Intelligence (AAAI). Honolulu, Hawaii, USA. 2019.

[9] Yandong Li, Lijun Li, Liqiang Wang, Tong Zhang, Boqing Gong. NATTACK: Improved Black-Box Adversarial Attack with Normal Distributions. In 36th International Conference on Machine Learning (ICML). Long Beach, CA, USA. 2019. [Code]

[1] Souly, Nasim and Spampinato, Concetto and Shah, Mubarak. (2017). Semi Supervised Semantic Segmentation Using Generative Adversarial Network.  IEEE International Conference on Computer Vision (ICCV).  5689 to 5697.  doi:10.1109/ICCV.2017.606

[2] Mazaheri, Amir and Gong, Boqing and Shah, Mubarak. (2018). Learning a Multi-Concept Video Retrieval Model with Multiple Latent Variables.  ACM Transactions on Multimedia Computing, Communications, and Applications. 14 (2) 1 to 21. doi:10.1145/3176647 [Code]

[3] Wei, Xiang and Gong, Boqing and Liu, Zixia and Lu, Wei and Wang, Liqiang. (2018). Improving the Improved Training of Wasserstein GANs: A Consistency Term and Its Dual Effect.  International Conference on Learning Representation (ICLR). [Code]

[4] Ding, Yifan and Wang, Liqiang and Fan, Deliang and Gong, Boqing. (2018). A Semi-Supervised Two-Stage Approach to Learning from Noisy Labels.  2018 IEEE Winter Conference on Applications of Computer Vision (WACV).  1215 to 1224. doi:10.1109/WACV.2018.00138. [Code]

[5] Mazaheri, A., & Shah, M. (2018). Visual Text Correction. In Proceedings of the European Conference on Computer Vision (ECCV) (pp. 155-171). [Project Page] [Code]

Presentations

[1] “View Invariant and Few Shot Human Action Recognition”, Invited Talk, Simula Metropolitan Center for Digital Engineering, Oslo, March 14, 2019.

[2] “View Invariant and Few Shot Human Action Recognition”, Invited Talk, Workshop on Human Activity Detection in Multi-Camera Video Streams at the IEEE Winter Conf. on Applications of Computer Vision (WACV), Hawaii, January 7, 2019.

[3] “Self Flying Drones: Wide Area Aerial Video Analysis”, Vision Meets Drone: A Challenge, ECCV 2018 workshop, September 8, 2018, Munich Germany.

Data (with documentation)

[1] Iterative Projection and Matching: Finding Structure-preserving Representatives and Its Application to Computer Vision [Data]

[2] Pay attention!-Robustifying a Deep Visuomotor Policy through Task-Focused Attention [Data]

[3] Visual Text Correction [Data]

Software Downloads (with documentation)

[1] Subspace Capsule Network [Code]

[2] Unsupervised Meta-Learning For Few-Shot Image and Video Classification [Code]

[3] Iterative Projection and Matching: Finding Structure-preserving Representatives and Its Application to Computer Vision [Code]

[4] Pay attention!-Robustifying a Deep Visuomotor Policy through Task-Focused Attention [Code]

[5] Photography and Exploration of Tourist Locations Based on Optimal Foraging Theory [Code]

[6] How Local is the Local Diversity? Reinforcing Sequential Determinantal Point Processes with Dynamic Ground Sets for Supervised Video Summarization [Code]

[7] NATTACK: Improved Black-Box Adversarial Attack with Normal Distributions [Code]

[8] Learning a Multi-Concept Video Retrieval Model with Multiple Latent Variables [Code]

[9] Improving the Improved Training of Wasserstein GANs: A Consistency Term and Its Dual Effect [Code]

[10] A Semi-Supervised Two-Stage Approach to Learning from Noisy Labels [Code]

[11] Visual Text Correction [Code]

Broader Impacts

Impact on the developments of the principal disciplines of the project: Our research on active-learning, multi-norm batch-normalization, and distributed semi-supervised training of deep neural networks will benefit many applications in machine learning and computer vision. These algorithms can be seen as some standard techniques to facilitate training procedure, specifically in big-data problems where the training time and data selection are always important factors. Our contributions in this project will bring new opportunities to advance the state of the art in various machine learning and computer vision tasks.

Impact on other disciplines: The proposed techniques can potentially benefit any field of research which directly or indirectly leverages machine learning and big data.

Impact on the development of human resources: The proposed work contributes in training and education of students by enriching the syllabus of courses and also encourages students to get familiar with the latest advances in the field through course projects. This work also forms the subject of many Ph.D. dissertations written by graduate students working under the supervision of the PIs. The PI will also continue leading the REU and RET programs with the goal of increasing the diversity among doctoral degrees in STEM and also inclusion of women and minority graduate students.

Impact on technology transfer: All of our research material including the codes and datasets are available to the public. Since our research has direct impact on deep learning and machine learning applications, it may help others to develop many related algorithms, software, and applications.

Impact on society beyond science and technology: Our research is useful for multimedia applications on big data, which may have value for social scientists to analyse digitised human behavior records such as big video data from social media, textual feeds from news agencies and blogs, and etc.

Educational Achievements

The PI, Mubarak Shah, taught “Advanced Computer Vision” in Spring 2018 and Spring 2019, in which, many recent papers on solving Big Video data analysis problems in computer vision as well as employing semi-supervised learning were discussed in the class. Graduate students implemented an interest point detection/matching deep neural network (related self-supervised learning), and also implemented a semi-supervised video object segmentation deep neural network on of the largest available datasets (YouTube-VOS) as the final project.

The Co-PI Liqiang Wang taught “Parallel and Cloud Computing” in Fall 2019, which involves topics about distributed machine learning and big data processing. Graduate students were trained and implemented distributed machine learning projects on the cutting edge multiple GPU/CPU platforms.

In addition, during summer 2019, 10 undergraduate students and 6 high school teachers participated in the Research Experience for Undergraduates (REU) and Research Experience for Teachers (RET) programs in Computer Vision which were both funded by NSF. REU students worked on computer vision projects related to big video data applying semi-supervised and weakly supervised methods. RET participants were exposed to several applications of big data.

Several Ph.D. students have been involved in the research funded by this grant. We highlight a few here:

• Amir Mazheri worked on big video data problem of visual text correction, and semi-supervised robot manipulation using visual attention and textual information.
• Mahdi Kalayeh worked on a novel formulation of multi-norm batch normalization for faster training of deep networks. He graduated in May 2019 and is currently working at Netflix as a senior research scientist.
• Rui Hou worked on video action localization and segmentation using deep neural networks trained on big video datasets. Rui finished his PhD in June 2019 and currently working at Niantic, Inc in California.
• Ehsan Kazemi and Yandong Li were also involved in the proposed research. Yandong published three papers in ECCV, AAAI, and ICML, and Ehsan published one paper on AAAI during the last academic year.

Awards, Prizes

[1] Trustee chair professor Dr. Mubarak Shah is the 2019 winner of the prestigious ACM SIGMM Award for Outstanding Technical Contributions to Multimedia Computing, Communications and Applications. Dr. Shah was selected for this award for his “outstanding and pioneering and continued research contributions in the areas of multimedia content analysis and multimedia applications, for leadership in education, and for outstanding and continued service to the community.” The award includes a $2000 honorarium and a 30-minute keynote talk at ACMMM 2019 in Nice, France.

[2] Amir Mazaheri won $5000 doctoral research fellowship award from the University of Central Florida, for the “Visual Text Correction” work.

Highlights, Press Release
ACM SIGMM Award for Outstanding Technical Contributions to Multimedia Computing, Communications and Applications

Point of Contact: Mubarak Shah

Date of Last Update:  September 23, 2019