Center for Research in Comptuer Vision
Center for Research in Comptuer Vision

Simultaneous Video Stabilization and Moving Object Detection in Turbulence


Turbulence mitigation refers to the stabilization of videos with non-uniform deformations due to the influence of optical turbulence. Typical approaches for turbulence mitigation follow averaging or de-warping techniques. Although these methods can reduce the turbulence, they distort the independently moving objects which can often be of great interest. In this paper, we address the novel problem of simultaneous turbulence mitigation and moving object detection. We propose a novel threeterm low-rank matrix decomposition approach in which we decompose the turbulence sequence into three components: the background, the turbulence, and the object. We simplify this extremely difficult problem into a minimization of nuclear norm, Frobenius norm, and L1 norm. Our method is based on two observations: First, the turbulence causes dense and Gaussian noise, and therefore can be captured by Frobenius norm, while the moving objects are sparse and thus can be captured by L1 norm. Second, since the objectís motion is linear and intrinsically different than the Gaussian-like turbulence, a Gaussian-based turbulence model can be employed to enforce an additional constraint on the search space of the minimization. We demonstrate the robustness of our approach on challenging sequences which are significantly distorted with atmospheric turbulence and include extremely tiny moving objects.

Decomposition Examples

Proposed Method

The figure below shows a diagram of the proposed approach. We first apply a pre-precessing step to improve the contrast of the sequence, and reduce the spurious and random noise. Consequently, we obtain an object confidence map using a turbulence model which utilizes both the intensity and the motion cues. Finally, we decompose the sequence into its components using three-term rank minimization, which employs the previuosly computed turbulence model as a prior in order to encourage object detection at regions observing non-Gaussian motion.

Three Term Decomposition

The frames of the sequence are stacked in a matrix F. Consequently, we use low-rank optimization to decompose the sequence into three components: The background (low-rank matrix A), the turbulence (dense error matrix E), and the moving objects (sparse matrix O). In practice, parts of the turbulence could also appear as sparse errors in the object matrix O. Therefore, we enforce an additional contraint on the objects using on a turbulence model (Pi in the equation below).

Turbulence Model

We model the turbulence using a Gaussian distribution in both motion and intensity domains. We capture the motion using particle advection as demonstrated in the figure below. The linear object motion is different than the Guassian-like turbulence; therefore, the object can be detected as illusratated in the figure.

Low-Rank Optimization

We apply convex relaxation to the decomposition. Consequently, we optimize the solution using the Augmented Lagrange Multiplier method. The algorithm below illusrates the involved steps.

Related Publication

Omar Oreifej, Xin Li, and Mubarak Shah. Simultaneous Video Stabilization and Moving Object Detection in Turbulence, IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI) 2012.


Seeing Through Turbulence

Dataset and Code

Presentation Video with Narration

Back to Object Detetion and Scene Segmentation Projects