## A Streakline Representation of Flow in Crowded Scenes

Paper: | A Streakline Representation of Flow in Crowded Scenes |

Contact: | Ramin Mehran, Brian E. Moore, Mubarak Shah |

**Introduction**

Based on the Lagrangian framework for fluid dynamics, a streakline representation of flow is presented to solve computer vision problems involving crowd and traffic flow. Streaklines are traced in a fluid flow by injecting color material, such as smoke or dye, which is transported with the flow and used for visualization. In the context of computer vision, streaklines may be used in a similar way to transport information about a scene, and they are obtained by repeatedly initializing a fixed grid of particles at each frame, then moving both current and past particles using optical flow. Streaklines are the locus of points that connect particles which originated from the same initial position. In this paper, a streakline technique is developed to compute several important aspects of a scene, such as flow and potential functions using the Helmholtz decomposition theorem. This leads to a representation of the flow that more accurately recognizes spatial and temporal changes in the scene, compared with other commonly used flow representations. Applications of the technique to segmentation and behavior analysis provide comparison to previously employed techniques, showing that the streakline method outperforms the state-of-the-art in segmentation, and opening a new domain of application for crowd analysis based on potentials.

We introduce a new representation of flow to computer vision community. The intuition comes from the resemblance of fluid flow to flow of pedestrians. Based on Lagrangian Framework of fluid dynamics, there are 3 representations of flow:

- Optical Flow or Vector Field
- Dense Particle Trajectories or Pathlines
- Streaklines

(a) Three representations of flow

- ChallengesIn scenes with dynamic motion, the flow changes continuously
- Optical Flow is instantaneous, ignores the continuity of motion
- Trajectories represent the continuity of motion, not imminent to capture behavior transitions

- Solution
- Streakline are instantaneous and capture the continuity of motion

- Advantages
- The Streakline are more effective in dynamic scenes in representing the changes in motion and behavior transitions

- DescriptionWith regard to flow visualization, streaklines are defined as the traces of a colored material in the low. To understand streaklines, consider a fluid flow with an ink dye injected at a particular point. If the ink is continuously injected, then a line will be traced out by the ink in the direction of the flow, this is a streakline. If the direction of flow changes, then the streaklines change accordingly. Streaklines are new to computer vision research. In this context, streaklines may be obtained by repeatedly initializing a grid of particles and moving all particles according to the optical flow, in the spirit of a Lagrangian fluid flow. In other words, place a particle at point
**p**, and move the particle one time step with the flow. In the next time step, the point**p**is initialized with a new particle, then both particles are moved with the flow. Repeating this process on some time interval**T**produces particle positions from which we obtain streaklines. Figure (b) gives and example of streaklines in a video. For illustration purposes, the streaklines are shown only on every 10th row and column.

(b) A visualization of the strealines for an example video. The curves represent the streaklines for every tenth row and column. Red and yellow colors represent the different directions of motion. - Difference between Streaklines, Pathlines, and Optical FlowIn a dynamic scene, optical flow would be time varying. To illustrate the effect of such an optical flow in different representations, we present the following toy example. Consider a spatially consistent but time varying vector field as:If we consider a certain point in this vector field it will generate a curve as:If we initiate new particles at the same point at continuously we will be able to generate a set of particle trajectories (pathlines) as:The streakline is the curve that connects the position of all the particles which are initiated at the certain point but at different times. Thus the streakline in the toy example will be the red curve in the following video:

Streaklines enable us to compute set of new features. In this section, we introduce two set of new dense features: Streakflow and flow Potentials. Streaklines provide a means to recognize spatial and temporal changes in the flow, that neither streamlines nor pathlines could provide directly. This point is made here using streak flow and potential functions. In essence, streak flow is obtained by time integration of the velocity field (i.e. optical flow), while potential functions are obtained from spatial integration, and each provides useful information concerning the dynamics in the scene.

- Streakflow
Streakflow is an instantaneous vector field which represent the accumulative motion of the scene. It resembles the temporal average of optical flow but it is more imminent. This vector field has following properties:

(c) Streak flow is computed as the weighted average of the imitating flow vectors of each particle.- Averages the velocities in direction of motion (Does not average blindly) therefore it relates to the group motion
- Imminently reactive to changes in the scene (Less lag than temporal average optical flow)
- Less noisy than optical flow (helpful in behavior recognition)

A comparison of Streak flow, Optical Flow, and average optical flow is give in Figure (d)

(d) The Streakflow stands in the middle of average optical flow and instantaneous optical flow in capturing changes in the scene. - PotentialsBuilding on the fluid dynamics approach to crowd motion, we employ another concept from fluids providing a different point of view. In simplified mathematical models of fluids, it is often assumed that the fluid is imcompressible, and irrotational. These assumptions imply several conservation properties of the fluid, but most importantly, they lead to potential functions, which are scalar functions that characterize the flow in a unique way. For this discourse, potential functions enable accurate classification of behaviors in a scene, which is not possible with streak flow alone. Since the optical flow denotes a planar vector field, the Helmholtz decomposition theorem states that , where and respectively denote the incompressible and irrotational parts of the vector field. To clarify, an incompressible vector field is divergence free , and an irrotational vector field is curl free . Thus, there are functions and , known respectively as the stream function and the velocity potential, satisfying and (see, for example [3]). Following [4], we use Fourier transforms to decompose incompressible and irrotational parts of the vector field and estimate the potential functions using Helmholtz decomposition theorem.

Figure (e) illustrates the computed stream function and velocity potential for a traffic scene where the cars are turning left.

(e) The two types of potentials: The Velocity Potential and Stream Function.

Streaklines are representations of motion and therefore, regions of similar motion would have similar streakline. Given the video we first compute the frame by frame optical flow and then we compute streaklines. We find the pixel by pixel similarity map for the neighboring streakline. Finally, a simple watershed segmentation separates the regions of coherent motion.

(f) The segmentation algorithm

(g) The shape similarity of streaklines

In addition to segmenting a frame into regions of consistent motion, we combine information from potentials to detect lanes in each segment. As stated above, the area between contours of corresponds to the steady flow, and the rate of the incompressible flow between a pair of contours is equal to the difference between the values of on those contours. Considering this, we detect lanes as parts of a segmented region that fall between two contours of the stream function by a simple intersection operation.

(g) The lane detection algorithm

**Behavior Recognition**

To detect abnormal behavior of crowds, it is necessary to have a global picture of the behaviorin a scene, for which we use potential fields. The surfaces and characterize article positions and velocities in a global sense, and abnormal behaviors are simply detected as large deviations from the expected. Here, we present an algorithm to detect abnormal behavior in crowds using potential functions for the flow (Figure h).

(h) The behavior recognition algorithm

**Divergent/Convergent Regions**

In addition to detecting abnormal behaviors, we incorporate streaklines and the velocity potential to provide a description of the anomaly based on divergent/convergent regions. The extrema on velocity potentials correspond to divergent or convergent regions. To robustly detect these regions, we find the major local extrema of , and then compute the average divergence factor , where **Vi** is defined in Figure (i). Simple thresholding of this factor distinguish divergent/convergent regions

(i) The computation of divergence factor

(h)Qualitative Results

(j) Quantitative Results

Please email me for passwords: ramin at cs dot ucf dot edu.

- Saad Ali and Mubarak Shah, A Lagrangian Particle Dynamics Approach for Crowd Flow Segmentation and Stability Analysis, IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), Minneapolis, June 19-21, 2007.
- UMN Dataset: Unusual Crowd Activity
- Helman, J., Hesselink, L.: Visualizing vector field topology in fluid flows. IEEE Comput. Graph. Appl. 11 (1991) 36-46
- Corpetti, T., Memin, E., Perez, P.: Extraction of singular points from dense motion fields: An analytic approach. Journal of Mathematical Imaging and Vision (2003)

Ramin Mehran, Brian E. Moore, Mubarak Shah, A Streakline Representation of Flow in Crowded Scenes, European Conference on Computer Vision (ECCV), Crete, 2010

**Bibtex Reference**

@inproceedings{MehranMooreShah_ECCV10, author=”Ramin Mehran and Brian E. Moore and Mubarak Shah”, title=”A Streakline Representation of Flow in Crowded Scenes”, booktitle=”European Conference on Computer Vision(ECCV)”, year=”2010″,