CSAIL Publications and Digital Archive header
bullet Research Abstracts Home bullet CSAIL Digital Archive bullet Research Activities bullet CSAIL Home bullet

link to publications.csail.mit.edu link to www.csail.mit.edu horizontal line


Research Abstracts - 2007
horizontal line

horizontal line

vertical line
vertical line

Scene Modeling for Far-Field Activity Analysis

Tomas Izo & Eric Grimson

Fig. 1: Example scene model--a sample of ~3000 moving object tracks (left) grouped into ~100 clusters (right). Tracks with different color belong to different clusters (colors are reused).
Example Scene Model

A key goal of any visual surveillance system is to automatically determine when an observed scene contains unusual or unexpected activity. In the past this task was performed by a human expert: someone familiar with the scene who was able to recognize when something out of the ordinary occurred. A typical surveillance site may have so many sensors in different locations that it is no longer feasible for a person to monitor all of them. Machine vision systems are needed to mine the collected data for potentially interesting activity. This has fostered a new area of machine vision research, aimed at building statistical models of the usual pattern of activity in scenes.


We define an activity as the movement of a single object, such as a car, person or group of people, through the scene. Various aspects of activities, to which we refer as tracks, include the paths the objects take, their sizes, velocities, direction of motion, etc. We propose a method for unsupervised, multi-featured learning of a statistical model of activities that can make use of a very large data set for learning, yet enables fast reasoning about new tracks, both partial and complete. We use a versatile similarity measure that groups together tracks only when they are similar along all of the various observed aspects. We group tracks using spectral clustering and estimate the spectral embedding efficiently from a sample of tracks using the Nystrom approximation [1]. Clusters are modeled as Gaussians in the embedding space and new tracks are projected into the embedding space and matched with the cluster models to detect anomalies. The ability to reason about partial tracks makes it possible to detect surprising moments, which occur when there is a sudden change in the belief distribution for a given track.


To learn a statistical model of activity in the scene, we use a set of approximately 40,000 moving object tracks, corresponding to an entire week of activity in a busy urban outdoor scene. Figure 2 shows two examples of unusual activities detected using the learned model by thresholding on the likelihood of each activity under the model. In addition to obtaining qualitative results such as the ones in Figure 2, we also demonstrate the validity of our scene modeling and anomaly detection framework by collecting human judgments for a sample of several hundred examples from the data set and showing that they are correlated with the likelihoods of the examples under the model.

Fig. 2: Examples of detected unusual activity. Left: A person walking along an unusual path and dropping off a large object (suddenly changing size). Right: A car driving into the pedestrian zone and turning around.
Example Detections
Research Support

This project is supported in part by a grant from DARPA.


[1] Charles F. Fowlkes, Serge Belongie, Fan Chung and Jitendra Malik. Spectral grouping using the Nystrom method. In IEEE Trans. Pattern Analysis and Machine Intelligence, 26(2):214-225, 2004.


vertical line
vertical line
horizontal line

MIT logo Computer Science and Artificial Intelligence Laboratory (CSAIL)
The Stata Center, Building 32 - 32 Vassar Street - Cambridge, MA 02139 - USA
tel:+1-617-253-0073 - publications@csail.mit.edu