Periodic motion
detection and classification
Object motions that repeat are common in both nature and the man-made environment in which we live. Perhaps the most prevalent periodic motions are the ambulatory motions made by humans and animals in their gaits (commonly referred to as "biological motion"). Other examples include a person walking, a waving hand, a rotating wheel, ocean waves, and a flying bird. Knowing that an object’s motion is periodic is a strong cue for object and action recognition. In addition, periodic motion can also aid in tracking objects. Furthermore, the periodic motion of people can be used to recognize individuals.
We analyze the periodicity of a moving object using image similarities of a tracked object. Let Ot be a tracked object at time t, segmented from the background. Objects with periodic motion will appear self-similar with period p. Let S(t1,t2) be a the normalized cross-correlation of Ot1 and Ot2. Then S(t,t+kp) is approximately 1, where k is an integer. Examples of S (which we call a similarity plot), are shown below. To detect and classify periodicities, we analyze the normalized autocorrelation A of S. If the tracked objects contain a periodic motion, then A will have regularly spaced peaks that fit a planar lattice. If the object does not contain periodic motion, then A will have no such peaks. The periodicity test we use is based on how well the peaks in A fit a planar lattice. The similarity plot S encodes the dynamics of the periodic motion. Different types of periodic motion yield qualitatively different types of features in S and A. We exploit this to classify periodic motion.
Below we show both synthetic and real image sequences for various types of periodic motion.
Pendulums
Below are four simple planar pendulum systems. While each system is
periodic, each has qualitatively different types of periodicities, which is
shown in their respective similarity plots an autocorrelations.
|
|
|
|
(left) Pendulum in zero gravity with a constant angular velocity. (middle) Similarity plot for pendulum. Darker pixels are more similar. (right) Autocorrelation of similarity plot.
|
|
|
|
(left) Pendulum in gravity with single angular direction. (middle) Similarity plot for pendulum. (right) Autocorrelation of similarity plot. Peaks are marked with red dots.
|
|
|
|
(left) Pendulum in gravity with an oscillating angular direction. (middle) Similarity plot for pendulum. (right) Autocorrelation of similarity plot.
|
|
|
|
(left) Two pendulum out of phase 180 degrees in gravity. (middle) Similarity plot for pendulum. (right) Autocorrelation of similarity plot.
Motion symmetry of
human walking
The motion symmetry of a walking human is similar to the two pendulum system. The similarity plot has lines parallel to the diagonal and cross diagonal. The autocorrelation has peaks in a 45° rotated square lattice.
|
|
|
|
(left) Person walking on a treadmill. (middle) Similarity plot for person walking. (right) Autocorrelation of similarity plot.
|
|
|
|
|
The walking phase is encoded in the similarity plot. The intersection of the lines in the similarity plot correspond to one of the poses: {A,A}, {B,B}, {C,C}, {C,A}, {A,C}. Note that the similarity of {C,A} and {A,C} is less that {A,A}, {B,B}, {C,C}.
Motion symmetry of
running dogs
The motion symmetry of a running dog is very different from a walking or running human. In the cycle of a running dog, no two images are significantly similar (see the cycle below). Therefore, the similarity plot has lines parallel to the diagonal, but no cross-diagonal lines. Similar to the pendulum in gravity and single angular direction, the autocorrelation has a square lattice structure.
|
|
|
|
(left) Running dog. (middle) Similarity plot for person walking. (right) Autocorrelation of similarity plot.
Cycle for a running dog. Note that unlike humans, no two images in the
cycle are similar.
Airborne video
surveillance video
The following image sequence was taken from an airborne video surveillance camera at an altitude of 2000 feet. The video is very poor quality, due to being digitized from a duplicated SVHS tape, and there is severe motion blur due to a long (automatic) exposure on the camera. The person running in the center of this image is shown below (digitally zoomed by 4X); he is approximate 7 pixels in height. Due to motion parallax, the telephone pole is also tracked. The size and aspect area of the tracked portion of the telephone pole and the person are similar, as is their ground speed. However, note the difference between their similarity plots and autocorrelation.
Airborne surveillance video (altitude about 2000', effective resolution 320x240)
|
|
|
|
(left) Person running. (middle) Similarity plot for runner. (right) Autocorrelation of similarity plot.
|
|
|
|
(left) Telephone pole. (middle) Similarity plot for pole. (right) Autocorrelation of similarity plot.
Related
Publications
Ross
Cutler and Larry Davis. “Robust Real-Time Periodic Motion Detection,
Analysis, and Applications.” Accepted for publication in IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI),
2000.
Ismail Haritaoglu, Ross Cutler, David Harwood, and Larry Davis. “Backpack: Detection of People Carrying Objects Using Silhouettes.” Accepted for publication in Computer Vision and Image Understanding, 2001.
Ross Cutler and Larry Davis. “Robust periodic motion and motion symmetry detection.” IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), June 2000, Hilton Head Island, South Carolina. PDF.
Ross Cutler, Chandra Shekhar, Brian Burns, Rama Chellappa, Robert Bolles, Larry Davis. “Monitoring Human and Vehicle Activities Using Airborne Video.” 28th Applied Imagery Pattern Recognition Workshop (AIPR), October 1999, Washington, D.C. PDF.
Ismail Haritaoglu, Ross Cutler, David Harwood, and Larry Davis. “Backpack: Detection of People Carrying Objects Using Silhouettes.” IEEE International Conference on Computer Vision (ICCV), September 1999, Kerkyra, Greece. PDF.
Ross Cutler and Larry Davis. “Real-Time Periodic Motion Detection, Analysis, and Applications.” IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), June 1999, Fort Collins, Colorado. PDF.
Ross Cutler and Larry Davis. “View-based Detection and Analysis of Periodic Motion.” IEEE International Conference on Pattern Recognition (ICPR), August 1998, Brisbane, Australia. PDF. Also published in DARPA Image Understanding Workshop, November 1998, Monterey, California.
Ross Cutler and Larry Davis. “Qualitative Analysis of Human Actions,” Image Understanding Workshop, May 1997, New Orleans.
Ross Cutler's home
page
Last updated on June 18, 2000