[Ref.] Eulerian video magnification for revealing subtle changes in the world.
Our goal is to reveal temporal variations in videos that are difficult or impossible to see with the naked eye and display them inan indicative manner. Our method, which we call Eulerian VideoMagnification, takes a standard video sequence as input, and applies
spatial decomposition, followed by temporal filtering to the
frames. The resulting signal is then amplified to reveal hidden information.
Using our method, we are able to visualize the flow
of blood as it fills the face and also to amplify and reveal small
motions. Our technique can run in real time to show phenomena
occurring at temporal frequencies selected by the user.
CR Categories: I.4.7 [Image Processing and Computer Vision]:
Scene Analysis—Time-varying Imagery;
Keywords: video-based rendering, spatio-temporal analysis, Eulerian
motion, motion magnification
The human visual system has limited spatio-temporal sensitivity,
but many signals that fall below this capacity can be informative.
For example, human skin color varies slightly with blood circulation.
This variation, while invisible to the naked eye, can be exploited
to extract pulse rate [Verkruysse et al. 2008; Poh et al. 2010;
Philips 2011]. Similarly, motion with low spatial amplitude, while
hard or impossible for humans to see, can be magnified to reveal
interesting mechanical behavior [Liu et al. 2005]. The success of
these tools motivates the development of new techniques to reveal
invisible signals in videos. In this paper, we show that a combination
of spatial and temporal processing of videos can amplify subtle
variations that reveal important aspects of the world around us.
Our basic approach is to consider the time series of color values at
any spatial location (pixel) and amplify variation in a given temporal
frequency band of interest. For example, in Figure 1 we automatically
select, and then amplify, a band of temporal frequencies
that includes plausible human heart rates. The amplification reveals
the variation of redness as blood flows through the face. For this
application, temporal filtering needs to be applied to lower spatial
frequencies (spatial pooling) to allow such a subtle input signal to
rise above the camera sensor and quantization noise.
Our temporal filtering approach not only amplifies color variation,
but can also reveal low-amplitude motion. For example, in the supplemental
video, we show that we can enhance the subtle motions
around the chest of a breathing baby. We provide a mathematical
analysis that explains how temporal filtering interplays with spatial
motion in videos. Our analysis relies on a linear approximation related
to the brightness constancy assumption used in optical flow
formulations. We also derive the conditions under which this approximation
holds. This leads to a multiscale approach to magnify
motion without feature tracking or motion estimation.
Previous attempts have been made to unveil imperceptible motions
in videos. [Liu et al. 2005] analyze and amplify subtle motions and
visualize deformations that would otherwise be invisible. [Wang
et al. 2006] propose using the Cartoon Animation Filter to create
perceptually appealing motion exaggeration. These approaches follow
a Lagrangian perspective, in reference to fluid dynamics where
the trajectory of particles is tracked over time. As such, they relyon accurate motion estimation, which is computationally expensiveand difficult to make artifact-free, especially at regions of occlusion
boundaries and complicated motions. Moreover, Liu et al. 
have shown that additional techniques, including motion segmentation
and image in-painting, are required to produce good quality
synthesis. This increases the complexity of the algorithm further.
In contrast, we are inspired by the Eulerian perspective, where
properties of a voxel of fluid, such as pressure and velocity, evolve
over time. In our case, we study and amplify the variation of pixel
values over time, in a spatially-multiscale manner. In our Eulerian
approach to motion magnification, we do not explicitly estimate
motion, but rather exaggerate motion by amplifying temporal color
changes at fixed positions. We rely on the same differential approximations
that form the basis of optical flow algorithms [Lucas and
Kanade 1981; Horn and Schunck 1981].
Temporal processing has been used previously to extract invisible
signals [Poh et al. 2010] and to smooth motions [Fuchs et al. 2010].
For example, Poh et al.  extract a heart rate from a video of a
face based on the temporal variation of the skin color, which is normally
invisible to the human eye. They focus on extracting a single
number, whereas we use localized spatial pooling and bandpass filtering
to extract and reveal visually the signal corresponding to the
pulse. This primal domain analysis allows us to amplify and visualize
the pulse signal at each location on the face. This has important
potential monitoring and diagnostic applications to medicine,
where, for example, the asymmetry in facial blood flow can be a
symptom of arterial problems.
Fuchs et al.  use per-pixel temporal filters to dampen temporal
aliasing of motion in videos. They also discuss the high-pass
filtering of motion, but mostly for non-photorealistic effects and for
large motions (Figure 11 in their paper). In contrast, our method
strives to make imperceptible motions visible using a multiscale
approach. We analyze our method theoretically and show that it
applies only for small motions.
In this paper, we make several contributions. First, we demonstrate
that nearly invisible changes in a dynamic environment can be
revealed through Eulerian spatio-temporal processing of standard
monocular video sequences. Moreover, for a range of amplification
values that is suitable for various applications, explicit motion estimation
is not required to amplify motion in natural videos. Our
approach is robust and runs in real time. Second, we provide an
analysis of the link between temporal filtering and spatial motion
and show that our method is best suited to small displacements and
lower spatial frequencies. Third, we present a single framework
that can be used to amplify both spatial motion and purely temporal
changes, e.g., the heart pulse, and can be adjusted to amplify particular
temporal frequencies—a feature which is not supported by
Lagrangian methods. Finally, we analytically and empirically compare
Eulerian and Lagrangian motion magnification approaches under
different noisy conditions. To demonstrate our approach, we
present several examples
Read more in:
Wu, H. Y., Rubinstein, M., Shih, E., Guttag, J. V., Durand, F., & Freeman, W. T. (2012). Eulerian video magnification for revealing subtle changes in the world. ACM Trans. Graph., 31(4), 65.