Maintaining Temporal Coherence in Video Retargeting Using Mosaic-Guided Scaling

Tzu-Chieh Yen, Chia-Ming Tsai and Chia-Wen Lin



Video retargeting from a full-resolution video to a lower-resolution display will inevitably cause information loss. Content-aware video retargeting techniques have been studied to avoid critical visual information loss while retargeting a video. Maintaining the spatio-temporal coherence of a retargeted video is very critical on visual quality. Camera motions and object motions, however, usually make it difficult to maintain temporal coherence with existing video retargeting schemes. In this thesis, we propose the use of a panoramic mosaic to guide the scaling of corresponding regions of video frames in a video shot to ensure good temporal coherence. In the proposed method, after aligning video frames in a shot to a panoramic mosaic constructed for the shot, a global scaling map for these frames is derived from the panoramic mosaic. Subsequently, the local scaling maps of individual frames are derived from the global map and is further refined according to spatial coherence constraints. Our experimental results show that the proposed method can effectively maintain temporal coherence so as to achieve good visual quality even a video contains camera motions and object motions.

Key words: Video Adaptation, Video Retargeting, Video Scaling, Spatio-temporal Coherence


Video Demo (download all original sequences )

We compare the proposed method with the uniform scaling, [Rubinstein et al.] , [Wolf et al.] and [Wang et al.] respectively.


The original resolution: 688x288

The target resolution: 344x288

(download files)


The original resolution: 688x288

The target resolution: 344x288

(download files)


The original resolution: 320x160

The target resolution: 160x160

(download files)


The original resolution: 720x352

The target resolution: 360x352

(download files)


The original resolution: 656x368

The target resolution: 328x368

(download files)

We show an example of integrating the sampling based image retargeting method with the proposed mosaic-guided scaling method [Kim et al.]. Obviously, the sampling based image retargeting method cannot maintain consistent sizes of the foreground objects in neighboring frames. However, by integrating the proposed method, the retargeting result does not only maintain the temporal coherence but also preserves the important content.


The original resolution: 776x436

The target resolution: 388x436

(download files)



* J.-S. Kim, J.-H. Kim and C.-S. Kim, "Adaptive image and video retargeting technique based on fourier analysis," in Proc. IEEE Int. Conf. Computer Vision and Pattern Recognition, pp. 1730-1737, Sept. 2009, Kyoto, Japan.

* M. Rubinstein, A. Shamir and S. Avidan, ˇ§Improved seam carving for video retargeting,ˇ¨ ACM Trans. Graphics, vol. 27, no. 3, pp. 16, 2008.

* Y.-S. Wang, H. Fu, O. Sorkine, T.-Y. Lee and H.-P. Seidel, ˇ§Motion-aware temporal coherence for video resizing,ˇ¨ ACM Trans. Graphics, vol. 28, no. 5, 2009.

* L. Wolf, M. Guttmann and D. Cohen-Or, "Non-homogeneous content-driven video-retargeting," in Proc. IEEE Int. Conf. Computer Vision, 2007, pp. 1-6.