Maintaining Temporal Coherence in Video Retargeting Using Mosaic-Guided Scaling
Tzu-Chieh Yen, Chia-Ming Tsai and Chia-Wen Lin
Abstract
Video retargeting from a full-resolution video to a lower-resolution display will inevitably cause information loss. Content-aware video retargeting techniques have been studied to avoid critical visual information loss while retargeting a video. Maintaining the spatio-temporal coherence of a retargeted video is very critical on visual quality. Camera motions and object motions, however, usually make it difficult to maintain temporal coherence with existing video retargeting schemes. In this thesis, we propose the use of a panoramic mosaic to guide the scaling of corresponding regions of video frames in a video shot to ensure good temporal coherence. In the proposed method, after aligning video frames in a shot to a panoramic mosaic constructed for the shot, a global scaling map for these frames is derived from the panoramic mosaic. Subsequently, the local scaling maps of individual frames are derived from the global map and is further refined according to spatial coherence constraints. Our experimental results show that the proposed method can effectively maintain temporal coherence so as to achieve good visual quality even a video contains camera motions and object motions.
Key words: Video Adaptation, Video Retargeting, Video Scaling, Spatio-temporal Coherence
Video Demo (download all original sequences )
We compare the proposed method with the uniform scaling, [Rubinstein et al.] , [Wolf et al.] and [Wang et al.] respectively.
TestSequence_01
The original resolution: 688x288
The target resolution: 344x288
TestSequence_02
The original resolution: 688x288
The target resolution: 344x288
TestSequence_03
The original resolution: 320x160
The target resolution: 160x160
TestSequence_04
The original resolution: 720x352
The target resolution: 360x352
TestSequence_05
The original resolution: 656x368
The target resolution: 328x368
We show an example of integrating the sampling based image retargeting method with the proposed mosaic-guided scaling method [Kim et al.]. Obviously, the sampling based image retargeting method cannot maintain consistent sizes of the foreground objects in neighboring frames. However, by integrating the proposed method, the retargeting result does not only maintain the temporal coherence but also preserves the important content.
Result_01
The original resolution: 776x436
The target resolution: 388x436
References
* J.-S. Kim, J.-H. Kim and C.-S. Kim, "Adaptive image and video retargeting technique based on fourier analysis," in Proc. IEEE Int. Conf. Computer Vision and Pattern Recognition, pp. 1730-1737, Sept. 2009, Kyoto, Japan.
* M. Rubinstein, A. Shamir and S. Avidan, ˇ§Improved seam carving for video retargeting,ˇ¨ ACM Trans. Graphics, vol. 27, no. 3, pp. 16, 2008.
* Y.-S. Wang, H. Fu, O. Sorkine, T.-Y. Lee and H.-P. Seidel, ˇ§Motion-aware temporal coherence for video resizing,ˇ¨ ACM Trans. Graphics, vol. 28, no. 5, 2009.
* L. Wolf, M. Guttmann and D. Cohen-Or, "Non-homogeneous content-driven video-retargeting," in Proc. IEEE Int. Conf. Computer Vision, 2007, pp. 1-6.