Temporal Fusion

Li-De Chen, Li-Qun Weng, Hao-Qian Cheng, An-You Cheng, and Chao-Tsung Huang

A factored display emits full-parallax dense-view light fields for a glasses-free 3D experience without sacrificing the spatial resolution of a liquid-crystal display (LCD). For static light fields, it achieves high-quality reconstruction by applying frame-based low-rank factorization to time-multiplexed sub-frame contents of stacked LCDs. However, for light field videos such frame-based factorization could introduce reconstruction artifacts and visual flickers and further cause human discomfort. The artifacts mainly come from incomplete constraints for the emitted light fields that are actually perceived in continuous time, instead of discrete frames. In particular, the perceived light fields are related to the persistence-of-vision (POV) effect of human eyes and the refresh rates of LCD displays, which is not well explored in previous work.

In this work, we introduce a light-field video factorization framework---temporal fusion (TF)---to resolve these issues. To begin with, we explicitly formulate the continuous-time POV effect into a global factorization objective functional to eliminate visual flickers and enhance image quality. We further show that this optimization problem can be solved by sequence-level iterative updates on LCD sub-frames. Then, to tackle the enormous requirement of memory access for the sequence-level processing flow, we devise an efficient cuboid-wise factorization algorithm which enables practical GPU implementation. We also devise another lightweight causal framework, TF-C, for supporting low-latency applications. Finally, extensive experiments are performed to verify the effectiveness. Compared to the plain frame-based factorization, TF/TF-C can improve temporal consistency by reducing flicker values by 85%/91% and enhance reconstruction quality by increasing PSNR values by 5.0dB/3.7dB. In addition, we present a prototype dual-layer factored display, which was built with two 240-Hz high-refresh-rate LCDs, to demonstrate the visual quality for real-life applications.

Frame Factorization	Backward Residue Removal
Temporal Fusion	Temporal Fusion - Causal

Dual-Layer Factored Display Prototype

Display Prototype.
Main Idea

Temporal modulation and persistence of vision (POV).

Supplement Materials

Justification for Cuboid-Based Factorization
Minimum Required Ranks for Light Field
Resources

Our source codes written in Python (CUDA required): [Hosted on GitLab]
Light Field Video Dataset: [Link]

Light Field Video Dataset

The dataset contains 5 synthesized sequences and 1 real scene sequence. Synthesized sequences are generated by Blender. The real scene sequence is captured with Lytro Illum.