Split-Lohmann Multifocal Displays

Yingsi Qin Wei-Yu Chen Matthew O'Toole Aswin C. Sankaranarayanan

Carnegie Mellon University

SIGGRAPH 2023

The Split-Lohmann Display is a near-eye 3D display capable of generating a dense set of focal planes simultaneously using a single exposure. Our key innovation is an optical arrangement, which we refer to as Split Lohmann, that enables a high degree of local control in depth selection on a traditional display. Given a scene in the form of an RGBD image, we use a RGB display to project the color image while a phase spatial light modulator (SLM) shows a simple analytical function of the depth map. This produces a 3D multifocal scene without requiring time multiplexing. Shown above in is focus stack acquired from our lab prototype using an observer Nikon Z5 camera that is not tethered to the display. Split-Lohmann displays enjoy a number of desirable features including the ability to handle scenes with complex depth maps over a large working range, at high spatial and depth resolutions, and a large étendue. The extreme simplicity of our computational pipeline also enables real-time operations for interactive 3D content.

Publications

SIGGRAPH 2023 Technical Paper

Yingsi Qin, Wei-Yu Chen, Matthew O'Toole, and Aswin C. Sankaranarayanan. 2023. Split-Lohmann Multifocal Displays. ACM Trans. Graph. 42, 4, Article 1 (August 2023), 18 pages.

Paper Supplementary 6-Minute Video Code Blender Model

SIGGRAPH 2023 Emerging Technologies

Yingsi Qin, Wei-Yu Chen, Matthew O'Toole, and Aswin C. Sankaranarayanan. 2023. Single-Shot VR. In Special Interest Group on Computer Graphics and Interactive Techniques Conference Emerging Technologies (SIGGRAPH '23 Emerging Technologies), August 06-10, 2023. ACM, New York, NY, USA, 2 pages.

Extended Abstract 3-Minute Video

Abstract

This work provides the design of a multifocal display that can create a dense stack of focal planes in a single shot. We achieve this using a novel computational lens that provides spatial tunability of its focal length, i.e, the lens appears to have different focal lengths across points on a display behind it. This enables a multifocal display via an appropriate selection of the spatially-varying focal length, thereby avoiding time multiplexing techniques that are associated with traditional focus tunable lenses. The idea central to this design is a modification of a Lohmann lens, a focus tunable lens created with two cubic phase plates that translate relative to each other. Using optical relays and a phase spatial light modulator, we replace the physical translation of the cubic plates with an optical one, while simultaneously allowing for different pixels on the display to undergo different amounts of translations, and consequently, different focal lengths. We refer to this design as the Split-Lohmann multifocal display. Split-Lohmann displays provide a large etendue as well as high spatial and depth resolutions; the absence of time multiplexing as well as the extremely light computational footprint for content processing makes it suitable for video and interactive experiences. Using a lab prototype, we show results over a wide range of static, dynamic and interactive 3D scenes, showcasing high visual quality over a large working range.

Results

The results presented here were acquired using a Nikon Z5 Camera with a 18-55 mm lens operated at focal length of 55 mm. In all results below, the camera was completely untethered to the display without any synchronization.

Static VR: Accommodation

Just like the physical world the human eye sees, when the camera changes focus, it sees sharp content at the focused depth and natural focal blur at other depths.
Both static scenes below occupy a depth range from 0.25 m to infinity. The focus stacks below were captured with remotely controlled focus settings of the camera lens at 100 ISO, 1.3 sec exposure, and f/5.6.

Split-Lohmann displays enjoy a large working depth range, at high spatial and depth resolutions, and a large étendue.

Static VR: Parallax

Observing parallax. In this scene, the foreground lotus is at 0.25 m and the background lotus is at infinity. To observe parallax, we horizontally moved the camera to the left and right by about 3 mm while capturing a video at 3200 ISO and f/11. We observe that scene points closer to the camera laterally translate a larger amount than scene points further from the camera.

Video and Interactive VR

Split-Lohmann displays has an extremely lightweight computational footprint for mapping RGBD content onto the the texture and phase patterns shown on the OLED and phase SLM, respectively. This allows to easily show videos and interactive VR scenes, like 3D games.
While the 3D movie was playing or the interactive game was in session, we remotely controlled the focus setting of the camera lens and captured the videos below using the video mode of the camera at 3200 ISO, f/5.6, and 24 fps.

3D Movie

3D Interactive Game

Methods

The Display

Derivation of Split-Lohmann multifocal displays. We begin with a traditional model for a multifocal display, which involves a translating display behind an eyepiece, as shown in (a). The physical motion of display can be avoided with a 4f relay and a focus tunable lens as shown in (b). Here, we obtain the focus tunablity with a Lohmann lens; relative translation of the two cubic phase plates of the Lohmann lens leads to an optical axial translation of the display. To avoid physical motion of the display, we can split the cubic phase plates using a 4f relay and use a linear phase ramp to induce optical translations instead. This setup is shown in (c); the phase ramp needs to be introduced in the Fourier plane of the cubic plates. Putting it all together in (d), we observe that the display is 4f away from the phase ramp, which we implement using a phase SLM. The 4f system also ensures that the display pixels are now resolved on the SLM, modulo the effect of the cubic phase plate; we can enable local control on the focal length induced on a display pixel. By implementing a phase pattern with piecewise constant slope, we enable spatially-varying axial shifts on the display; we refer to this system as the Split-Lohmann multifocal display.

The Hardware Setup

Schematic of Workflow

Lab Prototype

Citations

Technical Paper

@article{qin2023split,
      author = {Qin, Yingsi and Chen, Wei-Yu and O'Toole, Matthew and Sankaranarayananan, Aswin C.},
      journal = {ACM Transactions on Graphics / SIGGRAPH},
      number = {},
      pages = {},
      title = {Split-Lohmann Multifocal Displays},
      volume = {31},
      month = {Aug},
      year = {2023},
      url = {https://doi.org/10.1145/3592110},
      doi = {10.1145/3592110},
      pdf_supp = {files/paper/2023/Split_Lohmann_SIG23-Supp.pdf},
      url_web = {http://imaging.cs.cmu.edu/split_lohmann/},
     }

Emerging Technologies

@inproceedings{qin2023splitlohdemo,
      author = {Qin, Yingsi and Chen, Wei-Yu and O'Toole, Matthew and Sankaranarayananan, Aswin C.},
      title = {Single-Shot VR},
      year = {2023},
      isbn = {979-8-4007-0154-2/23/08},
      publisher = {Association for Computing Machinery},
      address = {New York, NY, USA},
      url = {https://doi.org/10.1145/3588037.3595396},
      doi = {10.1145/3588037.3595396},
      booktitle = {ACM SIGGRAPH 2023 Emerging Technologies},
      articleno = {},
      numpages = {2},
      location = {Los Angeles, CA, USA},
      series = {SIGGRAPH '23}
      }