DP sensors were introduced in commercial cameras, such as DSLRs, mirrorless, and smartphones, to improve autofocus. Nowadays, they are becoming increasingly commonplace. Each pixel of a dual-pixel sensor is split in two parts, and thus the sensor simultaneously captures two sub images per exposure.  We can think of it as a two-sample light field camera, or a stereo system with a tiny baseline.

To see why a dual-pixel sensor can help with depth from defocus, let’s consider its image formation model in a lens-based camera. This is very similar to the model for a regular sensor, except that each half pixel is integrating light from half the main lens aperture. An in-focus scene point produces identical left and right sub-images, whereas an out-of-focus point produces shifted sub-images. The shift, or disparity, depends on the distance of the scene point to the in-focus plane, and can be converted to depth in a calibrated system. Note that the sum of the two sub-images is the same as the image from a regular sensor. From a regular image alone, we cannot tell whether the point is in front of or behind the in-focus plane, whereas from the two sub-images, we can.

Our goal is to use the information available in a single dual-pixel image, and jointly recover the scene’s defocus map and all-in-focus image.

We test our method on a new dataset of 17 indoor and outdoor scenes we captured using a Google Pixel 4 camera. To generate ground truth, we use a commercial depth from focus software called Helicon Focus. This software computes all-in-focus images and defocus maps from input focal stacks. Note that we only use the generated ground truth for comparisons, and not for training.

Results on Defocus Map Estimation and Defocus Deblurring

We provide an interactive HTML viewer to facilitate comparisons across our full dataset. (Click on the image!)

For more details, please refer to our paper, supplementary material, poster, and video.

Shumian Xin, Neal Wadhwa, Tianfan Xue, Jonathan T. Barron, Pratul P. Srinivasan, Jiawen Chen, Ioannis Gkioulekas, and Rahul Garg. Defocus Map Estimation and Deblurring from a Single Dual-Pixel Image, ICCV 2021