Rendering radiative transfer through media with a heterogeneous refractive index is challenging because the continuous refractive index variations result in light traveling along curved paths. Existing algorithms are based on photon mapping techniques, and thus are biased and result in strong artifacts. On the other hand, existing unbiased methods such as path tracing and bidirectional path tracing cannot be used in their current form to simulate media with a heterogeneous refractive index. We change this state of affairs by deriving unbiased path tracing estimators for this problem. Starting from the refractive radiative transfer equation (RRTE), we derive a path-integral formulation, which we use to generalize path tracing with next-event estimation and bidirectional path tracing to the heterogeneous refractive index setting. We then develop an optimization approach based on fast analytic derivative computations to produce the point-to-point connections required by these path tracing algorithms. We propose several acceleration techniques to handle complex scenes (surfaces and volumes) that include participating media with heterogeneous refractive fields. We use our algorithms to simulate a variety of scenes combining heterogeneous refraction and scattering, as well as tissue imaging techniques based on ultrasonic virtual waveguides and lenses. Our algorithms and publicly-available implementation can be used to characterize imaging systems such as refractive index microscopy, schlieren imaging, and acousto-optic imaging, and can facilitate the development of inverse rendering techniques for related applications.

Media characterized by continuously-varying refractive index, occurring due to variations in temperature and pressure, or due to heterogeneous mixing of materials, are common in the real world. Examples of such ** heterogeneous refractive media ** include the atmosphere, biological tissue, crystals, minerals, and transparent plastics. When light travels through such media, it follows curved trajectories due to the continuous refraction. This bending of light can be observed by shining a laser beam on a heterogeneous refractive medium, e.g., as shown in figure to the right.

The aquarium shown to the left of the figure is filled with a sugar solution whose refractive index increases linearly with the liquid's depth. A laser beam propagating through this solution refracts and scatters continuously, resulting in a curved light trajectory. Multiple such trajectories appear due to Fresnel reflection of the beam on the aquarium's wall.

Light transport in heterogeneous refractive media is described using the ** refractive radiative transfer equation ** (RRTE) that, in addition to light bending due to continuous refraction, also models effects due to volumetric and surface scattering. The light bending effects make this equation significantly more challenging to simulate than its counterpart for homogeneous refractive media, the radiative transfer equation. Existing rendering algorithms are based on photon mapping techniques; these algorithms are efficient but biased, and can introduce significant artifacts in the output images. By contrast, unbiased algorithms such as path tracing, particle tracing, and bidirectional path tracing, are inefficient or even completely intractable for rendering refractive radiative transfer.

The main challenge in applying these unbiased algorithms to heterogeneous refractive media is the difficulty of performing **direct connections** between two points inside such a medium. Direct connections are needed, e.g., for next event estimation in path tracing and particle tracing, and to connect the source and sensor subpaths in bidirectional path tracing. Connecting two points inside a homogeneous refractive medium is trivial---one only needs to trace the linear segment between the two points. By contrast, connecting two points inside a heterogeneous refractive medium requires finding paths, generally curved, starting and ending at the two points that are solutions of the **eikonal equation**. This equation accounts for the variable (inversely proportional to refractive index) speed of light inside heterogeneous refractive media. The solutions to the eikonal equation are paths satisfying Fermat's principle, in that they are locally **stationary** with respect to the time it takes light to traverse them or, equivalently, with respect to **optical pathlength**.

In this paper, we address this challenge by developing techniques for computing these stationary paths, efficiently and without bias. Our technique is based on the fact that the eikonal equation, and the ray tracing equations derived from it, are differentiable. Therefore, we show that we can use use efficient gradient-based optimization algorithms to search for paths that connect two points, while also satisfying the eikonal equation. Special attention is required to account for the fact that, in heterogeneous refractive media, there may exist more than one stationary paths connecting two points: our optimization-based technique in its simple form would only find one of these paths, resulting in bias, and is not practical for enumerating all stationary paths. To ensure unbiasedness and maintain efficiency, we introduce a Monte Carlo technique that forms an unbiased estimate of the total throughput through multiple stationary paths, through repeated random reinitializations of our gradient-based optimization algorithm. To further improve efficiency, we show how to accelerate direct connections using techniques similar to sphere tracing of signed distance functions, and inside-outside tests based on fast winding numbers.

We use our implementation to perform a series of experiments on different scenes, including robustness tests for our direct connection procedure, visualization of volumetric caustics, time-of-flight and spectral rendering, as well as ultrasonic lensing. In the latter case, we also compare our renderings to real measurements.

For an in-depth description of the technology behind this work, please refer to our paper and accompanying videos.

Adithya Pediredla, Yasin Karimi Chalmiani, Matteo Giuseppe Scopelliti, Maysam Chamanzar, Srinivasa Narasimhan, and Ioannis Gkioulekas. *"Path Tracing Estimators for Refractive Radiative Transfer"*, SIGGRAPH Asia 2020

**GitHub repository** and you can also download our **Docker file**. It is based on the Mitsuba renderer and also supports a large variety of time-of-flight cameras, including transient, time-gated, and continuous-wave amplitude-modulated cameras. For any questions, please contact Adithya Pediredla.

This work was supported by NSF Expeditions award 1730147, NSF award 1935849, DARPA REVEAL contract HR0011-16-C-0028, and a gift from AWS Cloud Credits for Research.

Copyright © 2020 Adithya Pediredla