Adjoint Nonlinear Ray Tracing

Arjun Teh, Matthew O'Toole, Ioannis Gkioulekas


Reconstructing and designing media with continuously-varying refractive index fields remains a challenging problem in computer graphics. A core difficulty in trying to tackle this inverse problem is that light travels inside such media along curves, rather than straight lines. Existing techniques for this problem make strong assumptions on the shape of the ray inside the medium, and thus limit themselves to media where the ray deflection is relatively small. More recently, differentiable rendering techniques have relaxed this limitation, by making it possible to differentiably simulate curved light paths. However, the automatic differentiation algorithms underlying these techniques use large amounts of memory, restricting existing differentiable rendering techniques to relatively small media and low spatial resolutions.

We present a method for optimizing refractive index fields that both accounts for curved light paths and has a small, constant memory footprint. We use the adjoint state method to derive a set of equations for computing derivatives with respect to the refractive index field of optimization objectives that are subject to nonlinear ray tracing constraints. We additionally introduce discretization schemes to numerically evaluate these equations, without the need to store nonlinear ray trajectories in memory, significantly reducing the memory requirements of our algorithm. We use our technique to optimize high-resolution refractive index fields for a variety of applications, including creating different types of displays (multiview, lightfield, caustic), designing gradient-index optics, and reconstructing gas flows.

Continuously Refractive Materials

Computer graphics research has long been fascinated by the way light refracts through transparent materials to form beautiful, but intricate, caustics (e.g., the patterns at the bottom of a pool of water). Although typically such caustics are the result of light undergoing a discrete number of refraction events, it is also possible for light to bend continuously through certain objects and media. For example, the shimmer observed when looking through a mass of heated air (e.g., the heat haze observed when looking down an asphalt road on a hot day) is the result of light propagating along curved paths. Light deflects when it encounters a change in refractive index. Most simple lenses are made from material with a constant refrac- tive index, which results in light refracting only at the interface.

Luneburg lens
Transparent gas flow (10x)

Constant memory in step size

Comparison of memory usage

Our method has constant memory usage with respect to step size, whereas reverse-mode AD has a linear dependence on step size. Increasing the volume resolution also affects reverse-mode AD significantly more than our method. As resolution increases, the memory foot- print of the volume increases by N3 . However, the step size needs to decrease along with the increase in resolution so that traced rays sample the volume voxels properly. The decrease in step size domi- nates the memory resources more so than the cubic increase in the volume, which is why we see a linear dependence on resolution for reverse-mode AD as well.

Key idea

We achieve constant memory by observing that we do not need to store the ray locations traced during forward tracing. We only need to use the final position and velocity at the end of the forward tracing stage. As a result, the memory use of our combined forward tracing and back- ward tracing algorithms is constant, instead of scaling linearly with step-size and number of steps, as in conventional reverse-mode AD.

Forward tracing
Gradient calculation
Gradient index update


multiview display

einstein turing image

Luneburg and Maxwell lens reconstruction

Optimization of the Luneburg lens
Reconstructed Luneburg lens
Reconstructed Maxwell lens

light field display


More Details

For a detailed discussion on the theory behind our work, please refer to our paper and supplemental material.


Link to paper

supplemental material

Link to paper

Recorded talks



Our implementation is available on GitHub.


This work was supported by the National Science Foundation (NSF) under awards 1730147, 1900849, and 2008464. Ioannis Gkioulekas was supported by a Sloan Research Fellowship.

Copyright © 2022 Arjun Teh