Relight

How Relight works

Two pretrained vision models measure the photo once. Everything that moves when you drag the light is our own shading code, running in your browser. Here is each step on the Portrait sample, and who does it.

Once per photo

For the four samples this was done ahead of time and the two maps ship with the page. For your own photo, this demo runs the depth model in your browser and takes the normals from the slope of the depth map, which shades faces flatter: the normals model is too large for a browser. The photo is not uploaded.

Step 1: Estimate depth

Pretrained model: Depth Anything V2

A pretrained model estimates how near every pixel is to the camera. We did not train it; we scale its output into a height field that the shadows in step 4 are traced across.

An astronaut in an orange suit, head and shoulders
The photo
The same photo as a grey depth map: the astronaut and the helmet in front are light, the wall behind is dark
Depth map: lighter is nearer

Step 2: Estimate surface direction

Pretrained model: Marigold

A second pretrained model estimates which way every surface faces. It picks out a nose, cheeks and cloth folds that the depth map does not have.

The same photo as a normal map in purple, pink and green, with the nose, cheeks and folds of the suit clearly drawn
Normal map: colour is the way a surface faces

Every frame, in your browser

Our lighting code runs steps 3 and 4 again each time the light moves: in a WebGL2 shader where the device allows, otherwise on the device's processor. No AI model is involved in the dragging.

Step 3: Shade each pixel from the light

Our code

Diffuse shading: a pixel is bright when its surface faces the light. Brightness over distance is measured against the middle of the scene, so moving the light changes where it falls and not the overall exposure.

A grey image of the astronaut lit from the left like a plaster cast, with no colour and no cast shadows
Our shading by itself, before it is applied to the photo: light at the left

Step 4: Trace cast shadows in three layers

Our code

Each pixel walks toward the light across the height field to see whether anything stands in the way. What blocks it is sorted into a near, mid or far layer by how far in front it stands, and each layer gets its own blur, so a shadow softens with distance.

A white image with a soft grey shadow thrown to the right of the astronaut's head and shoulder
Our shadow map for the same light
The astronaut photo relit from the left: the left of the face and suit is bright, the right side is in shadow
Steps 3 and 4 multiplied into the photo
The astronaut photo relit from the right: the right of the face is bright and the left is in shadow
The same maps, light moved to the right
Moving the light only reruns steps 3 and 4. The stills in this panel were rendered with our Python version of the same maths; the shader is a port of it and the two are kept in agreement.

Only if you press Enhance

Switched off in this demo, because it needs the server.

Step 5: A photoreal pass

Hosted model, optional

Our shaded preview is sent to a hosted image model (the Gemini image model, or IC-Light v2 on fal.ai), with a short text prompt, for a more photographic finish. This is the only step that calls a service we did not write or run.

Honest limits

  • One photo gives a height field, not real 3D, so shadows are approximate: light from below can look blotchy on a face, and a faint dark column can remain on open sky.
  • Shading is diffuse only (no shine), and the light already in the photo is not removed. The live shader has been tested on five devices so far: a Mac, two Windows laptops and two phones.
demo

Move the light

Relight a photo by dragging the light.

Upload a photo or pick a sample. Its depth is estimated once, right here in your browser, which then shades it live: drag the light anywhere around the subject and the highlights and shadows follow instantly.

Drop a photo hereor click to browse, or paste an image or its address

Your photo stays on this device: its shape is worked out in your browser and nothing is uploaded. The first photo downloads a depth model of about 25 MB, once.

Or try a sample photo: