Skip to content

Add lumen-gi-enabled / lumen-reflections-enabled camera capture settings (captures match the viewport by default) - #181

Open
andrewjong wants to merge 1 commit into
iamaisim:mainfrom
castacks:feat/lumen-scene-capture-settings
Open

Add lumen-gi-enabled / lumen-reflections-enabled camera capture settings (captures match the viewport by default)#181
andrewjong wants to merge 1 commit into
iamaisim:mainfrom
castacks:feat/lumen-scene-capture-settings

Conversation

@andrewjong

@andrewjong andrewjong commented Aug 14, 2026

Copy link
Copy Markdown

About

TL;DR adds Lumen support to AirSim camera images.

Camera images captured through USceneCaptureComponent2D don't render with the global illumination / reflection method the viewport uses: a capture view seeds its method from the project CVars but never blends level PostProcessVolumes, and on hardware-ray-tracing projects the capture's rays have no acceleration-structure data unless the component opts in. The practical result is that a Lumen-lit level comes out flat and washed out in the sim's camera images — all indirect light missing — even though the game/editor viewport looks correct (see screenshots below).

Since UE 5.5 the engine natively supports Lumen in scene captures: when a capture's post-process settings resolve to Lumen, the renderer maintains a dedicated Lumen scene for the capture's persistent view state (which UnrealCamera already enables via bAlwaysPersistRenderingState).

This PR adds two per-capture tri-state settings, documented in docs/sensors/camera_capture_settings.md:

  • Key absent (the default): match the viewport. Scene (RGB) captures mirror the project's r.DynamicGlobalIlluminationMethod / r.ReflectionMethod, so Lumen projects get Lumen camera images out of the box. Every other image type (depth, segmentation, ...) resolves to None — their output comes from a replacement material, and previously they silently inherited the project CVar, paying Lumen cost for nothing.
  • lumen-gi-enabled / lumen-reflections-enabled = true: force Lumen for that capture regardless of the project default.
  • false: force the method to None. This is an active override — an un-overridden capture inherits the project CVar, so omitting the override is not an off switch.

The resolved methods are pinned as explicit post-process overrides on each capture, and bUseRayTracingIfEnabled is set whenever either method resolves to Lumen so hardware-ray-traced Lumen works (software Lumen works without it). The RGB wire format is unchanged (SCS_FinalColorLDR, 8-bit — in UE 5.5+ it runs the full viewport film tone curve). On engines older than 5.5 the overrides are ignored for scene captures, so this is a no-op there.

Cost note: each Lumen-enabled capture maintains its own Lumen scene (GPU time + VRAM) — numbers below; set the knobs to false per capture to opt out where frame rate matters more than fidelity.

How Has This Been Tested?

UE 5.7.4, Vulkan SM6, Linux (Epic unreal-engine:dev-5.7.4 container), photorealistic office scene (marketplace "QA Office" level, Lumen GI + reflections + HWRT in project settings), non-physics robot with two 640×480 RGB+depth-planar cameras at a 0.1 s capture interval:

  • Fixed-pose A/B across all three states (same build, only the jsonc differs): keys absent → captures match the viewport's lighting (Lumen); false → matches the pre-PR captures (verified visually and by luminance statistics); true → forces Lumen. The on/off pair was verified at four camera poses, the full tri-state matrix at one.
  • Quantitative: a ceiling lit only by GI bounce goes from 59/255 to 106/255 mean luminance (global mean is unchanged because auto-exposure renormalizes — the effect is in the light distribution and reflections).
  • Cost measured: two 640×480 Lumen RGB captures: 4.2 → 3.8 Hz capture rate (~25 ms/frame cycle) and +0.5 GiB VRAM.
  • Depth-planar output verified unchanged alongside the RGB runs.

Screenshots and videos:

Lobby — ceiling receives no direct light, only GI bounce; the marble floor gains reflections:

view1 off vs on

Lounge — ceiling, brass stanchions, chrome table legs:

view2 off vs on

Corridor — the trash can turns from flat matte to brushed metal with environment reflections:

view3 off vs on

Game-viewport reference at the same pose as the corridor pair — the Lumen-on capture matches its lighting (the yellow on-screen text is an unrelated engine debug message, viewport-only):

game viewport reference

🤖 Generated with Claude Code

…ings

UE scene captures don't render with the global illumination / reflection
method the viewport uses: a capture view seeds its method from the project
CVar but never blends level PostProcessVolumes, and on hardware-ray-tracing
projects its rays have no acceleration-structure data unless the component
opts in — so a Lumen-lit level comes out flat and washed out in captured
images, with all indirect light missing. Since UE 5.5 the engine supports
Lumen in scene captures when the capture's post-process settings resolve to
Lumen (the renderer then maintains a dedicated Lumen scene per capture's
persistent view state, which UnrealCamera already enables via
bAlwaysPersistRenderingState).

Add two per-capture tri-state settings, keyed off whether the jsonc key is
present:

- absent (default): match the viewport. Scene (RGB) captures mirror the
  project's r.DynamicGlobalIlluminationMethod / r.ReflectionMethod, so
  Lumen projects get Lumen captures out of the box; every other image type
  resolves to None — their output is a replacement material, so dynamic GI
  underneath is pure GPU/VRAM waste (and previously they silently
  inherited the project CVar).
- lumen-gi-enabled / lumen-reflections-enabled = true: force Lumen for
  this capture regardless of the project default.
- false: force the method to None. This is an active override — an
  un-overridden capture inherits the project CVar, so omitting the
  override is not a way to turn Lumen off.

The resolved methods are pinned as explicit post-process overrides, and
bUseRayTracingIfEnabled is set whenever either method resolves to Lumen so
hardware-ray-traced Lumen works when the project uses it (software Lumen
works without). On engines older than 5.5 the overrides are ignored for
captures.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant