Real-time ray tracing in a custom game engine rests on three architectural commitments: a programmable pipeline you control stage by stage, a two-level hierarchy of acceleration structures that turns raw triangle buffers into something rays can traverse quickly, and a denoising pass that lets you trace far fewer rays than the final image appears to need. Get the structure right and the rest of your renderer becomes a sequence of well-defined decisions. This article walks through those stages, explains how the bottom-level and top-level acceleration structures (BLAS and TLAS) fit together, and maps the key performance trade-offs you will face when your scenes move.
Understanding the Real-Time Ray Tracing Pipeline Architecture
The most important mental shift for an engine developer coming from rasterization is that ray tracing is a programmable system, not a fixed function. The NVIDIA OptiX engine, described in a design article in Communications of the ACM, builds on the observation that most ray tracing algorithms can be implemented with a small set of programmable operations: its core is a domain-specific just-in-time compiler that generates custom kernels by combining user-supplied programs for ray generation, material shading, object intersection, and scene traversal.
Crucially, OptiX's designers point out that such an engine is not a renderer. It focuses exclusively on the fundamental computations required for ray tracing and deliberately avoids embedding rendering-specific constructs such as lights, shadows, and reflectance. That distinction is exactly the one you face in a custom engine: the ray tracing layer is machinery that turns rays into hits, and all of your lighting models, materials, and effects live in the programmable programs you attach to it.
The pipeline is built for massively parallel architectures. Research surveys of hardware ray tracing, published in Applied Sciences, describe designs centered on parallel, pipelined traversal units alongside flexible, high-performance kernels for shading and ray generation — with some hardware designs pairing dedicated traversal hardware with a CPU for rebuilding trees in dynamic scenes. The architecture you build in software mirrors that shape: thousands of rays in flight, each stepping through a shared acceleration structure and invoking your programs when something interesting happens.
Programmable Stages of the Ray Tracing Pipeline
Your programmable surface breaks into three distinct responsibilities. Ray generation is where every trace begins: a custom program defines the origin and direction of rays cast into the scene — camera rays, shadow rays, reflection rays, whatever your effect needs. You decide the ray budget here, and that budget is the single biggest lever on your frame time.
Intersection and traversal are the fixed-function heart of the system, driven by the acceleration structure. Rays walk the hierarchy, and when one reaches actual geometry, your closest-hit shader runs to evaluate the surface. When a ray touches nothing, the miss shader runs instead. As NVIDIA's Martin-Karl Lefrancois explains in a technical blog series on practical real-time ray tracing, the miss shader "kicks in when you don't touch anything" — it is your environment or sky shader, and it also handles cases like shadow rays that fail to find an occluder.
From these primitives, effects compose recursively: a closest-hit shader can trace new rays for reflections or shadows, each of which may itself hit geometry and spawn further work. NVIDIA's pipeline diagram shows a TLAS and BLAS feeding a compiled shader pipeline bound together through a shader binding table that maps geometry instances to the right hit and miss programs. Your engine's job is to keep that binding coherent as objects, materials, and instances change.
Architectural Blueprint of a Real-Time Ray Tracing Pipeline
The table below deconstructs the pipeline into its major components, showing how data flows from raw scene geometry to a denoised final pixel — and where you write custom code versus where the structure does the work for you.
| Stage | Purpose | Key data | Who controls it | Main performance lever |
|---|---|---|---|---|
| Geometry input | Provide triangles and instances to the ray tracing system | Vertex/index buffers, AABBs of custom geometry | Engine importer and scene graph | Vertex count, instance count, geometry compaction |
| Acceleration structure build | Convert geometry into a traversable hierarchy (two-level BVH) | BLAS per object, TLAS over instances | Engine, using build flags (fast build vs. fast trace) | Build flags, update vs. rebuild, update frequency |
| Ray generation | Define ray origins and directions for each effect | Camera, material, and light parameters | Your custom shader program | Rays per pixel, effect resolution |
| Traversal and intersection | Find hits against scene geometry via the hierarchy | TLAS referencing BLASes, shader binding table | Fixed-function traversal invoking your programs | BVH quality and layout; letting miss rays exit early |
| Shading (closest-hit / miss) | Compute radiance at hits or for rays that escape | Material parameters, textures, environment data | Your custom shader programs | Shader cost, recursion depth |
| Denoising | Reconstruct a clean image from sparse, noisy traces | Radiance, motion vectors, depth, normals, roughness | Your post-processing integration of NRD, SVGF, or similar | Ray budget, denoiser mode, input precision |
The read of this blueprint is that the acceleration structure is the connective tissue: it stands between your scene data and your shading programs, and every design decision upstream — how you partition geometry, how often you rebuild, which build flags you choose — determines what the traversal stage costs downstream.
Bounding Volume Hierarchies: The Two-Level Acceleration Structure
To achieve high performance on complex scenes, ray tracing performs intersections against an optimized data structure called an acceleration structure. As the Khronos Group explains in its Vulkan ray tracing architecture, this structure is divided into a two-level hierarchy. The bottom level contains the triangles or the axis-aligned bounding boxes of custom geometry that make up the scene — and because each bottom-level build may correspond to what would be multiple draw calls in rasterization, a single BLAS can absorb multiple sets of geometry of a given type.
Why split it at all? Lefrancois offers a concrete example in the NVIDIA series: consider a city containing a car and a truck. Rather than one monolithic structure, one acceleration structure holds the city, and the moving vehicles get their own. The bottom-level structures hold geometry; the top level is built from instances of those BLASes, each carrying a transform. Move the car, and you rebuild only its small top-level entry — the city's BVH and the car's BLAS are untouched.
This is the architectural payoff for a custom engine: the split decouples geometry identity from scene placement. Objects that never deform get built once; animated transforms cost only a TLAS update. As NVIDIA's data-flow diagram shows, the TLAS and BLAS feed the shader pipeline together with a shader binding table that ties each instance to the right programs — so instancing, animation, and material assignment all flow through one coherent structure.
Performance Trade-offs: BVH Build Speed vs. Trace Speed
The trade-off at the center of acceleration structure management is simple to state: the same scene can be encoded in BVHs that build quickly or BVHs that trace quickly, and generally not both maximally at once. Intel's developer guide for real-time ray tracing on Arc graphics makes the recommendation concrete, following the Microsoft DirectX Raytracing specification's build flag guidance.
Use the fast-trace flag for static assets and for top-level BVHs, Intel advises, and reserve the fast-build flag for cases where profiling shows fast-trace construction is too slow to fit your frame budget. Updatable BLASes are likely to be more expensive in both memory footprint and ray-tracing performance, Intel cautions, so they should only be made updatable when genuinely needed.
There are also structural levers inside the hierarchy itself. Intel notes that good BVH design lets missed rays fall out of the top of the structure quickly rather than tracing all the way to a leaf node — meaning your hierarchy's shape directly affects the cost of shadow and reflection rays that hit nothing. For a custom engine, the practical process is: classify every object as static or dynamic; build static BLASes once with trace-optimized flags; make dynamic objects updatable or fast-build only if profiling demands it; and rebuild or update the TLAS each frame to reflect new transforms. The cost you are managing is the sum of build time, traversal time, and the memory both consume.
Integrating Denoising Algorithms into the Pipeline
Real-time ray tracing is only real time because engines trace a fraction of the rays a clean image needs and then denoise the result. Denoising is architecturally a post-processing stage: the shading outputs land in buffers, and a denoiser reconstructs a stable image from them using spatio-temporal information — accumulating evidence across neighboring pixels and previous frames rather than brute-forcing more rays.
NVIDIA's Real-Time Denoiser (NRD) denoises three signal types — diffuse lighting, specular reflections, and shadows from infinite light sources — using hit distance to guide the process and enabling free denoising of ambient and specular occlusion. In Watch Dogs Legion, NVIDIA reported that NRD improved image quality over the prior SVGF solution with softer, more realistic lighting and shadows while increasing performance by roughly 50% across 1080p, 1440p, and 4K resolutions. NRD integrates with DX11, DX12, and Vulkan and ships as full source code, which matters for a custom engine where you will likely need to rewire the integration to your own render graph.
The denoiser's input contract is what shapes your pipeline design. NVIDIA's NRD documentation describes the expected signals — radiance, view-space depth, normals and roughness, and motion vectors — including careful handling of virtual-space parameters for reflected hits, where a secondary bounce's hit is reprojected as if it were a primary one. In practice this means every ray traced in your engine must also write out motion vectors, normals, and depth consistent with what the denoiser expects, or the temporal accumulation will smear and ghost. NRD's own guidance also flags that when the denoiser runs at lower resolution, radiance denoising overhead in modes like REBLUR or RELAX must be weighed against the tracing savings. Denoising and ray budget are one decision, not two.
Key Decisions for Custom Engine Ray Tracing Implementation
Integrating real-time ray tracing into a custom engine means prioritizing efficient management of the two-level BVH structure and treating denoising as a first-class post-processing stage whose input contract drives your shader outputs. Concretely: classify geometry as static or dynamic before choosing build flags, default to trace-optimized builds and fast TLAS updates, reserve fast-build and updatable flags for cases profiling justifies, and design your ray generation and shading stages to emit the radiance, motion vectors, depth, and normals your denoiser consumes.
The measurable indicator of success is improved visual fidelity and stable frame rates in dynamic scenes, with minimal artifacts from denoising — the direct product of balancing BVH build and trace performance against effective noise reduction. The practical check for your engine is profiling both sides of that balance: time spent building and updating acceleration structures versus time spent traversing them, and residual denoising artifacts — ghosting, lagging shadows, over-blurred reflections — under camera and object motion. Both are observable, frame-by-frame, and together they tell you whether your architecture holds up.
Sources
- CACM
- MDPI
- Video Series: Practical Real-Time Ray Tracing With RTX — NVIDIA Technical Blog
- Ray Tracing In Vulkan — The Khronos Group
- Intel® Arc™ Graphics Developer Guide for Real-Time Ray Tracing in...
- NVIDIA Real-Time Denoiser Delivers Best-in-Class Denoising in Ubisoft’s Watch Dogs Legion — NVIDIA Technical Blog
- GitHub - NVIDIA-RTX/NRD: NVIDIA Real-time Denoising (NRD) library
