3D Gaussian Splatting for Real-Time Radiance Field Rendering

Store a scene as millions of tiny 3D Gaussians, and a GPU can render new views of it in real time.

NeRF made novel-view synthesis photorealistic but slow, because it queries a neural network millions of times per frame. This paper keeps the quality, throws the network out of the render loop, and hits over 100 frames per second by projecting sorted Gaussians onto the screen and blending them.

A high-quality neural radiance field renders one frame in about seventeen seconds. This paper renders the same scenes, at comparable quality, more than a thousand times faster, and it trains in forty minutes instead of two days.

Start with the problem the field is trying to solve, novel-view synthesis. You take a few dozen photos of a scene from different positions, and you want to produce an image from a new camera you never actually placed there, correct down to the reflections and the way a leaf occludes what is behind it. Solve it well and you can fly a virtual camera through a place that was captured with an ordinary phone.

NeRF cracked the quality by storing the scene implicitly: a small neural network that, given a 3D point and a viewing direction, returns a color and a density. To render one pixel you march a ray through the scene and query that network at dozens of points along it, then blend the results. The quality is the best in the field and the rendering is very slow. A ray per pixel, dozens of network calls per ray, millions of pixels per frame, which is why the best-quality method in this paper's comparison, Mip-NeRF360, renders a single frame in roughly seventeen seconds (0.06 frames per second) and takes 48 GPU-hours to train.

Later methods bought speed by replacing the network with a lookup grid (a voxel or hash grid you interpolate). Training drops to minutes, but rendering still marches rays through mostly-empty space, and quality slips. Before this paper, no method rendered unbounded, complete scenes at 1080p and real-time rates (30 frames per second or more) without giving up quality.

3D Gaussian Splatting closes that gap by dropping the implicit representation entirely. A scene becomes an explicit cloud of a few million little translucent 3D blobs, each a Gaussian with its own position, shape, color, and opacity. To render, you project every blob onto the screen as a fuzzy ellipse, sort the ellipses by depth, and blend them front to back. There is no network in that loop, and a GPU runs it fast: 134 frames per second at SOTA-competitive quality, trained in 41 minutes. The old choice was quality or speed. This gets both.

A handful of ideas carry the method, and none of them is heavy on its own: what a rendered pixel actually is (a front-to-back blend of translucent primitives), what the primitive is (an anisotropic 3D Gaussian) and how you keep its shape valid, how you project it to the screen, how you color it so it changes with the view, how a GPU blends millions of them fast, and how the optimizer creates and destroys Gaussians so the count fits the scene. Take them in order.

A pixel is a stack of translucent splats

Before any Gaussians, settle what a pixel's color even is when you render a scene made of translucent stuff. Look along the ray through one pixel. It passes through a sequence of primitives, each with a color $\mathbf{c}_i$ and an opacity $\alpha_i \in [0,1]$ (0 is invisible, 1 is fully solid). The pixel's color is the front-to-back accumulation over that ordered list:

The symbol to pin down is $T_i$, the transmittance: the fraction of light that survives all the way to primitive $i$ without being absorbed by anything in front of it. Each primitive $j$ ahead lets a fraction $(1-\alpha_j)$ of light through, and those survivals multiply, so $T_i$ is the running product of "one minus opacity" over everything nearer the camera. Primitive $i$ then contributes its color weighted by how visible it is, $T_i\,\alpha_i$. Think of a stack of tinted, faintly glowing glass panes: each pane passes $(1-\alpha)$ of the light and adds its own glow $\alpha\,\mathbf{c}$, and $T_i$ is how much light is still travelling when it reaches pane $i$. (The analogy holds for ordered, semi-transparent layers; it breaks for true scattering, where light would bounce sideways between panes rather than march straight through.)

Two consequences follow, and both matter later. Once enough opaque stuff has accumulated, $T_i$ drops to nearly zero and everything behind it contributes nothing, so a renderer can stop early the moment a pixel saturates. And because each layer only multiplies and adds, the sum is differentiable in every $\alpha_i$ and $\mathbf{c}_i$, which lets you optimize the scene by gradient descent.

This is not a new model, and the reuse is deliberate. It is exactly the image-formation model NeRF uses, which itself discretizes the classical emission-absorption integral from volume rendering. NeRF gathers the list by marching a ray and asking its network for a density at each step; 3D Gaussian Splatting gathers the list by projecting its Gaussians and sorting them. Same blend, different way of collecting the layers. In the figure below, a single ray runs through a teal front surface and an amber one behind it. Drag the front surface's density and watch two things: how far down the ray the transmittance T survives, and where along the ray the pixel's color actually comes from.

One thing the paper leaves as a trap for the careful reader: where does $\alpha_i$ come from? In NeRF it is $\alpha_i = 1 - e^{-\sigma_i\delta_i}$, built from a density $\sigma$ and a ray step $\delta$. 3D Gaussian Splatting has no $\sigma$ and no ray steps. Its $\alpha_i$ is a learned opacity times the primitive's 2D Gaussian falloff evaluated at the pixel, which the next two sections build. The blending algebra above is shared; the source of $\alpha$ is not. Keep the two apart and the rest of the method reads cleanly.

The primitive: an anisotropic 3D Gaussian

Now the primitive itself. Each one is a 3D Gaussian: a smooth blob centered at a point $\boldsymbol\mu$ in space, fading out with distance from that center. Written out, its value at a point $\mathbf{x}$ is

The $3\times 3$ matrix $\Sigma$, the covariance, is the blob's shape: it says how far the Gaussian reaches along each direction and how those directions tilt, which geometrically is an ellipsoid. Notice what is not here: the usual $(2\pi)^{-3/2}|\Sigma|^{-1/2}$ normalizing constant that would make $G$ integrate to one. That omission is intentional. This Gaussian is a spatial weight that peaks at 1, not a probability density; how solid the blob is comes from a separately learned opacity, so pinning the peak to a fixed height and letting opacity set the strength keeps the two jobs from fighting.

Crucially, the ellipsoid is anisotropic: it can be stretched and rotated, long in one direction and thin in another, rather than forced to a uniform sphere, and that fit to real geometry is why a Gaussian is the right primitive. A thin structure, the edge of a table or a railing or a blade of grass, is covered by a single long, flat splat aligned to it, where a pile of round blobs would either miss the ends or spill off the sides and blur it. The ablation is blunt about it: force every Gaussian to be an isotropic sphere and quality drops (average PSNR on the ablation scenes falls from 26.05 to 25.23, with visibly smeared fine detail) at the same Gaussian count. Below, shape one splat over a thin diagonal strip of geometry: stretch and rotate it, then flip between anisotropic and isotropic and watch the coverage and waste readouts respond.

So the optimizer needs to move $\Sigma$ around. One constraint shapes the design here: a covariance matrix must be positive semi-definite (its ellipsoid must have non-negative widths in every direction, or it is not a real shape at all). If you handed all six independent entries of $\Sigma$ to gradient descent, a single step would happily produce an invalid matrix with a negative variance, and the blob would become nonsense. The obvious fix, projecting back to the valid set after each step, is exactly the kind of constraint plain SGD handles badly.

The paper sidesteps it by never optimizing $\Sigma$ directly. It factors the shape the way you would describe an ellipsoid to someone: a rotation and a set of axis lengths.

It stores a scale vector $s$ (three positive stretches, one per axis) and a unit quaternion $q$ (a rotation), and rebuilds $\Sigma$ from them each step. This form is positive semi-definite for any $s$ and $q$ you throw at it, because $\mathbf{v}^\top\Sigma\mathbf{v} = \lVert S^\top R^\top \mathbf{v}\rVert^2 \ge 0$, so the optimizer is fenced inside the legal region without a projection step. It costs nothing in expressiveness: the eigenvalues of $\Sigma$ come out as the squared scales $s_i^2$ and its axes as the columns of $R$, so $(s, q)$ spans exactly the same six degrees of freedom as a general symmetric $3\times 3$. Store three stretch dials and one rotation dial, and you cannot dial in an impossible shape.

Where do the first Gaussians come from? Calibrating the cameras is itself done by Structure-from-Motion (COLMAP), which matches features across the photos and solves for the camera poses; as a by-product it drops a sparse point cloud roughly where the real surfaces are. 3D Gaussian Splatting seeds one small isotropic Gaussian at each of those points (its initial size set by the distance to its nearest neighbors), colored to match. That cloud is a scaffold, not the answer. The optimizer then stretches, recolors, splits, and prunes it into the final scene.

Projecting a Gaussian to the screen

To blend a Gaussian into a pixel you need its footprint on the screen, a 2D ellipse, not the 3D ellipsoid. This projection is where "splatting" gets its name: each blob is splatted onto the image as a soft elliptical stamp.

A Gaussian has a convenient property: push it through any linear map and it stays a Gaussian, with its covariance transformed as $A\Sigma A^\top$. The world-to-camera transform is linear, so that part is clean. Perspective projection is not linear, though (it divides by depth), and a perspective image of a Gaussian is not exactly a Gaussian. The fix, from EWA splatting, is to approximate the projection near each blob's center by its local linear part, the Jacobian $J$. Chaining the viewing transform $W$ and that Jacobian gives the screen-space covariance:

That gives a $3\times 3$ covariance in camera space; drop its third row and column and you have the $2\times 2$ covariance of the ellipse on the image plane. That last step is exact given the linearization; the only approximation in the projection is $J$, the local affine stand-in for perspective, and it is a good one because each splat is small, so perspective barely bends across it. Painting an ellipse on a rubber sheet and stretching the sheet is a fair picture: the ellipse's spread matrix transforms as $A(\cdot)A^\top$, and here the stretch is $A = JW$.

With the 2D ellipse in hand, the opacity from the last section is finally concrete. At a pixel offset $\mathbf{d}$ from the splat's projected center, the contribution is the learned opacity $o_i$ times the 2D Gaussian evaluated there:

That $\alpha_i$ feeds the blend of Figure 1. The splat is brightest at its center and fades with the elliptical falloff of $\Sigma'$, and $o_i$ (squashed through a sigmoid, so it always lands in $(0,1)$) scales how solid the splat is.

Color that depends on the view

A surface does not look the same from every angle. A waxed floor throws a highlight that slides as you move; a gilded frame flips from dark to bright. So each Gaussian's color has to be a function of the viewing direction, not one fixed RGB. NeRF handles this by feeding the direction into its network. 3D Gaussian Splatting will not put a network in the render loop, so it needs a cheaper trick, and it borrows the one Plenoxels and Instant-NGP used: spherical harmonics.

Spherical harmonics are a Fourier series for directions: a fixed set of basis functions on the sphere, ordered from smooth to wiggly, so that a few coefficients times those bases reconstruct a smooth function of direction. Give each Gaussian a handful of coefficients per color channel and evaluate them at the view direction, a plain dot product, and you get the RGB it shows from that angle:

The first basis, band 0, is a constant, so its coefficient is the flat base color the Gaussian shows from everywhere, its diffuse appearance. Each higher band adds gentler directional variation, a lobe that brightens as your view lines up with it, which is how a specular glint appears. The paper uses 4 bands (degrees 0 through 3), which is 16 coefficients per channel, 48 numbers in all, and it adds them one band at a time (a new band every 1000 iterations) so the base color is settled before the higher bands add shine. Below, orbit the view and raise the SH degree from 0 to 3 to watch the flat base color grow a view-dependent highlight.

Four bands buy the smooth, low-frequency view-dependence that most surfaces need, at almost no cost. What they cannot buy is a sharp mirror: a crisp reflection is high-frequency in direction and would need far more bands than this, so highly reflective surfaces stay a known weak spot. Keeping color a cheap dot product instead of a network call is what wins the speed and gives up the sharp mirror.

The fast tile rasterizer

Now assemble it: a few million splats, each with a screen ellipse, an opacity, and a color, blended front to back per Figure 1. Done naively, that is where the speed dies. Sorting every pixel's overlapping splats by depth, independently, is far too much work millions of times over. The rasterizer is the engineering that makes it real-time, and it turns on one decision: sort once, for the entire image, not per pixel.

Split the screen into $16\times 16$-pixel tiles. Duplicate each splat into every tile its ellipse touches, and give each copy a single 64-bit key with the tile ID in the high bits and the depth in the low bits. One GPU radix sort over those keys buckets everything by tile and, within each tile, orders it front to back in the same pass, like sorting mail by a combined ZIP-plus-house-number label. Each tile then walks its own short, already-sorted list, accumulating color and multiplying down the transmittance, and a pixel's thread quits the instant $T$ falls below $10^{-4}$, so the splats hidden behind a wall cost nothing. Pick a tile below and walk its depth-sorted list to see where the tile's color comes from.

Sharing one depth order across a whole tile makes the blend approximate, since two pixels in the same tile might technically want slightly different orders. It stops mattering once splats shrink toward pixel size, which the optimizer drives them to, and the paper reports no visible artifacts from it in converged scenes. That single approximation trades a small, invisible error for real-time speed.

Training needs the backward pass, and here the paper makes a pointed choice against the grain of earlier work. Pulsar and its kin cap how many front splats per pixel receive gradients (their default is a handful); the ablation shows that a cap of ten collapses quality by 11 dB, because a pixel deep in foliage needs gradients flowing to far more than ten overlapping blobs. 3D Gaussian Splatting puts no cap on it. To do that without storing a per-pixel list, it re-walks each tile's sorted splats back to front in the backward pass and recovers the transmittance in front of each splat from a single stored number: it keeps the final transmittance $T_\text{final} = \prod_i (1-\alpha_i)$ and divides back out one factor of $(1-\alpha_i)$ per step as it walks. One number per pixel, gradients for every splat.

Training: render, compare, correct

Every step so far is differentiable, so training is the obvious loop: render an image the scene predicts for a training camera, compare it to the real photo, and let the gradient nudge every Gaussian's position, scale, rotation, opacity, and color. The comparison is two graders blended:

$\mathcal{L}_1$ is the plain mean absolute error, pixel by pixel: is each dot roughly the right color? $\mathcal{L}_{\text{D-SSIM}} = 1 - \text{SSIM}$ scores the structural match over little $11\times 11$ windows: do the edges, contrast, and texture look right? The $0.2$ weight leans mostly on $\mathcal{L}_1$ and adds a fifth of structural pressure, which is enough to sharpen edges without letting the structural term dominate. (One number to keep straight: the D-SSIM term here is $1-\text{SSIM}$, not the $(1-\text{SSIM})/2$ you will see elsewhere; the released code uses the full difference.)

The activations are chosen so the optimizer stays in valid territory without constraints. Opacity passes through a sigmoid, so it is always in $(0,1)$; each scale passes through an exponential, so it is always positive and its gradient is smooth across orders of magnitude; the quaternion is renormalized to unit length after each step. Positions get a learning rate that decays over training (start coarse, settle fine); the other parameters use fixed rates. Every Gaussian carries roughly 59 numbers, its position (3), scale (3), rotation (4), opacity (1), and SH color (48), and a few million of them make up a scene.

# one optimization step (Adam), interleaved with density control
image = rasterize(gaussians, camera)     # project, sort once, blend
loss  = 0.8*l1(image, gt) + 0.2*(1 - ssim(image, gt))   # Eq (7)
loss.backward()                          # grads to every Gaussian
opt.step()                               # pos, scale, quat, opacity, SH

if step % 100 == 0 and 500 <= step <= 15000:  # density control
    g = mean_screen_space_grad(gaussians)     # per-Gaussian, in pixels
    clone(gaussians[(g > 2e-4) & small])      # under-reconstruction
    split(gaussians[(g > 2e-4) & large])      # over-reconstruction, s /= 1.6
    prune(opacity < 0.005 or footprint_too_big)
if step % 3000 == 0:
    opacity = min(opacity, 0.01)              # reset: re-earn visibility

Once trained, rendering a new view runs the top loop again, forward: project, sort once, blend. That inner loop has no network in it, which is the entire reason it runs at over 100 frames per second.

# render one frame: there is no neural network in this loop
for g in gaussians:
    mu2d, cov2d = project(g, camera)     # Eq (5): Sigma' = J W Sigma W^T J^T
    color       = eval_sh(g.sh, view_dir)   # view-dependent color, a dot product
    for tile in tiles_covered(mu2d, cov2d):    # 16x16 px tiles
        emit(key = (tile << 32) | depth_bits, g)   # one 64-bit key per splat

radix_sort(keys)                         # ONE global sort: by tile, then depth

for tile in screen:                      # each tile blends front to back
    T = 1
    for g in tile.sorted_splats:
        a  = g.opacity * gaussian_2d(pixel, mu2d, cov2d)   # per-pixel alpha
        C += T * a * color;  T *= (1 - a)
        if T < 1e-4: break               # saturated: skip the occluded rest

Creating and destroying Gaussians

Gradient descent can slide, stretch, and fade the Gaussians it has, but it cannot change how many there are. Start from a sparse SfM cloud and no amount of nudging will grow the splats needed to resolve a thin railing or fill a blank wall. So the paper interleaves adaptive density control: every 100 iterations (over iterations 500 to 15000) it adds and removes Gaussians based on which ones are poorly fit.

The tell that a region is under-reconstructed is a large gradient on a splat's screen-space position: the gradient keeps pushing the blob toward geometry it cannot cover, and that shows up as a big average positional gradient. Any Gaussian whose average exceeds $\tau_{\text{pos}} = 0.0002$ gets densified, and its size decides how. A small splat in an under-covered spot means missing geometry, so clone it: make a copy. (The paper's text says to offset the copy along the positional gradient; the released code places the clone at the exact same position and lets the subsequent optimization pull the two apart, which matters if you read the code expecting the offset.) A large splat sitting on top of fine detail means one blob is smeared across structure it should not own, so split it into two, each with its scale divided by $\phi = 1.6$ and its center drawn from the parent's own Gaussian. Below, pick a case and apply it to see the clone or split happen.

Splitting is described in the paper as conserving total volume, but two children at $1/1.6$ the scale hold only about $2/1.6^3 \approx 0.49$ of the parent's volume, so a split actually shrinks total volume, which is what you want when you are trying to resolve finer detail rather than fatten up. The factor $1.6$ is $0.8$ times the split count of 2 and was tuned by hand, not derived.

Left alone, this would only ever add Gaussians. Two culls keep the count in check. Any Gaussian whose opacity falls below $0.005$ is pruned, along with any that grows too large in world or screen space. And every 3000 iterations the optimizer runs a kind of amnesty: it forces every opacity down near zero (to $\min(\alpha, 0.01)$) and makes each Gaussian earn its visibility back from the images over the next steps. The blobs that are actually supporting the reconstruction climb back up; floaters and stray splats near the cameras, which nothing is asking for, never recover and get swept away. Between growth and these culls, the count breathes to wherever the scene needs it, 1 to 5 million Gaussians for the real scenes tested.

Real-time, and where it breaks

The headline is the top-right corner of a plot everyone in this field knows. Quality against render speed has always been a trade-off: Mip-NeRF360 sits top-left, high quality but 0.06 frames per second (one frame every seventeen seconds); the fast grid methods sit left of the real-time line and lower in quality. 3D Gaussian Splatting lands in the corner that was empty, SOTA-competitive quality and 134 frames per second, trained in 41 minutes instead of 48 hours. Drag the slider below to step through the methods and compare.

Be exact about "SOTA-competitive," because the paper is, and the popular retelling is not. On the Mip-NeRF360 dataset, 3D Gaussian Splatting at 30K iterations beats Mip-NeRF360 on SSIM (0.815 vs 0.792) and on LPIPS (0.214 vs 0.237, lower is better) but trails it on PSNR (27.21 vs 27.69). On the Tanks&Temples and Deep Blending datasets it matches or beats Mip-NeRF360 on all three metrics. So the precise claim is "matches or beats the best quality, and renders it a thousand-plus times faster," not "wins every metric everywhere."

The cost is hiding on a third axis nobody usually plots: storage. An explicit cloud of a few million Gaussians, each carrying 59 numbers, is big. That same Mip-NeRF360-dataset model is 734 MB on disk against 8.6 MB for Mip-NeRF360 and tens of megabytes for the Instant-NGP hash grids (Plenoxels, a dense voxel grid, is bigger still at 2.1 GB), and peak GPU memory during training can top 20 GB. So the method is dominant on the two axes people usually care about, quality and speed, and clearly loses on the one they usually ignore. It has other rough edges the paper owns: in regions the cameras barely saw it produces elongated or splotchy artifacts, and it can "pop" when large Gaussians suddenly switch depth order, both of which trace back to having no explicit smoothness or regularization on the cloud.

What is left is a change of representation, not a new kind of network. A scene is a pile of oriented, colored, translucent blobs; rendering it is a sort and a blend a GPU finishes in milliseconds; and the blobs are trained by comparing each render to a photo and adjusting them to fit. The neural network that NeRF put at the center of radiance fields is optional. Take it out of the render loop, put a rasterizer there, and the field has its real-time renderer.