Efficiently Reconstructing Dynamic Scenes One D4RT at a Time

Encode a video once, then ask for the 3D position of any pixel at any moment.

D4RT replaces the separate depth, camera and tracking models of a 4D reconstruction pipeline with one transformer. Depth maps, point clouds, 3D point tracks and camera parameters all come from one kind of query (a pixel plus three frame indices), decoded against a single encoding of the video.

A billion-parameter encoder runs once per video. After that, each 3D point costs one pass through a 144M-parameter decoder, so the work grows with the number of points you ask for.

Picture a video of a swan gliding across a pond, filmed by someone walking along the bank. Behind the pixels is a 3D scene that changes over time: the pond and the reeds stay put, the swan moves, and the camera moves too. Researchers call the combination of 3D geometry and its change over time 4D. Reconstructing it from one ordinary video means answering three questions for every frame: how far away each surface is (depth), where the camera stood and which way it pointed (camera pose), and where each physical point goes as time passes (tracking).

Most systems split those questions across separate models. MegaSaM combines off-the-shelf models for monocular depth, metric depth and motion segmentation, then runs an expensive optimization at test time to make their outputs agree. Feedforward models such as VGGT (Visual Geometry Grounded Transformer) skip the optimization but attach a separate decoder head for each output. Neither MegaSaM nor VGGT produces correspondences for the moving parts of a scene. In the paper's qualitative comparison, reconstruction models of this kind (MegaSaM and π³) get the pond and the reeds right but draw the swan as repeated copies along its path, or drop it.

D4RT (Dynamic 4D Reconstruction and Tracking), from Google DeepMind with Oxford and UCL, uses one model and one output type. An encoder turns the whole video into a set of feature vectors. A small decoder then answers queries of the form: take the point seen at this pixel in this frame, and return its 3D position at a chosen moment, in the coordinates of a chosen frame's camera. Depth, point clouds, tracks and camera parameters are all read off patterns of such queries.

The page follows the paper's order. First the encoder and what it produces, then the query and its three frame indices, then how each task becomes a query pattern (camera pose and focal length need a little closed-form geometry on top). Then the training loss, the algorithm that tracks every pixel, and the results with their caveats.

The encoder reads the video once

The two-stage shape comes from the Scene Representation Transformer (SRT), an earlier model from the same group. SRT encodes a handful of photos of a scene into a set of latent tokens, then renders a new view by sending one query per camera ray into those tokens with cross-attention and reading back a colour. D4RT keeps the encode-once, query-many structure and changes what a query asks for.

Stage one is the encoder $\mathcal{E}$. It reads all frames of the video $V$ together and outputs the Global Scene Representation $F$:

The input is $T$ frames of height $H$ and width $W$ with three colour channels. The output is $N$ tokens, each a vector of $C$ numbers. The encoder is a Vision Transformer that cuts the video into spatio-temporal patches of 2 frames × 16 × 16 pixels. At the training resolution of 48 frames at 256×256, that gives 48/2 × 256/16 × 256/16 = 24 × 16 × 16 = 6,144 video tokens, each summarizing 2 × 16 × 16 × 3 = 1,536 input numbers. (The video is resized to a square before tokenization; the original aspect ratio goes in as one extra token.)

Inside the encoder, two kinds of self-attention layer alternate, a design taken from VGGT. A frame-wise layer lets each token attend only to tokens from its own frame, which builds a description of that frame alone. A global layer lets every token attend to every other token in the video, and correspondence across time forms in these layers: the token covering the swan's head in frame 3 can attend to the tokens covering it in frame 40. The final model uses a ViT-g encoder with 40 layers and about 1 billion parameters.

Once computed, $F$ is fixed. The decoder reads from it and never writes to it, and the expensive pass over the video happens exactly once no matter how many questions follow. A model that decodes a dense map per frame per task repeats heavy work for every output; D4RT pays the encoder once and each query after that costs one light decoder pass. Drag the number of queries in Figure 1 and compare the two cost lines:

A query: one pixel, three frame indices

Stage two is the decoder $\mathcal{D}$, a cross-attention transformer with 8 layers and 144M parameters. Cross-attention here means the query token computes attention weights over the 6,144 tokens of $F$ and pulls in a weighted mix of them, layer after layer (the mechanism is the one in the original Transformer's decoder). A final linear projection turns the result into three numbers. The query has five fields:

The first three fields identify a physical point by where it appears in one frame: $(u, v) \in [0,1]^2$ is a normalized pixel position (0.5, 0.5 is the image centre) in the source frame $t_{\text{src}}$. The last two fields are frame indices in $1, \dots, T$ and ask two separate things about that point:

• $t_{\text{tgt}}$ (target time) chooses when. The speck of wing you pointed at in frame 1 is somewhere else in frame 30; setting $t_{\text{tgt}} = 30$ asks for that frame-30 position.
• $t_{\text{cam}}$ (camera time) chooses which coordinate system. A 3D position needs an origin and axes. D4RT uses a camera's: in the usual computer-vision convention, the origin at the camera centre, x to the right, y down, and z straight ahead along the viewing direction. Because the camera moves, each frame has its own such system, and $t_{\text{cam}}$ picks one.

For a concrete query, take $\mathbf{q} = (0.62,\ 0.40,\ 1,\ 30,\ 1)$: the point seen at 62% of the way across and 40% of the way down frame 1, where it is at frame 30, written in frame 1's camera coordinates. It decodes to three numbers such as $\mathbf{P} = (0.31,\ -0.12,\ 2.40)$ (made up for illustration): 0.31 units right of frame 1's camera, 0.12 up, and 2.40 ahead of it.

The two indices are independent, and the paper calls this the disentanglement of space and time. Consider the alternative where they are forced equal. Every answer about frame 30 would then come back in frame 30's camera coordinates, every answer about frame 31 in frame 31's, and so on. Each moment would be correct in its own frame, but the frames themselves move with the camera, so there would be no common system in which to compare the swan at frame 30 with the swan at frame 31. Pinning $t_{\text{cam}}$ to one reference frame for every query puts every moment of the video into that frame's coordinates, which is what a single 4D point cloud needs. Figure 2 shows the difference from above.

TAPVid-3D, the standard 3D tracking benchmark, scores tracks in each frame's own camera coordinates, which is exactly $t_{\text{tgt}} = t_{\text{cam}}$. It also has a world-coordinate variant that keeps one reference frame throughout; D4RT handles both by changing one field of the query.

The simplest query pattern is a point track. Pin a pixel in a source frame and sweep both remaining indices together, $t_{\text{tgt}} = t_{\text{cam}} = 1, \dots, T$. The answers are the 3D positions of one physical point at every frame, which is its trajectory. Figure 3 plays one back:

The query token itself is built from five pieces. The continuous coordinates $(u, v)$ pass through a Fourier feature embedding: a vector of sines and cosines of $u$ and $v$ at many frequencies, which lets a network distinguish nearby coordinates finely (the same device as the positional encoding in NeRF). Each of $t_{\text{src}}$, $t_{\text{tgt}}$ and $t_{\text{cam}}$ gets a learned embedding, and these four vectors are added. The fifth piece is an embedding of the 9×9 block of raw RGB pixels around $(u, v)$ in the source frame, covered in the training section.

Each query is decoded independently: it attends to $F$ and to nothing else, so no query sees any other. Consistency between answers comes from $F$; two queries about the same surface read the same tokens. The obvious alternative, letting queries attend to each other so neighbouring points could coordinate, was tried: the authors report major performance drops when they enabled self-attention between queries in early experiments, which they attribute to out-of-distribution effects when queries are chosen freely at inference. Independence also means training needs only a few thousand decoded queries per clip for its supervision signal, and inference can run any number of queries in parallel. The paper's conclusion puts the resulting cost as linear in the number of points reconstructed.

Each task is a query pattern

With one query type, choosing a task means choosing which fields to pin and which to sweep. The paper's Table 1 lists five patterns, and Figure 4 shows four of them on the five query fields:

• Point track: pin $(u, v, t_{\text{src}})$, sweep $t_{\text{tgt}} = t_{\text{cam}}$ over all frames.
• Point cloud: sweep $(u, v)$ over every pixel and $t_{\text{src}} = t_{\text{tgt}}$ together over every frame, and pin $t_{\text{cam}}$ to one reference frame. Each pixel is placed where it was at the moment it was seen, and all of them land in the reference camera's coordinates directly. Other pipelines get there by predicting a camera for each frame and transforming each frame's points with it, which carries any error in those camera estimates into the cloud.
• Depth map: for frame $t$, set $t_{\text{src}} = t_{\text{tgt}} = t_{\text{cam}} = t$ for every pixel and keep only the $Z$ coordinate of each answer, since depth in camera coordinates is the distance along the viewing axis.
• Camera extrinsics (position and orientation): a grid of pixels in one frame, decoded twice with only $t_{\text{cam}}$ changed. Next section.
• Camera intrinsics (focal length): a grid of pixels with all three times tied, then a formula per point. Also next section.

In code, with the query token spelled out:

# encode the whole video once, then ask cheap point questions
F = encoder(video)          # (48, 256, 256, 3) -> (6144, C), computed ONCE

def P(u, v, t_src, t_tgt, t_cam):          # one query -> one 3D point
    tok = fourier(u, v) + E_src[t_src] + E_tgt[t_tgt] + E_cam[t_cam]
    tok = add_patch(tok, rgb_9x9(video[t_src], u, v))  # local appearance
    return decoder(tok, F)  # 8 cross-attention layers into F; no other query

track = [P(u, v, s, k, k) for k in frames]            # pin pixel, sweep time
cloud = [P(u, v, t, t, ref) for (u, v, t) in pixels]  # every pixel, one frame
depth = [P(u, v, t, t, t)[2] for (u, v) in frame(t)]  # keep only Z

No task has its own head or its own weights. The table in the paper that compares capabilities marks D4RT as the only method among MegaSaM, DUSt3R, VGGT, π³, St4RTrack and SpatialTrackerV2 with all of: dynamic correspondence, a freely chosen reference frame, sparse decoding, and a single decoder.

Camera pose and focal length

Camera extrinsics describe where the camera sat and which way it pointed. D4RT recovers them as a relative pose: the rotation and translation that take frame $i$'s camera coordinates to frame $j$'s. Take a small grid of $k$ pixels in frame $i$ (with an 8 × 8 grid, 64 points) and decode each twice, changing only $t_{\text{cam}}$:

Both queries name the same point at the same moment ($t_{\text{src}} = t_{\text{tgt}} = i$), so the two answers are the same 3D point written in two coordinate systems. The whole set of 64 first answers and the whole set of 64 second answers therefore differ by one rigid motion, a rotation $R$ and a translation $\mathbf{t}$, and that motion is the relative camera pose.

Finding the rigid motion that best maps one set of matched points onto another has a closed-form answer, Umeyama's algorithm from 1991. Subtract each set's centroid. Multiply the centred sets together to form their 3×3 cross-covariance matrix, and take its singular value decomposition. The rotation is a product of the two singular-vector matrices, with one sign flipped when needed so that a proper rotation comes out and not a mirror image. The translation then follows as the offset that carries the rotated centroid of one set onto the centroid of the other. The SVD is always of a 3×3 matrix, however many points went in. Drag Figure 5's slider to apply the recovered transform:

Camera intrinsics describe the lens, chiefly the focal length. A pinhole camera projects a 3D point $\mathbf{P} = (p_x, p_y, p_z)$ in its own coordinates to the pixel $u = c_x + f_x\, p_x / p_z$, and likewise for $v$ with $f_y$ and $p_y$. Assume the principal point $(c_x, c_y)$, where the optical axis meets the image, sits at the centre $(0.5, 0.5)$ in normalized coordinates, and solve for the focal length:

To use it, query a grid of pixels in frame $i$ with all three times set to $i$, so each answer is in that frame's own camera coordinates. Say the pixel $(u, v) = (0.62, 0.46)$ decodes to $\mathbf{P} = (0.30,\ -0.10,\ 2.0)$. Then $f_x = 2.0 \times 0.12 / 0.30 = 0.80$ and $f_y = 2.0 \times (-0.04) / (-0.10) = 0.80$. In normalized units a focal length of 0.80 means 0.80 image widths, which is 205 pixels on a 256-pixel frame and a horizontal field of view of 2·arctan(0.5/0.8) ≈ 64°.

Every grid point gives its own estimate, and D4RT takes the median. Figure 6 shows why the median and not the mean. Near the image centre, $u - 0.5$ and $p_x$ are both close to zero, so a small error in the decoded $p_x$ produces a large error in the ratio, sometimes with the wrong sign. A mean follows those few wild estimates; a median ignores them as long as they are fewer than half.

The formula assumes a centred principal point and no lens distortion. For distorted lenses such as fisheye, the paper suggests adding a non-linear refinement step on top of this initial estimate, in the style of Zhang's classic calibration method. Because intrinsics are computed per frame, they can change within a video (a zoom, for example); the paper lists fixed intrinsics as a limitation of several prior models.

The training loss

Training is supervised regression on synthetic and real datasets where the true 3D answer to a query is known (BlendedMVS, Co3Dv2, Dynamic Replica, Kubric, MVS-Synth, PointOdyssey, ScanNet++, ScanNet, TartanAir, Virtual KITTI, Waymo Open, and internal data). Sample queries, decode them, and penalize the error. The main term is an $L_1$ loss on the 3D point with two additions, a normalization and a confidence weight. Averaged over $N$ sampled queries, with $\lambda_{3D} = 1$:

The normalization divides the predicted points $\hat{\mathbf{P}}$ by their own mean depth $\hat{\bar z}$, and the ground-truth points $\mathbf{P}$ by theirs, $\bar z$, following DUSt3R. This makes the loss blind to overall scale: if the model predicts every point exactly twice as far away as the truth, both sets become identical after division and the loss is zero. The loss has to work this way because one video cannot reveal absolute scale; the same pixels come from a toy swan a metre away or a real one ten metres away.

After normalization, $\psi$ compresses large coordinates so that distant points do not dominate. A one-unit error on a point near the mean depth, from 1 to 2, costs $\psi(2) - \psi(1) = 0.41$. The same one-unit error on a point ten times farther out, from 10 to 11, costs $\psi(11) - \psi(10) = 0.087$, almost five times less.

For the confidence weight, the decoder outputs a second quantity per query, $c_i > 0$, following Kendall and Gal. It multiplies the point's error, and the $-\lambda_{\text{conf}} \log c_i$ term (with $\lambda_{\text{conf}} = 0.2$) charges for low confidence. Without that term the model could set every $c_i$ to zero and pay nothing. To see what the two terms do together, fix a point's error at $e$ and ask which confidence minimizes $c\,e - 0.2 \log c$. Setting the derivative $e - 0.2/c$ to zero gives:

This assumes $c$ can take any positive value; the paper does not say how $c$ is parameterized. At the optimum, a point with error 0.1 gets $c^* = 2$ and a loss of 0.061; a point with error 1 gets $c^* = 0.2$ and a loss of 0.52; a point with error 2 gets $c^* = 0.1$ and a loss of 0.66, where plain $L_1$ would charge 2. The loss grows with the logarithm of the error instead of linearly, and the gradient reaching a point's prediction is scaled by $c$, so points the model cannot get right (sky, reflections, heavy motion blur) pull on the weights much less than points it can. Figure 7 plots both effects:

In the paper's ablation (Table 8), removing the confidence term barely moves the depth error (AbsRel(S), the mean of |predicted − true| / true over pixels after one global scale alignment; lower is better) by +0.002, but hurts camera pose badly: the absolute trajectory error (ATE) goes from 0.091 to 0.217, more than doubling, and the relative rotation error rises by 0.115. The paper does not explain the asymmetry. A plausible reading is that pose comes from fitting one rigid transform to many points at once, so a model trained to spend its effort on points it can place accurately yields a better fit, while a depth map is scored pixel by pixel.

Four auxiliary losses come from extra linear projections on the decoder output: an $L_1$ loss on the point's 2D position in the target image, a cosine-similarity loss on the surface normal, a binary cross-entropy on the point's visibility at $t_{\text{tgt}}$, and an $L_1$ loss on its motion vector. Each applies only where the dataset has that ground truth. Their weights are 0.1, 0.5, 0.1 and 0.1. In the ablation, dropping the 2D-position loss raises depth AbsRel(S) by 0.071 and dropping normals by 0.043; both slightly improve rotation error when removed (−0.028 and −0.022), a small depth-versus-pose trade-off the paper notes.

The fifth part of the query token is the local 9×9 RGB patch. The encoder sees the video in 16×16 pixel patches, so $F$ has one token per 16×16 block per pair of frames, too coarse to place an object boundary that falls inside a block. The patch hands the decoder the raw pixels at the query location. The paper gives two reasons it helps: the appearance makes it easier for the query to find its match among the tokens of $F$, and the low-level edges let the decoder separate an object from its background. Most dense predictors get fine detail through skip connections from encoder to decoder, as in DPT (the Dense Prediction Transformer); here it is 243 extra input numbers per query. The paper calls the improvement dramatic, and its patch-size sweep finds the best size between 9 and 12 pixels.

Because $(u, v)$ is continuous, the decoder is not tied to the encoder's 256×256 grid. In an appendix experiment with the ViT-g encoder at 256×256, adding the patch lowered depth AbsRel from 0.254 to 0.218 and a boundary-error metric from 3.32 to 2.25; decoding at the original video resolution and taking the patches from the full-resolution frames brought the boundary error down to 2.19.

The recipe: 500k steps with AdamW (weight decay 0.03), a learning rate warmed up over 2,500 steps to $10^{-4}$ and cosine-decayed to $10^{-6}$, gradients clipped at norm 10, one clip per chip on 64 TPU chips, just over 2 days in total. Each clip has 48 frames at 256×256 and contributes 2,048 queries. Queries are drawn from ground-truth point trajectories; 30% of the pixel locations are sampled near depth discontinuities or motion boundaries (found with a Sobel filter on the depth maps), the three time indices are uniform at random, and $t_{\text{tgt}} = t_{\text{cam}}$ is forced with probability 0.4. One step looks like this, with every loss term flowing back through both decoder and encoder:

# one training step (48-frame clip, 2048 queries)
F = encoder(clip)                       # gradient flows into the encoder too
q = sample_queries(clip, n=2048)        # 30% near depth or motion edges;
                                        # t_tgt = t_cam forced with prob 0.4
P_hat, c, aux = decoder(q, F)           # each query decoded on its own
P = ground_truth(q)
err = l1(psi(P_hat / mean_z(P_hat)), psi(P / mean_z(P)))   # per query
loss = mean(c * err - 0.2 * log(c)) + aux_losses(aux, q)   # eq. (5) + aux
loss.backward()                         # updates encoder AND decoder

Tracking every pixel

A 3D tracker that follows points from one starting frame leaves holes: anything hidden in that frame never gets a track. The paper shows SpatialTrackerV2's reconstructions with gaps in exactly those regions. Tracking every pixel, from every frame, static or moving, removes those holes.

Occlusion itself is not a problem for a D4RT query. Say a speck on the swan's wing is visible in frames 1 to 12, hidden behind a reed in frames 13 to 18, and visible again from frame 19. A tracker that carries a point forward by matching it from frame to frame loses the speck at frame 13 and has no evidence at frame 19 that the reappearing speck is the same one. D4RT asks for frame 15 directly: pin the speck's pixel in frame 1, set $t_{\text{tgt}} = 15$, and decode against $F$, which the encoder built from all 48 frames including those after the speck reappears. The decoder returns a position for frame 15, and its visibility output, trained with the visibility loss, can flag the point as hidden there.

Cost limits the naive version. Starting a full-length track at every pixel of every frame takes $T \cdot H \cdot W$ tracks of $T$ queries each, $O(T^2 H W)$ in total: for a 48-frame 256×256 clip, 48 × 48 × 256 × 256 ≈ 151 million queries. Most are redundant. A track started at a pixel of frame 1 already says where that physical point is in every later frame, so starting new tracks from the pixels it passes through repeats work.

Algorithm 1 removes that redundancy with an occupancy grid $G \in \{0,1\}^{T \times H \times W}$, one bit per pixel of every frame, set once some track has explained that pixel. New tracks start only from pixels whose bit is still 0. Each finished track sets the bit of every pixel it visibly passes through; pixels where the point sits behind something stay unclaimed, since the surface in front of it needs its own track. The loop ends when every bit is set:

# Algorithm 1: track all pixels without redundant work
F = encoder(video)                  # encode once
G = zeros(T, H, W, dtype=bool)      # occupancy grid: pixel already explained?
tracks = []
while not G.all():
    for (u, v, t_src) in sample_unvisited(G):       # a batch, in parallel
        q = [(u, v, t_src, k, k) for k in range(T)]  # one full-video track
        pts = [decoder(qk, F) for qk in q]
        G[visible_pixels(pts)] = True  # every pixel it visibly passes through
        tracks.append(pts)
return tracks

The algorithm depends on two properties of the decoder: it can be asked for a single point (sparse), and asking is cheap (lightweight). Models that decode dense per-frame maps cannot skip pixels, and trackers with heavy per-query decoders pay too much per track for the full video.

Results, speed, and limits

For 3D tracking on TAPVid-3D, the primary metric is 3D Average Jaccard (AJ), which combines position accuracy and visibility prediction averaged over several distance thresholds; higher is better. In camera coordinates without ground-truth intrinsics, D4RT scores 0.257 on DriveTrack (next best 0.129) and 0.138 on PStudio (next best 0.119), while SpatialTrackerV2 leads on ADT with 0.260 against D4RT's 0.240. With ground-truth intrinsics D4RT leads all three subsets: 0.304, 0.307 and 0.372, against SpatialTrackerV2's 0.195, 0.303 and 0.175. In world coordinates, the test of placing every moment in one shared frame, D4RT's mean L1 error on DriveTrack is 0.020 against SpatialTrackerV2's 0.139 without intrinsics.

For speed, Table 3 counts how many full-video tracks each model can produce while keeping up with a target frame rate on one A100 GPU, where each D4RT track is $T$ independent decoder queries. At 24 FPS, D4RT produces 1,570 tracks, SpatialTrackerV2 84 and DELTA 5. The paper summarizes the gap as 18 to 300 times. Against SpatialTrackerV2 the ratio stays between 17.5× and 19.0× at every rate; against DELTA it is 314× at 24 FPS and 7× at 1 FPS, and at 60 FPS DELTA produces none. Pick a rate:

On point-cloud reconstruction, scored by mean L1 distance after aligning the clouds by mean-shift, D4RT reaches 0.768 on Sintel, where the next best, π³, is at 1.139 and VGGT at 1.582. On the static ScanNet scenes the margin is small: 0.028 against π³'s 0.030. Sintel is rendered animation with fast character motion, the hardest of these datasets for methods that assume a static scene.

For video depth, the metric is AbsRel, defined above, with scale (S) or scale-and-shift (SS) alignment. D4RT is best on Sintel (0.171 / 0.148) and ScanNet (0.020 / 0.018), ties π³ on KITTI with scale alignment (0.055), and trails π³ on Bonn (0.036 against 0.033 with scale alignment). The paper calls this top-tier across all four, with the clearest lead on the dynamic Sintel sequences.

On camera pose, after Sim(3) alignment to ground truth, D4RT has the lowest ATE on all three pose datasets (Sintel, ScanNet and Re10K), for example 0.065 on Sintel against MegaSaM's 0.074 and 0.014 on ScanNet against π³'s 0.015. On Re10K it also has the best pose AUC (area under the curve of pose accuracy against error threshold; higher is better), 83.5 against π³'s 78.7. The relative errors are competitive rather than uniformly best: it ties MegaSaM on Sintel rotation error (0.126), ties π³ on ScanNet translation error (0.010), and trails π³ on ScanNet rotation error (0.302 against 0.291). On throughput, the paper reports more than 200 pose estimates per second on an A100, 9 times faster than VGGT and 100 times faster than MegaSaM, with the baselines' unrelated decoding heads removed.

Scaling the encoder through ViT-B (90M parameters), ViT-L, ViT-H and ViT-g (1B) on the ablation setup improves depth steadily (AbsRel(S) 0.319, 0.256, 0.226, 0.191 in that order, a 40% reduction) and rotation error likewise (0.266, 0.191, 0.186, 0.160). ATE does not follow: ViT-H has the best, 0.070, and ViT-g's is 0.078.

Because one video cannot fix absolute scale, the output is correct up to one global scale per video, never in metres, and every number above was computed after an alignment that removes that scale. The intrinsics formula assumes a centred principal point and no distortion. A clip is 48 frames at training time; longer videos are cut into overlapping segments, each segment is reconstructed separately, and neighbours are aligned by a Sim(3) transform fitted with Umeyama's algorithm on the 85% most confident points in the overlap, the first stage of VGGT-Long without its loop closure.