ControlNet: Adding Conditional Control to Text-to-Image Diffusion Models

Freeze a text-to-image model, train a copy of its encoder on edge maps or poses, and join the two through layers that start at zero.

ControlNet steers Stable Diffusion with an image such as an edge map, a depth map or a pose skeleton. A trainable copy of the U-Net's encoder reads that image and adds its outputs to the frozen model's skip connections. The convolutions that carry those outputs start with all-zero weights, so training begins from the unmodified model.

The architectural change fits in one equation. Most of this page is about the initialization in that equation: what a layer with all-zero weights does to the first few gradient steps, and why the paper credits it with keeping the pretrained model intact.

Stable Diffusion turns a sentence into a 512×512 image, and a sentence is a poor way to say where things go. "A dancer, left arm raised, seen from slightly below" leaves the exact pose, the framing and the camera height to the model, and the user's only recourse is to reword the prompt and sample again. Image-to-image models solve the placement problem directly by learning a mapping from one kind of image to another (edge map to photo, pose skeleton to person, depth map to room), but the training sets for any one of those tasks are small. The paper puts the largest at about 100,000 examples, 50,000 times smaller than the LAION-5B collection Stable Diffusion was trained on.

ControlNet, posted by Lvmin Zhang, Anyi Rao and Maneesh Agrawala of Stanford in February 2023, adds such an image as an extra input to a pretrained text-to-image model without changing any of the model's weights. It works with Canny edge maps (the output of a classic edge-detection algorithm), straight-line maps, user scribbles, pose skeletons, segmentation maps, depth and surface-normal maps, with or without a text prompt. The ideas, in order:

1. freeze each block of the network and give it a trainable twin that also reads the condition image;
2. connect the twin to the frozen network through convolutions whose weights start at zero, and check that they still train;
3. apply this to Stable Diffusion's U-Net, which fixes every tensor shape;
4. train with Stable Diffusion's own loss, unchanged;
5. at sampling time, decide how the condition interacts with classifier-free guidance, and add several ControlNets together.

Why not fine-tune Stable Diffusion directly

The obvious way to add an input is to widen the model's first layer to accept extra channels and continue training everything on pairs of (condition, image). Stability AI built its depth-to-image model this way, and the v1 of the ControlNet paper uses the same recipe as its baseline. With a large dataset it works. With tens of thousands of examples, continued training of an 860-million-parameter model runs into two problems the paper names: overfitting, where the model memorizes the small dataset, and catastrophic forgetting, where updating the weights for the new task overwrites what they encoded about everything else.

One known remedy is to train only a few new parameters, as LoRA (Hu et al., 2021) does by learning a low-rank change $BA$ to each weight matrix (see the LoRA explainer). The paper argues that this is too little capacity for in-the-wild condition images: reading a hand-drawn scribble, or a segmentation map with 150 classes, takes a deep network. The paper instead reuses the pretrained model's own encoder, which already turns an image into a stack of features at four resolutions, so a copy of it, initialized with the pretrained weights, makes a deep encoder for the condition image that starts out knowing what images look like.

How ControlNet works on one block

The construction is defined for any network block: a ResNet block, an attention block, a transformer layer. Call the block $\mathcal{F}$, its trained parameters $\Theta$, its input feature map $x$ and its output $y$:

In Stable Diffusion, $x$ and $y$ are 2D feature maps of shape $h \times w \times c$ (height, width, channels), for example 32×32×640 in the second level of the U-Net. ControlNet freezes $\Theta$, makes a trainable copy with parameters $\Theta_c$ initialized to the same values, and connects the two with two convolution layers $\mathcal{Z}$ whose weights and biases are all zero. A condition feature map $c$, of the same shape as $x$, enters through the first one:

Read it from the inside out. The condition passes through the first zero convolution, $\mathcal{Z}(c;\Theta_{z1})$, and is added to the block's normal input. The trainable copy runs on that sum. Its output passes through the second zero convolution and is added to the frozen block's output. The frozen path, $\mathcal{F}(x;\Theta)$, is computed exactly as before; everything ControlNet contributes arrives through the second term.

Each $\mathcal{Z}$ is a 1×1 convolution: at every pixel it multiplies the channel vector by a square matrix (as many output channels as input channels) and adds a bias, the same dense layer applied independently at each position. With its weights and biases at zero it outputs zero whatever its input, so before any training step both $\mathcal{Z}$ terms vanish and

The copy is not idle at this point. Its input is $x + 0 = x$ and its weights equal $\Theta$, so it computes exactly $y$ internally; the second zero convolution discards the result. A software engineer would recognize the arrangement as a new code path shipped behind a feature flag set to off: the production path is untouched on day one, and the new path is fully built and running. The analogy breaks at the flag: here it is the weights of $\mathcal{Z}(\cdot;\Theta_{z2})$, which gradient descent grows continuously from zero as training proceeds.

Why zero convolutions still learn

Initializing a layer to zero is normally avoided. If every weight in a network starts at the same value, every hidden unit computes the same function and receives the same gradient, and the units never differentiate. In ControlNet the zeros sit only at the two ends of the branch, and the units between them, the copy, have distinct pretrained weights, so different channels receive different gradients. The remaining question is whether any gradient reaches the branch, and the answer comes from the three partial derivatives of a 1×1 convolution.

Write the convolution at pixel $p$ and output channel $i$, with input $I$, weight matrix $W$ and bias $B$:

At $W = 0$ the third derivative is zero: no gradient passes backward through the layer to its input. The first two do not depend on $W$, so the layer's own weights and bias get a nonzero gradient as long as its input is nonzero and the loss has some gradient at its output. Summing over pixels with the upstream gradient $g_{p,i} = \partial L / \partial \mathcal{Z}_{p,i}$:

Apply this to the two zero convolutions of equation (2) on the first training step:

1. Z2, the output convolution. Its input is the copy's output, which equals $y$ and is nonzero. The upstream gradient is the loss gradient at $y_c$, nonzero unless the frozen model already fits the training target. So $W_{z2}$ and $B_{z2}$ get a nonzero gradient and move on step 1.
2. The trainable copy. Its gradient arrives through Z2, whose derivative with respect to its input (the third one above) is $W_{z2} = 0$. The copy gets exactly zero gradient on step 1.
3. Z1, the input convolution. Its own weight gradient is (upstream gradient) times $c$, and its upstream gradient comes through the copy and Z2, so it is also zero on step 1.

After step 1, $W_{z2}$ is no longer zero (with plain gradient descent at learning rate $\eta$ it equals $-\eta$ times the first expression above), so from step 2 on gradient flows through it into the copy and on into Z1. The paper's first version derives the first-step update for a single zero convolution; the ordering across the two of them follows from the same derivatives. LoRA's initialization behaves the same way: with $B = 0$ the product $BA$ is zero, $B$ gets a gradient on the first step, and $A$ gets none until $B$ has moved. The zero-initialized layer itself predates both papers; the ADM code of Dhariwal and Nichol wraps the last convolution of every residual block in a function called zero_module, which the ControlNet paper cites.

Figure 1 trains a toy ControlNet block in your browser: a 4-channel block standing in for $\mathcal{F}$, a 2-channel condition, and 24 training pairs whose target is built from the pretrained block's own features. At init scale 0 and step 0, read the gradient under each box: Z2 has one and the other two print exactly 0. Step to 1 and all three are nonzero, while the copy's distance from the pretrained weights (right chart) is still 0, because its first update had a zero gradient. Then raise the init scale, which draws the two connecting layers from a standard random initialization scaled by the slider.

With a random initialization, the copy's first gradient is computed through random weights in Z2, so the direction it is pushed in depends on that random draw rather than on what the condition requires, and it is pushed from the very first step, before Z2 has learned anything. At scale 1.5 in the toy, the loss at step 0 is twice the zero-init loss and the copy ends 40 steps about 2.7 times further from its pretrained weights than the zero-init run does. The paper calls this effect harmful noise reaching the pretrained backbone. Its evidence is an ablation on Stable Diffusion itself, in Section 4.2: replacing both zero convolutions with standard convolutions initialized with Gaussian weights dropped the results on a sketch condition to about the level of ControlNet-lite, a variant whose trainable copy is replaced by a single convolution layer per block. The authors attribute the drop to the pretrained backbone of the copy being destroyed during fine-tuning. The ablation is qualitative (random batches of six images for each setting, no metric), and the toy is too small to reproduce it; at small init scales the toy trains as well as zero init or better.

Attaching it to Stable Diffusion

Stable Diffusion is a latent diffusion model. A variational autoencoder compresses a 512×512×3 image into a 64×64×4 latent, and a U-Net learns to remove noise from latents; the text prompt enters through cross-attention as 77 token embeddings of size 768 from the CLIP text encoder, and the noise level enters as a sinusoidal timestep embedding. The U-Net has an encoder of 12 blocks that halves the resolution three times (64, 32, 16, 8), a middle block at 8×8, and a decoder of 12 blocks that doubles it back. Each encoder block's output is also sent across a skip connection and concatenated onto the input of the matching decoder block.

ControlNet treats the whole encoder plus middle block as the block $\mathcal{F}$ of equation (2). The trainable copy is a copy of all 13 (348,712,960 parameters, 40.6% of the U-Net's 859,520,964), receiving the same noisy latent, timestep and text embedding as the frozen encoder. Instead of one output convolution there are 13, one per copied block, each a 1×1 zero convolution with as many input as output channels. Their outputs are added to the 12 skip connections and to the output of the middle block, so every resolution of the decoder receives a condition-dependent correction.

The condition image has to be brought to the copy's working shape first. A 512×512×3 edge map goes through a small convolutional encoder $\mathcal{E}$:

In the released code $\mathcal{E}$ is eight 3×3 convolutions with SiLU activations, three of them with stride 2 (512 → 256 → 128 → 64), ending in a zero-initialized 3×3 convolution that outputs 320 channels. Its output, 64×64×320, is added to the copy's features right after the copy's first convolution, which has the same shape. That final zero layer plays the part of Z1 in equation (2), and it is the only place the condition enters. (The paper's text describes $\mathcal{E}$ differently, with four stride-2 layers; four halvings of 512 give 32, not 64, and the Provenance panel has the details.)

Figure 2 lays out the 13 connections. Each row is one encoder block of the frozen U-Net, its skip connection to the decoder, the ControlNet's copy of the same block, and the zero convolution between them. Click a row for its shapes and the size of its zero convolution, and switch to the gradients view to see which parts of the network the backward pass touches.

The frozen encoder never needs a backward pass. Nothing upstream of it is trainable, so the official code runs it inside torch.no_grad() and stores none of its activations for the backward pass. The frozen decoder is different: the control is added to its inputs, so the backward pass has to run through the decoder to reach the ControlNet, even though no decoder weight is updated. The paper measures the net cost on one A100 PCIe 40GB against training Stable Diffusion itself: about 23% more GPU memory and 34% more time per iteration. The whole ControlNet for SD 1.5, copy plus zero convolutions plus $\mathcal{E}$, has 361,279,120 parameters, all of them trained. The 13 zero convolutions account for 11,479,680 of those, from 102,720 each at 320 channels to 1,639,680 each at 1,280.

Training: the same loss, and a sudden jump

ControlNet is trained with the loss Stable Diffusion was trained with. Take a training image's latent $z_0$, pick a timestep $t$ between 0 and 999, and mix in Gaussian noise $\epsilon$ at the level the schedule assigns to $t$:

Here $\bar\alpha_t$ falls from nearly 1 at $t = 0$ to 0.0047 at $t = 999$ under Stable Diffusion's schedule; at $t = 500$ the latent is 0.526 parts signal to 0.851 parts noise. The network predicts the noise, given the timestep, the text prompt $c_t$ and the condition $c_f$, and is scored by squared error, as in DDPM:

This is equation (5) of the paper (renumbered here because this page numbers the 1×1 convolution separately). Only $\epsilon_\theta$ has changed: it is now the frozen U-Net plus the ControlNet, and only the ControlNet's parameters are updated. A dataset is a set of triples (image, caption, condition) where the condition is computed from the image by an off-the-shelf detector: Canny edges, HED soft edges, MiDaS depth, OpenPose skeletons, and so on. For the Canny model, the v1 paper reports 3 million such triples and 600 A100 GPU-hours; for surface normals, 25,452 triples from the DIODE dataset and 100 GPU-hours.

The training data differs from Stable Diffusion's in one respect: 50% of the captions are replaced by the empty string. With no prompt, the only evidence about what the image shows is the condition image, so the ControlNet must learn to recognize that an edge map depicts a house or that a depth map depicts a room. The paper reports that this is what lets the model produce sensible images from a condition with no prompt at all.

The authors also report an observation they call the sudden convergence phenomenon. Because the zero convolutions leave the model unchanged at the start, samples drawn during training are ordinary Stable Diffusion images from the first step on, and for thousands of steps they ignore the condition. Then, within a short span, they start following it. In the run the paper shows (learning rate 1e-5, AdamW, no exponential moving average), samples at 6,100 steps still ignore the condition and samples at 6,133 steps follow it; the paper says this usually happens before 10,000 steps. It is an observation from sample grids, with no loss curve or explanation offered.

Guidance, strength and stacking ControlNets

Stable Diffusion samples with classifier-free guidance. Each denoising step runs the network twice, once with the prompt ($\epsilon_c$) and once with an empty or negative prompt ($\epsilon_{uc}$), and extrapolates from the second past the first:

The paper's experiments use $\beta_{cfg} = 9$ with 20 steps of DDIM, a deterministic sampler that skips most of the 1,000 noise levels used in training. A ControlNet raises the question of which of the two passes receives the condition image. If both do, the condition is present on both sides of the difference $\epsilon_c - \epsilon_{uc}$, which then measures the prompt's effect with the condition held fixed, and the condition's own effect is applied once, unamplified. If only $\epsilon_c$ does, the condition's effect sits inside the difference and is multiplied by $\beta_{cfg}$. With an empty prompt and the condition in both passes, the two passes get identical inputs, so $\epsilon_c = \epsilon_{uc}$ exactly and guidance does nothing, which the paper describes as completely removing guidance (its Figure 5b). With an empty prompt and the condition only in $\epsilon_c$, the condition's effect is multiplied by 9, which the paper describes as very strong guidance (Figure 5c).

Figure 3 makes this concrete with a toy linear model in two dimensions, where the prompt pushes the prediction right and the condition pushes it up. It opens with a prompt and the condition in both branches; switch the prompt to empty and move the guidance slider, and nothing moves. Then put the condition in $\epsilon_c$ only and watch the condition's multiplier track $\beta_{cfg}$.

The paper's fix, CFG resolution weighting, feeds the condition to $\epsilon_c$ only and multiplies each of the 13 connections by a weight $w_i = 64/h_i$, where $h_i$ is the block's resolution: 1 for the 64×64 connections, 2, 4, and 8 for the 8×8 ones. The released demo implements the same idea under the name guess mode, with a different formula: the condition is dropped from the unconditional pass and connection $i$ (0 to 12, the middle block last) is scaled by $0.825^{12-i}$, from 0.0994 at the first 64×64 block to 1.0 at the middle block. Both give the low-resolution connections roughly 8 to 10 times the weight of the high-resolution ones. The absolute scales differ, since the paper's weights are at least 1 and the code's at most 1, and the paper gives no reason for its choice of 64/h beyond the images in its Figure 5d. The inference-scales view of Figure 2 plots both. Outside guess mode, the demo puts the condition in both passes and multiplies all 13 outputs by a single control strength, 1.0 by default.

Several ControlNets combine by addition. A pose ControlNet and a depth ControlNet each produce their 13 maps; the maps are summed per connection and added to the same skip connections. The paper reports that no extra weighting is needed for this. The same additivity lets a ControlNet trained on SD 1.5 drive community models fine-tuned from SD 1.5, which keep its architecture: the paper shows this for Comic Diffusion and Protogen 3.4 without retraining, as images only.

What the experiments show

The paper's quantitative evaluation is three tables and a user study. Figure 4 plots the three tables.

• User ranking (Table 1). Twelve users ranked five methods' outputs for each of 20 unseen hand-drawn sketches, from 1 (worst) to 5 (best), once for image quality and once for fidelity to the sketch. ControlNet averaged 4.22 for quality and 4.28 for fidelity. Sketch-guided diffusion trades one for the other: raising its guidance scale from 1.6 to 3.2 raised fidelity from 2.31 to 3.28 and lowered quality from 3.21 to 2.52.
• Layout recovered (Table 2). Generate images from ADE20K segmentation maps, run the OneFormer segmenter on them, and measure the overlap (IoU) with the input map. ControlNet scores 0.35. The ceiling is OneFormer's own score on the real images, 0.58, so about 60% of the achievable overlap.
• Image quality (Table 3). FID compares the statistics of generated and real images in a pretrained network's feature space; lower is better. Segmentation-conditioned ControlNet scores 15.27, against 17.92 for ControlNet-lite and 19.74 for PITI. Stable Diffusion with no segmentation input scores 6.09; it is a reference, not a competitor, since it is free to draw any layout. ControlNet's CLIP text-image score, 0.26, and aesthetic score, 6.31, match plain Stable Diffusion's 0.26 and 6.32, consistent with the claim that the frozen model's quality is kept.

The paper also compares against an industrial model. Stability AI's SD 2 depth-to-image model was trained on more than 12 million images with thousands of A100 GPU-hours. The authors trained a depth ControlNet for SD 2 on 200,000 images on one RTX 3090 Ti for 5 days, showed 12 users 100 images from each model to learn the difference, then asked them to attribute 200 new images. The users' average precision was 0.52 (standard deviation 0.17), about what guessing gives.

On training-set size, ControlNets trained on as few as 1,000 images do not collapse and still produce a recognizable lion for a lion edge map, and results improve as the subset grows; the paper shows this as sample images, without a metric.

What the paper leaves open

The sudden convergence is unexplained. The zero initialization accounts for the clean images before the jump (the model is plain Stable Diffusion until the connections grow), but the paper offers no reason for the transition being abrupt rather than gradual, and it shows no loss curve.

The human evaluations are small. Table 1 rests on 12 users and 20 sketches; the text calls the metric Average Human Ranking and the table caption calls it Average User Ranking. The text says the study yields 100 rankings per criterion, a count equal to 20 sketches times 5 methods; it does not say how the 12 users' rankings combine into it. The key ablations, removing the zero convolutions or shrinking the copy, are shown as image grids without a number.

The ControlNet adds inference cost. It reads the current noisy latent and timestep, so it runs at every denoising step, adding a 361-million-parameter encoder pass to each call of an 860-million-parameter U-Net, and with the condition in both guidance branches it runs twice per step; stacking two ControlNets doubles the added cost.

When the condition is ambiguous and the prompt does not resolve it, the model guesses: the paper's Figure 11 shows it interpreting a bare shape as different objects, and v1 lists generating the wrong content after a wrong reading of the condition as its limitation.

Training and sampling in pseudocode

The pseudocode below is one training step with the shapes of the official SD 1.5 setup. The VAE and the CLIP text encoder are frozen and run without gradients; the weight update touches only the ControlNet.

# One ControlNet training step (SD 1.5, 512x512, batch of 4)
img, edges, caption = batch()          # img [4,512,512,3] in [-1,1]
                                       # edges [4,512,512,3] in [0,1]
if rand() < 0.5: caption = ""          # 50% empty prompts (Sec. 3.3)
with no_grad():
    z0  = 0.18215 * vae.encode(img)    # [4,64,64,4]  latent image
    txt = clip_text(caption)           # [4,77,768]
t   = randint(0, 1000, size=4)         # one noise level per image
eps = randn(4, 64, 64, 4)
zt  = sqrt(abar[t]) * z0 + sqrt(1 - abar[t]) * eps
# ControlNet: copy of the encoder + middle block, reads zt, t, txt and edges
ctrl = controlnet(zt, t, txt, hint=edges)   # 13 maps, 64x64x320 ... 8x8x1280
# locked U-Net: encoder under no_grad, ctrl added to its 13 connections
eps_hat = unet(zt, t, txt, control=ctrl)    # [4,64,64,4]
loss = mean((eps - eps_hat) ** 2)           # Eq. (5), SD's own loss
loss.backward()            # weight gradients reach the ControlNet only
adamw_step(controlnet.parameters(), lr=1e-5)

Sampling runs the ControlNet inside every guided step; the two branches below are the demo's default and its guess mode.

# Sampling with one ControlNet: DDIM, 20 steps, guidance 9.0
txt_c  = clip_text(prompt)
txt_uc = clip_text(negative_prompt)          # often ""
z = randn(1, 64, 64, 4)
for t in ddim_timesteps(20):                 # 999 ... 0
    ctrl = [s * m for s, m in zip(scales, controlnet(z, t, txt_c, edges))]
    eps_c = unet(z, t, txt_c, control=ctrl)
    if guess_mode:                            # control in eps_c only
        eps_uc = unet(z, t, txt_uc, control=None)
    else:                                     # default: control in both
        ctrl_u = [s * m for s, m in zip(scales,
                  controlnet(z, t, txt_uc, edges))]
        eps_uc = unet(z, t, txt_uc, control=ctrl_u)
    eps = eps_uc + 9.0 * (eps_c - eps_uc)     # classifier-free guidance
    z = ddim_update(z, eps, t)
image = vae.decode(z / 0.18215)
# scales: [strength] * 13 by default;
#         strength * 0.825 ** (12 - i) for i in 0..12 in guess mode

In the code, the condition image is scaled to [0, 1] while the training image is scaled to [−1, 1] before the VAE, and the optimizer is AdamW over control_model.parameters() only. The v1 paper describes a "large-scale training" option for datasets of a million pairs or more: train the ControlNet alone for more than 50,000 steps, then unlock Stable Diffusion's weights and train jointly. In the code, setting sd_locked = False adds the decoder's weights (output blocks and final layer) to the optimizer; the encoder stays frozen.