SAM: Segment Anything

A segmentation model that outlines any object from a click or a box.

SAM runs a large image encoder once per image, then turns each prompt into three candidate masks in about 50 milliseconds. Its 1.1 billion training masks came from a loop in which the model labeled images, people corrected it, and the retrained model took over more of the work until it labeled 11 million images on its own.

Before SAM, a segmentation model was trained on one dataset for a fixed list of classes. SAM is trained on a single task instead: given any prompt, return a valid mask.

Segmentation means labeling, for every pixel in an image, whether it belongs to a given object. The output is a mask: a grid the size of the image, 1 on the object and 0 elsewhere. For most of the 2010s the standard way to get one was to choose a dataset, fix its class list (COCO has 80), train a model for exactly those classes, and repeat for the next dataset. A model trained to find cars and people had no way to outline a coffee mug.

Language models had moved past that pattern. One model is pre-trained on a broad objective (predict the next token) and then pointed at new jobs with a prompt instead of new training. Segment Anything, from Meta's FAIR lab in April 2023, applies that pattern to segmentation. It has three parts designed together: a task (promptable segmentation), a model (SAM, the Segment Anything Model), and a dataset (SA-1B, 1.1 billion masks on 11 million images) produced by a data engine in which the model helped label the data it was trained on.

The page follows the paper's order. First the task and the ambiguity problem it creates, and how three output masks plus a min-over-three loss solve it. Then the model: why it is split into a heavy encoder and a light decoder, what tensors flow through the decoder for one click, what a mask is numerically, and the losses. Then the three-stage data engine, the filters that decided which automatic masks to keep, and the quality audit of SA-1B. Last, the zero-shot results, including the ones where SAM loses.

A foundation model for segmentation

A foundation model is pre-trained once on a broad task and then adapted to many narrower ones. In vision, the best-known example before SAM was CLIP, which trains an image encoder and a text encoder so that a photo and its caption land near each other in one embedding space. After training you can classify images into categories you name in words, with no labeled examples of those categories. The paper calls that zero-shot transfer: evaluating a model on data distributions and tasks it was not trained on, with no fine-tuning.

CLIP names what is in an image; it does not say which pixels belong to it. Segmentation also lacks what made CLIP possible: the web has billions of image-caption pairs and almost no pixel-accurate outlines. The authors therefore set the goal as a promptable model pre-trained on a task broad enough that, afterwards, a new segmentation problem can be solved by choosing the right prompt, for image distributions the model never saw. Section 7 of the paper tests that claim on 23 datasets and five tasks.

The promptable task

A prompt is anything that indicates what to segment: one or more foreground or background points, a rough box, a coarse mask, or, experimentally, a phrase of text. The promptable segmentation task asks for a valid mask for any prompt.

"Valid" is defined for ambiguous prompts. A single click on a shirt pocket could mean the pocket, the shirt, or the person wearing it. The task does not require the model to pick the one the user had in mind; it requires the output to be a reasonable mask for at least one of those interpretations. A pocket-shaped mask is valid, a shirt-shaped mask is valid, and a mask covering half the shirt and part of the background is not.

The same definition serves as the pre-training objective and as the interface for downstream use. For pre-training, each ground-truth mask generates a sequence of simulated prompts (a point inside it, a noisy box around it, corrective clicks) and the model's outputs are scored against that mask. For downstream use, a new problem is solved by generating prompts: a detector's boxes for instance segmentation, a grid of points for "segment everything". The simulated-prompt training comes from interactive segmentation, where a user clicks until the mask is right, but the goal differs. An interactive segmenter needs to be right after enough clicks. SAM needs a valid mask after the first click, because the data engine's last stage prompts it with single points and no human checks each answer.

Three masks per click

Suppose the model outputs one mask per prompt, and training shows it the pocket click paired with the pocket mask a third of the time, the shirt mask a third of the time, and the person mask a third of the time. The per-pixel loss is minimized by predicting the average of the three targets: probability near 1 on the pocket, near 2/3 on the rest of the shirt, near 1/3 on the rest of the person. Thresholded at 0.5 that is the shirt, and the pocket and person are never produced. As a soft map it is a blurred compromise that matches none of the three.

SAM predicts three masks for a single prompt. The paper's reason for three: when a click is ambiguous, the candidate objects are usually nested (subpart inside part inside whole) and the nesting is rarely more than three levels deep. Figure 1 shows the pocket example. The point sits inside all three regions, so all three masks are valid for it.

Training has one ground-truth mask per prompt and three predictions. SAM computes the loss of each prediction against the ground truth and back-propagates only the smallest:

Here $m_i$ are the three predicted masks, $g$ is the ground truth, and $\mathrm{FL}$ and $\mathrm{DL}$ are the focal and dice losses defined in the losses section. If the ground truth is the shirt and the three losses come out 0.90, 0.12 and 0.60, only output 2 receives a gradient, and it moves further toward the shirt. Outputs 1 and 3 are left where they are, so the next time the ground truth is the pocket, one of them is likely to have the lowest loss and be the one that moves. Over many prompts the three outputs divide the interpretations between them. The technique is multiple-choice learning (Guzman-Rivera et al., 2012): an ensemble of outputs covers the plausible answers and only the closest is graded. Summing the loss over all three outputs would pull each of them to the same average mask as a single output.

Figure 2 runs that argument on a toy. Three nested discs of radius 4, 10 and 18 are the possible ground truths for one central click. Each training step samples one of them. Three heads each have 24 free parameters, one logit per ring. With the gradient going only to the lowest-loss head, init A ends with each head matching one disc (best IoU 1.00 for all three targets). Switch to "all heads" and every head converges to the same staircase of probabilities: about 1.00, 0.62 and 0.34 across the three zones at the last step (the exact average of the targets is 1, 2/3 and 1/3; each noisy step moves the values around it). Thresholded at 0.5 that is the part disc for every head, and the subpart (best IoU 0.16) and the whole (0.31) are never produced.

Init B is a known weakness of winner-take-all training: if two outputs start closer to every target than the third, the third never has the lowest loss and receives no gradient. Rupprecht et al. (2017) proposed passing a small share of the gradient to the losing outputs to fix it. The SAM paper does not report dead outputs; in the real network the three outputs share the decoder and differ only in their learned tokens and final MLPs, which is a different starting point from three independent heads.

An application that wants a single mask needs a way to choose among the three. SAM adds a small head that predicts, for each mask, its estimated IoU: the overlap the model expects that mask to have with the object it covers (IoU is defined below). The head is trained by regression against the true IoU of each predicted mask, and the mask with the highest estimate is returned. The estimate ranks the model's own candidates; it is not a calibrated probability, and on an image unlike its training data it can be high for a wrong mask.

A fourth output token covers prompts that are not ambiguous. With two points, or a box plus a point, the three outputs would usually be near-duplicates, so whenever more than one prompt is given SAM returns only the mask from this fourth token, trained with an ordinary loss; for a single prompt the fourth mask is never returned.

Encode once, decode per prompt

The authors wanted SAM usable interactively in a web annotation tool: click, see the mask, click again. That requires roughly 50 milliseconds per prompt, and a large vision Transformer on a 1024 x 1024 image takes far longer than that. SAM is split along the line between computation that depends only on the image and computation that depends on the prompt.

• Image encoder. A ViT-H/16 (the Huge size of the Vision Transformer, which cuts the image into 16 x 16-pixel patches), pretrained with MAE (reconstructing randomly hidden patches) and adapted to a $1024 \times 1024$ input. It outputs a $64 \times 64$ grid of $256$-dimensional vectors, the image downscaled $16\times$. It runs once per image.
• Prompt encoder. Each point or box corner becomes a $256$-dimensional vector: a positional encoding of its location (sines and cosines of a fixed random projection of $(x, y)$, so nearby points get similar vectors) plus a learned vector for its type (foreground point, background point, top-left corner, bottom-right corner). A mask prompt goes through a few convolutions and is added to the image embedding. Text, in the experimental variant, goes through CLIP's text encoder.
• Mask decoder. Two small Transformer blocks in which the prompt tokens attend to the image embedding and the image embedding attends back to the tokens, then an upsampling step and a per-pixel product that produces the masks. It uses less than $1\%$ of the image encoder's compute. Given a precomputed image embedding, prompt encoder plus decoder run in about $50$ ms in a web browser, on CPU.

The first prompt on a new image pays for the encoder. Every later prompt on the same image reuses the embedding and costs only the decode, so K prompts on one image cost (encoder + 50K) ms in total, or (encoder + 50K)/K ms per prompt. Drag the prompt count in Figure 3 and read the amortized cost; Figure 4 shows the same budget as a timeline.

This split is also why SAM is not real-time end to end. The 50 ms covers the prompt encoder and decoder. The project's FAQ quotes about 0.15 s for the ViT-H encoder on an NVIDIA A100 GPU; that number is not in the paper, and the paper's own limitations section says SAM's overall performance is not real-time with a heavy image encoder.

Inside the mask decoder

Figure 5 steps through one foreground click with the shapes of the released ViT-H model. The decoder works on tokens, a short list of 256-dimensional vectors. For one click there are 7: the click itself, a padding point the code appends when no box is given, four learned mask tokens (one per possible output mask) and one learned IoU token. The image embedding is treated as a second set of 4,096 vectors, one per location of the $64 \times 64$ grid.

Each decoder block does four things in order: self-attention among the 7 tokens, cross-attention from the tokens to the 4,096 image vectors (so the output tokens gather information about the clicked region), an MLP on each token, and cross-attention from the image vectors back to the tokens (so the image embedding is updated with the prompt). The positional encoding is added to the image vectors in every attention layer, and the original prompt tokens are re-added to the updated tokens, so the click location stays available at every step. The MLP is wide (2048) but runs on only 7 vectors; the cross-attention layers, which touch all 4,096 image vectors, work at 128 dimensions instead of 256 to save compute.

After two blocks the image embedding is upsampled 4x by two transposed convolutions, from $64 \times 64 \times 256$ to $256 \times 256 \times 32$. Each of the four mask tokens goes through its own 3-layer MLP and comes out as a 32-dimensional vector $v$. The mask logit (the raw score before a sigmoid turns it into a probability) at every location of the upsampled grid is the dot product of $v$ with the 32-dimensional feature $e_{x,y}$ there, and the probability is its sigmoid:

Read (2) as a linear classifier applied at every pixel. The weights $v$ are produced by the decoder from this prompt, so the paper calls it a dynamic linear classifier: each prompt gets a new 32-weight classifier, and the mask is that classifier evaluated at all $256 \times 256 = 65{,}536$ locations in one matrix multiply. Pixels whose features point the same way as $v$ get a large positive logit; pixels whose features point away get a negative one. The logit maps are upsampled to the input resolution and thresholded at 0 (probability 0.5). Figure 6 reduces this to 2-dimensional features so the direction of $v$ can be a dial.

The IoU token goes through its own MLP to four estimated IoUs, one per mask token. With one prompt, the code returns masks 1 to 3 and their scores; with more than one, it returns only mask 0. The paper describes mask 0 as a "fourth" token added for the multi-prompt case; the code stores it first, which changes the indexing and not the behavior.

Losses and simulated clicks

Mask quality is measured by intersection over union: the area where the predicted mask and the ground truth overlap, divided by the area covered by either.

It runs from 0 (no overlap) to 1 (identical). A worked case: ground truth 1,000 pixels, prediction 900 pixels, 850 of them on the object. The union is 1,000 + 900 − 850 = 1,050, so IoU = 850 / 1,050 = 0.81. Averaged over a benchmark it is reported as mIoU, the main metric in the paper's point-prompt experiments. In Figure 7, drag the prediction and watch the value; the dashed mark at 0.90 is the level most of SA-1B clears in the quality audit described later.

IoU of a thresholded mask is a step function of the logits and has no useful gradient, so training uses two differentiable losses in a $20{:}1$ ratio of focal to dice, a recipe SAM takes from DETR and MaskFormer. Focal loss (Lin et al., 2017) is per-pixel cross-entropy with a factor that shrinks the loss on pixels the model already gets right:

where $p_t$ is the predicted probability of the correct label for a pixel. The factor $(1-p_t)^{\gamma}$ does the down-weighting. With $\gamma = 2$ (the default in Lin et al.; the SAM paper does not state its value), a pixel predicted correctly at $p_t = 0.9$ has its cross-entropy multiplied by 0.01, while an uncertain pixel at $p_t = 0.5$ keeps a quarter of it. Most pixels of a mask are easy interior or background pixels, so the gradient concentrates on the uncertain ones along the boundary. The separate weight $\alpha_t$ balances foreground against background and plays no part in the easy-versus-hard weighting.

Dice loss (Milletari et al., 2016) is one minus the Dice overlap coefficient:

In the worked case, Dice = 2 · 850 / (1,000 + 900) = 0.89, and in general Dice = 2 IoU / (1 + IoU). In training, $A$ is the soft probability map rather than a binary mask, which makes (5) differentiable. Because it is a ratio of areas, dice gives a 50-pixel object the same weight as a 50,000-pixel one, which a per-pixel average does not. (V-Net's original loss squares the terms in the denominator; (5) is the plain coefficient.)

The prompts come from a simulated interactive session with 11 rounds per mask. The first prompt is, with equal probability, a point sampled uniformly inside the mask or its bounding box with noise (standard deviation 10% of the box side, capped at 20 pixels). Each of the next 8 rounds adds a point sampled from the error region between the current prediction and the ground truth: a foreground point if the prediction missed that area, a background point if it covered extra area. The previous round's best mask is fed back as a mask prompt, as raw logits rather than a thresholded mask. Two rounds add no new point, one inserted at random among the 8 and one at the end, so the model learns to improve its own mask from the mask prompt alone. From the second round on the prompt has more than one element, so those rounds use the single fourth mask.

# Sketch of one training step on (image, ground-truth mask g), after App. A.
# The public repository ships inference code only.
emb = image_encoder(img)                 # once; reused by all 11 rounds
prompt = [rand_point_in(g) if coin() else noisy_box(g)]  # box noise: 10%, <=20px
rounds = ["click"] * 8
rounds.insert(randint(0, 8), "none")     # one empty round at a random spot
rounds.append("none")                    # one at the end: 1 + 8 + 2 = 11 rounds
prev, total = None, 0
for r in ["first"] + rounds:
    masks, iou_hat = decoder(emb, prompt, prev)  # 3 masks for one prompt, else 1
    per_mask = [20 * focal(m, g) + dice(m, g) for m in masks]
    total += min(per_mask)               # eq. (1): only the best mask is graded
    total += mse(iou_hat, [iou(m > 0, g) for m in masks])   # weight 1.0
    prev = masks[argmax(iou_hat)]        # its raw logits become the mask prompt
    if r == "click":
        prompt.append(rand_point_in_error(prev, g))  # fg if missed, bg if extra
total.backward()                         # one optimizer step for all 11 rounds

Eleven decoder passes per mask per step are affordable because the decoder costs under 1% of the encoder, whose output all 11 rounds reuse. The rest of the recipe, from Appendix A: AdamW with learning rate 8e−4 after a 250-iteration warmup, 90k iterations (about 2 epochs of SA-1B) with the rate cut 10x at 60k and 86,666, batch of 256 images with up to 64 masks per GPU, weight decay 0.1, drop path 0.4, layer-wise learning-rate decay 0.8, no data augmentation, and initialization from MAE weights. The paper's model card gives the cost as 256 A100 GPUs for 68 hours.

The data engine

Training SAM needs a very large set of masks, and producing masks cheaply needs a good segmentation model. The paper breaks this circle with a data engine: the current model assists annotators, the new masks retrain the model, and the better model takes over more of the labeling in the next round. It ran in three stages:

• Assisted-manual. Annotators clicked foreground and background points in a browser tool running SAM and fixed the result with brush and eraser tools. SAM started from public segmentation datasets and was then retrained 6 times on the new masks, with the encoder scaled from ViT-B to ViT-H along the way. Time per mask fell from $34$ to $14$ seconds (the paper notes 14 s is 6.5x faster than COCO's mask annotation) and masks per image rose from $20$ to $44$. Output: $4.3$ million masks on $120$k images.
• Semi-automatic. To add less prominent objects, images arrived pre-filled with confident masks (found by a box detector trained on the stage-1 masks with a single generic "object" class), and annotators labeled only what was missing. Output: $5.9$ million more masks on $180$k images ($10.2$ million in total), with masks per image rising from $44$ to $72$ counting the automatic ones. Time per hand-drawn mask rose back to $34$ seconds, because the objects left for people were the harder ones.
• Fully automatic. SAM, by now ambiguity-aware and trained on the diverse stage-2 masks, was prompted with a $32 \times 32$ grid of points on each image. Each point yields up to three masks; the engine keeps the confident and stable ones and removes duplicates. Run on all $11$ million images, this produced $1.1$ billion masks.

Automatic masks and the stability filter

The fully automatic stage (Appendix B) prompts more than the 32 x 32 grid. It also cuts 20 zoomed-in crops from the original high-resolution image, 4 from a 2 x 2 layout with a 16 x 16 point grid each and 16 from a 4 x 4 layout with an 8 x 8 grid each. That is 1,024 + 4 · 256 + 16 · 64 = 3,072 points per image, and up to three masks per point, about 9,000 candidates before filtering. SA-1B averages about 100 masks per image, so roughly 99% of candidates are discarded by these steps:

1. Confidence. Keep masks whose estimated IoU exceeds 0.88.
2. Stability. Keep masks whose stability score, defined below, is at least 0.95.
3. Coverage. Drop masks covering 95% or more of the image.
4. Duplicates. Neighboring grid points on the same object return near-identical masks. Non-maximum suppression (NMS) sorts masks by score and drops any whose bounding box overlaps an already-kept box with IoU above 0.7. It runs within each crop (ranked by estimated IoU) and then across crops (preferring the more zoomed-in crop).
5. Cleanup. Remove connected components and fill holes smaller than 100 pixels. The paper estimates about 4% of masks had spurious small components and another 4% small holes.

The stability score uses the fact that SAM outputs logits, not a binary mask. Threshold the logit map at 0 and you get the mask. Threshold it at +1 and at −1 and you get two more: the +1 mask is inside the −1 mask, and the band between them holds the pixels whose logit is within 1 of the decision boundary (probability between 0.27 and 0.73). The stability score is their IoU, which for nested masks is the area of the +1 mask divided by the area of the −1 mask. If the logits fall steeply at the object edge, the band is thin and the score is close to 1. If they fall slowly, the model is unsure where the object ends, the band is wide, and the score drops. The paper's main text phrases this as thresholding the probability map at 0.5 ± $\delta$; the released setting of ±1 in logits corresponds to $\delta \approx 0.23$.

For a disc of radius $R$ whose logit drops from +1 to −1 across a band $b$ pixels wide, the two masks are discs of radius $R - b/2$ and $R + b/2$:

At $R = 40$ pixels, a 0.8-pixel band gives $(39.6/40.4)^2 = 0.961$ and the mask is kept; a 2-pixel band gives $(39/41)^2 = 0.905$ and it is dropped. The same 0.8-pixel band on a disc of radius 10 gives $(9.6/10.4)^2 = 0.852$, also dropped. For a fixed band width, small and thin objects score lower, because the band is a larger share of their area. Equation (6) is this page's disc calculation, not a formula from the paper; the code computes the ratio of pixel counts for whatever shape the mask has.

Stability and estimated IoU are separate signals. The estimated IoU is a learned output of the decoder; the stability score is computed from the logit map with no learned parameters. The engine requires both. The model that generated SA-1B was also a special version of SAM: trained only on the manual and semi-automatic masks, for 177,656 iterations instead of 90k, with large-scale jitter augmentation, point and mask prompts only, and a 3-layer decoder instead of 2.

SA-1B and how good its masks are

SA-1B has $1.1$ billion masks on $11$ million licensed, privacy-protecting images (faces and license plates blurred), about $100$ masks per image. That is $400\times$ more masks and $11\times$ more images than Open Images, the largest segmentation dataset before it. The images average 3300 x 4950 pixels and are released downsampled to a shortest side of 1500 pixels. 99.1% of the masks were generated fully automatically, and SA-1B ships only the automatic masks; the 10.2 million hand-drawn masks from stages 1 and 2 would be 0.9% of 1.1 billion.

To check the automatic masks, the authors took $500$ images (about $50$k masks) and asked professional annotators to correct every mask with brush and eraser tools. Comparing each automatic mask with its corrected version, 94% had IoU above $0.90$ and 97% above $0.75$. For comparison the paper cites prior work putting agreement between two human annotators at 85 to 91% IoU; that range is a baseline from other studies, not a measurement on SA-1B. In an ablation, training on the automatic masks alone scores only about $0.5$ mIoU below training on all masks from the three stages, which is why the release includes only the automatic ones.

Zero-shot transfer

The paper tests five tasks with no fine-tuning: segmenting from a single point on 23 datasets, edge detection, object proposals, instance segmentation, and text-to-mask. The last four differ from the training task and are each solved by constructing prompts.

Single point

The comparison is against RITM, a strong interactive segmenter, on 23 datasets covering egocentric, microscopy, X-ray, underwater, aerial, simulated, driving and painting images. Scoring only SAM's highest-ranked mask against the ground truth, SAM beats RITM on $16$ of the $23$. With an "oracle" that picks whichever of SAM's three masks best matches the ground truth, SAM beats RITM on all $23$. The gap between the two scorings is the cost of choosing the wrong nesting level: the right mask was among the three, but the estimated IoU ranked another one first.

A human study on the same task rated masks from 1 to 10. SAM's mean ratings fell between 7 and 9, above RITM, and above an ablated SAM with a single output mask (which in turn rated above RITM). This held on datasets such as DRAM and IBD, where SAM scores worse than RITM on the automatic metric.

Instance segmentation

Here SAM is one component in a pipeline. A detector (ViTDet-H) finds boxes; each box goes to SAM as a prompt; SAM returns the mask. No part of SAM is trained on COCO or LVIS. Against ViTDet's own masks, trained directly on those datasets, SAM trails on mask AP (average precision, the standard detection metric): $46.5$ versus $51.0$ on COCO and $44.7$ versus $46.6$ on LVIS. In a human study, raters preferred SAM's masks, which have crisper boundaries. The authors hypothesize that ViTDet learns the specific biases of COCO's masks, whose ground-truth quality the human study rates relatively low, while SAM, trained on neither dataset, cannot exploit them; AP rewards matching the annotations, biases included.

Edge detection

On BSDS500, SAM is prompted with a 16 x 16 grid of foreground points (768 masks, 3 per point), and edges are read off the boundaries of the resulting masks, although SAM was never trained on edges. It reaches a recall at 50% precision of 0.928 (ODS 0.768, OIS 0.786, AP 0.794): it finds most annotated edges but also marks many the benchmark did not annotate, so precision is lower, and it trails methods trained on BSDS500.

Object proposals

Object proposals are class-agnostic candidate regions that a later stage classifies; the benchmark is LVIS v1. Using the automatic mask generator as a proposal source, SAM beats ViTDet-H on medium, large, rare and common objects, and trails it on small and frequent ones.

Text to mask

This experiment needed its own training run. For every manually collected mask larger than $100^2$ pixels, the CLIP image embedding of that region was used as the prompt. At inference the CLIP text embedding of a phrase such as "a wheel" goes in its place. This works to the extent that CLIP maps a picture of a wheel and the words "a wheel" to nearby vectors. The authors call it exploratory and not entirely robust, and the released models take points, boxes and masks.

Encoder size

In the ablations, ViT-H improves substantially over ViT-B but only marginally over ViT-L, and the authors concluded that scaling the image encoder further did not appear fruitful at the time.

What SAM does not do

SAM's masks are class-agnostic. It outputs a mask for an object without a label saying what the object is. Semantic segmentation (a class for every pixel) and panoptic segmentation (a class and an instance for every pixel) cannot be obtained by prompting it, and the paper says it is unclear how to design simple prompts that would do so. Labels have to come from whatever SAM is combined with, such as the detector in the instance-segmentation experiment.

It does not top the specialist benchmarks. It trails ViTDet on mask AP, trails edge detectors that learn BSDS500's annotation style, and the authors expect dedicated interactive segmenters to beat it when many clicks are given. The limitations section lists missed fine structures, occasional small disconnected components that should not be there, and boundaries less crisp than slower methods that zoom in on the object. The estimated IoU it reports ranks its own masks and does not certify them.

The paper leaves open whether SAM is a foundation model; the authors write that it "remains to be seen by how it is used in the community". The paper does show a recipe that works at this scale: a task general enough to pre-train on and to reuse through prompts, a model split so that the per-prompt part costs 50 ms, and a data engine that turned about 10 million human-drawn masks into 1.1 billion. Its successor, SAM 2 (2024), applies the same promptable interface to video.