Mask R-CNN

A detector that also outlines each object: Faster R-CNN plus a small network that predicts a 28 × 28 mask inside every detected box.

Mask R-CNN adds a third output to each region Faster R-CNN examines: one binary mask per class, from a few convolution layers. Two details make it work. The mask loss is applied only to the mask of the region's own class, and the region features are sampled at exact positions by a new layer, RoIAlign, instead of being rounded to the feature grid.

In 2016 a detector could put a box around each person in a photo, and a segmentation network could colour every person pixel. Producing a separate outline for each of two people standing side by side was harder, and the best systems for it were long chains of hand-tuned stages.

This paper, from Facebook AI Research in March 2017, solves that task with a short extension of an existing detector. It takes Faster R-CNN, keeps its two stages, and gives the second stage one more output: a small grid of per-pixel probabilities for each region. On COCO the result beat every single-model entry of the 2016 competition at instance segmentation, box detection and human pose estimation, while running at 5 frames per second. The authors released the code as part of Facebook's Detectron library.

A few ideas explain it: what instance segmentation outputs and how it is scored; the Faster R-CNN detector it extends; the mask branch and its loss; why that loss uses an independent sigmoid per class; what the old RoIPool layer rounds and how RoIAlign removes the rounding; and how a feature pyramid decides which feature map each region reads. The training recipe, the results and the keypoint extension come after.

Instance segmentation and how it is scored

Instance segmentation asks for every object in an image as a class label, a confidence score, and a binary mask: the set of pixels that belong to that one object. It differs from semantic segmentation, the task of networks like FCN and U-Net, which label each pixel with a class and do not separate objects, so two people who touch come out as one connected person region. In an instance segmentation output the two people are two masks, and where one stands in front of the other, the front person's mask covers the overlap.

The benchmark is COCO: 80 object classes, every object outlined by a polygon. The paper trains on 115,000 images (trainval35k) and runs its ablations on 5,000 held-out validation images (minival). A predicted mask matches a ground-truth mask if their intersection over union (IoU, the overlap area divided by the combined area) passes a threshold. For one class, sorting all detections by score and walking down the list gives a precision-recall curve; its area is the class's average precision (AP). COCO's headline number, written AP, averages that over the 80 classes and over ten IoU thresholds, 0.50, 0.55, up to 0.95. AP₅₀ and AP₇₅ use one threshold each, and AP_S, AP_M, AP_L restrict the count to small objects (under 32 × 32 pixels), medium ones, and large ones (over 96 × 96).

For masks, IoU is computed pixel by pixel. A 30 × 30 square mask predicted with the right shape but 3 pixels to the side overlaps the true one by 27 × 30 = 810 pixels out of a union of 990, an IoU of 0.82: it counts as correct at thresholds up to 0.80 and as a miss at 0.85, 0.90 and 0.95. Half of COCO's ten thresholds are 0.75 or stricter, so boundary errors of a few pixels cost AP directly. Box AP, written AP^bb in the paper, is the same computation with box IoU.

The starting point: Faster R-CNN

Faster R-CNN works in two stages on a shared convolutional feature map. The first stage, the region proposal network (RPN), slides a small network over the map and proposes a few hundred to a few thousand boxes that might contain an object. The second stage takes each proposal, which is called a region of interest or RoI, cuts the matching window out of the feature map, resamples it to a fixed 7 × 7 grid, and passes that through a head that outputs class scores and four box-refinement offsets per class. The resampling layer in the original is RoIPool, from Fast R-CNN. The Faster R-CNN explainer covers the anchors, the proposal loss, and non-maximum suppression (NMS, which deletes boxes that overlap a higher-scoring box by more than a threshold).

The paper names two parts of the network. The backbone computes features over the whole image once. The head runs once per RoI. Mask R-CNN is tested with two backbone families, each built on a ResNet of 50 or 101 layers, or on ResNeXt, a ResNet variant with grouped convolutions:

• ResNet-C4: features from the end of ResNet's fourth stage, one map with 1,024 channels at stride 16 (one feature cell per 16 × 16 image pixels). The fifth stage, res5, becomes part of the per-RoI head, which makes the head expensive.
• ResNet-FPN: a Feature Pyramid Network (Lin et al., 2017, three of whose authors wrote this paper) adds a top-down path with lateral connections and produces four maps of 256 channels at strides 4, 8, 16 and 32. Each RoI reads the map that matches its size. The head is two fully connected layers, much lighter.

How Mask R-CNN works

Mask R-CNN keeps the RPN unchanged and adds a third branch to the second stage. For each RoI the branch outputs $K$ masks of $m \times m$ logits, one per class, where $K = 80$ on COCO and $m$ is 28 with FPN (14 with C4). A logit is an unbounded score; the sigmoid $\sigma(z) = 1/(1+e^{-z})$ turns it into a probability that the pixel belongs to the object.

The branch is a small fully convolutional network (FCN): every layer is a convolution, so every layer keeps a spatial grid and the output pixel at (i, j) is computed from features near (i, j) of the RoI. The class and box outputs, by contrast, come from fully connected layers that flatten the RoI into a vector. With FPN the mask branch reads its own 14 × 14 × 256 RoI features, applies four 3 × 3 convolutions of 256 channels, a 2 × 2 deconvolution with stride 2 (a learned upsampling that doubles the grid to 28 × 28), and a 1 × 1 convolution to 80 channels. That is about 2.6 million weights. The paper also tried predicting the masks with fully connected layers; the last layer alone would need 1,024 × 80 × 28² = 64.2 million weights, and Table 2e shows the FCN head is 2.1 AP more accurate.

Toggle to inference and the order changes. At test time the box head runs first on 1,000 proposals, NMS removes duplicates, and the mask head runs only on the 100 highest-scoring detections, using their refined boxes. Of the 80 masks for each detection, only the one for the predicted class is kept; its probabilities are resized to the box and thresholded at 0.5. Masking 100 boxes instead of 1,000 (300 for C4) keeps the added cost at about 20% of the Faster R-CNN model, and the refined boxes fit the objects better than the raw proposals, which the paper says also improves accuracy.

The mask loss

During training every sampled RoI gets the same three-part loss:

$L_{cls}$ and $L_{box}$ are exactly Fast R-CNN's: a log loss over the 81 outputs (80 classes and background), and a smooth-L1 loss on the box offsets of the true class, switched off for background RoIs. The mask term is new. For an RoI whose ground-truth class is $k$, with mask logits $z^k_{ij}$ and binary target $y_{ij}$, it is the binary cross-entropy averaged over the $m^2$ pixels:

Three rules decide what enters (2). Only positive RoIs, those with IoU of at least 0.5 against some ground-truth box, have a mask loss; background RoIs have nothing to outline. Only channel $k$ is scored; the other 79 masks of that RoI contribute nothing. And the target is the ground-truth mask as seen through the RoI: in the released code each positive RoI is matched to the ground-truth polygon whose box overlaps it most, and that polygon is rasterized in the RoI's own coordinates onto an $m \times m$ grid. If the RoI is a little too tight, the target is cut off at the RoI's edge, which is what "the intersection between an RoI and its associated ground-truth mask" means in the paper.

A worked example with the FPN head. A proposal around a standing person, in an image resized to 800 pixels on its short side, spans x from 117.3 to 345.9 and y from 210.8 to 742.4, and has IoU 0.62 with the person's ground-truth box. It is positive with class person. Its target is a 28 × 28 grid of 0s and 1s, the person's polygon drawn into the proposal's frame, so each target cell covers 228.6 / 28 ≈ 8.2 pixels horizontally and 531.6 / 28 ≈ 19 vertically. The mask head outputs 80 × 784 = 62,720 logits for this RoI, and (2) reads 784 of them, 1.25%. A pixel whose logit is 0 has probability 0.5 and costs $-\log 0.5 = 0.693$ whichever its label.

The gradient of (2) per logit has the familiar sigmoid-cross-entropy form:

For the logit-0 pixel above with label 1 the gradient is $-0.5/784$, which pushes the logit up. The person channel of the final 1 × 1 convolution learns from person RoIs only, the dog channel from dog RoIs only, and the shared layers below receive the sum. One step of training, in shapes:

# one training step, ResNet-50-FPN, one image (Detectron's recipe)
P      = fpn(image)                     # P2..P5, 256 channels, strides 4..32
props  = rpn(P)                         # proposal boxes
rois   = sample(props, n=512, pos_frac=0.25, pos_iou=0.5)
lvl    = clip(floor(4 + log2(sqrt(area(rois)) / 224)), 2, 5)
f_box  = roi_align(P, lvl, rois, out=7, samples=2)      # [512, 256, 7, 7]
cls, d = box_head(f_box)                                # [512, 81], [512, 324]
pos    = rois[labels > 0]                               # at most 128 RoIs
f_mask = roi_align(P, lvl, pos, out=14, samples=2)      # [n, 256, 14, 14]
z      = mask_head(f_mask)                              # [n, 80, 28, 28]
y      = rasterize(gt_polygon[match(pos)], pos, 28)     # [n, 28, 28], 0 or 1
z_k    = z[arange(n), labels[pos] - 1]                  # own class only
L_mask = bce_with_logits(z_k, y).mean()                 # over n * 784 pixels
loss   = ce(cls, labels) + smooth_l1(d, box_targets, pos) + L_mask
loss.backward()   # the 79 other mask channels of each RoI get zero gradient

Why a sigmoid per class instead of a softmax

The obvious alternative is the standard semantic-segmentation loss, used by FCN (Long et al., 2015): at each pixel, a softmax across the classes, trained with a multinomial cross-entropy. The probability of class $c$ at pixel (i, j) is then

and the gradient with respect to every channel is $p^c_{ij} - [c = k]$, nonzero for all classes. Each pixel's classes compete: raising the horse logit at a pixel lowers the person probability there, even if the person logit stays fixed. A mask head trained this way must, at every pixel, decide which of the classes it is looking at, and do that well enough that the true class wins.

Mask R-CNN already knows the class. The box branch classifies the whole RoI from pooled features, with the benefit of seeing the entire object, and its prediction picks which mask to read at test time. With (2) the mask head only answers a yes-or-no question per pixel: given that this RoI is a person, is this pixel part of the person? The paper calls this decoupling mask and class prediction.

The figure below sets toy logits for a 14 × 14 person RoI with a horse overlapping it. The person channel is +3 on person pixels and −3 elsewhere, and the slider sets how strongly the horse channel fires on horse pixels. It starts at +4, one unit above the person logit.

At +4 the softmax mask has lost the 19 pixels where the horse overlaps the person (IoU 0.67), although nothing in the person channel changed; drag the slider to +3 or below and the two panels agree. The sigmoid mask depends on the person channel alone and stays exact. In training, the gradient of (4) also pushes the horse channel down on person pixels, so a softmax mask head is trained to separate classes that the box branch already separates; the paper says the decoupled loss makes the model easier to train.

Table 2b measures it on ResNet-50-C4: the softmax version scores 24.8 mask AP, the sigmoid version 30.3, a gap of 5.5 points (7.1 at AP₅₀, 6.4 at AP₇₅). A class-agnostic variant that predicts a single mask whatever the class scores 29.7, only 0.6 below the 80-mask default. The paper reads the two results together: once the box branch has classified the instance, a binary mask predicted without regard to category is enough.

What RoIPool rounds away

A mask is a pixel-level output, so the RoI features must line up with the image: the feature at row i, column j of the RoI grid must come from the location that row i, column j of the mask describes. RoIPool breaks that in two places. Take the person proposal again on the stride-16 C4 map, pooled to the C4 head's 7 × 7 grid.

First, RoIPool rounds the RoI's edges to whole feature cells. In feature coordinates the box runs from x = 117.3 / 16 = 7.33 to 345.9 / 16 = 21.62, and RoIPool rounds these to 7 and 22. In image pixels it now reads from 112 to 352 instead of 117.3 to 345.9: the left edge moves 5.3 pixels left and the right edge 6.1 pixels right. Vertically, 13.18 and 46.40 become 13 and 46, so the top moves 2.8 pixels and the bottom 6.4. Each edge can move by up to half a cell, 8 pixels at stride 16 and 16 at stride 32.

Second, RoIPool splits the rounded RoI into bins along whole-cell boundaries. The rounded box covers cells 7 to 22, 16 cells wide, so each of 7 bins should be 16 / 7 = 2.29 cells. The code takes cells from floor(p * 2.29) to ceil((p + 1) * 2.29): the first bin max-pools cells 7 to 9, the second 9 to 11, the third 11 to 13. Neighbouring bins share a cell, and a bin's nominal centre and the cells it reads can differ by a fraction of a cell.

Classification is barely affected, because a dog shifted by 6 pixels is still a dog; the paper notes classification is robust to small translations. A mask drawn from features that sit up to 8 pixels away from where the mask is pasted has its boundary off by that much. Rounding also makes RoIPool's output a step function of the box coordinates: moving an edge by 3 pixels changes nothing until a rounding boundary is crossed, then the output jumps. The next figure shows that staircase.

How RoIAlign works

RoIAlign removes every rounding. It keeps the box at x / 16 instead of round(x / 16), splits it into exactly equal bins, and in each bin reads the feature map at 4 regularly spaced points (a 2 × 2 grid at a quarter and three quarters of the bin's width and height). The feature map only has values at integer cells, so each point's value is computed by bilinear interpolation from the four cells around it. With $x_0 = \lfloor x \rfloor$, $y_0 = \lfloor y \rfloor$ and fractional parts $l_x = x - x_0$, $l_y = y - y_0$:

Each weight is the area of the rectangle opposite its corner, so the four weights sum to 1 and a point that lands on a cell gets exactly that cell's value. The four samples in a bin are then averaged (the paper found max and average equivalent, 30.2 and 30.3 AP, and uses average).

On the FPN model the person proposal goes to the stride-16 level, for reasons the next section gives, and the mask head pools it to 14 × 14. The box starts at x = 7.331 and y = 13.175 in feature cells and spans 14.29 × 33.23 cells, so each bin is 1.021 cells wide and 2.373 tall. The first sample of the top-left bin sits a quarter of the way into it, at x = 7.586, y = 13.768. Its neighbours are cells (7, 13), (8, 13), (7, 14), (8, 14), with $l_x = 0.586$, $l_y = 0.768$, and weights 0.096, 0.136, 0.318 and 0.451. Every one of the 14 × 14 × 4 = 784 samples is computed this way for each of the 256 channels.

In RoIPool mode the violet RoI jumps from grid point to grid point as you drag, and the bin's output in the plot is flat between jumps. In RoIAlign mode the teal sample points move with the box and the output changes continuously. RoIWarp, from the MNC system of Dai et al., sits between them: it samples bilinearly, like RoIAlign, but inside the rounded RoI, so its output is a staircase too: between rounding flips the rounded RoI, and with it every sample point, stays put.

RoIWarp isolates the cause. Table 2c, on ResNet-50-C4 at stride 16, scores RoIPool at 26.9 mask AP, RoIWarp at 27.2, and RoIAlign at 30.3. Bilinear sampling without alignment buys 0.3 points; removing the rounding buys the remaining 3.1. The gap grows with the stride. On stride-32 features (ResNet-50-C5, Table 2d), RoIAlign raises mask AP from 23.6 to 30.9 and AP₇₅ from 21.6 to 32.1, a 50% relative gain, and lifts box AP too, from 28.2 to 34.0. The stride-32 model with RoIAlign is more accurate than the stride-16 model (30.9 against 30.3).

The continuity also changes the gradient. RoIAlign's output is a continuous, piecewise-differentiable function of the box coordinates, so in principle gradients can flow into the box. The paper does not use that: its end-to-end variant in Appendix B follows Faster R-CNN's "approximate" joint training and ignores the gradient with respect to the RoI coordinates.

The released Detectron kernel treats feature cell j as sitting at coordinate j, while under the convention that a cell covers $[16j, 16j+16)$ its centre is at $j + 0.5$. Every RoI is therefore read half a cell, 8 pixels at stride 16, away from its nominal place. Detectron2 later subtracted the 0.5 (its aligned=True option) and reports no change in accuracy. A plausible reason is that the shift is the same for every RoI, so the convolutions that follow can learn to compensate, while RoIPool's error depends on where each box happens to fall relative to the grid, so no fixed correction undoes it.

Which pyramid level an RoI reads

With a feature pyramid there are four maps to read from, and the choice decides how many cells an RoI covers. FPN assigns an RoI of width $w$ and height $h$ (in pixels of the resized image) to level $P_k$ by

clipped to the range 2 to 5; $P_k$ has stride $2^k$. The constant 224 is the ImageNet training image size, and a 224 × 224 RoI goes to $P_4$, the stride-16 level a C4 model would use. Each halving of the RoI's side moves it one level finer. For the person proposal:

Between 56 and 896 pixels, where the clipping does not apply, the teal sawtooth stays between 14 and 28 cells: every RoI is read from a map on which it spans at least as many cells as the 14 × 14 mask grid has bins. On a single stride-16 map a 32-pixel object spans 2 cells, so its 14 × 14 grid interpolates within 2 cells, and an 800-pixel object spans 50. Objects under 56 pixels clip to $P_2$ and cover fewer cells there, which is one reason small objects remain the weakest category (AP_S 15.5 against AP_L 52.4 for ResNet-101-FPN on test-dev). FPN's finer strides shrink RoIAlign's advantage but do not remove it: the paper reports +1.5 mask AP and +0.5 box AP from RoIAlign on FPN.

Training and inference

The training recipe follows Fast R-CNN and FPN. Images are resized so the shorter side is 800 pixels. Each GPU takes 2 images, and from each image samples RoIs at a 1:3 ratio of positives to negatives: 64 per image for C4, 512 for FPN, so at most 16 or 128 positives per image carry a mask loss. Eight GPUs make an effective mini-batch of 16 images. SGD runs for 160,000 iterations at learning rate 0.02, divided by 10 at iteration 120,000, with weight decay 0.0001 and momentum 0.9. The RPN uses anchors at 5 scales and 3 aspect ratios. For the ablations the RPN is trained separately and does not share features with the second stage, which lets the paper change the second stage without retraining the proposals.

Training ResNet-50-FPN on COCO's 115,000-image trainval35k set takes 32 hours on 8 GPUs, 0.72 seconds per 16-image mini-batch, and ResNet-101-FPN takes 44 hours. A ResNet-101-FPN model that shares features between the RPN and the heads runs in 195 ms per image on an Nvidia Tesla M40, plus 15 ms of CPU time to resize the masks to the original image, about 5 frames per second. The C4 variant takes about 400 ms because res5 runs once per RoI, and the paper recommends against it in practice.

# inference, ResNet-101-FPN
props  = rpn(P)[:1000]
cls, d = box_head(roi_align(P, props, out=7))
dets   = nms_per_class(decode(props, d), cls)[:100]     # boxes, scores, class k
z      = mask_head(roi_align(P, dets.boxes, out=14))     # [100, 80, 28, 28]
prob   = sigmoid(z[arange(100), dets.k - 1])             # [100, 28, 28]
masks  = [resize(p, box) > 0.5 for p, box in zip(prob, dets.boxes)]

Results and ablations

On COCO test-dev (Table 1), ResNet-101-FPN reaches 35.7 mask AP and ResNeXt-101-FPN 37.1. The previous winners were MNC (24.6, COCO 2015) and FCIS (29.2 with online hard example mining, which trains on the highest-loss RoIs; 33.6 as FCIS+++, which adds multi-scale training and testing and flipped test images). Mask R-CNN beats FCIS+++ with none of those additions. The paper's Figure 6 shows FCIS producing systematic artifacts where instances overlap, which Mask R-CNN's box-then-mask order avoids: two overlapping people are two RoIs with two separate masks.

On box detection (Table 3), the same model with its mask output ignored reaches 38.2 box AP with ResNet-101-FPN and 39.8 with ResNeXt-101-FPN, 3.0 above the best earlier single model. The paper splits the gain over the FPN paper's Faster R-CNN (36.2): +1.1 from RoIAlign, +0.9 from training with the mask branch, and +1.6 from ResNeXt. The middle term is measured directly by training Mask R-CNN without its mask branch (37.3), so the 0.9 comes from multi-task training alone. The gap between mask AP and box AP for the ResNeXt model is 2.7 points (37.1 against 39.8).

The tabs follow the order of this page. The mask-loss tab is the decoupling result (5.5 points). The two RoI-layer tabs show the gain from alignment growing from 3.4 to 7.3 points as the stride doubles. The mask-head tab shows the FCN beating two MLP heads by 2.1 points; the paper chose ResNet-50-FPN for this comparison so that neither head starts from pre-trained convolutions. The backbone tab shows that deeper networks, FPN and ResNeXt all help, from 30.3 for ResNet-50-C4 to 36.7 for ResNeXt-101-FPN on minival.

On Cityscapes (Appendix A), with 2,975 finely annotated street images, Mask R-CNN scores 26.2 AP on test without the 20,000 coarsely annotated images that the previous best entries, DIN and SGN, trained on, and 32.0 after initializing from a COCO model. The largest gains are on the classes with many overlapping instances: person rises from 21.8 to 30.5 and car from 39.4 to 46.9 against the best earlier entries. Appendix B stacks later improvements (longer training, end-to-end training, ImageNet-5k pre-training, scale augmentation, ResNeXt-152, non-local blocks, test-time augmentation) for 41.8 mask AP and 47.3 box AP. The three winning teams of the COCO 2017 instance segmentation challenge all built on Mask R-CNN.

Keypoints as one-hot masks

Human pose estimation asks for the image positions of 17 keypoints per person in COCO: nose, eyes, ears, shoulders, elbows, wrists, hips, knees and ankles. Mask R-CNN treats each keypoint type as a mask with exactly one foreground pixel. For each person RoI the keypoint head predicts 17 maps of 56 × 56 logits (eight 3 × 3 convolutions of 512 channels, a deconvolution and a 2× bilinear upsampling). The target for a visible keypoint is a one-hot 56 × 56 map, and the loss is a softmax cross-entropy over the 3,136 positions of that map:

where $p^*$ is the labelled position. Here the softmax runs over positions within one map, because a wrist sits at one place; the 17 maps are still independent, as the class masks are. The paper found this higher resolution, compared with masks, necessary for keypoint-level accuracy: on a 200-pixel-tall person box one of 56 rows is 3.6 pixels tall.

With ResNet-50-FPN the keypoint model reaches 62.7 keypoint AP on test-dev, 0.9 above the COCO 2016 winner, which used a multi-stage pipeline (Table 4), and 63.1 when a person-mask branch is trained alongside it. Alignment matters even more here: on FPN, which already has stride-4 features, RoIAlign improves keypoint AP over RoIPool by 4.4 points, 59.8 to 64.2 (Table 6). Training with the mask branch helps keypoints, but adding the keypoint branch lowers person box AP from 53.6 to 52.0 and mask AP from 45.8 to 45.1 (Table 5), so the extra task does not help the others in turn.

Limits and what came next

Every mask lives inside its box: an error in the box head truncates the mask, and nothing outside the box can be recovered. The mask is 28 × 28, so large objects lose fine boundary detail when the mask is stretched to their size. Small objects remain hard, as the AP_S numbers show. The model is not tuned for speed; the paper notes that image size and proposal count trade accuracy for time and leaves that out of scope.

On Cityscapes the paper reports a gap between validation and test AP for truck, bus and train (for example 53.5 against 32.2 for bus), classes with only about 200 to 500 training instances each, and reads it as a domain shift that COCO pre-training narrows but does not remove.

Detectron2 reimplemented it in PyTorch, with the half-cell RoIAlign offset corrected. Self-supervised vision papers such as MAE report detection and segmentation by fine-tuning a Mask R-CNN on COCO with their pretrained backbone, and the Segment Anything paper compares its masks against ViTDet, a detector built on the Mask R-CNN framework with a plain Vision Transformer backbone. Detectors without proposals or NMS, such as DETR, came later and added their own mask heads.