Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

A small network that runs on the detector's own feature map proposes object boxes in 10 ms instead of 1.5 seconds.

Faster R-CNN replaces the hand-built proposal step of a two-stage detector with a region proposal network: one 3 × 3 convolution and two 1 × 1 convolutions that score and adjust nine reference boxes at every position of the feature map. With VGG-16 the whole detector runs at 5 frames per second and is more accurate than the same detector on Selective Search boxes.

In 2015 the best detectors worked in two steps: list a few thousand boxes that might hold an object, then classify each one. Listing the boxes took 1.5 seconds per image on a CPU. Classifying them took 0.32 seconds on a GPU.

This paper, from Microsoft Research in June 2015, moved the listing step onto the GPU by computing it from the same convolutional features the classifier already uses. The new component, the region proposal network (RPN), adds 10 ms per image. The paper reports that the detector built from it, Faster R-CNN, was the base of first-place entries in four tracks of the ILSVRC and COCO 2015 competitions, and that its framework had by then been adapted to 3D detection, part-based detection, instance segmentation and image captioning.

Six pieces explain it: what a detector outputs and how it is scored; why proposals became the bottleneck; the RPN itself and its anchors; how the RPN is trained (labels, box offsets, loss); how its boxes are pruned to 300 and handed to the Fast R-CNN head; and how the two networks are trained to share one backbone. The results come last, with the ablations that show which parts matter.

What a detector outputs, and how it is scored

An object detector takes an image and returns a list of detections. Each detection is an axis-aligned rectangle, a class label from a fixed set, and a confidence score. PASCAL VOC, the main benchmark in this paper, has 20 classes (person, car, dog, bottle and so on) and about 5,000 training and 5,000 test images in its 2007 edition. MS COCO has 80 classes and, in the split used here, 80,000 training images.

Boxes are compared by intersection over union (IoU): the area where two boxes overlap divided by the area they cover together. Two 100 × 100 boxes shifted 50 pixels sideways overlap in a 50 × 100 strip, so the intersection is 5,000 px², the union is 10,000 + 10,000 − 5,000 = 15,000 px², and the IoU is 0.33. Identical boxes score 1; disjoint boxes score 0. The same measure appears everywhere below: to decide which training boxes count as objects, to remove duplicate boxes, and to grade the output.

On PASCAL a detection is correct if its class matches a ground-truth box it overlaps with IoU of at least 0.5, and that box has not already been claimed by a higher-scoring detection. Sorting one class's detections by score and walking down the list traces precision (the fraction of detections so far that are correct) against recall (the fraction of ground-truth objects found so far). Average precision (AP) summarizes that curve; the VOC 2007 version averages the best precision reached at recall 0, 0.1, …, 1.0, eleven points in all. mAP is the mean of AP over the 20 classes, and every PASCAL number on this page is an mAP in percent. COCO's main metric, mAP@[.5, .95], averages mAP over ten IoU thresholds from 0.5 to 0.95, so a detector only scores well on it if its boxes are tight.

Proposals became the slow step

A region proposal is a box that might contain an object of any class. Detectors of this family first generate about 2,000 of them per image, cheaply and with high recall, and then spend the expensive computation only on those. The most popular generator, Selective Search, over-segments the image into small regions of similar colour and texture and greedily merges neighbours, emitting a box around each merged group. It uses no learning and runs on the CPU, at about 2 seconds per image in the paper's introduction; the paper's timing table (Table V) measures 1.5 s, the number this page uses. EdgeBoxes, which the paper calls the best trade-off between quality and speed, scores boxes by the edge contours that lie wholly inside them and takes 0.2 seconds.

The classification step had been getting faster. R-CNN (2014) cropped each of the 2,000 proposals out of the image and ran a whole CNN on every crop. SPPnet and then Fast R-CNN (April 2015) run the convolutional layers once over the entire image and cut each proposal's features out of the resulting feature map. The feature map is a grid of vectors, one per image location at a coarser resolution; for VGG-16 its last convolutional layer, conv5_3, has 512 channels and one position per 16 × 16 pixels. Fast R-CNN takes the part of that grid under each proposal and max-pools it to a fixed 7 × 7 grid (RoI pooling, where RoI means region of interest), so proposals of any size become the same 512 × 7 × 7 input to a few fully connected layers. Those layers output class scores and a refined box.

With VGG-16, Fast R-CNN takes 320 ms per image on 2,000 Selective Search boxes. Selective Search itself takes about 1.5 s. The proposal step, which contains no learning, cost 4.7 times as much as the network. Re-implementing Selective Search on the GPU would speed it up, but the paper sets that aside: a re-implementation would still ignore the detection network and could share none of its computation. Instead the paper computes proposals from the conv5_3 features that Fast R-CNN computes anyway, with a few extra convolutional layers trained for the purpose. The figure puts the three systems in Table V side by side.

With VGG-16 the total falls from 1,830 ms to 198 ms, from 0.5 to 5 frames per second, and the proposal stage from 1,510 ms to 10 ms. The region-wise stage also shrinks, from 174 to 47 ms, because the head now runs on 300 proposals instead of 2,000. The convolutional layers, 141 ms, are now most of the cost. With the smaller ZF network (5 convolutional layers from Zeiler and Fergus, 2013) the whole system takes 59 ms, 17 frames per second.

The region proposal network

The RPN takes the conv5_3 feature map and slides a small network over it. At each position the small network looks at a 3 × 3 window of feature vectors, maps it to a 512-dimensional vector followed by a ReLU (the function max(0, x) applied to each number), and feeds that vector to two sibling layers: a box-classification layer (cls) that says whether an object is present and a box-regression layer (reg) that says where exactly its box is. Because the same weights run at every position, the whole thing is a convolution: a 3 × 3 convolution with 512 outputs, then two 1 × 1 convolutions.

# RPN head on VGG-16's last conv layer, one 600 x 1000 image
f   = conv5_3(image)               # [1, 512, 38, 63]  stride 16
h   = relu(conv3x3(f, out=512))    # [1, 512, 38, 63]  the sliding window
cls = conv1x1(h, out=2 * 9)        # [1, 18, 38, 63]   object / not, per anchor
reg = conv1x1(h, out=4 * 9)        # [1, 36, 38, 63]   (tx, ty, tw, th) per anchor
# 38 * 63 = 2,394 positions, * 9 anchors = 21,546 candidate boxes

The shapes come from the input size. Images are rescaled so their shorter side is 600 pixels, and VGG-16's four pooling layers each halve the resolution, a total stride of 16. A 600 × 1000 image gives a 38 × 63 feature map (the pooling rounds up), 2,394 positions, which is the paper's "typically ∼2,400". Each output channel group belongs to one of the nine anchors described next, so the 18 cls channels hold 2 scores per anchor and the 36 reg channels hold 4 numbers per anchor, at every position.

A 3 × 3 window on a stride-16 map is small in feature space but not in the image. The paper gives its effective receptive field, the patch of input pixels that can influence one output, as 228 pixels for VGG-16 and 171 for ZF. For VGG-16 the number comes from conv5_3 seeing 196 pixels and the extra 3 × 3 window adding 2 × 16 on each axis.

Why a sliding window and not a fully connected layer over the whole image, as in the MultiBox proposal networks from Google (2014)? The paper gives two reasons, parameter count and translation invariance. The RPN's output layers have 512 × (4 + 2) × 9 = 27,648 weights for VGG-16; MultiBox's fully connected output layer has 1536 × (4 + 1) × 800 = 6.1 million. Counting the 3 × 3 layer too, the RPN has 2.4 million parameters against MultiBox's 27 million, which the paper expects to reduce overfitting on a dataset of 5,000 training images. For translation invariance, the RPN applies the same function at every position, so an object moved across the image gets the same proposal moved with it, up to the 16-pixel stride. MultiBox's 800 reference boxes come from k-means on training boxes and each is tied to its own output units, so it has no such guarantee.

Anchors: nine reference boxes per position

One 512-dimensional vector per position has to describe objects of very different sizes and shapes: a bottle 40 pixels wide, a bus filling the frame. The RPN handles this with anchors, a fixed set of k reference boxes centred on each position. Every anchor has its own pair of cls scores and its own four reg outputs. The cls scores say whether this anchor, once adjusted, covers an object; the reg outputs say how to adjust it. The paper uses three scales, with areas 128², 256² and 512² pixels, times three aspect ratios, 1:1, 2:1 and 1:2, so k = 9.

The released code builds them from a 16 × 16 base window and rounds the side lengths to whole pixels, which gives these nine boxes (width × height):

Nine anchors at each of 2,394 positions makes 21,546 anchors for a 600 × 1000 image (the paper rounds this to "roughly 20,000, ≈ 60 × 40 × 9"). Every one of them gets a score and four offsets on every forward pass, for the cost of two 1 × 1 convolutions. Anchors work like a tailor's nine stock sizes, each altered to fit: the scale comes from the stock size and the fit from the alteration. The analogy fails at the choosing: a tailor picks one stock size first, while the RPN alters and scores all nine at once, at every position, and keeps the best-scoring results.

Move the position toward any edge and the large anchors go dashed first. At the default position in the middle of the image the 352 × 704 anchor already sticks out of the 600-pixel-tall image. Those anchors are left out of training (the labelling section gives the reason) but still produce proposals at test time, which are clipped to the image.

The paper calls this a pyramid of anchors and contrasts it with two older ways to handle scale. An image pyramid resizes the image to several scales and computes features at each, which multiplies the convolutional cost. A pyramid of filters runs windows of several sizes over one feature map, as in the DPM detector. Anchors keep one image scale and one 3 × 3 filter and put the scale variation into the output channels. Fast R-CNN also works on a single image scale, so the RPN can share its feature map without extra cost. Table VIII measures what the anchor set buys: one anchor per position loses 3 to 4 mAP points, and three scales with one aspect ratio (69.8%) do about as well as all nine (69.9%).

Predicting a box as an offset from its anchor

The reg layer does not output pixel coordinates. For each anchor it outputs four numbers $t = (t_x, t_y, t_w, t_h)$ that say how to move and stretch that anchor. The parameterization comes from R-CNN. With $x, y$ a box's centre and $w, h$ its width and height, and the subscript $a$ marking the anchor, the prediction is encoded as

and the target $t^*$ is the same four expressions with the ground-truth box $x^*, y^*, w^*, h^*$ in place of the prediction. The paper describes this as "bounding-box regression from an anchor box to a nearby ground-truth box". At test time the network's $t$ is turned back into a box by inverting (2):

Dividing the centre shift by the anchor's width measures it in anchor widths, so a target of 0.1 means the same relative move for a 128-pixel anchor as for a 512-pixel one, and the regression targets have similar ranges at every scale. The logarithm makes size changes symmetric: twice as wide is +0.693 and half as wide is −0.693, where a plain ratio would give 2 and 0.5. And whatever $t_w$ the network outputs, the decoded width $w_a e^{t_w}$ is positive.

For a worked example, take the feature-map position (25, 18), whose centre is pixel (407.5, 295.5). Its 256², 2:1 anchor is 368 × 192. A dog's ground-truth box is centred at (430, 280) and is 300 × 200. Then

The anchor and the box have IoU 0.708. The 256 × 256 anchor at the same position has IoU 0.687 with the dog, and the other seven anchors there are below 0.41. The regressor for this anchor has to learn a shift of 6% of the anchor's width and 8% of its height, a width about 18% smaller ($e^{-0.204} = 0.82$) and a height 4% larger.

Stretch the ground-truth box to twice the anchor's width and $t_w$ reads about +0.69; shrink it to half and it reads about −0.69. Slide the prediction from 0 to 1 and the teal box moves from the anchor onto the target along the path (2) defines, with the centre moving in a straight line and the width growing geometrically.

The nine regressors at a position do not share weights. Each is a separate group of four output channels, and each only ever sees targets relative to its own anchor shape, so the 512 × 512 regressor specializes in large objects and the 88 × 176 regressor in small upright ones. Fast R-CNN's box regression works differently: it pools features from proposals of arbitrary size to a fixed grid and uses one set of weights for all sizes. In the RPN the input features are always the same 3 × 3 window, and the anchors supply the scale.

Which anchors count as objects

To train the cls layer, every anchor needs a label: object or background. The RPN compares each anchor with every ground-truth box in the image and applies three rules:

• Positive if its IoU with some ground-truth box is at least 0.7, or if it is the anchor with the highest IoU for some ground-truth box.
• Negative if its IoU with every ground-truth box is below 0.3.
• Ignored otherwise. An anchor between 0.3 and 0.7 contributes nothing to the loss, so the classifier is never trained on half-overlapping boxes as either class.

The second positive rule exists because the first can find nothing. An object of an unusual size or shape may have no anchor at 0.7 or above; without the rule it would produce no positive training example. The code applies it to every anchor that ties for the highest IoU, which happens when a box is taller or wider than the anchor and the anchor can slide along it without changing the overlap.

A positive anchor's regression target $t^*$ is computed against the ground-truth box it overlaps most. Two more restrictions shape what is trained. Anchors that cross the image border are dropped from training: the paper reports that with them the objective gains large error terms that are hard to correct, and training does not converge. And because the loss would otherwise be dominated by background, each training step samples 256 anchors from one image, with up to 128 positives and the rest negatives.

At the default, the 300 × 200 dog from the previous section, 14 anchors are positive, 504 are ignored and 7,534 are negative, and 13,494 anchors are excluded for crossing the border. The mini-batch is therefore 14 positives and 242 negatives. Shrink the box below about 115 on the slider (the square root of its area) and no anchor reaches 0.7, so every positive comes from the highest-IoU rule; below about 70 every IoU is under 0.3 and nothing is ignored. The tall 1:3 shape, a standing person, has a best IoU of 0.69 at the default size, so its 10 positives are all anchors that tie for that maximum.

On this 1000 × 600 grid, 8,052 of the 21,546 anchors lie fully inside the image. The paper says "about 6000" for its typical image; that figure matches an 800 × 600 image, the size a typical 500 × 375 PASCAL photo becomes after rescaling, which has 17,100 anchors of which about 5,900 lie inside by the code's test.

The RPN loss

With labels in hand, the RPN is trained on the multi-task loss of Fast R-CNN, adapted to anchors. For one image it is

Here $i$ runs over the 256 sampled anchors. $p_i$ is the predicted probability that anchor $i$ is an object, and $p_i^*$ is its label, 1 for positive and 0 for negative. $L_{cls}$ is the log loss over the two classes, $-\log p_i$ for a positive and $-\log(1-p_i)$ for a negative. $t_i$ and $t_i^*$ are the predicted and target offsets from (2). The factor $p_i^*$ in the second sum switches the box loss on for positive anchors only: a background anchor has no box to regress toward.

The box loss is $L_{reg}(t_i, t_i^*) = R(t_i - t_i^*)$, applied to each of the four coordinates and summed, with $R$ the smooth L1 function from Fast R-CNN:

Near zero it is quadratic, so its gradient shrinks as the prediction approaches the target. Beyond 1 it is linear, so its gradient never exceeds 1 in magnitude, and a badly placed anchor early in training cannot produce a huge update. Fast R-CNN introduced it because training box regression with a squared loss needed careful tuning of the learning rate to avoid exploding gradients.

The normalizers and $\lambda$ set the balance between the two terms. The paper divides the classification sum by $N_{cls} = 256$, the mini-batch size, and the box sum by $N_{reg} \approx 2{,}400$, the number of anchor positions, and sets $\lambda = 10$. Each sampled anchor then carries a weight of 1/256 ≈ 0.0039 in the first term and each positive anchor a weight of 10/2,400 ≈ 0.0042 in the second, which is what the paper means by "roughly equally weighted". Table IX varies $\lambda$: from 1 to 100 the mAP stays within 1.0 point of the default (68.9% to 69.9%), and 0.1 costs 2.7 points. The paper adds that the normalization is not required. The released Python code uses a different but nearly equivalent setting: both sums divided by 256, $\lambda = 1$, and a smooth L1 whose quadratic zone ends at |x| = 1/9 rather than 1.

Continuing the example: the new layers start from zero-mean Gaussian weights with standard deviation 0.01, so an untrained RPN outputs $t \approx 0$ and the predicted box is the anchor itself. For the 368 × 192 anchor all four errors are below 1, so each is in the quadratic zone, and its box loss is 0.5 × (0.061² + 0.081² + 0.204² + 0.041²) = 0.027 before the 10/2,400 weight.

# one RPN training step on one image (py-faster-rcnn's anchor_target_layer)
A     = anchors(38, 63)                    # [21546, 4], nine per position
A     = A[inside_image(A)]                 # drop border-crossing anchors
iou   = box_iou(A, gt)                     # [n_anchors, n_gt]
lab   = full(len(A), -1)                   # -1 = ignored, no loss
lab[iou.max(1) < 0.3]  = 0                 # negative
lab[argmax_per_gt(iou)] = 1                # best anchor for each gt box
lab[iou.max(1) >= 0.7] = 1                 # positive
lab   = subsample(lab, total=256, max_pos=128)
t_gt  = encode(gt[iou.argmax(1)], A)       # equation (2)
L_cls = log_loss(cls[lab >= 0], lab[lab >= 0]).sum() / 256
L_reg = smooth_l1(reg[lab == 1] - t_gt[lab == 1]).sum() / 2400
loss  = L_cls + 10 * L_reg                 # equation (1), lambda = 10
loss.backward()                            # into the RPN layers and conv3_1+

The rest of the recipe: SGD with momentum 0.9 and weight decay 0.0005, learning rate 0.001 for 60,000 mini-batches and 0.0001 for the next 20,000 on PASCAL VOC, one image per mini-batch. The shared convolutional layers start from an ImageNet-trained classifier. All of ZF's layers are fine-tuned; for VGG-16, only conv3_1 and above, to save memory.

From 21,546 anchors to 300 proposals

At test time the RPN scores and adjusts every anchor, including the ones that cross the border. The released code then turns them into proposals in five steps: apply the predicted offsets with the inverse of (2); clip each box to the image; drop boxes with a side shorter than 16 pixels of the original image; keep the 6,000 highest object scores; and run non-maximum suppression at IoU 0.7, keeping the top 300 survivors.

Non-maximum suppression (NMS) removes near-duplicates. Several neighbouring anchors usually regress onto the same object, so the list holds clusters of almost identical boxes. Greedy NMS sorts the boxes by score, keeps the first, deletes every remaining box whose IoU with it exceeds a threshold, and repeats on what is left. The threshold sets how similar two boxes must be to count as duplicates: at 1 nothing is deleted, and at 0 any overlap deletes the lower-scoring box.

At 0.7, the RPN's setting, NMS keeps 18 of the 47 boxes, and most objects keep a few near-duplicates. At 0.5 it keeps 11, and the two people standing close together still have separate boxes. At 0 it keeps 5, and the two people share a single box. A loose threshold suits proposals: an object whose boxes are all deleted here cannot be recovered later, while a surviving near-duplicate is removed by the head's own NMS. The Fast R-CNN head later runs a stricter NMS at 0.3, separately for each class, on its final detections.

The paper reports that NMS at 0.7 leaves about 2,000 proposals per image. Fast R-CNN is trained on 2,000 RPN proposals per image and tested on the top 300. Table II checks that NMS costs no accuracy: skipping it and passing the top 6,000 raw boxes gives 55.2% mAP, against 56.8% with NMS and 300 boxes.

The detection head: RoI pooling on the same features

The second stage is Fast R-CNN, unchanged except for where its boxes come from. For each of the 300 proposals it divides the box coordinates by 16 to find the box on the conv5_3 map, splits that region into a 7 × 7 grid of bins, and takes the maximum of each channel in each bin. The 300 × 200 dog proposal covers about 18.8 × 12.5 feature positions, so each bin spans about 2.7 × 1.8 positions. The result, 512 × 7 × 7 = 25,088 numbers per proposal, goes through two fully connected layers of 4,096 units and then two outputs: a softmax (scores normalized into probabilities that sum to 1) over 21 classes (the 20 PASCAL classes and background) and 84 box offsets, one set of four per class, in the parameterization of (2) relative to the proposal.

Its training mirrors the RPN's with different constants. Fast R-CNN samples 128 proposals per mini-batch from two images, a quarter of them with IoU of at least 0.5 against a ground-truth box (labelled with that box's class) and the rest with IoU in [0.1, 0.5) (labelled background). Its loss is a log loss over the 21 classes plus the smooth-L1 box loss for the true class, with weight 1, switched off for background. On COCO the paper widens the background range to [0, 0.5), which improved mAP@0.5 there and made a negligible difference on PASCAL.

The paper borrows a word from the attention models of the time, such as Bahdanau attention for translation, and says the RPN tells Fast R-CNN "where to look". The mechanism is a hard choice of 300 boxes by score and NMS, with no weighted average over positions.

Why keep a second stage at all, if the RPN already scores and places boxes? The RPN's features for every anchor come from the same 3 × 3 window, whatever the anchor's size, and it only separates object from background. The head pools features from the region the proposal actually covers and is trained to tell 20 classes apart. The paper tests this in Table X by emulating a one-stage detector (in the style of OverFeat): Fast R-CNN applied directly to 20,000 dense sliding windows at the anchor scales and ratios. It scores 53.9% mAP on a five-scale image pyramid, against 58.7% for the two-stage system, and is slower because it processes many more boxes.

Training two networks to share one backbone

The RPN and Fast R-CNN are two networks on top of the same convolutional layers. Trained separately, each would modify those layers its own way, and the system would need two backbones at test time, doubling the convolutional cost that sharing is meant to save. The paper discusses three ways to train one shared set of layers:

1. Alternating training. Train the RPN, train Fast R-CNN on its proposals, use the result to initialize the RPN again, and repeat. Every experiment in the paper uses it.
2. Approximate joint training. Merge the two into one network. In each SGD step the forward pass produces proposals, which Fast R-CNN then treats as fixed, precomputed boxes; in the backward pass the shared layers receive the sum of the gradients from the RPN loss and the Fast R-CNN loss. This ignores the derivative of the Fast R-CNN loss with respect to the proposal coordinates, which are also network outputs. The paper found it gives close results and cuts training time by about 25 to 50%. It is included in the released Python code.
3. Non-approximate joint training. Include that derivative, which needs an RoI pooling layer that is differentiable with respect to the box coordinates. The paper points to the "RoI warping" layer of Dai et al. and leaves it out of scope.

The missing derivative comes from how RoI pooling works. The layer rounds the proposal's coordinates to feature-map cells and takes maxima over whole cells, so a small change in a coordinate changes nothing until it crosses a cell boundary. The layer passes gradients to the feature values it selected and none to the box. In approximate joint training the RPN's box outputs are therefore trained only by the RPN's own loss (1), never by how useful their boxes turned out for classification.

The paper's alternating scheme has four steps, and after step 4 the two networks run on the same frozen layers:

# 4-step alternating training (every experiment in the paper)
W0   = imagenet_pretrained()                     # VGG-16 or ZF
rpn1 = train_rpn(init=W0)                        # 1: RPN, all layers tuned
P1   = rpn1.propose(train_images)                # 2,000 boxes per image
det2 = train_fast_rcnn(init=W0, proposals=P1)    # 2: detector, own backbone
rpn3 = train_rpn(init=det2, frozen=det2.convs)   # 3: only RPN layers move
P3   = rpn3.propose(train_images)
det4 = train_fast_rcnn(init=det2, proposals=P3, frozen=det2.convs)  # 4: head only
# rpn3 and det4 now run on det2's conv layers: one network at test time

Sharing helps the proposals as well as the speed. Stopping after step 2, with two separate backbones, gives 58.7% mAP with ZF against 59.9% for the shared network; with VGG-16, 68.5% against 69.9%. The paper attributes the gain to step 3: fine-tuning the RPN on features already tuned for detection improves its proposals. More rounds of alternation gave negligible further improvement.

Results and ablations

On PASCAL VOC 2007 test with VGG-16 (Table III), the Selective Search baseline scores 66.9% mAP when trained on VOC 2007 and 70.0% when trained on VOC 2007 and 2012 together. Faster R-CNN, with 300 proposals instead of 2,000, scores 69.9% and 73.2% on the same data, 3.0 and 3.2 points higher. Pre-training on COCO first and then fine-tuning on VOC raises it to 78.8%, and the paper notes this model has the best AP of the compared methods on every one of the 20 classes. On VOC 2012 test (Table IV) the numbers are 70.4% against 68.4% for Selective Search, and 75.9% with COCO. A COCO-trained model applied to VOC 2007 with no VOC training scores 76.1%, higher than the 73.2% trained on VOC.

On COCO test-dev (Table XI), trained on the 80,000-image training set, Faster R-CNN reaches 42.1% mAP@0.5 and 21.5% mAP@[.5, .95], against 39.3% and 19.3% for the authors' own Fast R-CNN with Selective Search. The gains are 2.8 and 2.2 points; relative to the baseline the strict metric gains more (11% against 7%), which the paper reads as better localization at high IoU. Replacing VGG-16 with the 101-layer network from the ResNet paper, and changing nothing else, raises COCO validation mAP from 41.5% / 21.2% to 48.4% / 27.2%; with further additions, that system won the COCO 2015 detection challenge.

Table II runs one fixed Fast R-CNN detector with ZF, trained on 2,000 Selective Search boxes, and changes only the proposals it receives at test time. Tables VIII, IX and X vary the anchors, $\lambda$ and the number of stages.

Three results in the first tab explain the design. Removing the cls scores and sampling proposals at random costs 10.5 points at 100 proposals, 5.4 at 300 and 0.5 at 1,000, each against the scored RPN list of the same length, so the cls scores matter most when the list is short, and a short list is what keeps the head at 47 ms. Removing the reg outputs and passing raw anchors costs 4.7 points at 300 proposals and 5.0 at 1,000, so the offsets of (2) matter for box quality at any budget. And an RPN trained on VGG-16 feeding the same ZF detector raises its mAP from 56.8% to 59.2%, above Selective Search's 58.7%. The paper reads this as evidence that learned proposals improve with a stronger network.

The paper also plots proposal recall against IoU for 300, 1,000 and 2,000 proposals (its Figure 4). The RPN's recall changes little from 2,000 to 300 proposals, while Selective Search and EdgeBoxes lose recall faster as the list gets shorter. The paper treats recall as a diagnostic rather than a target, because recall at a fixed IoU is only loosely related to final detection mAP.

Limits and what came next

The title says "towards" real time: 5 frames per second with VGG-16 and 17 with ZF, the latter at 59.9% mAP on VOC 2007 against VGG-16's 69.9%. After the RPN, the 141 ms of convolutions is most of the remaining cost, and the head still runs once per proposal.

The stride of 16 is coarse for small objects. On a typical PASCAL image it corresponds to about 10 pixels of the original image, and the paper notes that a smaller stride might improve accuracy. For COCO, which has many small objects, the authors added a fourth anchor scale of 64² pixels. The anchor sizes and ratios themselves are hand-set ("not carefully chosen for a particular dataset", in the paper's words), and the training scheme needs four stages or an approximation.

Later work addressed these points. Feature Pyramid Networks (Lin et al., 2017) attach RPN heads to feature maps at several strides, assigning small anchors to fine maps. Mask R-CNN (He et al., 2017) adds a mask branch to the head and replaces RoI pooling with RoIAlign, which samples the feature map with bilinear interpolation instead of rounding to cells. One-stage detectors, YOLO (posted to arXiv four days after this paper) and SSD (December 2015), drop the second stage and classify dense boxes directly; SSD keeps anchor-style default boxes on several feature maps.